A quick retrieval method for language structure big data

Abstract

The application provides a quick retrieval method for language structure big data, according to the type of language structure data, the basic index unit to be constructed and the index content corresponding to the index unit are determined, when retrieving, the retrieval expression input by the user is analyzed, the retrieval unit corresponding to the index unit and the relationship between the retrieval units are analyzed, the result of each retrieval unit is obtained in the way of single chain binary tree postorder traversal and is combined according to the relationship between the analyzed retrieval units. In the application, the design of the index unit can utilize the lexical and syntactic information of the language data itself, realize the retrieval requirement of multi-directional language knowledge of the language structure data, the design of the structure retrieval language can enable the user to learn the retrieval language at a low cost and write the retrieval expression which is standard and meets the retrieval requirement.

Description

Technical Field

[0001] This invention relates to a corpus retrieval method, and more particularly to a rapid retrieval method for large amounts of language structure data. Background Technology

[0002] Natural language exhibits hierarchical composition in its grammar. Based on the grammatical units and the grammatical structural features presented after grammatical analysis of language data, index units and retrieval processes can be designed to achieve rapid retrieval of large amounts of language structure data.

[0003] 1. Hierarchical Combination of Grammar: Sentences in natural language are composed of grammatical units at various levels in a hierarchical manner, where higher-level grammatical units are combined with lower-level grammatical units according to certain rules, such as words being composed of morphemes. This hierarchical combination characteristic of natural language sentences can be reflected through grammatical analysis.

[0004] 2. Grammatical Units: In natural language sentences, any segment of language that can be replaced in a certain position within a combination is a grammatical unit. The largest grammatical unit is the sentence, and other grammatical units, from largest to smallest, are: phrase, word, and morpheme.

[0005] 3. Language Structure Data: Language structure data refers to corpora that have undergone syntactic or semantic analysis and present the analysis results in a tree or graph structure. This mainly includes syntactic structure tree corpora, syntactic dependency tree corpora, syntactic dependency graph corpora, semantic structure tree corpora, and semantic dependency graph corpora. This invention primarily focuses on the rapid retrieval of syntactic structure tree corpora. Syntactic structure tree corpora refer to tree-structured corpora formed after syntactic structure analysis of sentences, such as phrase structure syntactic analysis, that mark the properties and functions of each grammatical unit in the sentence and reflect the hierarchical grammatical relationships between grammatical units.

[0006] 4. Index: An index is a pre-created storage structure based on the target information content to accelerate information retrieval; it is a common retrieval optimization technique. For large-scale corpus systems, indexes can significantly improve corpus query efficiency and enhance the concurrency capabilities of the corpus retrieval system. Indexing language data involves extracting the objects to be retrieved from the corpus, building an index on them, and constructing a relationship between the objects to be retrieved and the sentences, paragraphs, or documents containing those objects, organizing them into a specific structure to support rapid querying of large-scale language data.

[0007] 5. Index Unit: During indexing, an index unit refers to a node used to locate the indexed content. During retrieval, it refers to a unit parsed from the atomic search expression used for comparison with content in the index database; it is also called a retrieval unit. An index unit can be a character, word, or grammatical marker. The index content corresponding to the index unit stores the offset of the sentence containing the index unit in the corpus, the position of the index unit within the sentence, and related information about the index unit. During retrieval, the corresponding index content is found through the index unit. The index content allows locating the index unit's position and related information in the corpus, reconstructing the corpus content, completing the comparison, and obtaining the retrieval results.

[0008] Index units reveal the content or formal features of the corpus that are relevant to retrieval. Therefore, it is necessary to select appropriate index units based on the specific circumstances of the indexed data and the user's retrieval needs, and equip them with suitable indexes. The design of corpus index units should be based on corpus features and guided by retrieval requirements. Features that are frequently retrieved, and combinations of features that improve the accuracy and efficiency of knowledge extraction, should be prioritized for indexing, and related index units should be constructed accordingly. This aims to achieve more efficient corpus retrieval performance by maximizing the use of limited resources.

[0009] Typical indexing units include characters and words. Raw corpora often use characters as indexing units, independent of word segmentation, making implementation relatively easy; retrieval is limited to searching by character or combinations of characters. Word-segmented and part-of-speech-tagged corpora typically use words and part-of-speech tags as indexing units to support word and part-of-speech retrieval. For syntactic structure tree corpora, which contain characters, words, phrases, part-of-speech tags, attribute tags, and grammatical structural relationships between different language units, the indexing units should be adjusted accordingly to support retrieval of these contents and relationships.

[0010] Meanwhile, with the deepening of corpus linguistics research, higher demands are being placed on the accuracy and functional diversity of corpus retrieval. Furthermore, the development of natural language processing technology has made the acquisition of large-scale language structure data a realistic possibility. Based on this, this invention, considering the characteristics of language structure data and examining the retrieval needs of such data, designs the following basic index units: combinations of part-of-speech and the character immediately preceding the left end of the word; combinations of part-of-speech and the character immediately preceding the right end of the word; and combinations of syntactic function attribute markers and the last character of the syntactic unit. In addition to these three basic index units, five types of composite index units are also designed, including: part-of-speech index units, syntactic function attribute marker index units, and character index units. These composite index units reuse the index data of the basic index units to support fast queries using part-of-speech, syntactic markers, and characters, without generating additional index data, effectively compressing the size of the index database.

[0011] 6. Retrieval Language: The retrieval language is a crucial component of a corpus system. While corpus indexing enables rapid searching of large linguistic datasets, the retrieval language carries the retrieval functionality provided by the corpus retrieval system. For a corpus retrieval system, during the computer's retrieval operation, the system can only provide the required results if the search input matches the system's internal data feature identifiers. The accuracy of the results depends not only on the degree of matching between the search input and the data feature identifiers but also, to a large extent, on the descriptive power of the search input specifications—that is, the descriptive power of the retrieval language. Complex retrieval formula designs can support powerful retrieval functions but can burden users in terms of learning and use. For example, if the retrieval system uses regular expressions for querying, although the statements are standard and powerful, they are difficult to understand, require a greater learning cost, and are prone to errors. This invention defines a custom retrieval language that supports arbitrary assembly of retrieval units, can describe complex linguistic phenomena, and strikes a balance between retrieval functionality and ease of use.

[0012] 7. With the deepening of corpus linguistics research, higher demands are being placed on corpus retrieval. Traditional keyword indexes or keyword and part-of-speech indexes can only provide word and part-of-speech retrieval, without considering the syntactic structure information of language data. The resulting retrieval results contain a large number of ambiguous structures and have low accuracy. At the same time, the rapid development of natural language processing technology in recent years has made it possible to obtain large-scale language structure data. Summary of the Invention

[0013] This invention provides a rapid retrieval method for large-scale language structure data, solving the problem of corpus retrieval. The technical solution is as follows:

[0014] A rapid retrieval method for large datasets of language structures includes the following steps:

[0015] S1: Based on the type of language structure data, determine the basic index unit to be constructed and the index content corresponding to the index unit. The basic index unit includes the part of speech and the last character combination of the word immediately to the left of the part of speech, the first character combination of the part of speech and the word immediately to the right of the part of speech, the syntactic property function tag, and the last character combination of the syntactic unit.

[0016] S2: Design a structured retrieval language to support writing retrieval expressions and realize the query of structured data. The retrieval expression written in the structured retrieval language consists of three parts: query object, constraints, and functional operation.

[0017] S3: During retrieval, the user-input retrieval expression is parsed to extract the retrieval units corresponding to the index units and the relationships between the retrieval units. The query obtains the results of each retrieval unit by post-order traversal of a singly linked binary tree and merges them according to the relationships between the parsed retrieval units. The process includes setting constraints, querying, performing functional operations on the query results, and outputting the retrieval results.

[0018] Furthermore, in step S1, the part-of-speech tag and the first character combination of the right-hand neighboring word, the part-of-speech tag and the last character combination of the left-hand neighboring word, the syntactic property function tag, and the last character combination of the syntactic unit are formally represented as: |POS_HZ, HZ_POS|, $TAG_HZ; where POS represents the part-of-speech tag, HZ represents any single character, and TAG represents the syntactic property function tag. The symbols “|” and “$” are used to identify different index unit types, and “_” is used to separate different index elements.

[0019] The |POS_HZ is used to retrieve words whose right adjacent character is HZ and whose part of speech is POS. It supports specifying the string to the right of the part of speech, so as to extend the query to the right starting from the part of speech.

[0020] The HZ_POS| is used to retrieve words whose left adjacent character is HZ and whose part of speech is POS. It supports specifying the adjacent string to the left of the part of speech, so as to extend the query to the left starting from the part of speech.

[0021] The index unit represented by $TAG_HZ is "$attribute tag_block tail character", where the attribute tag includes block property function tag, clause tag and sentence tag. It is used to retrieve language units with the attribute tag TAG and the tail character HZ. The language unit where the attribute tag is located may be a phrase, block, clause or sentence. It supports specifying the suffix string of the language unit where the TAG is located and querying language units that match the suffix string.

[0022] Furthermore, step S1 also includes five types of composite index units: part-of-speech tags, syntactic property function tags, and characters, formally represented as: |POS, POS|, $TAG,<HZ、HZ> The symbols “<” and ">” are also used to identify different index unit types.

[0023] The index unit represented by |POS is “|part of speech”, which is used to retrieve words whose right adjacent character is any and whose part of speech is POS. It shares the same set of index data with the index unit of type “|POS_HZ”.

[0024] The index unit represented by POS| is “part-of-speech|”, which is used to retrieve words whose left adjacent character is any and whose part-of-speech is POS. It shares the same set of index data with the index unit of type “HZ_POS|”.

[0025] The index unit represented by the said $TAG is "$Attribute Tag", which is used to retrieve the language unit where the attribute tag is TAG, and shares the same set of index data with the index unit of the type "$TAG_HZ".

[0026] The index unit represented by <HZ is "<Character", which is used to retrieve the words starting with this character, and shares the same set of index data with the index unit of the type "|POS_HZ".

[0027] The index unit represented by HZ> is "Character>", which is used to retrieve the words ending with this character, and shares the same set of index data with the index unit of the type "HZ_POS|".

[0028] Further, in step S2, the query object is a description of the basic retrieval content, denoted by Query, which consists of a Chinese character string or a word string, a词性标记符号 (positional tag symbol), a句法标记符号 (syntactic tag symbol), and some special symbols. The syntactic tag symbol refers to the syntactic annotation information of phrases, chunks, or higher-level language units in the syntactic structure tree corpus.

[0029] Further, in step S2, the restrictive conditions are divided into three categories: constraints on default variables, corpus interval constraints, and corpus meta-information constraints.

[0030] The limiting conditions for constraining the default variables include: default variables ($1, $2, $B, $E, $Q), comparison operators (>、<、=、!=), content restriction operators, length restriction operators, and output control operators, which are used to restrict the range, content, length, position of the limited object, and the output range of the retrieval formula.

[0031] The corpus interval constraint is used to set the corpus query interval of the retrieval formula, and set zero or more intervals.

[0032] The "meta-information" in the corpus meta-information constraint refers to the meta-information possessed by the documents constituting the corpus, or the position paragraph information of the text.

[0033] Further, in step S2, the types of functional operations include: instance retrieval, frequency statistics, and diachronic statistics. When the retrieval formula does not specify the window size or does not specify the type of functional operation, instance retrieval is performed by default and the entire sentence instance where the query object is located is returned.

[0034] It should be noted that some terms like "词性标记符号" and "句法标记符号" in the original text are not clearly defined in the context. Here, rough translations are provided for the overall understanding of the text structure. You may need to adjust them according to the specific definitions in your relevant field.Further, in step S3, the process of setting the constraint conditions is as follows: After receiving the retrieval expression, the corpus system first analyzes the query object Query, the constraint conditions, and the functional operations in the retrieval expression, and then parses the query object Query into a retrieval unit that can be directly searched in the index library. After the retrieval expression is parsed, if the retrieval expression contains conditional constraint statements, constraint conditions are set for the current retrieval.

[0035] Further, in step S3, the query is divided into a basic query and a combined query, which are completed by the GetAS function and the JoinAS function respectively; the GetAS function is used to locate and find the index data information corresponding to a retrieval unit in the index library, and the JoinAS function is used to merge the index data of two retrieval units according to the given relationship, and is executed in the path order of post-order traversal, and finally obtains the index data information that matches all retrieval units and conforms to the relationship between the retrieval units.

[0036] Further, in step S3, the functional operations are completed by three functions: Context, Freq, and Count, which respectively represent obtaining the instances of the retrieval results and their contexts, obtaining the instances of the retrieval results and their statistical frequencies, and obtaining the instances of the retrieval results and their historical statistical frequencies.

[0037] Further, in step S3, the output of the retrieval results is completed by functions such as Output and Save. Output means output through a network form, and Save means directly saving the retrieval results to a local file.

[0038] The fast retrieval method for the big data of language structures, considering the research needs of corpus linguistics and the characteristics of language data at the present stage, is a fast retrieval method that can utilize the morphological, syntactic, and structural information in language structure data. By designing reasonable index units and index contents, while controlling the scale of index data, the retrieval function of the corpus is ensured; through the design of the retrieval process, while controlling the retrieval time, the memory space occupied during retrieval is compressed; through the design of the retrieval language, while controlling the user's learning cost, the description ability of the retrieval language is ensured, and the retrieval function of the corpus system is exerted. Generally speaking, that is, by using the grammatical features of language structure data, index units are designed to improve the retrieval effect; the algorithm process during retrieval is designed to reduce the space-time consumption during big data retrieval; the retrieval language is designed to reduce the user's learning cost and ensure support for the retrieval function. Description of the Drawings

[0039] Figure 1 It is a tree diagram of the syntactic structure of an example sentence;

[0040] Figure 2 It is an example diagram of the index data organization form of "|d_认"

[0041] Figure 3 It is an example diagram of the indexing data organization form of "识_n|";

[0042] Figure 4 It is an example diagram of the indexing data organization form of "|d";

[0043] Figure 5 It is an example diagram of the indexing data organization form of "n|";

[0044] Figure 6 It is an example diagram of the indexing data organization form of "$NP-OBJ_响";

[0045] Figure 7 It is an example diagram of the indexing data organization form of "$NP-OBJ_响";

[0046] Figure 8 It is an example diagram of the indexing data organization form of "<认";

[0047] Figure 9 It is an example diagram of the indexing data organization form of "识>";

[0048] Figure 10 It is the diagram corresponding to the default variable in the retrieval formula "打击(n){len($1)=1}";

[0049] Figure 11 It is the retrieval flow chart;

[0050] Figure 12 It is the call flow chart of the basic query and the combined query;

[0051] Figure 13 It is the execution flow chart of the example script. Specific implementation manners

[0052] The fast retrieval method for the big data of the language structure includes the following steps:

[0053] S1: According to the type of the language structure data, determine the basic index unit to be constructed and the index content corresponding to the index unit;

[0054] If it is the corpus of word segmentation and词性标注 (positional tagging), use the combination of the词性 (positional) and the tail character of the word immediately to the left of the词性 (positional) as the basic index unit, and use the combination of the词性 (positional) and the head character of the word immediately to the right of the词性 (positional) as the basic index unit to establish an index library;

[0055] If it is the corpus of syntactic structure analysis, in addition to the above two basic index units, use the combination of the syntactic property function marker of the syntactic unit and the tail character of the syntactic unit as the basic index unit to establish an index library.

[0056] S2: During retrieval, the user-input retrieval expression is parsed to extract the retrieval units corresponding to the index units and the relationships between the retrieval units. The results of each retrieval unit are obtained by post-order traversal of a singly linked binary tree and then merged according to the relationships between the parsed retrieval units.

[0057] Based on the design of the index units and the query requirements of the language structure data, a structured retrieval language was designed. The retrieval language supports the assembly of arbitrary structured retrieval expressions using retrieval units and special symbols representing the relationships between retrieval units.

[0058] In this invention, the index unit design leverages the lexical and syntactic information inherent in the language data itself to fulfill the retrieval needs of multi-faceted language knowledge from language structure data. It utilizes three basic index units and five composite index units that share index data with the basic units to enable queries based on words, parts of speech, syntactic structures, and their arbitrary combinations. The single-linked binary tree post-order traversal retrieval process ensures query speed while minimizing memory usage during retrieval, thus supporting rapid retrieval of large amounts of language structure data. The structure retrieval language is designed to allow users to learn the retrieval language at a low cost and write standardized retrieval expressions that meet their retrieval needs. This will be described in detail below.

[0059] S1 Index Unit Design

[0060] For a syntactic structure tree corpus, the corpus contains characters, words, phrases, part-of-speech tags, syntactic attribute tags, and grammatical structural relationships between different language units. To enable fast querying of these corpus contents and formal features, this method designs three basic index unit types: a combination of the part of speech and the first character of the word immediately to the right of that part of speech; a combination of the part of speech and the last character of the word immediately to the left of that part of speech; and a combination of a syntactic property / function tag and the last character of the syntactic unit. These are formally represented as |POS_HZ, HZ_POS|, and $TAG_HZ, respectively. Here, POS represents a part-of-speech tag, HZ represents any single character (in languages ​​such as English, it represents a word), and TAG represents a syntactic property / function tag. The symbols “|” and “$” are used to identify different index unit types, and “_” is used to separate different index elements; neither has any substantive meaning. It is worth noting that these special symbols (“|” and “$”) used to identify index unit types appearing in the corpus content have been converted to their corresponding full-width characters. Therefore, to retrieve these characters from the corpus, the full-width form must be used for querying.

[0061] The design of the basic index units "|POS_HZ" and "HZ_POS|" reflects the working principle of the language model, namely that certain parts of speech are highly likely to appear before and after a word. For example, "I" is usually followed by a verb, in which case "I" often acts as the subject of the sentence, acting as the agent of the following verb. If the retrieval requirement is to obtain all predicate verbs with "I" as the agent, the target result can be quickly obtained through the atomic search "I_v|". The design of the two basic index unit types "|POS_HZ" and "HZ_POS|" takes into account both forward and backward sequence information of the language. "|POS_HZ" carries backward language information of the index unit, while "HZ_POS|" carries forward language information of the index unit, so as to make full use of the language sequence features.

[0062] The design of “$TAG_HZ” originates from the characteristic that in Chinese, noun phrases and predicate phrases other than verb complements mostly have their headwords at the end. It uses the syntactic property function markers of syntactic units (phrases are one type of syntactic unit) combined with the syntactic unit's tail character as the basic index unit, and the tail word of the syntactic unit as the index item. It distinguishes each type of syntactic property function marker according to the headword at the end of the syntactic unit, thereby enabling more accurate and faster location of the block structure to be retrieved and reducing noise processing work in subsequent research.

[0063] In addition to the three basic index units, there are five composite index units: part-of-speech tags, syntactic property function tags, and characters, formally represented as: |POS, POS|, $TAG,<HZ、HZ> The symbols “<” and ">” are also used to identify different index unit types, but they have no real meaning.

[0064] Five types of composite index units can support more retrieval functions. For example, retrieving the part-of-speech tag "v" can be achieved by finding all contiguously stored basic index units "|v_HZ", where HZ can be any character. Similarly, retrieving the syntactic property function tag "VP" can be achieved by finding all contiguous storage block ranges of basic index units "$VP_HZ". In the index database, basic index units of the same type are stored in a contiguously allocated storage space, while composite index units point to the address range of their corresponding basic index units. Therefore, composite index units share the index entries and inverted entries of the basic index units, which greatly compresses the size of the entire index data. Figure 1 As shown, taking the sentence from EXAMPLE 1 as an example. Original sentence: A preliminary understanding of the influence of the state of matter. Syntactic structure analysis results:

[0065] [ROOT[IP[VP-PRD[NULL-MOD[preliminary]][VP-PRD[understand]]][NP-OBJ[substance][state][of][influence]][w[x.]]]]

[0066] The following table shows the specific forms and functions of some index units taking EXAMPLE 1 as an example.

[0067] Table 1 Index Unit Example Table of Example Sentence EXAMPLE 1

[0068]

[0069] 1) |POS_HZ

[0070] The index unit represented is "|pos_ the first character of the word immediately to the right of this pos". This type of index unit can be used to retrieve words whose immediately right character is HZ and whose pos is POS. For example, the index unit "|d_recognize" retrieves adverb instances whose immediately right character is "recognize". The result examples are as follows:

[0071] <q> Preliminary< / q> recognize; <q> According to law< / q> determine; <q> Once< / q> determine;

[0072] <q> new< / q> determine; <q> already< / q> subscribe; <q> further< / q> recognize clearly;

[0073] <q> No< / q> think;

[0074] In the above examples, <q> and< / q> the bold content between is the query result of "|d_recognize", that is, an adverb instance. The words on the right side of the adverb all start with the character "recognize". These are listed here only to illustrate the context where the query result is located.

[0075] Inside the index library, the index data content of the |POS_HZ type index unit is the immediately adjacent string on the right side of the pos, and these strings are sorted in character order from left to right. Taking "|d_ recognize" as an example, its index data content and organization form are as Figure 2 shown. The index data content all starts from "recognize" and is sorted from left to right. Based on this content and organization form, this type of index unit can support the string on the right side of the specified pos, and realize the query extending to the right starting from the pos. For example, if it is specified that the string on the right side of the adverb d in the retrieval unit "|d_recognize" is "determine", the retrieval result examples at this time are as follows:

[0076] <q> Preliminary assessment< / q> ; <q> Legal determination< / q> ; <q> Once identified< / q> ;

[0077] <q> New determination< / q> ; <q> Confirmed< / q> ; <q> Self-identification< / q> ;

[0078] 2) HZ_POS|

[0079] The indexed unit represented is "the end character of the left adjacent word _ POS|". This type of indexed unit can be used to retrieve words where the left adjacent character is HZ and the POS is POS. For example, "识_n|" retrieves noun instances where the left adjacent character is "识". The result examples are as follows:

[0080] 辨识 <q> equipment< / q> ; 标识 <q> information< / q> ; 标识 <q> Signage< / q> ;

[0081] 标识 <q> system< / q> ; 常识 <q> Skill< / q> ; 认识 <q> enterprise< / q> ;

[0082] 知识 <q> Lecture< / q> ;......

[0083] <q> and< / q> The bold content between is the query result of "识_n|". The words on the left of the noun all end with the character "识".

[0084] The indexed data content of this type of indexed unit is the string immediately adjacent to the left of POS. These strings are sorted in character order from right to left. Taking "识_n|" as an example, its indexed data content and organization form are as Figure 3 shown. The indexed data content starts from "识" and is sorted from right to left. This type of indexed unit can support the string immediately adjacent to the left of the specified POS, enabling query extension to the left starting from the POS. For example, when specifying that the string to the left of the POS n in the retrieval unit "识_n|" is "充分认识", the retrieval result examples are as follows:

[0085] <q> Fully understand the city< / q> ; <q> Fully understand the enterprise< / q> ; <q> Fully understand food< / q> ; <q> Fully understand rural areas< / q> ; <q> Fully understand nature< / q> ; <q> Fully understand foreign trade< / q> ;......

[0086] 3) |POS

[0087] The indexed unit represented is "|POS". This type of indexed unit can be used to retrieve words where the right adjacent character is any character and the POS is POS. For example, the indexed unit "|d" retrieves adverb instances where the right adjacent character is any character. The result examples are as follows:

[0088] <q> real< / q> 实现;<q> must< / q> according to; <q> now< / q> propose; <q> again< / q> Processing; <q> not yet< / q> according to; <q> Comparison< / q> many;......

[0089] In the above query results using "|d", the right side of the adverb can be any character.

[0090] Within the index database, this type of index unit shares the same set of index data as the "|POS_HZ" type index unit. Taking "|d" as an example again, the relationship between this index unit and the "|POS_HZ" type index unit is as follows: Figure 4 As shown, the index unit "|d" points to all index units of type "|POS_HZ" with part-of-speech tag d. Therefore, the "|POS" type index unit also supports specifying the string immediately to the right of the part-of-speech tag, enabling queries to extend to the right starting from the part-of-speech tag.

[0091] 4) POS

[0092] The index unit is represented by "part of speech |". This type of index unit can be used to retrieve words whose left immediate contiguous character is any and whose part of speech is POS. For example, index unit "n|" retrieves noun instances whose left immediate contiguous character is any, as shown in the following example:

[0093] , <q> Dust storm< / q> ;of <q> overall< / q> ;Province <q> science and technology< / q> Business <q> Talent< / q> ;

[0094] wait <q> facility< / q> ; Advance <q> energy< / q> ;......

[0095] In the above query results using "n|", the left side of the noun can be any character.

[0096] In the index library, this type of index cell shares the same set of index data as the "HZ_POS|" type index cell. The relationship between index cell "n|" and the "HZ_POS|" type index cell is as follows: Figure 5 As shown, the index unit "n|" points to all index units of type "HZ_POS|" with part-of-speech tag n. The "POS|" type index unit can support specifying the string immediately to the left of the part-of-speech tag, enabling queries to extend to the left starting from the part-of-speech tag.

[0097] 5)$TAG_HZ

[0098] The indicated index unit is "$Attribute Marker_Tail Character", where the attribute marker includes chunk property function markers, clause markers, and sentence markers. This type of index unit is used to retrieve language units with an attribute marker of TAG and a tail character of HZ. The language unit where the attribute marker is located may be a phrase, chunk, clause, or sentence. For example, "$NP-OBJ_ring" retrieves noun object chunks with "ring" as the tail character. The result examples are as follows:

[0099] <q> Market impact< / q> ; <q> Significant impact< / q> ; <q> Significant impact< / q> ;

[0100] <q> Potential impacts of various situations on power operation< / q> ;

[0101] <q> Impact on enterprise production and operation activities< / q> ;

[0102] <q> The impact of rising prices on the lives of low-income groups< / q> ;......

[0103] Above <q> and< / q> The content between is all noun components and acts as the object in the sentence.

[0104] The index data content of this type of index unit is the language unit where the attribute marker is located. These language units will also be sorted from right to left in character order. Taking "$NP-OBJ_ring" as an example, the content and organization form of its index data are as Figure 6 shown. The index data content starts from "ring" and is sorted from right to left. The $TAG_HZ type index unit can support specifying the suffix string of the language unit where TAG is located and querying the language units that match the suffix string. For example, if the suffix string of the specified retrieval unit "$NP-OBJ_ring" is "adverse impact", the retrieval result examples are as follows:

[0105] <q> Significant adverse effects< / q> ; <q> Adverse effects of large reservoir capacity regulation on aquatic life< / q> ; <q> Adverse environmental impacts of the construction process< / q> ; <q> Adverse environmental impacts of the plan's implementation< / q> ; <q> The implementation of planned projects may be affected by meteorological disasters, adverse climate factors, and potential adverse impacts on the local climate.< / q> ; <q> Adverse effects< / q> ; <q> Already have adverse effects< / q> ;......

[0106] 6)$TAG

[0107] The indicated index unit is "$Attribute Marker". This type of index unit is used to retrieve the language unit where the attribute marker is TAG, which may be a phrase, chunk, clause, or sentence. For example, "$NP-OBJ" retrieves all noun object chunks in the corpus. The result examples are as follows:

[0108] <q> Creation of Information Technology-Based Science Popularization Products< / q> ; <q> Ice and Snow Literature Creation< / q> ;

[0109] <q> Straw mulching and conservation tillage< / q> ; <q> Selection of sports coaches< / q> ;

[0110] <q> Corresponding types and severity of penalties< / q> ; <q> Coverage of improved varieties in the production of Chinese medicinal herbs< / q> ;

[0111] <q> Funds review progress< / q> ;......

[0112] The query results of the above "$NP - OBJ" can be any language unit with the attribute tag NP - OBJ in the corpus.

[0113] In the index library, this type of index unit shares the same set of index data with the index units of the type "$TAG_HZ". The relationship between the index unit "$NP - OBJ" and the index units of the type "$TAG_HZ" is as Figure 7 shown. The index unit "$NP - OBJ" points to the index units of the type "$TAG_HZ" that end with any character and have the attribute tag "NP - OBJ". The index units of the type "$TAG" can support specifying the suffix string of this language unit.

[0114] 7) <HZ

[0115] Indicates that the index unit is "< character" and is used to retrieve words starting with this character. This type of index unit can be used to retrieve words that are immediately adjacent to any part of speech on the left and start with the character "HZ". For example, the index unit "<认" retrieves instances that are immediately adjacent to any part of speech on the left and start with the character "认". The result examples are as follows:

[0116] qualification <q> Determination< / q> ; shareholder <q> Subscription< / q> ; aptitude <q> Approval< / q> ;

[0117] authority <q> claim< / q> ; thought <q> know< / q> ; function <q> know< / q> ;

[0118] society <q> Agree< / q> ;.....

[0119] For the query results of the above "<认", the left side of the word starting with "认识" can be any part of speech.

[0120] In the BCC corpus index library, this type of index unit shares the same set of index data with the index units of the type "|POS_HZ". The relationship between the index unit "<认" and the index units of the type "|POS_HZ" is as Figure 8As shown in the figure. The index unit "<Ren" points to the index unit of the "|POS_HZ" type that is adjacent to the left and has any part of speech and the first character is "Ren". The "<HZ" type index unit can support specifying the string immediately adjacent to the right of this character, and can query by extending to the right starting from this character.

[0121] 8)HZ>

[0122] Indicates that the index unit is "Character>", and is used to retrieve words with this character at the end. This type of index unit can be used to retrieve words that are adjacent to the right and have any part of speech and the last character is "HZ". For example, the index unit "Shi>" retrieves examples where it is adjacent to the right and has any part of speech and the last character is "Shi". The result examples are as follows:

[0123] <q> know< / q> 、; <q> know< / q> Development; <q> know< / q> Measurement;

[0124] <q> know< / q> And; <q> know< / q> Strengthen; <q> know< / q> Implement;

[0125] <q> know< / q> Speed up;....

[0126] For the above query results of "Shi>", the words to the right of the words with "Shi" as the last character can have any part of speech.

[0127] In the index database, this type of index unit shares the same set of index data with the index unit of the "HZ_POS|" type. The relationship between the index unit "Shi>" and the index unit of the "HZ_POS|" type is as Figure 9 shown. The index unit "Shi>" points to the index unit of the "HZ_POS|" type that is adjacent to the right and has any part of speech and the last character is "Shi". The "HZ>" type index unit can support specifying the string immediately adjacent to the left of this character, and can query by extending to the left starting from this character.

[0128] S2 Structure Retrieval Language Design

[0129] To balance the relationship between user-friendliness and retrieval function, this method designs a structure retrieval language to support writing retrieval expressions and implement the query of language structure data.

[0130] For different corpus forms, the structure retrieval language needs to meet different retrieval requirements. For raw corpus, it needs to implement character or string retrieval; for segmented and词性标注语料 (corpus with word segmentation and part-of-speech tagging), it needs to support the retrieval of basic language units and language features such as characters, words, and parts of speech; for syntactic structure tree corpus, in addition to basic content features such as characters, words, and parts of speech, it also needs to meet the query of content and form features such as syntactic attributes and syntactic structure relationships.

[0131] A search query written in Structured Search Language consists of three parts: the query object, the constraints, and the operation, in the form of: Query{Condition1; Condition2; ...}Operation

[0132] Each search query must contain a Query, while Condition and Operation are optional. When you want to specify an Operation but do not want to add a Condition, you must retain the curly braces {} surrounding the Condition; in this case, the inside of the curly braces can be empty.

[0133] The following sections will explain the syntax rules and functions of the structured search language in three parts: query objects, restrictions, and functional conditions.

[0134] 1) Query object

[0135] The query object (Query) describes the basic search content and mainly consists of Chinese character strings (or word strings), part-of-speech tagging symbols, syntactic tagging symbols, and some special symbols. For example, to search for language instances of a numeral followed by a quantifier and then the string "apple", the query object can be written as: "mqapple", where m is the part-of-speech tag for the numeral and q is the part-of-speech tag for the quantifier.

[0136] Syntactic markers primarily refer to the syntactic annotation information of phrases, chunks, or higher-level language units in the syntactic structure tree corpus, such as the syntactic attribute markers VP and NP for phrases in the phrase structure annotation corpus. In the query, the description method for these syntactic attribute markers differs from that of part-of-speech tagging. These markers are written as "marker[internal description]", and the internal description method is consistent with the query's description rules. For example, to retrieve a verb followed immediately by a noun chunk, where the noun chunk consists of an adjective and the word "system," the search query could be written as: "v NP[a system]".

[0137] The query object section can also support some symbols with special meanings to express more complex matching patterns. See Table 2 for an explanation of special characters in the query.

[0138] Table 2. List of special characters in the query object Query

[0139]

[0140]

[0141] 2) Restrictions

[0142] The content within curly braces {} is the conditional statement. Condition is the restriction condition for the query object Query. Multiple restrictions are separated by semicolons ";".

[0143] The constraints can be divided into three categories: constraints on default variables, constraints on corpus intervals, and constraints on corpus metadata.

[0144] (1) Constraints of default variables

[0145] In a query, the portion enclosed in parentheses () is considered the qualified content. A query can only have two qualified elements, meaning only two parentheses can appear. The order in which the parentheses appear is used to represent the content, indicated by the dollar sign ($) and a sequence number, within the qualified statement. That is, $1 in a query represents the content enclosed in the first parenthesis, and $2 represents the content enclosed in the second parenthesis. In structured search languages, this is called a default variable.

[0146] In addition to $1 and $2, which are used to refer to the content qualifiers in the query, there are three other default variables: $Q, $B, and $E. $Q represents the entire query object, i.e., the query; $B represents the context to the left of $Q; and $E represents the context to the right of $Q. $Q, $B, and $E are independent of whether parentheses are used to qualify the query; each search query can use these three default variables in its conditional statements.

[0147] The constraints on default variables consist of: default variables ($1, $2, $B, $E, $Q), comparison operators (>, <, =, !=), content constraint operators, length constraint operators, output control characters, etc., and are used to restrict the scope, content, length, position of the constrained object and the output range of the search query.

[0148] For example, to search for "strike" followed by a monosyllabic noun, the search query could be written as: "strike(n) {len($1)=1}". Figure 10 This is a diagram illustrating the default variables in this search query. This search query can be used to study the linguistic phenomenon of "strike" followed by a monosyllabic noun. Table 3 shows the types of constraints supported by the query language.

[0149] Table 3. Limitation Condition Types

[0150]

[0151]

[0152] Note: The default variables for content and length limits in the table can be replaced by any one of $1, $2, $B, $E, and $Q.

[0153] (2) Corpus Interval Constraint

[0154] The corpus interval constraint is used to set the corpus query interval of the retrieval formula, and zero or more intervals can be set. The structured retrieval language provides two reserved keywords, AddLimit and ClearLimit, to set the constraints of the corpus retrieval interval. The interval indicators used for the constraints can be pre-exported from the index data through the corpus tool. The usage method of the corpus interval limit is in the form of: "AddLimit(lower bound index of the interval, upper bound index of the interval)", where the upper and lower bounds of the interval are integer values. Using "ClearLimit()" can cancel all corpus interval constraints. For example: To retrieve the statistical result of the occurrence of the word "development" in the People's Daily corpus in 1978, assuming that the upper and lower bound index intervals of this part of the corpus in the index data are [10,000, 50,000), the retrieval formula can be written as "development{AddLimit(10000,50000)}Freq".

[0155] (3) Corpus Meta-Information Constraint

[0156] The "meta-information" in the corpus meta-information constraint refers to the meta-information possessed by the documents constituting the corpus, such as the author, publication time, publishing time, etc., or the position and paragraph information of the text, such as the title, abstract, body text, etc. When the document attribute information of the language structure corpus is indexed, attribute constraint conditions can be set in the restriction statement to perform attribute constraints on the query object Query. For example: To retrieve the statistical result of the occurrence of the word "youth" in Lu Xun's works corpus in 1918, the retrieval formula can be written as "youth{YEAR=1918;AUTHOR=Lu Xun}Freq". For the language structure corpus, the attribute information indexed during the indexing stage can all be used as restriction conditions in the restriction statement.

[0157] 3) Functional Operations

[0158] The third part Operation of the retrieval formula is the functional operation, which is used to specify which type of operation will be performed based on the previous query object Query and restriction condition Condition. The types of functional operations are: instance retrieval, frequency statistics, and diachronic statistics, which are respectively implemented by the three keywords Context, Freq, and Count.

[0159] Instance Retrieval:

[0160] For example, to retrieve the instances of the disyllabic verb v and its context, the retrieval formula can be written as: (v){len($1)=2}Context.

[0161] When the function operation type is Context, the window size of the context can be specified through the default variable $WinSize. Continuing with the previous example, to specify a maximum window size of 10 words, the search query is: "(v){len($1)=2; $WinSize=10}Context".

[0162] When the search query does not specify a window size or a function operation type, it defaults to performing an instance search and returning the entire instance of the query object.

[0163] Frequency statistics:

[0164] For example, to count the frequency of all instances of disyllabic verbs, the search query can be written as: (v){len($1)=2}Freq.

[0165] Historical statistics:

[0166] Durational statistics are generally used in conjunction with the "corpus interval constraint" in the limitations conditions. Use the AddLimit keyword to set the duration interval for the search. If no interval is set, Count will function the same as Freq, that is, to count the frequency of the query object across all corpora; if a duration interval is set, Count will count the data in partitions according to the interval division.

[0167] For example, to retrieve statistical results of the occurrence of disyllabic verbs between 1990 and 1999 and between 2000 and 2010, assuming the upper and lower bounds of the 1990-1999 corpus are [100,000, 500,000) and the 2000-2010 corpus are [500,000, 1,000,000), the search expression can be written as:

[0168] "(v){AddLimit(100000,500000);AddLimit(500000,1000000)}Count"

[0169] The term "duration" here is not limited to the concept of time; it can also be divided into intervals based on author, document category, etc., as long as the upper and lower bounds of the corresponding corpus interval are known. If time is regarded as the vertical "duration" statistical indicator, then other dimensions, such as author, domain, and chapter, can be regarded as the horizontal "duration" statistical indicator. Count can be used to achieve "duration" partition statistics for different dimensions.

[0170] S3: Search Process Design

[0171] The overall retrieval process of each retrieval requirement described by the structure retrieval language in the corpus system is as follows: Figure 11 .

[0172] The overall search process consists of four parts: setting constraints, querying, function operations, and outputting search results. These are executed sequentially from beginning to end.

[0173] After receiving a search expression, the corpus system first analyzes the query object (Query), constraints, and functional operations within the search expression. Then, it parses the query object into search units that can be directly found in the index.

[0174] After the search query is parsed, if the query contains conditional statements, then constraints are set for the current search. Constraint settings are mainly accomplished by four functions: Condition, AddLimit, ClearLimit, and SetBase, each used to set different types of constraints. The SetBase function sets the result of the previous query as the base point (…). Figure 11 (The query process on the left) In the same script, the next query will be a second query based on the result of the previous query. When executing the second query, constraints can be applied again.

[0175] After setting the constraints, the search terms are parsed to extract the search units for querying. Queries are divided into basic queries and combined queries, performed by the GetAS and JoinAS functions respectively. The GetAS function locates and finds the corresponding index data for a search unit in the index database, while the JoinAS function merges the index data of two search units according to a given relationship. The entire query process follows... Figure 12 The dashed arrows shown in the "Flowchart of Basic and Combined Queries" execute the queries in the order of the post-order traversal path, ultimately obtaining index data information that matches all retrieval units and conforms to the relationships between retrieval units.

[0176] After obtaining all index data information matching all search units, the corresponding result format can be obtained according to the functional operation specified in the search query. The functional operation is completed by three functions: Context, Freq, and Count, which respectively represent obtaining the instance of the search result and its context, obtaining the instance of the search result and its statistical frequency, and obtaining the instance of the search result and its historical statistical frequency. Based on the index data information obtained during the query phase, the functional operation is executed to obtain the final search results.

[0177] Once the final search results are obtained, they can be output according to the specified output format. Output and Save functions are used to output the search results. Output indicates output via network, while Save indicates saving the search results directly to a local file.

[0178] Each script with corpus query functionality must include three steps: query, function operation, and output. The query part is used to implement basic atomic queries and their combined queries to obtain result information. The function operation obtains the target result format based on the result information, including instances and their context, instances and their statistical frequencies, and instances and their historical statistical frequencies. The output is used to output the final search results in a specified manner, which can be output via network, or the results can be directly saved to a file, or the search results can be further processed and analyzed.

[0179] The following is a detailed explanation using a specific search requirement as an example:

[0180] Search Requirement Description: Retrieve verb-object structures and output their context, where the verb is a complex verb containing an adverbial (NULL-MOD), and the object is a noun phrase ending with "work". The lengths of the verb and object are limited to 2 and 4, respectively.

[0181] Search query:

[0182] (VP-PRD[NULL-MOD[]VP-PRD[]])(NP-OBJ[*job]){len($1)=2;len($2)=4}Context

[0183] Example of search results:

[0184] I <q> Do it first< / q> <q> Preparation< / q> .

[0185] And my editor is like... <q> All< / q> <q> His work< / q> .

[0186] Now <q> I love it< / q> <q> I work< / q> I really don't want to lose it.

[0187] this <q> Search again< / q> <q> A job< / q> To earn wages.

[0188] Because of financial difficulties, I <q> Do< / q> <q> This job< / q> .

[0189] I went to Shanghai a few days ago. <q> Do< / q> <q> Filming< / q> .

[0190] Second <q> Dare to do< / q> <q> Mass work< / q> . ...

[0192] Example retrieval flowchart and data flow in corpus system: Figure 13As shown, the execution flow of the example script can be divided into four parts: setting constraints, querying, performing functional operations on the query results, and outputting the search results. These four steps are executed sequentially from bottom to top.

[0193] Setting constraints: This is done by the Condition function. The constraints set in this search query are: the lengths of the predicate and the object are 2 and 4 respectively. Once the constraints are set, they will be effective for the subsequent execution of the script.

[0194] Queries: mainly include basic queries and combined queries, which are completed by the GetAS function and the JoinAS function respectively. The GetAS and JoinAS function calls in the script form a single-branch binary tree. The entire query is executed according to the post-order traversal of the binary tree to obtain the results that satisfy all basic queries and combined queries.

[0195] Performing functional operations on the query results: The results obtained after the query process is completed are not the actual corpus content, but only contain some information that can be used to obtain the real corpus content. This step can obtain the instance and context of the result, or the instance of the result and its statistical frequency, or the historical statistical frequency of the instance of the result. The Context function in the example script is used to obtain the instance of the result and its context.

[0196] Output search results: This script outputs the search results generated by the function operation steps. The Output function is used to return the search results to the corpus search requester as a web service.

[0197] This invention achieves corpus retrieval functionality while controlling the scale of indexed data through the design of reasonable index units and index content; it reduces memory usage during retrieval while controlling retrieval time through the design of the retrieval process; and it ensures the descriptive power of the retrieval language while controlling user learning costs through the design of the retrieval language, thus maximizing the retrieval capabilities of the corpus system. In summary, it utilizes the syntactic features of language structure data to design index units to improve retrieval results; it designs the algorithm flow during retrieval to reduce time and space consumption during large-scale data retrieval; and it designs the retrieval language to reduce user learning costs while ensuring support for retrieval functions.

Claims

1. A fast retrieval method for language structure big data, comprising the following steps: S1: According to the type of language structure data, determine the basic index units to be constructed and the index content corresponding to the index units. The basic index units include the combination of word type and the tail character of the word immediately to the left of this word type, the combination of word type and the first character of the word immediately to the right of this word type, the syntactic property function marker and the tail character combination of this syntactic unit; The combination of word type and the first character of the word immediately to the right of this word type, the combination of word type and the tail character of the word immediately to the left of this word type, the syntactic property function marker and the tail character combination of this syntactic unit are formally represented as: |POS_HZ, HZ_POS|, $TAG_HZ; Where POS represents the word type marker, HZ represents any character, TAG represents the syntactic property function marker, the symbols "|" and "$" are used to identify different index unit types, and "_" is used to separate different index elements; It also includes five types of composite index units, namely: word type, syntactic property function marker and character, formally represented as: |POS, POS|, $TAG, <HZ, HZ>, and the symbols "<" and ">" are also used to identify different index unit types; The index unit represented by |POS is "|word type", which is used to retrieve words with any character immediately to the right and the word type being POS, and shares the same set of index data with the index unit of the "|POS_HZ" type; The index unit represented by POS| is "word type|", which is used to retrieve words with any character immediately to the left and the word type being POS, and shares the same set of index data with the index unit of the "HZ_POS|" type; The index unit represented by $TAG is "$attribute marker", which is used to retrieve the language unit where the attribute marker is TAG, and shares the same set of index data with the index unit of the "$TAG_HZ" type; The index unit represented by <HZ is "<character", which is used to retrieve words starting with this character, and shares the same set of index data with the index unit of the "|POS_HZ" type; The index unit represented by HZ> is "character>", which is used to retrieve words ending with this character, and shares the same set of index data with the index unit of the "HZ_POS|" type; S2: Design a structure retrieval language to support writing retrieval expressions and implement the query of language structure data. The retrieval expression written in the structure retrieval language consists of three parts: query object, restriction condition and function operation; S3: During retrieval, parse the retrieval expression input by the user, parse out the retrieval units corresponding to the index units and the relationships between the retrieval units. The query obtains the results of each retrieval unit in the post-order traversal manner of a single-chain binary tree and merges them according to the relationships between the parsed retrieval units. The process includes setting constraint conditions, querying, performing function operations on the query results, and outputting the retrieval results.

2. The method for rapid retrieval of large amounts of language structure data according to claim 1, characterized in that: The |POS_HZ is used to retrieve words with the character immediately to the right being HZ and the word type being POS, and supports specifying the string on the right side of the word type, so as to implement querying by extending to the right starting from the word type; The HZ_POS| is used to retrieve words whose left adjacent character is HZ and whose part of speech is POS. It supports specifying the adjacent string to the left of the part of speech, so as to extend the query to the left starting from the part of speech. The index unit represented by $TAG_HZ is "$attribute tag_block tail character", where the attribute tag includes block property function tag, clause tag and sentence tag. It is used to retrieve language units with the attribute tag TAG and the tail character HZ. The language unit where the attribute tag is located may be a phrase, block, clause or sentence. It supports specifying the suffix string of the language unit where the TAG is located and querying language units that match the suffix string.

3. The method for rapid retrieval of large amounts of language structure data according to claim 1, characterized in that: In step S2, the query object is a description of the basic search content, represented by Query, which consists of Chinese character strings or word strings, part-of-speech tagging symbols, syntactic tagging symbols, and some special symbols. The syntactic tagging symbols refer to the syntactic annotation information of phrases, chunks, or higher-level language units in the syntactic structure tree corpus.

4. The method for rapid retrieval of large amounts of language structure data according to claim 1, characterized in that: In step S2, the constraints are divided into three categories: constraints on default variables, corpus interval constraints, and corpus metadata constraints. The constraints on the default variables include: default variables ($1, $2, $B, $E, $Q), comparison operators (>, <, =, !=), content constraint operators, length constraint operators, and output control characters, which are used to restrict the scope, content, length, position of the restricted object and the output range of the search query; The corpus interval constraint is used to set the corpus query interval for retrieval, and can set 0 or more intervals; The "meta-information" in the corpus meta-information constraint refers to the meta-information of the documents that constitute the corpus, or the positional paragraph information of the text.

5. The method for rapid retrieval of large amounts of language structure data according to claim 1, characterized in that: In step S2, the types of functional operations include: instance retrieval, frequency statistics, and historical statistics. When the search query does not specify the window size or the type of functional operation, instance retrieval is performed by default and the whole sentence instance where the query object is located is returned.

6. The method for rapid retrieval of large amounts of language structure data according to claim 1, characterized in that: In step S3, the process of setting constraints is as follows: After receiving the search expression, the corpus system first analyzes the query object, constraints, and functional operations in the search expression, and then parses the query object into a search unit that can be directly searched in the index. After the search expression is parsed, if the search expression contains conditional statements, then constraints are set for the current search.

7. The method for rapid retrieval of large amounts of language structure data according to claim 1, characterized in that: In step S3, the query is divided into a basic query and a combined query, which are completed by the GetAS function and the JoinAS function, respectively. The GetAS function is used to locate and find the index data information corresponding to a retrieval unit in the index library. The JoinAS function is used to merge the index data of two retrieval units according to the given relationship, and executes them in the order of the subsequent traversal path, so as to finally obtain the index data information that matches all retrieval units and conforms to the relationship between the retrieval units.

Images