A Trie tree structure, creation method, word segmentation method and system for a scanning pen

Abstract

The present invention provides a Trie tree structure, creation method, word segmentation method and system for a scanning pen. By redesigning the structure of the Trie tree, each child node of the Trie tree only occupies 14 bytes of space, thereby greatly reducing the overall space occupied by the Trie tree, and simplifying the operation logic of the AC automaton, so that it can quickly match the characters required to be matched. When performing a word segmentation operation, it is only necessary to determine whether the next child node exists, and directly locate the target node according to the obtained remainder, without traversing the entire Trie tree or completely searching multiple branches, which greatly saves matching time and improves word segmentation efficiency. In addition, through the Trie tree structure and the improved AC automaton provided by the present invention, the Trie tree can be saved in the form of a file. On this basis, word segmentation can still be achieved at a faster speed, improving the efficiency and accuracy of word segmentation of small embedded devices.

Description

Technical Field

[0001] The present invention relates to the field of computers, and in particular to a Trie tree structure, a creation method, a word segmentation method and a system for a scanning pen. Background Art

[0002] The Trie tree structure used by AC automata, also known as a dictionary tree or prefix tree, is a special data structure designed for string processing. In existing AC automata, the Trie tree is used to store multiple pattern strings (i.e., strings that need to be matched) and perform efficient multi-pattern matching on this basis. Trie trees are currently widely used in character matching, especially word segmentation technology. The basic idea of ​​​​the Trie tree is to use the common prefix of the string to reduce the query time overhead, thereby achieving the purpose of improving efficiency. Each node in the Trie tree represents a prefix in a string. All the characters on the path from the root node to a certain node are connected to form the string corresponding to the node.

[0003] Trie trees are widely used in English. Trie trees are a tree data structure in which each node represents a character or character sequence in a string. In English, since words often have common prefixes, Trie trees can significantly reduce the waste of storage space. Therefore, using Trie trees to search for English words can achieve fast matching, accurate word segmentation, and support automatic completion. The space occupied by Trie trees is not very large.

[0004] In the field of Chinese word segmentation, the Trie tree has a fatal flaw. Chinese characters and words often do not have an order, and there is no common prefix between words. If a Trie tree is designed in the Chinese field, the structure of the Trie tree will be extremely complex, with a large number of branches, resulting in a huge amount of space occupied. At present, there are mature solutions for Chinese word segmentation, such as dictionary matching-based methods and statistical model-based methods. However, these solutions often require a large amount of memory and CPU frequency. In order to maintain a faster running speed and higher accuracy, it is usually necessary to set a larger memory for the device and save the Trie tree in the memory to solve the technical problem that the Trie tree is too large and the search speed is too slow.

[0005] However, the above solution is not applicable to all fields. When it is necessary to use the Trie tree on small devices (such as scanning pens, reading pens, etc.), due to device limitations or cost considerations, it is impossible to configure a larger memory and a higher CPU frequency, and speed and accuracy can only be sacrificed. Especially in embedded devices such as scanning pens, the running memory is usually less than 100K, and the CPU frequency is 300Mhz, which is difficult to meet the needs of fast word segmentation.

[0006] Therefore, a new Trie tree structure and word segmentation method are urgently needed to meet the word segmentation needs of small embedded devices. Summary of the invention

[0007] The present invention proposes a Trie tree structure, a creation method, a word segmentation method and a system for a scanning pen, so as to solve the problem that the existing Trie tree structure is complex and occupies a large space.

[0008] In order to achieve the above object, the present invention provides a Trie tree structure for a scanning pen, comprising a root node and a child node, wherein the root node has first structure data, and the child node has second structure data;

[0009] The first structure data includes: a sub-node number field and a sub-node index field;

[0010] The second structure data includes: a coding field, an end identification field, a fail index field, a collision field, a child node number field, and a child node index field.

[0011] Furthermore, the encoding field includes 3 bytes of Chinese UTF-8 encoding;

[0012] The end mark field includes a 1-byte mark indicating whether the node is ended.

[0013] The fail index field includes a 3-byte fail index value;

[0014] The collision field includes a 2-byte hash collision next node position;

[0015] The subnode number field includes the number of subnodes of 2 bytes;

[0016] The child node index field includes a 3-byte first child node index value.

[0017] On the other hand, the present invention provides a method for creating a Trie tree structure for a scanning pen as described above, the method comprising:

[0018] S101: Create the Trie tree in the memory, and add a root node and child nodes to the Trie tree;

[0019] S102: Update the mismatch pointers of all the child nodes;

[0020] S103: recursively traverse the Trie tree according to the breadth, and update the second structure data of all the child nodes;

[0021] S104: recursively traverse the Trie tree according to the breadth, update the index of the child node and the first structure data of the root node, and write the Trie tree into a file in the order of the index.

[0022] Furthermore, the S101: creating the Trie tree in the memory and adding a root node and a child node to the Trie tree includes:

[0023] S1011: Create a blank Trie tree in memory;

[0024] S1012: Add one of the root nodes and at least one of the child nodes;

[0025] S1013: Acquire a matching word, where the matching word is the first word of the target word / target sentence in the dictionary;

[0026] S1014: Compare the matching word with at least one of the child nodes. If they do not match, execute S1015; if they match, execute S1016;

[0027] S1015: inserting a new child node, wherein the encoding field of the second structure data of the new child node corresponds to the matching word;

[0028] S1016: The matching child node is used as the first matching child node, the matching word is updated to the next word of the target word / target sentence, and the matching word is compared with the child node of the first matching child node. If there is no match, S1015 is executed; if there is a match, S1017 is executed;

[0029] S1017: Update the child node index field of the first matching child node, and determine whether the matching word is the last word of the target word / target sentence. If not, repeat S106; if so, update the end identifier field of the second structure data of the child node corresponding to the last word of the target word / target sentence, and continue to add other target words / target sentences of the dictionary until all the words of all the target words / target sentences have matching child nodes.

[0030] Furthermore, the S102: updating the mismatch pointers of all the child nodes includes:

[0031] Set any child node as the target mismatch node, find the parent node of the target mismatch node, obtain the parent node mismatch node, and find out whether the coding field of all child nodes of the parent node mismatch node is the same as that of the target mismatch node. If they are the same, the mismatch pointer of the target mismatch node points to the node with the same coding field; if they are different, the mismatch pointer points to the root node; and update the fail index field of the child node.

[0032] Furthermore, the step S103: recursively traversing the Trie tree according to the breadth to update the second structure data of all the child nodes includes:

[0033] Update the number of child nodes of all the child nodes, create an array with the same length as the number of child nodes, loop all child nodes of the current node, calculate the remainder of the value of the encoding field of the current child node and the number of child nodes, and place the child node in the hash index position corresponding to the remainder; if it is found that the hash index position is not empty, it indicates that there is a collision, set the collision field of the child node to other empty positions closest to the hash index position, and update the collision field.

[0034] On the other hand, the present invention provides a Chinese word segmentation method, the method adopts the Trie tree structure as described above, and the method comprises:

[0035] S201: Read the sentence to be segmented character by character;

[0036] S202: starting from the starting character of the sentence, using the optimized AC automaton and the Trie tree structure, searching for a matching word in the Trie tree structure;

[0037] S203: If a matching word is found, add it as a segmented word to the segmented word array, and continue the next round of search from the end position of the word;

[0038] S204: If no matching word is found, the current character is added to the word segmentation array as a separate word segmentation, and the search for the next character continues;

[0039] S205: Repeat steps S202-S204 until the entire sentence is traversed, and finally the word segmentation array of the sentence is obtained.

[0040] Furthermore, the step S202: starting from the starting character of the sentence, searching for matching words in the Trie tree structure using the optimized AC automaton and the Trie tree structure, includes:

[0041] S2021: Set the starting character as the character to be matched, search for a target child node that matches the character to be matched, if not found, execute S204, if the target child node exists, determine the end identification field of the target child node; if the end identification field is 1, execute S203; otherwise, execute S2022;

[0042] S2022: updating the character to be matched to the next character of the starting character, setting the target child node to the remainder child node, calculating the remainder according to the encoding field and the child node number field of the remainder child node; and searching for the next target child node according to the child node index field of the remainder child node;

[0043] S2023: Compare the encoding field of the next target child node with the character to be matched; if they match, execute S2024; if they do not match, execute S2025;

[0044] S2024: Determine the end identification field of the next target child node; if the end identification field is 1, execute S203; otherwise, execute S2022;

[0045] S2025: Determine the collision field of the next target child node. If the collision field is 0, execute S204; if the collision field is not 0, find the child node at the index position corresponding to the collision field, use it as the next target child node, and repeat S2023.

[0046] On the other hand, the present invention provides a Chinese analysis system, the system comprising a server, and the server executes the steps of the Chinese word segmentation method as described above.

[0047] On the other hand, the present invention provides a system for constructing a Trie tree structure, the system comprising a server, the server executing the steps of the method for creating the Trie tree structure for a scanning pen as described above.

[0048] Compared with the problems existing in the prior art, the present invention provides a Trie tree structure, creation method, word segmentation method and system for a scanning pen. By redesigning the structure of the Trie tree, each child node of the Trie tree only occupies 14 bytes of space, thereby greatly reducing the overall space occupied by the Trie tree, and simplifying the operation logic of the AC automaton, so that it can quickly match the characters required to be matched. When performing a word segmentation operation, it only needs to determine whether there is a next child node, and directly locate the target node according to the obtained remainder, without traversing the entire Trie tree or completely searching multiple branches, which greatly saves matching time and improves word segmentation efficiency. In addition, due to the Trie tree structure and the improved AC automaton provided by the present invention, the Trie tree can be saved in the form of a file. On this basis, word segmentation can still be achieved at a faster speed, improving the efficiency and accuracy of word segmentation of small embedded devices. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art are briefly introduced below.

[0050] Figure 1 A schematic diagram of a Trie tree root node structure for a scanning pen;

[0051] Figure 2 A schematic diagram of a Trie tree subnode structure for a scanning pen;

[0052] Figure 3 A flow chart of a method for creating a Trie tree structure for a scanning pen;

[0053] Figure 4 The figure is a flowchart of a Chinese word segmentation method. DETAILED DESCRIPTION

[0054] The following will be combined with the accompanying drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than all of the embodiments. Based on the embodiments in the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention. In order to explain the present invention in more detail, the following is a specific description of a Trie tree structure, creation method, word segmentation method and system for a scanning pen provided by the present invention in combination with the accompanying drawings.

[0055] Embedded devices refer to those special computer systems that are application-centric, based on computer technology, with tailorable software and hardware to meet the strict requirements of application systems on functions, reliability, cost, volume, and power consumption. They are usually composed of specific functional modules and computer control modules, mainly composed of embedded microprocessors, peripheral devices, embedded operating systems, and application software.

[0056] As an embedded device, the scanning pen is small, light and easy to carry. Users can perform scanning operations anytime and anywhere without being restricted by fixed locations or large devices. Scanning pens usually have fast scanning capabilities and can complete the scanning of large amounts of text in a short period of time, improving work efficiency. At the same time, they can also support instant translation, OCR recognition and other functions, further improving the speed and accuracy of information processing. However, the realization of the above functions requires word segmentation technology.

[0057] Word segmentation, also known as text segmentation or word segmentation, is a basic and important technical link in natural language processing (NLP). It refers to the process of recombining continuous word sequences into word sequences according to certain specifications. In natural language, words are the smallest meaningful language components that can act independently, but in languages ​​such as Chinese, there are no obvious separators between words (such as spaces in English), so word segmentation technology is needed to identify word boundaries in text.

[0058] Trie tree, also known as dictionary tree, prefix tree or key tree, is a data structure specially designed for fast retrieval of elements in a string collection. Trie tree is a tree data structure in which each node represents a character in a string, and the path (the sequence of characters represented by the edge from the root node to a node) represents a string. Trie tree effectively organizes and stores data by sharing common prefixes, thereby achieving higher efficiency in search, insertion and deletion operations. Trie tree is often combined with AC automaton to achieve accurate word segmentation.

[0059] AC automaton is a multi-pattern matching algorithm based on Trie tree (dictionary tree) structure and KMP algorithm. It can search for multiple pattern strings (or "keywords") in a main text string at the same time and complete the search in linear time. AC automaton adds a fail pointer to the Trie tree, so that when the match fails, it can quickly jump to other branches in the Trie tree that may contain matching items, thereby avoiding repeated matches and improving search efficiency.

[0060] like Figure 1-2 As shown, the present invention provides a Trie tree structure for a scanning pen, including a root node (ROOT node) and a child node, wherein the root node has first structure data, and the child node has second structure data;

[0061] The first structure data includes: a sub-node number field and a sub-node index field;

[0062] The second structure data includes: a coding field, an end identification field, a fail index field, a collision field, a child node number field, and a child node index field.

[0063] Generally, in the commonly used English Trie tree and the existing Chinese Trie tree, the root node does not store any data. When using the Trie tree structure, it is necessary to traverse each branch structure of the Trie tree to achieve accurate word segmentation. However, this method has the disadvantages of slow search speed and heavy processor burden. The present invention provides a Trie tree structure for a scanning pen, including a root node and child nodes. The root node stores first structure data, which includes: a child node number field (such as Figure 1 child_count as shown), child node index fields (such as Figure 1 By setting the first structure data and combining the second structure data of the Trie tree child node of the present invention, the query words can be read quickly to achieve the purpose of fast word segmentation.

[0064] Specifically, the first structure data includes: a sub-node number field and a sub-node index field; the second structure data includes: a coding field (such as Figure 2 word shown), end mark field (such as Figure 2 is_end as shown), fail index fields (such as Figure 2 fail_index as shown), collision fields (such as Figure 2 next_collion as shown), the number of child nodes field (such as Figure 2 child_count as shown), child node index fields (such as Figure 2 It can be understood that, in the current embodiment, the first structure data only includes: a child node number field and a child node index field; the second structure data only includes: a coding field, an end identifier field, a fail index field, a collision field, a child node number field, and a child node index field; through the above settings, the space occupied by each child node is minimized, and the format is fixed, so that it can be quickly located when performing a word query. However, in other embodiments, the first structure data and the second structure data are not limited to this, and other fields can be set as needed to improve query efficiency.

[0065] Specifically, the subnode number field is used to indicate the number of subnodes of the node, which can be understood as the number of branches that may exist in the node. When the node data matches, it is necessary to further search for its subnodes, that is, to further search for its branches, and to improve the accuracy of word segmentation by querying multiple branches. The subnode index field is used to indicate the index position of the subnode of the current node, so as to quickly locate the position of the subnode; the encoding field is used to achieve data matching between the characters to be matched and the subnodes; since there are many encoding methods for Chinese, the encoding field can store the encoding data corresponding to any Chinese character, however, it can be understood that in the same Trie tree, the encoding method of Chinese should be unified. In this embodiment, it can store 3-byte Chinese utf-8 encoding, and compare the utf-8 encoding of Chinese characters to achieve character matching. The end identification field is used to indicate whether the child node is the end point of this word segmentation; when the end identification field is 1, it means that the child node is the end point of the branch, and its subsequent characters will not form a word or sentence with it. The child node can be used as the end point of this word segmentation, and the query operation is performed again on the next character. The fail index field includes the fail index value, which is used to indicate whether to jump to the child node of another branch for subsequent matching when the match fails; it can be understood that the fail index value corresponds to the mismatch pointer; when the match fails, the search process is guided to jump to the next possible matching position. The collision field is used to indicate whether the child node determined by the remainder has the same remainder as other child nodes. If so, it means that there is a collision. The data stored in the collision field is used to point to the position of the child node that needs to be found after the collision; it can be understood that when performing query matching, it is not only necessary to match the current child node, but also to query the child node with collision to ensure the comprehensiveness and accuracy of the match. The child node index field is used to indicate the location of the next child node. The index value can be used to quickly locate the next child node, thereby improving search efficiency.

[0066] Further, in the second structure data, specifically: the encoding field includes 3-byte Chinese utf-8 encoding; the end flag field includes 1-byte whether it is an end node flag; the fail index field includes 3-byte fail index value; the collision field includes 2-byte hash collision next node position; the child node number field includes 2-byte number of child nodes; the child node index field includes 3-byte index value of the first child node. Through the above settings, the space occupied by each child node is controlled to 14 bytes, greatly simplifying the structure of the child node and reducing the space occupied by the final Trie tree. Moreover, due to the existence of the above second data structure, the Trie tree of the present invention can be stored in a storage device in the form of a file, rather than being stored in memory. In an embedded device, such as a scanning pen device, the performance of the memory and the processor is usually limited. If the Trie tree is stored in memory, it will occupy a large amount of resources and cause the embedded device to run stuck. The Trie tree structure of the present invention is stored in the form of a file and read when needed. Through the seek method, it can be quickly searched without occupying too much memory resources. Moreover, due to the above structure setting of the child node, it does not need to traverse all branch structures during word query, greatly improving the query efficiency.

[0067] It should be noted that Figure 1-2 is only a schematic diagram of the root node (ROOT), one of the child nodes and its child nodes of the Trie tree of an embodiment of the present invention, and the data of each field is not detailed therein. For example, the data of fail_index is default. Figure 2 The 4 nodes in are 4 exemplary nodes. Among them, "中" corresponds to one of the child nodes, and "国", "间", and "介" respectively correspond to the 3 child nodes of the "中" node. Specifically, when the sentence to be matched is "...中介是一个...", first, according to the word segmentation method of the Trie tree structure of the present invention, the word "中" will be searched first. When the child node corresponding to the word "中" is found, it will be judged according to its respective fields. The judgment result is to locate the node at index 16. The word after the word "中" is the word "介", which corresponds to the node at index 16, and the is_end of this node shows 1. At this time, the word "中介" can be segmented. If the matched sentence is "...中间是一种烟...", since the word after the word "中" is the word "间", which does not match the data in the node at index 16. However, since its hash_index is 2, it can be seen that a hash collision occurs at this time, and it is necessary to find the position corresponding to the next_collion field, that is, to find the node at index 15 for matching. If the match is successful and is_end is 1, then "中间" is used as a word segment. Thus, it can be seen that Figure 2Tokenize the exemplary child node and the child nodes of this child node. It is not necessary to compare the three nodes of "country", "interval", and "medium" in sequence after matching the "zhong" character node.

[0068] It can be understood that the schematic diagram of the node structure in the figure is only used to intuitively display the node structure and to assist in understanding the creation method and tokenization method of the nodes, and does not constitute a limitation on the node structure. A complete trie tree has a huge branching structure, and the number of its branching structures directly determines the accuracy of its tokenization. However, for an overly large Trie tree, if the existing technology is used, the tokenization efficiency will be affected because in the existing technology, it is necessary to poll each child node under the child node in sequence; through the Trie tree structure and tokenization method of the present invention, based on the complex Trie tree, the position of the next child node can be accurately located for fast tokenization.

[0069] In one embodiment, the present invention provides a method for creating a Trie tree structure for a scanning pen as described above, as Figure 3 shown, the method includes:

[0070] S101: Create the Trie tree in memory, and add a root node and child nodes to the Trie tree;

[0071] S102: Update the mismatch pointers of all the child nodes;

[0072] S103: Traverse the Trie tree recursively in breadth-first order, and update the second structure data of all the child nodes;

[0073] S104: Traverse the Trie tree recursively in breadth-first order, update the index for the child nodes and the first structure data of the root node, and write the Trie tree to a file in the order of the index.

[0074] Due to the structural characteristics of the above-mentioned Trie tree, the existing creation method of the Trie tree cannot be used. In this embodiment, a method for creating the above-mentioned Trie tree structure is provided, by creating a Trie tree in a memory, adding a root node and a child node to it, the root node and the child node can be created according to resources such as dictionaries, lexicons, and ancient poems, and the specific number of the root node and the child node can be dynamically adjusted and selected as needed. After adding the root node and the child node, it is necessary to update the mismatch pointers of all the child nodes. Through the added mismatch pointers, the branch structure of the same prefix can be quickly found when searching for words, thereby improving the query efficiency. When creating the Trie tree structure, it is necessary to recursively traverse the Trie tree according to the breadth, update the second structure data of all the child nodes, and then update the index for the child node and the first structure data of the root node, so that the data of the root node and the child node are complete. It can be understood that the creation of the Trie tree can select its basis as needed, such as all or part of the materials such as the Xinhua Dictionary, the Idiom Dictionary, and the Three Hundred Tang Poems. The root node and child nodes are updated according to the selected data to make the data of the Trie tree complete so that the subsequent word search can proceed smoothly. The selected material determines the complexity of the structure of the Trie tree and the size of the space occupied.

[0075] After updating the index for the child node and the first structure data of the root node, the Trie tree is written into the file in the order of the index. It can be understood that the strategy adopted for the creation of the Trie tree structure in the present invention is to traverse the Trie tree in the memory, fill its data, and after the filling is completed, write the Trie tree into the file in the order of the index. By establishing the corresponding index and writing the Trie tree in the order of the index, when searching for words, the child node that needs to be matched can be quickly located through the index, thereby improving the query efficiency. And writing the Trie tree into the file does not occupy the limited memory of the embedded device, avoiding jamming when the device is used.

[0076] In one embodiment, the S101: creating the Trie tree in the memory and adding a root node and a child node to the Trie tree includes:

[0077] S1011: Create a blank Trie tree in memory; when creating the Trie tree, first create a blank Trie tree to fill in data later. It can be understood that there are multiple nodes in the blank Trie tree, but the nodes do not contain any data.

[0078] S1012: Add a root node and at least one child node; after creating a blank Trie tree, add a root node and at least one child node to it, so that when traversing the Trie tree later, its data can be quickly filled. It can be understood that a root node and at least one child node are only exemplary, and the number of root nodes and child nodes can be added as needed, and is not limited to this. For example, when the data covered by the created Trie tree is relatively simple, its node structure is correspondingly simple; when the data covered by the created Trie tree is relatively simple, for example, it covers a variety of data such as Xinhua Dictionary and Three Hundred Tang Poems at the same time, it can include more complex nodes and their branch structures. At this time, the number of its root nodes and child nodes can be expanded as needed.

[0079] S1013: Obtain matching words, which are the first words of the target words / target sentences in the dictionary; it is understandable that after adding the root node and the child nodes, the data of the nodes needs to be filled. At this time, the data target needs to be extracted, and this data target can be a word or sentence in a dictionary, ancient poetry, etc. Due to the characteristics of the tree structure of the Trie tree, the first word of the target word / target sentence needs to be extracted in the first filling step.

[0080] S1014: Compare the matching word with at least one of the child nodes. If there is no match, execute S1015; if there is a match, execute S1016. In this step, after setting the first word of the target word / target sentence as the matching word, match it with the data of the child node to determine whether the data in the child node matches the matching word. It can be understood that when creating the structure of the Trie tree, it is necessary to traverse the required filled data and match each word in each target word / target sentence to ensure the integrity of the final Trie tree. During the creation process, the matching word is constantly matched with the data in the node to determine whether the matching word updates the node data or whether a new child node needs to be added.

[0081] S1015: insert a new child node, the encoding field of the second structure data of the new child node corresponds to the matching word. It is understandable that when the matching word cannot match the data in the existing child node, the data in the existing Trie tree does not contain the matching word, then it is necessary to add a new child node according to the matching word, and the encoding field of the second structure data of the newly added child node corresponds to the matching word.

[0082] S1016: Use the matched child node as the first matched child node, update the matched word to the next word of the target word / target sentence, compare the matched word with the child node of the first matched child node, if there is no match, execute S1015; if there is a match, execute S1017. If there is a child node identical to the matched word during the matching operation on the matched word, it is necessary to query the next word of the target word / target sentence, then at this time, update the matched word to the next word of the target word / target sentence; since the current matched word and the previous matched word are the same target word / target sentence, the matched word at this time continues to be compared with the child node of the child node corresponding to the previous matched word, so as to meet the tree structure characteristics of the Trie tree, so as to form a branching structure of the tree.

[0083] S1017: Update the subnode index field of the first matching subnode, and determine whether the matching word is the last word of the target word / target sentence. If not, repeat S106; if so, update the end identifier field of the second structure data of the subnode corresponding to the last word of the target word / target sentence, and continue to add other target words / target sentences of the dictionary until all the words of all the target words / target sentences have matching subnodes. It can be understood that in the matching process, it is necessary to query and match each word of the target word / target sentence, and continuously update the second structure data of the subnode during the matching process to facilitate further updates of subsequent target words / target sentences, and add more branch structures to the Trie tree so that it can continuously use words or sentences with the same prefix to maximize the query speed in the subsequent word query process.

[0084] In one embodiment, the step S102: updating the mismatch pointers of all the child nodes includes:

[0085] Set any child node as the target mismatch node, find the parent node of the target mismatch node, obtain the parent node mismatch node, and find out whether the coding field of all child nodes of the parent node mismatch node is the same as that of the target mismatch node. If they are the same, the mismatch pointer of the target mismatch node points to the node with the same coding field; if they are different, the mismatch pointer points to the root node; and update the fail index field of the child node.

[0086] It is understandable that when a target word / target sentence is queried during the query process of the Trie tree, it is usually not just a query for one of the branches. In the query process, if a branch query fails, if the subsequent multiple branches are directly traversed, the query efficiency will be low. Therefore, in the process of creating the Trie tree, it is necessary to create a mismatch pointer for the child node. The so-called mismatch pointer is used to guide the search process to jump to the next possible matching position when the match fails; the construction of the mismatch pointer is one of the cores of the AC automaton algorithm. Through the update of the above-mentioned mismatch pointer, when a branch query fails, it can quickly and accurately jump to the next branch with the same prefix without traversing all the branches of the Trie tree. In this way, the query efficiency of the Trie tree can be improved, especially in the present invention, the Trie tree is not directly stored in the memory, so its query efficiency is particularly important.

[0087] In one embodiment, the S103: recursively traversing the Trie tree according to the breadth to update the second structure data of all the child nodes includes:

[0088] Update the number of child nodes of all the child nodes, create an array with the same length as the number of child nodes, loop all child nodes of the current node, calculate the remainder of the value of the encoding field of the current child node and the number of child nodes, and place the child node in the hash index position corresponding to the remainder; if it is found that the hash index position is not empty, it indicates that there is a collision, set the collision field of the child node to other empty positions closest to the hash index position, and update the collision field.

[0089] It can be understood that when creating a Trie tree, the strategy adopted by the present invention is to match and update at the same time; and when traversing the Trie tree, the number of child nodes of all child nodes is updated; because the Trie tree structure of the present invention corresponds to the created index, the corresponding position can be quickly located through the index. Therefore, in the process of traversing the Trie tree, it is necessary to calculate the remainder of the value of the encoding field of the current child node and the number of child nodes, and place the child node in the hash index position corresponding to the remainder; by calculating the remainder, combined with the index and its corresponding position, it can be realized that when searching, the corresponding position of the corresponding branch can be directly located without traversing all the child nodes of the branch, which greatly improves the query efficiency. However, due to the uncertainty of the target words saved, when creating the Trie tree, there may be multiple child nodes under the same branch with the same remainder, resulting in a collision. At this time, it is only necessary to set the colliding child nodes to adjacent empty child nodes and update the collision field of the child nodes, so that when the query fails during the query process, the next node position when the collision occurs can be quickly located, so that the query process can accurately determine the child nodes while completely covering all possible child nodes to avoid omissions, thereby improving the accuracy of the query.

[0090] On the other hand, the present invention provides a Chinese word segmentation method, the method adopts the Trie tree structure as described above, such as Figure 4 As shown, the method includes:

[0091] S201: Read the sentence to be segmented character by character;

[0092] S202: starting from the starting character of the sentence, using the optimized AC automaton and the Trie tree structure, searching for a matching word in the Trie tree structure;

[0093] S203: If a matching word is found, add it as a segmented word to the segmented word array, and continue the next round of search from the end position of the word;

[0094] S204: If no matching word is found, the current character is added to the word segmentation array as a separate word segmentation, and the search for the next character continues;

[0095] S205: Repeat steps S202-S204 until the entire sentence is traversed, and finally the word segmentation array of the sentence is obtained.

[0096] It can be understood that the Trie tree structure created in the above-mentioned embodiment of the present invention can be used in embedded devices, especially in scanning pens. Combined with the optimized AC automaton, fast and accurate word segmentation is achieved. In the word segmentation method, the sentence to be segmented is first extracted for character extraction, and the characters of the sentence are analyzed one by one, while the Trie tree is queried and the word segmentation is determined. Specifically, the starting character of the sentence is queried, and the optimized AC automaton and Trie tree structure are used to find the matching word in the Trie tree structure; if it is found, the next character of the starting character is queried, and the subnodes of the subnodes corresponding to the next character and the previous character and the subnodes after the mismatch pointer jump are queried until the subsequent characters have a query failure, then the characters before the query failure are used as a word segmentation; and the query of subsequent characters is continued. Accurate word segmentation is achieved by traversing and querying each character in the sentence in turn, and confirming the word segmentation according to the matching results of the Trie tree.

[0097] In one embodiment, the step S202: starting from the starting character of the sentence, using the optimized AC automaton and the Trie tree structure, searching for a matching word in the Trie tree structure includes:

[0098] S2021: Set the starting character to the character to be matched, and search for the target child node that matches the character to be matched. If not found, execute S204. If the target child node exists, determine the end identification field of the target child node; if the end identification field is 1, execute S203; otherwise, execute S2022. It can be understood that when a character is queried, it is found that there is no child node that matches it, which means that it cannot form a word with other characters in this Trie tree. Therefore, it can be used as a separate word segment, and other subsequent characters can be queried. When there is a child node that matches it, although it is proved that it exists in the Trie tree, in the process of creating the Trie tree, the entire Trie tree is traversed and the second structure data of the child node is updated. There is an end identification field in the second structure data, which is used to determine whether the node can end. Therefore, when searching and matching characters, even if the corresponding child node is found, it is not necessary to further match its child node backwards; when the end identification field is 1, it proves that it will not form a word or sentence with other subsequent characters. At this time, it can be used as the result of this word segmentation and continue to match other characters.

[0099] S2022: Update the character to be matched to the next character of the starting character, set the target child node to the remainder child node, calculate the remainder according to the encoding field of the remainder child node and the child node number field; and find the next target child node according to the child node index field of the remainder child node. It can be understood that due to the existence of the unique Trie tree structure of the present invention, it is not necessary to match all branches of the Trie tree when matching; in this embodiment, when matching the next character, only the remainder needs to be calculated, and the specific position of the next child node can be quickly located according to the calculated remainder and index field, and other child nodes of the branch do not need to be matched, which greatly improves the matching efficiency.

[0100] S2023: Compare the encoding field of the next target child node with the character to be matched; if they match, execute S2024; if they do not match, execute S2025. It can be understood that when the encoding field of the next target child node matches the character to be matched, it is necessary to query its end mark field, and determine whether to end the word segmentation according to the end mark field and enter the query of the next character.

[0101] S2024: Determine the end identification field of the next target child node; if the end identification field is 1, execute S203; otherwise, execute S2022. It can be understood that if the end identification field is 1, the current word segmentation ends and the query of the next character begins. If the end identification field is not 1, it means that the character may have a next child node that can form a word or sentence with it, so at this time, execute S2022.

[0102] S2025: Determine the collision field of the next target child node. If the collision field is 0, execute S204; if the collision field is not 0, find the child node at the index position corresponding to the collision field, use it as the next target child node, and repeat S2023. It can be understood that when the encoding field of the next target child node does not match the character to be matched, it does not necessarily mean that it cannot form a word or sentence with other words. Due to the characteristics of the Trie tree structure of the present invention, a collision may occur because the remainders obtained during the query are the same. Therefore, when the match fails, it is necessary to query the collision field of the child node to determine whether there is a collision. When there is a collision, it is necessary to continue to search for the collision child node with the same remainder to ensure the accuracy of word segmentation.

[0103] In order to achieve the purpose of the present invention, the present invention also provides a Chinese analysis system, the system includes a server, and the server executes the steps of the Chinese word segmentation method as described above.

[0104] In order to achieve the purpose of the present invention, the present invention also provides a system for constructing a Trie tree structure, the system comprising a server, and the server executes the steps of the method for creating the Trie tree structure for a scanning pen as described above.

[0105] Through the description of the above implementation modes, those skilled in the art can clearly understand that the present invention can be implemented by means of software plus necessary general hardware, and of course by hardware, but in many cases the former is a better implementation mode. Based on such an understanding, the technical solution of the present invention is essentially or the part that contributes to the prior art can be embodied in the form of a software product, which is stored in a readable storage medium, such as a computer floppy disk, hard disk or optical disk, etc., and includes a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment of the present invention.

[0106] The above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some of the technical features therein by equivalents. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for creating a Trie tree structure for a scanning pen, the method include: S101: Create the Trie tree in the memory, and add a root node and child nodes to the Trie tree; S102: Update the mismatch pointers of all the child nodes; S103: recursively traverse the Trie tree according to the breadth, and update the second structure data of all the child nodes; S104: recursively traverse the Trie tree by breadth, update the index of the child node and the first structure data of the root node, and write the Trie tree to a file in the order of the index; The Trie tree structure includes a root node and a child node, the root node has first structure data, and the child node has second structure data; The first structure data includes: a subnode number field and a subnode index field; the second structure data includes: a coding field, an end identifier field, a fail index field, a collision field, a subnode number field and a subnode index field; The encoding field includes 3 bytes of Chinese UTF-8 encoding; the end mark field includes 1 byte of whether the node is ended; the fail index field includes 3 bytes of fail index value; the collision field includes 2 bytes of hash collision next node position; the sub-node number field includes 2 bytes of sub-node number; the sub-node index field includes 3 bytes of the first sub-node index value; The step S101: creating the Trie tree in the memory and adding a root node and child nodes to the Trie tree includes: S1011: Create a blank Trie tree in memory; S1012: Add one of the root nodes and at least one of the child nodes; S1013: Acquire a matching word, where the matching word is the first word of the target word / target sentence in the dictionary; S1014: Compare the matching word with at least one of the child nodes. If they do not match, execute S1015; if they match, execute S1016; S1015: inserting a new child node, wherein the encoding field of the second structure data of the new child node corresponds to the matching word; S1016: The matching child node is used as the first matching child node, the matching word is updated to the next word of the target word / target sentence, and the matching word is compared with the child node of the first matching child node. If there is no match, S1015 is executed; if there is a match, S1017 is executed; S1017: Update the child node index field of the first matching child node, and determine whether the matching word is the last word of the target word / target sentence. If not, repeat S106; if so, update the end identifier field of the second structure data of the child node corresponding to the last word of the target word / target sentence, and continue to add other target words / target sentences of the dictionary until all the words of all the target words / target sentences have matching child nodes.

2. The method according to claim 1, It is characterized in that The step S102: updating the mismatch pointers of all the child nodes includes: Set any child node as the target mismatch node, find the parent node of the target mismatch node, obtain the parent node mismatch node, and find out whether the coding field of all child nodes of the parent node mismatch node is the same as that of the target mismatch node. If they are the same, the mismatch pointer of the target mismatch node points to the node with the same coding field; if they are different, the mismatch pointer points to the root node; and update the fail index field of the child node.

3. The method according to claim 1, It is characterized in that The S103: recursively traversing the Trie tree according to the breadth to update the second structure data of all the child nodes includes: Update the number of child nodes of all the child nodes, create an array with the same length as the number of child nodes, loop all child nodes of the current node, calculate the remainder of the value of the encoding field of the current child node and the number of child nodes, and place the child node in the hash index position corresponding to the remainder; if it is found that the hash index position is not empty, it indicates that there is a collision, set the collision field of the child node to other empty positions closest to the hash index position, and update the collision field.

4. A Chinese word segmentation method, Features: The method comprises: S201: Read the sentence to be segmented character by character; S202: starting from the starting character of the sentence, using the optimized AC automaton and the Trie tree structure, searching for a matching word in the Trie tree structure; The Trie tree structure includes a root node and a child node, the root node has a first structure data, and the child node has a second structure data; the first structure data includes: a child node number field and a child node index field; the second structure data includes: a coding field, an end identifier field, a fail index field, a collision field, a child node number field, and a child node index field; The encoding field includes 3 bytes of Chinese UTF-8 encoding; the end mark field includes 1 byte of whether the node is ended; the fail index field includes 3 bytes of fail index value; the collision field includes 2 bytes of hash collision next node position; the sub-node number field includes 2 bytes of sub-node number; the sub-node index field includes 3 bytes of the first sub-node index value; S203: If a matching word is found, add it as a segmented word to the segmented word array, and continue the next round of search from the end position of the word; S204: If no matching word is found, the current character is added to the word segmentation array as a separate word segmentation, and the search for the next character continues; S205: Repeat steps S202-S204 until the entire sentence is traversed, and finally the word segmentation array of the sentence is obtained; Wherein, the step S202: starting from the starting character of the sentence, using the optimized AC automaton and the Trie tree structure, searching for matching words in the Trie tree structure includes: S2021: Set the starting character as the character to be matched, search for a target child node that matches the character to be matched, if not found, execute S204, if the target child node exists, determine the end identification field of the target child node; if the end identification field is 1, execute S203; otherwise, execute S2022; S2022: updating the character to be matched to the next character of the starting character, setting the target child node to the remainder child node, calculating the remainder according to the encoding field and the child node number field of the remainder child node; and searching for the next target child node according to the child node index field of the remainder child node; S2023: Compare the encoding field of the next target child node with the character to be matched; if they match, execute S2024; if they do not match, execute S2025; S2024: Determine the end identification field of the next target child node; if the end identification field is 1, execute S203; otherwise, execute S2022; S2025: Determine the collision field of the next target child node. If the collision field is 0, execute S204; if the collision field is not 0, find the child node at the index position corresponding to the collision field, use it as the next target child node, and repeat S2023.

5. A Chinese analysis system, comprising a server, wherein the server executes the method steps as claimed in claim 4.

6. A system for constructing a Trie tree structure, the system comprising a server, the server executing the method steps as claimed in any one of claims 1 to 3.

Images