Chinese word segmentation method, device and medium based on hash table and binary search tree
By using a combination of hash tables and binary search trees in Chinese word segmentation, the problem of word segmentation accuracy and inefficiency in traditional methods is solved, and a more efficient and accurate word segmentation process is achieved.
Patent Information
- Application Number
- CN202410847566.1
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-06-27
- Publication Date
- 2025-04-04
- Estimated Expiration
- 2044-06-27
AI Technical Summary
Traditional Chinese word segmentation methods have problems such as low word segmentation accuracy and low efficiency when processing large-scale text data.
The Chinese word segmentation method based on hash table and binary search tree is adopted to quickly locate vocabulary by combining the first word hash table and the vocabulary length hash table, and use the binary search tree to search for word segmentation to achieve word segmentation.
Improve word segmentation efficiency, reduce unnecessary search range, save memory space, and improve word segmentation accuracy.
Smart Images
Figure CN118734850B_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the technical field of text processing, and in particular to a Chinese word segmentation method, device and medium based on a hash table and a binary search tree. Background Art
[0002] With the rapid development of information technology, Chinese text processing plays an increasingly important role in natural language processing, information retrieval, data mining and other fields. As a basic task in Chinese text processing, the accuracy and efficiency of Chinese word segmentation directly affect the results of subsequent tasks. Traditional Chinese word segmentation methods mainly include rule-based word segmentation and statistics-based word segmentation, but these methods often have problems such as low word segmentation accuracy and low efficiency when processing large-scale text data. Summary of the invention
[0003] In order to solve the above problems, this application proposes a Chinese word segmentation method based on a hash table and a binary search tree, including:
[0004] Obtain a sentence to be segmented, determine the first character in the sentence to be segmented, and determine its index value in a preset first-character hash table according to a first unicode code corresponding to the first character;
[0005] Determine the string length corresponding to the sentence to be segmented, obtain the index pointing to the word length hash table corresponding to the first character according to the index value, and determine the word length hash table matching the first character according to the index and the string length; wherein the word length hash table is composed of words with the same first character and the same length;
[0006] Obtain a binary search tree corresponding to the vocabulary length hash table, and segment the sentence to be segmented according to a forward maximum matching principle and the binary search tree to obtain a segmentation result.
[0007] In one implementation of the present application, determining a vocabulary length hash table matching the first character according to the index and the string length specifically includes:
[0008] Subtract 1 from the length of the string to obtain the length of the hash array corresponding to the vocabulary length hash table;
[0009] According to the index, a vocabulary length hash table whose corresponding length matches the length of the hash array is obtained.
[0010] In one implementation of the present application, before obtaining the binary search tree corresponding to the vocabulary length hash table, the method further includes:
[0011] For any word length hash table consisting of words with the same first character and the same length, determine the second unicode code corresponding to the first subsequent character in the word after the first character, and arrange the words in ascending order of the second unicode code;
[0012] In the process of arranging the words according to the second unicode code, determining whether there is a designated word with the same second unicode code;
[0013] If so, for the next subsequent character after the first subsequent character in the specified vocabulary, arrange the specified vocabulary in ascending order of the third unicode code corresponding to the next subsequent character;
[0014] Repeat the above process of arranging the words according to the ascending order of the Unicode codes corresponding to the subsequent characters when the words have the same Unicode code, until all the words are arranged to obtain the corresponding word sequence;
[0015] A binary search tree is constructed according to the order of each word in the vocabulary sequence.
[0016] In one implementation of the present application, the sentence to be segmented is segmented according to the forward maximum matching principle and the binary search tree, specifically including:
[0017] According to the binary search method, the binary search tree is traversed to determine whether there is a word matching the sentence to be segmented in the binary search tree;
[0018] If not, the last character is deleted from the end of the sentence to be segmented in sequence, and the binary search tree is searched according to the remaining sentence to be segmented obtained after the deletion, until the target vocabulary is matched according to the binary search tree or the remaining sentence to be segmented includes only one character, and then the character deletion operation on the sentence to be segmented is stopped.
[0019] In one implementation of the present application, after stopping the character deletion operation on the sentence to be segmented, the method further includes:
[0020] When the target vocabulary is obtained by matching the binary search tree, the target vocabulary is extracted, and the binary search tree is continued to be searched for the remaining sentences to be segmented after the target vocabulary is extracted, until all characters in the remaining sentences to be segmented are matched, and the extracted target vocabulary is used as the segmentation result.
[0021] In one implementation of the present application, before determining the index value of the first character in a preset first-word hash table according to the first unicode code corresponding to the first character, the method further includes:
[0022] Obtain a preset Chinese dictionary, determine the unicode codes corresponding to the first characters of words in the Chinese dictionary, and filter out the maximum code and the minimum code from the unicode codes;
[0023] According to the maximum code and the minimum code, the hash table capacity corresponding to the first word hash table is determined, and according to the first character of the vocabulary corresponding to the unicode code, the first word hash table composed of the first characters of the vocabulary is constructed.
[0024] In one implementation of the present application, determining the index value of the first character in a preset first-character hash table according to the first unicode code corresponding to the first character specifically includes:
[0025] Convert the first unicode code corresponding to the first character and the minimum code into corresponding decimal codes, and determine the difference between the decimal code corresponding to the first unicode code and the decimal code corresponding to the minimum code;
[0026] The difference value and 1 are summed to obtain the index value of the first character in the preset first-character hash table.
[0027] In one implementation of the present application, the binary search tree is a balanced binary tree.
[0028] The embodiment of the present application provides a Chinese word segmentation device based on a hash table and a binary search tree, the device comprising:
[0029] at least one processor;
[0030] and, a memory communicatively coupled to the at least one processor;
[0031] The memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the Chinese word segmentation method based on a hash table and a binary search tree as described in any one of the above items.
[0032] The embodiment of the present application provides a non-volatile computer storage medium storing computer executable instructions, wherein the computer executable instructions are configured as follows:
[0033] A Chinese word segmentation method based on a hash table and a binary search tree as described in any of the above items.
[0034] The Chinese word segmentation method based on hash table and binary search tree proposed in this application can bring the following beneficial effects:
[0035] By combining the first-letter hash table and the word length hash table, the words related to the current character and string length can be quickly located. In addition, only words of the same length are matched during the word matching process, avoiding the retrieval of all words with the same first letter during the word matching process, reducing unnecessary search ranges, and thus improving word segmentation efficiency. Using a binary search tree for word search has a faster search speed, further improving word segmentation efficiency. Compared with the traditional word segmentation method based on complete matching, only the first character and length information of the word need to be stored, rather than the entire word, which can save a lot of memory space. BRIEF DESCRIPTION OF THE DRAWINGS
[0036] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:
[0037] Figure 1 A flowchart of a Chinese word segmentation method based on a hash table and a binary search tree provided in an embodiment of the present application;
[0038] Figure 2 A schematic diagram of a binary search tree provided in an embodiment of the present application;
[0039] Figure 3 A schematic diagram of the structure of a Chinese word segmentation device based on a hash table and a binary search tree provided in an embodiment of the present application. DETAILED DESCRIPTION
[0040] In order to make the purpose, technical solution and advantages of the present application clearer, the technical solution of the present application will be clearly and completely described below in combination with the specific embodiments of the present application and the corresponding drawings. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present application.
[0041] The technical solutions provided by various embodiments of the present application are described in detail below in conjunction with the accompanying drawings.
[0042] like Figure 1 As shown, the Chinese word segmentation method based on a hash table and a binary search tree provided in the embodiment of the present application includes:
[0043] S101: Obtain a sentence to be segmented, determine the first character in the sentence to be segmented, and determine its index value in a preset first-character hash table according to a first unicode code corresponding to the first character.
[0044] Chinese word segmentation is a common function in full-text search, natural language processing and other technologies. The efficient word segmentation efficiency provides important support for text processing effects. The embodiment of the present application provides a word segmenter applied to the server side, which can perform structured analysis and storage of data in a given dictionary library, so that the retrieval rate of the dictionary can be improved when performing word segmentation operations, and there is no restriction on the dictionary library, and a standard Chinese dictionary or an extended dictionary customized based on the standard dictionary can be used. The word segmenter supports dictionary data initialization, that is, the Chinese dictionary directory is saved according to a certain data structure, and its purpose is to support fast retrieval of the dictionary, thereby improving the word segmentation efficiency of the word segmenter. The data structure of the word segmenter specifically includes a first word hash table, a vocabulary length hash table, and a binary search tree. Dictionary data initialization can be completed before word segmentation, or it can be executed synchronously with the word segmentation logic after the word segmentation process begins.
[0045] This application takes the synchronous execution method as an example to explain the word segmentation logic of the word segmenter in detail. The sentence to be segmented can be obtained from files, databases, network resources and other channels. After obtaining the sentence to be segmented, the first character in the sentence to be segmented is determined, and according to the first unicode code corresponding to the first character, its index value in the preset first word hash table is determined.
[0046] The first-word hash table refers to converting the first words of all words in a given Chinese dictionary into unicode codes, and establishing an array based on the range length of the maximum code and the minimum code. The array elements are pointers or references, which point to the length hash table corresponding to the first-word words. Therefore, when constructing a first-word hash table, you first need to obtain a preset Chinese dictionary, determine the unicode code corresponding to the first character of the word in the Chinese dictionary, and filter out the maximum and minimum codes from the unicode codes. The maximum and minimum codes determine the hash table capacity corresponding to the first-word hash table. Assuming that the maximum code is N and the minimum code is M, the corresponding hash table capacity should be N-M+1. At this time, you can construct a first-word hash table s[N-M+1] consisting of the first characters of the words according to the unicode code.
[0047] The minimum encoding corresponds to the minimum array subscript 0 of the first-character hash table, and the maximum encoding corresponds to the maximum array subscript in the first-character hash table. For the remaining words, the corresponding position in the array of the first character of the word is calculated according to the difference between the Unicode encoding of the word and the minimum Unicode encoding of the first characters of the words. After determining the first character of the sentence to be segmented, it is also necessary to determine its position in the preset first-character hash table according to the first Unicode encoding corresponding to the first character, that is, the index value. First, convert the first Unicode encoding corresponding to the first character and the minimum encoding into the corresponding decimal encodings, and determine the difference between the decimal encoding corresponding to the first Unicode encoding and the decimal encoding corresponding to the minimum encoding. Then, sum the difference and 1 to obtain the index value of the first character in the preset first-character hash table. Assume that the first Unicode encoding of the first character of any word is x (x must satisfy M <= x <= N), then its position in the first-character hash table is s[x - M + 1]. For example, given the Unicode encodings of the first characters of all words in the dictionary, the minimum value is u4E00, which is 19968 in decimal, representing the Chinese character "一", and the maximum value is u9F9F, which is 40863 in decimal, representing the Chinese character "龟". Then the length of the first-character hash table S is 40863 - 19968 + 1 = 20896. Among them, S[0] represents the Chinese character "一", and S
[20895] represents the Chinese character "龟". Another word has the first character "墨", whose Unicode encoding is u58A8 and the decimal is 22696, then its corresponding index value in the first-character hash table is 22696 - 19968 + 1 = 2729, that is, S
[2729] .
[0048] S102: Determine the string length corresponding to the sentence to be segmented. According to the index value, obtain the index pointing to the vocabulary length hash table corresponding to the first character, and determine the vocabulary length hash table that matches the first character according to the index and the string length; wherein, the vocabulary length hash table is composed of words with the same first character and the same length.
[0049] The array element in the first-character hash table points to the hash table of word lengths corresponding to the first character. That is to say, after determining the index value, according to this index value, the index pointing to the hash table of word lengths corresponding to the first character can be obtained. The hash table of word lengths is composed of words with the same first character and the same length. For example, "one zhang", "one ding", "one shang", the above words have the same first character and the length of each word is 2, then they form a hash table of word lengths. The index only points to the entire hash table of word lengths corresponding to the first character. If you want to obtain the hash table of word lengths with a specific length, it also needs to be determined according to the actual word segmentation requirements. Determine the string length of the sentence to be segmented. According to the string length and the index, the hash table of word lengths that matches the first character can be determined. Specifically, for a sentence to be segmented with a string length of 1, that is, the Chinese character itself, there is no need to perform word segmentation. Therefore, the length of the hash array of the hash table of word lengths needs to be subtracted by 1 based on the string length. Assuming the string length is L, L - 1 obtained by subtracting 1 from the string length is the length of the hash array corresponding to the hash table of word lengths. In this way, after clarifying the length of the hash array, according to the index, the hash table of word lengths with a corresponding length matching the length of the hash array can be obtained. For example, if the sentence to be segmented is 10 characters long, at this time, a hash table of word lengths t with a hash array length of 9 should be established.
[0050] S103: Obtain the binary search tree corresponding to the hash table of word lengths, and segment the sentence to be segmented according to the forward maximum matching principle and the binary search tree to obtain the word segmentation result.
[0051] The array element in the hash table of word lengths is a pointer or reference, pointing to the root node of the binary search tree composed of words with the same first character and the same length. After determining the length of the hash array, according to the length of the hash array, the binary search tree corresponding to the hash table of word lengths can be obtained. For example, if the length of the hash array is 9, t[7] points to the binary search tree with a word length of 9. In this way, according to the forward maximum matching principle and the binary search tree, the sentence to be segmented is segmented to obtain the word segmentation result.
[0052] It should be noted that the binary search tree is an AVL tree, which is sorted according to the size of the unicode encoding. Specifically, for any vocabulary length hash table composed of words with the same first character and the same length, determine the second unicode encoding corresponding to the first successor character after the first character in the vocabulary, and arrange the words in ascending order of the second unicode encoding. During the process of arranging the words according to the second unicode encoding, determine whether there are specified words with the same second unicode encoding. If so, for the next successor character after the first successor character in the specified word, that is, the character in the third position, arrange the specified word in ascending order of the third unicode encoding corresponding to the next successor character. Repeat the above process of arranging words in ascending order of the unicode encoding corresponding to the successor character when there are words with the same unicode encoding. Once the situation where the unicode encodings corresponding to the characters in the same position are the same appears during the sorting process, continue to arrange the above words with the unicode encoding corresponding to the next character after the current character position until the arrangement of all words is completed, and the corresponding word sequence is obtained. Construct a binary search tree according to the order of each word in the word sequence. As Figure 2 shown in the schematic diagram of a binary search tree, for the vocabulary length hash table with the first character being '一', the length of its hash array is 2, and the first element in the vocabulary length hash table points to the root node of the binary search tree. The structure of the binary search tree is determined according to the unicode encoding corresponding to the second character in each word.
[0053] When segmenting the sentence to be segmented according to the binary search tree, it needs to be implemented based on the forward maximum matching principle. Specifically, according to the dichotomy method, traverse the binary search tree to determine whether there is a word in the binary search tree that matches the sentence to be segmented. If so, the data is matched and the segmentation of the current round ends. If not, the last character needs to be deleted sequentially from the end of the sentence to be segmented, and the remaining sentence to be segmented after deletion is retrieved in the binary search tree. If the target word is retrieved or there is only one character left in the remaining sentence to be segmented, stop the operation of deleting characters from the sentence to be segmented. If there is only one character left in the remaining sentence to be segmented, it is considered that the character is a single character that cannot form a word, and the segmentation can be stopped at this time. If the target word is retrieved according to the binary search tree, the target word needs to be extracted from the sentence to be segmented, and the remaining sentence to be segmented after the extraction of the target word is retrieved in the binary search tree until all characters in the remaining sentence to be segmented are matched, and then all the extracted target words can be used as the segmentation result.
[0054] In one embodiment, w[0…9] is a sentence to be segmented, which needs to be matched and retrieved. Take out the first character located at w[0], and calculate the position of the Chinese character in the first character hash table according to its unicode encoding, where sl=getUnicode(w[0])-19968+1. Calculate the length of w[0…9] to be 10, and then use the position t[10-2] in the word length hash table pointed to by s[sl]. After obtaining the binary search tree of length 10 pointed to by t[8], match w[0…9] with the corresponding vocabulary in the binary tree according to the binary search method. If the data is matched, the segmentation ends. If no data is matched, remove the last character in the sentence to be matched, and use the remaining sentence w[0…8] to continue matching in t[10-2]. Repeat this step until the vocabulary is matched or only one Chinese character remains in the sentence to be segmented. The matched words are recorded, and the remaining part of w[0…9] is intercepted and continued to match until all Chinese characters are matched, and the final word segmentation result can be obtained.
[0055] The above are embodiments of the method proposed in this application. Based on the same idea, some embodiments of this application also provide devices and non-volatile computer storage media corresponding to the above methods.
[0056] Figure 3 This is a schematic diagram of the structure of a Chinese word segmentation device based on a hash table and a binary search tree provided in an embodiment of the present application. Figure 3 As shown, including:
[0057] at least one processor; and,
[0058] at least one processor is communicatively connected to a memory; wherein,
[0059] The memory stores instructions executable by at least one processor, the instructions being executed by at least one processor to enable the at least one processor to:
[0060] Obtain a sentence to be segmented, determine the first character in the sentence to be segmented, and determine its index value in a preset first-character hash table according to a first unicode code corresponding to the first character;
[0061] Determine the string length corresponding to the sentence to be segmented, obtain the index pointing to the word length hash table corresponding to the first character according to the index value, and determine the word length hash table matching the first character according to the index and the string length; wherein the word length hash table is composed of words with the same first character and the same length;
[0062] Obtain a binary search tree corresponding to the vocabulary length hash table, and segment the sentence to be segmented according to the forward maximum matching principle and the binary search tree to obtain a segmentation result.
[0063] The embodiment of the present application provides a non-volatile computer storage medium storing computer executable instructions, wherein the computer executable instructions are configured as follows:
[0064] Obtain a sentence to be segmented, determine the first character in the sentence to be segmented, and determine its index value in a preset first-character hash table according to a first unicode code corresponding to the first character;
[0065] Determine the string length corresponding to the sentence to be segmented, obtain the index pointing to the word length hash table corresponding to the first character according to the index value, and determine the word length hash table matching the first character according to the index and the string length; wherein the word length hash table is composed of words with the same first character and the same length;
[0066] Obtain a binary search tree corresponding to the vocabulary length hash table, and segment the sentence to be segmented according to the forward maximum matching principle and the binary search tree to obtain a segmentation result.
[0067] Each embodiment in this application is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the device and medium embodiments, since they are basically similar to the method embodiments, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiments.
[0068] The devices and media provided in the embodiments of the present application correspond one-to-one to the methods. Therefore, the devices and media also have similar beneficial technical effects as the corresponding methods. Since the beneficial technical effects of the methods have been described in detail above, the beneficial technical effects of the devices and media will not be repeated here.
[0069] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented in one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that include computer-usable program code.
[0070] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.
[0071] These computer program instructions may also be stored in a computer readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture including an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.
[0072] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.
[0073] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.
[0074] The memory may include non-permanent storage in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. The memory is an example of a computer-readable medium.
[0075] Computer readable media include permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. Information can be computer readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic tape magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer readable media does not include temporary computer readable media (transitory media), such as modulated data signals and carrier waves.
[0076] It should also be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of more restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the process, method, commodity or device including the elements.
[0077] The above is only an embodiment of the present application and is not intended to limit the present application. For those skilled in the art, the present application may have various changes and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application should be included in the scope of the claims of the present application.
Claims
1. A Chinese word segmentation method based on a hash table and a binary search tree, characterized in that: The method comprises: Obtain a sentence to be segmented, determine the first character in the sentence to be segmented, and determine its index value in a preset first-character hash table according to a first unicode code corresponding to the first character; Determine the string length corresponding to the sentence to be segmented, obtain the index pointing to the word length hash table corresponding to the first character according to the index value, and determine the word length hash table matching the first character according to the index and the string length; wherein the word length hash table is composed of words with the same first character and the same length; the word length hash table is an array, and the array elements in the word length hash table are pointers or references, pointing to the root node of the binary search tree composed of words with the same length as the first character; Obtain a binary search tree corresponding to the vocabulary length hash table, and segment the sentence to be segmented according to the forward maximum matching principle and the binary search tree to obtain a segmentation result; According to the forward maximum matching principle and the binary search tree, the sentence to be segmented is segmented, specifically including: According to the binary search method, the binary search tree is traversed to determine whether there is a word matching the sentence to be segmented in the binary search tree; If not, the last character is deleted from the end of the sentence to be segmented in sequence, and the binary search tree is searched according to the remaining sentence to be segmented obtained after the deletion, until the target vocabulary is matched according to the binary search tree or the remaining sentence to be segmented includes only one character, and then the character deletion operation on the sentence to be segmented is stopped.
2. The Chinese word segmentation method based on hash table and binary search tree according to claim 1 is characterized in that: Determining a vocabulary length hash table matching the first character according to the index and the string length, specifically comprising: Subtract 1 from the length of the string to obtain the length of the hash array corresponding to the vocabulary length hash table; According to the index, a vocabulary length hash table whose corresponding length matches the length of the hash array is obtained.
3. The Chinese word segmentation method based on hash table and binary search tree according to claim 1 is characterized in that: Before obtaining the binary search tree corresponding to the vocabulary length hash table, the method further includes: For any word length hash table consisting of words with the same first character and the same length, determine the second unicode code corresponding to the first subsequent character in the word after the first character, and arrange the words in ascending order of the second unicode code; In the process of arranging the words according to the second unicode code, determining whether there is a designated word with the same second unicode code; If so, for the next subsequent character after the first subsequent character in the specified vocabulary, arrange the specified vocabulary in ascending order of the third unicode code corresponding to the next subsequent character; Repeat the above process of arranging the words according to the ascending order of the Unicode codes corresponding to the subsequent characters when the words have the same Unicode code, until all the words are arranged to obtain the corresponding word sequence; A binary search tree is constructed according to the order of each word in the word sequence.
4. The Chinese word segmentation method based on hash table and binary search tree according to claim 1 is characterized in that: After stopping the character deletion operation on the sentence to be segmented, the method further includes: When the target vocabulary is obtained by matching the binary search tree, the target vocabulary is extracted, and the binary search tree is continued to be searched for the remaining sentences to be segmented after the target vocabulary is extracted, until all characters in the remaining sentences to be segmented are matched, and the extracted target vocabulary is used as the segmentation result.
5. The Chinese word segmentation method based on hash table and binary search tree according to claim 1 is characterized in that: Before determining the index value of the first character in a preset first-word hash table according to the first unicode code corresponding to the first character, the method further includes: Obtain a preset Chinese dictionary, determine the unicode codes corresponding to the first characters of words in the Chinese dictionary, and filter out the maximum code and the minimum code from the unicode codes; According to the maximum code and the minimum code, the hash table capacity corresponding to the first word hash table is determined, and according to the first character of the vocabulary corresponding to the unicode code, the first word hash table composed of the first characters of the vocabulary is constructed.
6. The Chinese word segmentation method based on hash table and binary search tree according to claim 5 is characterized in that: According to the first unicode code corresponding to the first character, determining its index value in a preset first-word hash table specifically includes: Convert the first unicode code corresponding to the first character and the minimum code into corresponding decimal codes, and determine the difference between the decimal code corresponding to the first unicode code and the decimal code corresponding to the minimum code; The difference value and 1 are summed to obtain the index value of the first character in the preset first-character hash table.
7. The Chinese word segmentation method based on hash table and binary search tree according to claim 1 is characterized in that: The binary search tree is a balanced binary tree.
8. A Chinese word segmentation device based on a hash table and a binary search tree, characterized in that: The device comprises: at least one processor; and, a memory communicatively coupled to the at least one processor; The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the Chinese word segmentation method based on a hash table and a binary search tree as described in any one of claims 1-7.
9. A non-volatile computer storage medium storing computer executable instructions, characterized in that: The computer executable instructions are configured to: A Chinese word segmentation method based on a hash table and a binary search tree as described in any one of claims 1 to 7.