Language data processing method, device, computer equipment and storage medium

By sorting the length of the array of words to be identified in natural language processing and dynamic programming ideas, the problem of low efficiency in searching combined words is solved, and the method of quickly identifying combined words in large-scale data is realized, which improves the search efficiency and reduces complexity Spend.

CN118333053BActive Publication Date: 2024-11-12SUZHOU MAGIC VISION INTELLIGENT TECH CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202410758314.1
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2024-06-13
Publication Date
2024-11-12
Estimated Expiration
2044-06-13

AI Technical Summary

Technical Problem

In natural language processing, the process of finding combination words from an array composed of a large number of strings takes a long time. Existing methods such as recursion + brute force search and dictionary tree + depth-first search have problems of low efficiency and high complexity.

Method used

By obtaining the array of words to be identified, sorting them from small to large in length, and using dynamic programming ideas, establishing a target array to determine whether each word is composed of the previous words, and by updating the element values ​​in the target array, we determine whether the words are combined words, reduce time complexity.

Benefits of technology

It improves the efficiency of combined word search, reduces time complexity, and makes the process of finding combined word in large-scale data faster and more efficient.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118333053B_ABST
    Figure CN118333053B_ABST
Patent Text Reader

Abstract

The present invention relates to the technical field of natural language processing, and discloses a language data processing method, device, computer equipment and storage medium, the method comprising: obtaining a word array to be identified, wherein the elements in the word array are mutually different words; sorting the words in ascending order of length; for each of the words, determining whether it is composed of two or more words preceding the word; if so, determining that the word is a combination word. The present invention reduces the time complexity of finding all combination words in a given word list, and improves the search efficiency of combination words.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the technical field of natural language processing, and in particular to a language data processing method, device, computer equipment and storage medium. Background Art

[0002] Given a word array (if the word is a word, it is a string array) without repeated words (usually words), a word (string) composed of at least two shorter words (words) in the given word array (string array) is a compound word. For example: given a string array of ["cat", "dog"], then "dogcat" is a legal compound word, which is composed of "dog" + "cat". In addition, "catcat" is also a legal compound word, because it is composed of two "cat".

[0003] In Natural Language Processing (NLP), compound words have quite a few usage scenarios:

[0004] 1. Data preprocessing: In NLP, data preprocessing is a very important step that can affect the performance of the model. Compound words can be seen as a special type of data preprocessing, which is to identify and extract long words that are composed of other words. For example, if you are processing text data containing compound words (such as German), a compound word solution may be useful. By identifying and decomposing these compound words, you can better understand the meaning of the text.

[0005] (ii) Data augmentation: Data augmentation is a technique used to increase the generalization ability of a model by creating new data that does not exist in the training set. Combining words can be seen as a way of data augmentation, that is, creating new words by combining existing words. For example, if there is a text classification task, the diversity of the data can be increased by combining existing words into new words, which may help improve the performance and generalization ability of the model.

[0006] It is not difficult to imagine the following demand scenario: given a string array consisting of tens of thousands of strings, some of which are composite words and are composed entirely of at least two shorter strings in the given array. A method is needed to find all these composite words while ensuring a low time complexity (the lower the time complexity, the faster the execution speed). Summary of the invention

[0007] In view of this, the present invention provides a language data processing method, apparatus, computer device and storage medium to solve the problem that it takes a lot of time to find a combination word from a string array composed of a large number of strings.

[0008] In a first aspect, the present invention provides a language data processing method, the method comprising:

[0009] Obtain an array of words to be recognized, where the elements in the array are different words;

[0010] Sort the words in ascending order of length;

[0011] For each word, determine whether it is composed of two or more words preceding the word;

[0012] If so, it is determined that the word is a compound word.

[0013] The language data processing method provided in this embodiment is a simpler and more intuitive method for searching for combination words, which reduces the time complexity of finding all combination words in a given word list and improves the efficiency of searching for combination words.

[0014] In an optional implementation, for each word, determining whether it is composed of two or more words preceding the word includes:

[0015] For the target word currently being judged, a target array is created, and the length of the target array is 1 more than the length of the target word;

[0016] Set each element in the target array to its initial value;

[0017] Determine in order whether each subword of the target word is a word that precedes the target word; if so, update the value of the corresponding first target element in the target array;

[0018] According to the value of the element in the updated target array, it is determined whether the target word is composed of two or more words preceding the target word.

[0019] In an optional implementation, determining in order whether each subword of the target word is a word that precedes the target word; if so, updating the value of the corresponding first target element in the target array includes:

[0020] Obtain the first subword starting from the first word of the target word in order from short to long;

[0021] For the first sub-word obtained, determine whether it is a word that precedes the target word;

[0022] If yes, then update the value of the first target element corresponding to the first subword in the target array;

[0023] Acquire the second sub-word starting with the second character of the target word and the second sub-word starting with other characters of the target word in order from short to long; other characters are characters other than the first character and the second character of the target word;

[0024] For the obtained second sub-word, determine whether it is a word that precedes the target word;

[0025] If so, update the value of the first target element corresponding to the second sub-word in the target array.

[0026] In an optional implementation, updating the value of the first target element corresponding to the second sub-word in the target array includes:

[0027] When the first character of the second sub-word is the Mth character in the target word and the last character of the second sub-word is the Nth character in the target word, determining the first target element corresponding to the second sub-word in the target array to be the N+1th element;

[0028] Update the value of the N+1th element in the target array to the value of the Mth element plus the preset value.

[0029] In an optional implementation, the initial values ​​of the second element to the last element in the target array are the same;

[0030] According to the value of the element in the updated target array, determine whether the target word is composed of two or more words preceding the target word, including:

[0031] Get the value of the last element in the target array;

[0032] According to the value of the last element, determine whether the target word is composed of two or more words preceding the target word.

[0033] In an optional implementation, the initial value of the first element in the target array is 0, and the initial value of each of the other elements is -1; the preset value is 1;

[0034] According to the value of the last element, determine whether the target word is composed of two or more words preceding the target word, including:

[0035] Determine whether the value of the last element is greater than or equal to 2;

[0036] If so, it is determined that the target word is composed of two or more words preceding the target word.

[0037] In an optional implementation, obtaining the second sub-word starting with the second character of the target word and the second sub-word starting with other characters of the target word in order from short to long includes:

[0038] According to the latest value of the second target element in the target array, it is determined whether to obtain the second sub-word starting from the Lth word of the target word; the second target element is an element corresponding to the L-1th word of the target word, and the latest value of the second target element can indicate whether the sub-word composed of the first word to the L-1th word of the target word is the word before the target word, or is composed of two or more words before the target word; L is an integer greater than 1;

[0039] If yes, then obtain the second subword starting with the Lth word of the target word;

[0040] Otherwise, skip obtaining the second sub-word starting from the Lth word of the target word.

[0041] In a second aspect, the present invention provides a language data processing device, the device comprising:

[0042] An acquisition module is used to acquire a word array to be recognized, where the elements in the word array are different words;

[0043] A sorting module is used to sort the words in ascending order of length;

[0044] A judgment module is used to judge, for each word, whether it is composed of two or more words preceding the word;

[0045] The determination module is used to determine that a word is a compound word when the word is composed of two or more words preceding the word.

[0046] In a third aspect, the present invention provides a computer device, comprising: a memory and a processor, the memory and the processor being communicatively connected to each other, the memory storing computer instructions, and the processor executing the language data processing method of the first aspect or any corresponding embodiment thereof by executing the computer instructions.

[0047] In a fourth aspect, the present invention provides a computer-readable storage medium having computer instructions stored thereon, the computer instructions being used to enable a computer to execute the language data processing method of the first aspect or any corresponding embodiment thereof. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] In order to more clearly illustrate the specific implementation methods of the present invention or the technical solutions in the prior art, the drawings required for use in the specific implementation methods or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some implementation methods of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0049] Figure 1 is a flow chart of a language data processing method according to an embodiment of the present invention;

[0050] Figure 2 is a flow chart of another language data processing method according to an embodiment of the present invention;

[0051] Figure 3 is a flow chart of another language data processing method according to an embodiment of the present invention;

[0052] Figure 4 is a structural block diagram of a language data processing device according to an embodiment of the present invention;

[0053] Figure 5 It is a schematic diagram of the hardware structure of a computer device according to an embodiment of the present invention. DETAILED DESCRIPTION

[0054] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present invention.

[0055] Given an array of thousands of strings, some of which are composite words that are composed entirely of at least two shorter strings in the given array. Currently, there are several ways to find all the composite words in the above string array:

[0056] 1. Recursive + brute force search: for each string, traverse all other strings to see if the current string can be composed of other strings. This method runs very slowly, which is extremely time-consuming. It is only a feasible solution, but it has no application value.

[0057] Second, the dictionary tree + depth-first search, the core idea is to use the dictionary tree data structure to record the order of characters in all traversed strings, which can greatly reduce the occurrence of repeated searches compared to brute force search. However, most people do not understand the dictionary tree as a data structure. At the same time, this method is difficult to construct a depth-first search, and there are many details to consider, which makes it difficult to write code.

[0058] The embodiment of the present invention provides a language data processing method, which can improve the search efficiency of compound words.

[0059] According to an embodiment of the present invention, an embodiment of a language data processing method is provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of executable computer instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0060] In this embodiment, a language data processing method is provided, which can be used in a computer device. Figure 1 is a flow chart of a language data processing method according to an embodiment of the present invention. Figure 1 As shown, the process includes the following steps:

[0061] Step S101, obtaining a word array to be recognized, where the elements in the word array are mutually different words.

[0062] The word array can be in English, German, or Chinese, and there is no limitation here. For example, the word array can be words=["cat", "catsdog", "cats", "dog"].

[0063] Step S102, sorting the words in ascending order of length.

[0064] A compound word (word) is composed of multiple words whose length is shorter than the compound word. Assuming that the compound word is composed of two words A and B, if the corresponding words A and B can be found in the given word array, then it can be proved that word is a legal compound word. Therefore, the first step is to sort the words in the given word array from small to large according to their length.

[0065] For the word array in the above example, the order is "cat" -> "dog" -> "cats" -> "catsdog".

[0066] Step S103, for each word, determine whether it is composed of two or more words preceding the word.

[0067] Specifically, all the split words of a word can be listed separately, and then it is judged one by one whether the sub-words obtained by each split are words arranged in front of the word. The split words include sub-words obtained by splitting the word into two words, sub-words obtained by splitting the word into three words, and so on, and sub-words obtained by splitting the word into H words (H is equal to the number of words (number of characters) of the word). When judging, if one of the sub-words obtained by a split is judged not to be a word arranged in front of the word, then the other sub-words obtained by this split do not need to be judged whether they are words arranged in front of the word.

[0068] Step S104: when the word is composed of two or more words preceding the word, determine that the word is a compound word.

[0069] The language data processing method provided in this embodiment is a simpler and more intuitive method for searching for combination words, which reduces the time complexity of finding all combination words in a given word list and improves the efficiency of searching for combination words.

[0070] In this embodiment, a language data processing method is provided, which can be used in a computer device. Figure 2 is a flow chart of a language data processing method according to an embodiment of the present invention. Figure 2 As shown, the process includes the following steps:

[0071] Step S201, obtain a word array to be recognized, where the elements in the word array are different words. Figure 1 Step S101 of the illustrated embodiment will not be described in detail here.

[0072] For example, the words array is words=["cat", "catsdog", "cats", "dog"].

[0073] Step S202, sorting the words in ascending order of length.

[0074] After sorting the words in the above word array, we get: "cat" -> "dog" -> "cats" -> "catsdog".

[0075] Step S203, for each word, determine whether it is composed of two or more words preceding the word.

[0076] In an optional specific implementation, the above step S203, i.e., for each word, determining whether it is composed of two or more words preceding the word, includes:

[0077] Step S2031, for the target word currently being judged, a target array is established, and the length of the target array is 1 more than the length of the target word.

[0078] For example, the target word is catsdog in the above word array, and accordingly, a target array dp with a length of 8 is established.

[0079] Step S2032, setting each element in the target array to an initial value.

[0080] The target array dp corresponding to the target word "catsdog" is [0, -1, -1, -1, -1, -1, -1, -1].

[0081] Step S2033, determining in order whether each sub-word of the target word is a word that precedes the target word; if so, updating the value of the corresponding first target element in the target array.

[0082] Specifically, step S2033, determining in order whether each sub-word of the target word is a word that precedes the target word; if so, updating the value of the corresponding first target element in the target array, including:

[0083] Step S20331, obtaining the first sub-word starting from the first character of the target word in order from short to long;

[0084] The characters (which may be characters such as letters) contained in the first sub-word here and the second sub-word mentioned below are one character or multiple consecutive characters in the target word.

[0085] For the target word "catsdog", the first subwords starting from the first word are: "c", "ca", "cat", "cats", "catsd", "catsdo".

[0086] Step S20332, for the first sub-word obtained, determine whether it is a word that precedes the target word;

[0087] Step S20333: If yes, then update the value of the first target element corresponding to the first sub-word in the target array;

[0088] "cat" and "cats" in the first sub-word are both in front of the target word, so the fourth element dp[3] in the target array dp is updated to dp[0]+1=1, and the fifth element dp[4] is updated to dp[0]+1=1. The updated target array dp=[0, -1, -1, 1, 1, -1, -1, -1].

[0089] Step S20334, obtaining, in order from short to long, the second sub-word starting with the second character of the target word and the second sub-word starting with other characters of the target word; the other characters are characters other than the first character and the second character of the target word;

[0090] Specifically, the second sub-word starting with the second character of the target word and the second sub-word starting with other characters of the target word are obtained in order from short to long, including:

[0091] According to the latest value of the second target element in the target array, it is determined whether to obtain the second sub-word starting from the Lth word of the target word; the second target element is an element corresponding to the L-1th word of the target word, and the latest value of the second target element can indicate whether the sub-word composed of the first word to the L-1th word of the target word is the word before the target word, or is composed of two or more words before the target word; L is an integer greater than 1;

[0092] If yes, then obtain the second subword starting with the Lth word of the target word;

[0093] Otherwise, skip obtaining the second sub-word starting from the Lth word of the target word.

[0094] Here, after obtaining all the first sub-words starting with the first word in the target word, each time before replacing the next word as the initial word (i.e., the first word) to obtain the second sub-word, it is necessary to determine whether the sub-word composed of all the words (excluding the initial word) before the initial word in the target word is a word that is ranked before the target word, or whether it is composed of words that are ranked before the target word, based on the current latest element value in the target array. Only when it is determined that the sub-word composed of the words before the initial word in the target word is a word that is ranked before the target word, or is composed of words that are ranked before the target word, it is necessary to obtain the second sub-word starting with the initial word, otherwise it is not necessary to obtain the second sub-word starting with the initial word, which further reduces the amount of calculation, thereby further improving the efficiency of identifying all combined words. Note that here, if it is determined that the sub-word composed of the words before the initial word in the target word is not a word that is ranked before the target word, and is not composed of words that are ranked before the target word, it is necessary to skip obtaining the second sub-word starting with the initial word, or the target array cannot be updated based on the second sub-word starting with the initial word being a word that is ranked before the target word, otherwise it may cause an error in judging the combined word.

[0095] For example, for the target word "catsdog", after obtaining all the first subwords starting with the first word: "c", "ca", "cat", "cats", "catsd", "catsdo", and judging whether each subword is after the word in front of the target word, the updated target array is dp=[0, -1, -1, 1, 1, -1, -1, -1]. At this time, it can be determined whether it is necessary to continue to use "a" as the initial word to obtain the second subword according to whether the element dp[1] corresponding to "c" in the target array dp is -1 or 0. Here, dp[0]=0, dp[1]=-1, so the second subword with "a" as the initial word is skipped. Similarly, since dp[2]=-1, the second subword with "t" as the initial word is skipped. Since dp[3]=1, the second subword with "s" as the initial word is obtained. Similarly, since dp[4]= 1, the second subword with "d" as the initial word is obtained.

[0096] Step S20335, for the acquired second sub-word, determine whether it is a word that precedes the target word;

[0097] Step S20336: If yes, update the value of the first target element corresponding to the second sub-word in the target array.

[0098] Specifically, step S20336, i.e. updating the value of the first target element corresponding to the second sub-word in the target array, includes:

[0099] Step S203361, when the starting character of the second sub-word is the Mth character in the target word and the ending character of the second sub-word is the Nth character in the target word, determining that the first target element corresponding to the second sub-word in the target array is the N+1th element;

[0100] In this embodiment, the number of elements in the target array is 1 more than the number of characters in the target word, wherein the ranking of the first target element corresponding to the second sub-word in the target array is the ranking of the last character of the second sub-word in the target word plus 1. For example, if the last character of the second sub-word is ranked 3rd in the target word, then the corresponding first target element is ranked 4th in the target array. If the elements in the target array are numbered from 0, then it is dp[3].

[0101] Step S203362, update the value of the N+1th element in the target array to the value of the Mth element plus the preset value.

[0102] For example, for the target word "catsdog", according to the above, after obtaining all the first subwords starting from the first word: "c", "ca", "cat", "cats", "catsd", "catsdo", and judging whether each subword is after the word before the target word, the updated target array is dp=[0, -1, -1, 1, 1, -1, -1, -1]. Then, based on the second element dp[1]=-1 and the third element dp[2]=-1 in the target array, it is determined that it is not necessary to obtain the second subword with "a" as the initial word, nor is it necessary to obtain the second subword with "t" as the initial word. Based on the fourth element dp[3]= 1 in the target array, "s" is obtained as the second subword of the initial word: "s", "sd", "sdo", "sdog". However, since these subwords are not the words before the target word, the target array is still dp=[0, -1, -1,1, 1, -1, -1, -1]. Based on the 5th element dp[4]= 1 in the target array, get the second subword with "d" as the initial word: "d", "do", "dog". At this time, since "dog" is the word before the target word, according to the above rules, the starting word of "dog" is the M=5th word in the target word, and the last word of "dog" is the N=7th word in the target word. Update the value dp[7] of the (7+1)th element in the target array to the value dp[4]=1 of the M=5th element plus the preset value 1, that is, dp[7]=2. So the updated target array is dp=[0, -1, -1, 1, 1, -1, -1, 2].

[0103] Step S2034: Determine whether the target word is composed of two or more words preceding the target word according to the value of the element in the updated target array.

[0104] Specifically, the initial values ​​of the second to last elements in the target array are the same;

[0105] Step S2034, i.e., determining whether the target word is composed of two or more words preceding the target word according to the value of the element in the updated target array, includes:

[0106] Get the value of the last element in the target array;

[0107] According to the value of the last element, determine whether the target word is composed of two or more words preceding the target word.

[0108] Specifically, the initial value of the first element in the target array is 0, and the initial value of each other element is -1; the preset value is 1;

[0109] According to the value of the last element, determine whether the target word is composed of two or more words preceding the target word, including:

[0110] Determine whether the value of the last element is greater than or equal to 2;

[0111] If so, it is determined that the target word is composed of two or more words preceding the target word.

[0112] According to the above, for the target word "catsdog", based on the sixth element dp[5]= -1 in the target array, it is determined that "o" does not need to be obtained as the second subword of the initial word, and based on the seventh element dp[6]=-1 in the target array, it is determined that "g" does not need to be obtained as the second subword of the initial word. Therefore, after determining whether all subwords are words preceding the target word, the final target array is dp=[0, -1, -1, 1, 1, -1, -1,2]. At this time, based on the last element dp[7]=2 of the target array, it is determined that the target word is composed of two or more words preceding the target word.

[0113] Step S204: When the word is composed of two or more words preceding the word, determine that the word is a compound word.

[0114] The language data processing method provided in this embodiment utilizes the idea of ​​dynamic programming, greatly improves the efficiency of combination word search, and reduces the difficulty of programming.

[0115] like Figure 3 As shown, the above language data processing method is illustrated below by taking the string array (i.e., word array, also called word array) of words=["cat", "catsdog", "cats", "dog"] as an example.

[0116] A compound word is composed of multiple strings whose length is less than word. Assume that word is composed of two strings A and B. If the corresponding strings A and B can be found in the given string array, then it can be proved that word is a legal compound word. Therefore, the first step is to sort the strings in the given string array from small to large according to their length. The logic here is: a compound word can only be composed of strings shorter than itself, so each search can exclude strings longer than itself, thereby narrowing the scope. After that, after determining whether a word is a compound word in order, store it in the hash table seen (a hash table is a basic data structure that can implement the search function very quickly). Each time you determine whether a substring of word (i.e., a subword, also called a substring) is the word that comes before word, you only need to determine whether it is in the hash table seen. After step 1, words are reordered to ["cat", "dog", "cats", "catsdog"].

[0117] Traverse the sorted words; for example, the current traversal order is "cat"->"dog"->"cats"->"catsdog". Additional note: If s="abcd", define s[i:j] to intercept the substring between subscripts i~j from s (subscripts start at 0), for example, s[1:3]="bc", s[0:1]="a". The specific traversal steps for each string in words are as follows:

[0118] Ⅰ. Initialize a dynamic programming array (i.e., target array) dp, whose length is the length of the string plus one. dp[i] indicates whether the substring ending with index i can be composed of other words. The first element in dp is initialized to 0, and the remaining elements are initialized to -1 to indicate that they have not been processed yet. Taking the first string "cat" as an example, dp should be initialized to dp=[0, -1, -1, -1].

[0119] Ⅱ. Traverse each character of the string: For each character of the string word (index is i, starting from 0), perform the following operations:

[0120] If the value of dp[i] is -1, it means that the substring has not been processed yet, so the inner loop is skipped and the next character is processed.

[0121] Iterate over all substrings starting at index i: For all substrings starting at index i (ending with index j), do the following:

[0122] If the substring word[i:j] is in the set of words seen, then update the value of dp[j] to dp[i] + 1, indicating that the substring ending with index j can be composed of other strings.

[0123] If the value of the last element of the dp array is greater than 1, it means that the original word can be composed of at least two other words and is a compound word, that is, a compound word is found.

[0124] Add the current word to the set of strings seen.

[0125] The following is a detailed explanation by traversing the three strings "cat", "cats" and "catsdog" in words=["cat", "dog", "cats", "catsdog"].

[0126] 1. Process the string "cat". At this time, the hash table seen={}, which is empty:

[0127] Ⅰ. Initialize the dynamic programming array dp = [0, -1, -1, -1]. The length is the length of "cat" plus one.

[0128] Ⅱ. For each character of "cat", index is i, traverse all substrings starting from index i (end index is j):

[0129] When i=0, j starts from i+1 and traverses backwards, making word[i:j] "c", "ca", "cat" respectively. Since these three substrings are not in seen, dp does not change.

[0130] When i=1, dp[i]=dp[1]=-1, skipping the inner loop traversal.

[0131] When i=2, dp[i]=dp[2]=-1, skipping the inner loop traversal.

[0132] III. Check the last element of dp, i.e. dp[3], whose value is -1, indicating that "cat" is not composed of other strings, so word="cat" is not a compound word.

[0133] IV. Add "cat" to seen, and get seen={"cat"}.

[0134] 2. The process of processing the string "dog" is similar to that of "cat", so it will not be repeated.

[0135] 3. Process the string "cats". Note that seen={"cat", "dog"}:

[0136] Ⅰ. Initialize the dynamic programming array dp = [0, -1, -1, -1, -1]. The length is the length of "cats" plus one.

[0137] II. For each character of "cats", index is i, traverse all substrings starting from index i (end index is j):

[0138] When i=0, j starts from i+1 and traverses backward, so that word[i:j] is "c", "ca", "cat", "cats". When traversing to "cat", because "cat" is in seen, update dp[3]=dp[0]+1=1. At this time, dp=[0, -1, -1, 1, -1].

[0139] When i=1, dp[i]=dp[1]=-1, skipping the inner loop traversal.

[0140] When i=2, dp[i]=dp[2]=-1, skipping the inner loop traversal.

[0141] When i=3, j starts from i+1 and traverses backwards, making word[i:j] equal to "s". Because this substring is not in seen, dp does not change.

[0142] III. Check the last element of dp, i.e. dp[4], whose value is -1, which means "cats" is not composed of other words. Therefore, word="cats" is not a compound word.

[0143] IV. Add "cats" to seen, and get seen={"cat", "dog", "cats"}.

[0144] 4. Process the string "catsdog". Note that seen={"cat", "dog", "cats"}:

[0145] Ⅰ. Initialize the dynamic programming array dp = [0, -1, -1, -1, -1, -1, -1, -1]. The length is the length of "catsdog" plus one.

[0146] Ⅱ. For each character of "catsdog", index is i, traverse all substrings starting from index i (end index is j):

[0147] When i=0, j starts from i+1 and traverses backward, making word[i:j] "c", "ca", "cat", "cats", "catsd", "catsdo", "catsdog". When traversing to "cat", because "cat" is in seen, update dp[3]=dp[0]+1=1. When traversing to "cats", because "cats" is also in seen, update dp[4]=dp[0]+1=1. At this time dp=[0, -1, -1, 1, 1, -1, -1, -1].

[0148] When i=1, dp[i]=dp[1]=-1, skipping the inner loop traversal.

[0149] When i=2, dp[i]=dp[2]=-1, skipping the inner loop traversal.

[0150] When i=3, j starts from i+1 and traverses backward, making word[i:j] "s", "sd", "sdo", "sdog". These substrings are not in seen, so dp does not change.

[0151] When i=4, j starts traversing from i+1 and goes backwards, making word[i:j] "d", "do", "dog" respectively. When traversing to "dog", because "dog" is in seen, update dp[7]=dp[4]+1=2. At this time, dp=[0, -1, -1,1, 1, -1, -1, 2].

[0152] When i=5, dp[i]=dp[5]=-1, skipping the inner loop traversal.

[0153] When i=6, dp[i]=dp[6]=-1, skipping the inner loop traversal.

[0154] III. Check the last element of dp, dp[7], whose value is 2, indicating that "cat" is composed of at least 2 other words, so word="catsdog" is a compound word.

[0155] IV. Add "catsdog" to seen, and get seen={"cat", "dog", "cats", "catsdog"}.

[0156] The language data processing method provided in the embodiment of the present invention is used to find compound words. A dynamic programming algorithm is added on the basis of sorting, which improves the search efficiency and reduces the difficulty of the algorithm. Compared with the recursive + brute force search method, the time complexity of the algorithm is greatly reduced, thereby greatly improving the search efficiency. Compared with the dictionary tree + depth-first search method, which has high requirements on the algorithm and data structure and the algorithm is not easy to understand, the algorithm of this embodiment is simpler and easier to understand, and the search efficiency is also improved.

[0157] In the present embodiment, a language data processing device is also provided, which is used to implement the above-mentioned embodiments and preferred implementation modes, and the descriptions that have been made will not be repeated. As used below, the term "module" can implement a combination of software and / or hardware of a predetermined function. Although the device described in the following embodiments is preferably implemented in software, the implementation of hardware, or a combination of software and hardware is also possible and conceived.

[0158] This embodiment provides a language data processing device, such as Figure 4 As shown, including:

[0159] The acquisition module 401 is used to acquire a word array to be recognized, where the elements in the word array are different words;

[0160] Sorting module 402, used to sort the words in ascending order of length;

[0161] The judging module 403 is used to judge, for each word, whether it is composed of two or more words preceding the word;

[0162] The determination module 404 is used to determine that a word is a compound word when the word is composed of two or more words preceding the word.

[0163] In some optional implementations, the determination module 403 includes:

[0164] An establishing unit is used to establish a target array for the target word currently being judged, wherein the length of the target array is 1 more than the length of the target word;

[0165] Initialization unit, used to set each element in the target array to an initial value;

[0166] An updating unit, used to determine in order whether each sub-word of the target word is a word that precedes the target word; if so, updating the value of the corresponding first target element in the target array;

[0167] The first determination unit is used to determine whether the target word is composed of two or more words preceding the target word according to the value of the element in the updated target array.

[0168] In some optional implementations, the updating unit includes:

[0169] A first sequential acquisition unit is used to sequentially acquire the first sub-word starting from the first character of the target word in order from short to long;

[0170] A first judging unit, configured to judge whether the first sub-word obtained is a word that precedes the target word;

[0171] A first updating unit, configured to update the value of a first target element corresponding to the first sub-word in the target array when the acquired first sub-word is a word that precedes the target word;

[0172] The second sequential acquisition unit is used to sequentially acquire, in order from short to long, a second sub-word starting with the second character of the target word and a second sub-word starting with other characters of the target word; the other characters are characters other than the first character and the second character of the target word;

[0173] A second judgment unit is used to judge whether the acquired second sub-word is a word that precedes the target word;

[0174] The second updating unit is configured to update the value of the first target element corresponding to the second sub-word in the target array when the acquired second sub-word is a word that precedes the target word.

[0175] In some optional implementations, the second updating unit includes:

[0176] a first target element determination unit, configured to determine that the first target element corresponding to the second sub-word in the target array is the N+1th element when the starting character of the second sub-word is the Mth character in the target word and the ending character of the second sub-word is the Nth character in the target word;

[0177] The value updating unit is used to update the value of the N+1th element in the target array to the value of the Mth element plus a preset value.

[0178] In some optional implementations, the initial values ​​of the second element to the last element in the target array are the same;

[0179] The first determining unit includes:

[0180] The element value acquisition unit is used to obtain the value of the last element in the target array;

[0181] The combination word determination unit is used to determine whether the target word is composed of two or more words preceding the target word according to the value of the last element.

[0182] In some optional implementations, the initial value of the first element in the target array is 0, and the initial value of each of the other elements is -1; the preset value is 1;

[0183] The combination word determination unit includes:

[0184] A third judging unit, used to judge whether the value of the last element is greater than or equal to 2;

[0185] The combination word recognition unit is used to determine that the target word is composed of two or more words preceding the target word when the value of the last element is greater than or equal to 2.

[0186] In some optional implementations, the second sequential acquisition unit includes:

[0187] a second determination unit, for determining whether to obtain a second sub-word starting from the Lth word of the target word according to the latest value of the second target element in the target array; the second target element is an element corresponding to the L-1th word of the target word, and the latest value of the second target element can indicate whether the sub-word composed of the first word to the L-1th word of the target word is a word preceding the target word, or is composed of two or more words preceding the target word; L is an integer greater than 1;

[0188] The second sub-word acquiring unit is used to acquire the second sub-word starting from the Lth word of the target word when it is determined to acquire the second sub-word starting from the Lth word of the target word; otherwise, skip acquiring the second sub-word starting from the Lth word of the target word.

[0189] The further functional description of each of the above modules and units is the same as that of the above corresponding embodiments and will not be repeated here.

[0190] The language data processing device in this embodiment is presented in the form of a functional unit, where the unit refers to an ASIC (Application Specific Integrated Circuit) circuit, a processor and memory that executes one or more software or fixed programs, and / or other devices that can provide the above functions.

[0191] The embodiment of the present invention also provides a computer device having the above Figure 4 The language data processing device shown.

[0192] See also Figure 5 , Figure 5 is a schematic diagram of the structure of a computer device provided by an optional embodiment of the present invention, such as Figure 5As shown, the computer device includes: one or more processors 10, a memory 20, and interfaces for connecting various components, including high-speed interfaces and low-speed interfaces. Various components are connected to each other using different buses for communication, and can be installed on a common mainboard or installed in other ways as needed. The processor can process the instructions executed in the computer device, including instructions stored in or on the memory to display the graphical information of the GUI on an external input / output device (such as, a display device coupled to the interface). In some optional embodiments, if necessary, multiple processors and / or multiple buses can be used together with multiple memories and multiple memories. Similarly, multiple computer devices can be connected, and each device provides some necessary operations (for example, as a server array, a group of blade servers, or a multi-processor system). Figure 5 A processor 10 is taken as an example.

[0193] The processor 10 may be a central processing unit, a network processor or a combination thereof. The processor 10 may further include a hardware chip. The hardware chip may be a dedicated integrated circuit, a programmable logic device or a combination thereof. The programmable logic device may be a complex programmable logic device, a field programmable gate array, a general purpose array logic or any combination thereof.

[0194] The memory 20 stores instructions executable by at least one processor 10, so that at least one processor 10 executes the method shown in the above embodiment.

[0195] The memory 20 may include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required for at least one function; the data storage area may store data created according to the use of the computer device, etc. In addition, the memory 20 may include a high-speed random access memory, and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some optional embodiments, the memory 20 may optionally include a memory remotely arranged relative to the processor 10, and these remote memories may be connected to the computer device via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0196] The memory 20 may include a volatile memory, such as a random access memory; the memory may also include a non-volatile memory, such as a flash memory, a hard disk or a solid state drive; the memory 20 may also include a combination of the above types of memory.

[0197] The computer device also includes an input device 30 and an output device 40. The processor 10, the memory 20, the input device 30 and the output device 40 may be connected via a bus or other means. Figure 5 The example of connecting through bus is taken in the following.

[0198] The input device 30 can receive input digital or character information, and generate key signal input related to the user settings and function control of the computer device, such as a touch screen, a keypad, a mouse, a track pad, a touch pad, an indicator bar, one or more mouse buttons, a trackball, a joystick, etc. The output device 40 may include a display device, an auxiliary lighting device (e.g., an LED) and a tactile feedback device (e.g., a vibration motor), etc. The above-mentioned display device includes but is not limited to a liquid crystal display, a light emitting diode, a display and a plasma display. In some optional embodiments, the display device may be a touch screen.

[0199] The computer device also includes a communication interface, which is used for the computer device to communicate with other devices or a communication network.

[0200] The embodiment of the present invention also provides a computer-readable storage medium. The method according to the embodiment of the present invention can be implemented in hardware, firmware, or can be implemented as a computer code that can be recorded in a storage medium, or can be implemented as a computer code that is originally stored in a remote storage medium or a non-temporary machine-readable storage medium and will be stored in a local storage medium through a network download, so that the method described herein can be stored in such software processing on a storage medium using a general-purpose computer, a dedicated processor, or programmable or dedicated hardware. Among them, the storage medium can be a magnetic disk, an optical disk, a read-only storage memory, a random access memory, a flash memory, a hard disk or a solid-state hard disk, etc.; further, the storage medium can also include a combination of the above types of memories. It can be understood that a computer, a processor, a microprocessor controller, or programmable hardware includes a storage component that can store or receive software or computer code. When the software or computer code is accessed and executed by a computer, a processor, or hardware, the method shown in the above embodiment is implemented.

[0201] Although the embodiments of the present invention have been described in conjunction with the accompanying drawings, those skilled in the art may make various modifications and variations without departing from the spirit and scope of the present invention, and such modifications and variations are all within the scope defined by the appended claims.

Claims

1. A language data processing method, characterized in that: The method comprises: Obtaining a word array to be recognized, wherein the elements in the word array are mutually different words; Sort the words in ascending order of length; For each of the words, determine whether it is composed of two or more words preceding the word; If so, it is determined that the word is a combination word; For each of the words, determining whether it is composed of two or more words preceding the word includes: For the target word currently being judged, a target array is established, wherein the length of the target array is 1 more than the length of the target word; Setting each element in the target array to an initial value; Determine in order whether each sub-word of the target word is a word that precedes the target word; if so, update the value of the corresponding first target element in the target array; Determining whether the target word is composed of two or more words preceding the target word according to the updated value of the element in the target array; The step of determining in order whether each sub-word of the target word is a word that precedes the target word; if so, updating the value of the corresponding first target element in the target array comprises: Acquire the first subword starting from the first character of the target word in order from short to long; For the first sub-word obtained, determining whether it is a word that precedes the target word; If yes, then updating the value of the first target element corresponding to the first sub-word in the target array; Acquire, in order from short to long, a second sub-word starting with the second character of the target word and a second sub-word starting with other characters of the target word; the other characters are characters in the target word other than the first character and the second character; For the acquired second sub-word, determining whether it is a word that precedes the target word; If yes, then updating the value of the first target element corresponding to the second sub-word in the target array; The updating the value of the first target element corresponding to the second sub-word in the target array includes: When the starting character of the second sub-word is the Mth character in the target word and the ending character of the second sub-word is the Nth character in the target word, determining that the first target element corresponding to the second sub-word in the target array is the N+1th element; Update the value of the N+1th element in the target array to the value of the Mth element plus a preset value; The step of sequentially acquiring the second sub-word starting with the second character of the target word and the second sub-word starting with other characters of the target word in order from short to long includes: According to the latest value of the second target element in the target array, it is determined whether to obtain the second sub-word starting from the Lth word of the target word; the second target element is an element corresponding to the L-1th word of the target word, and the latest value of the second target element can indicate whether the sub-word composed of the first word to the L-1th word of the target word is the word before the target word, or is composed of two or more words before the target word; L is an integer greater than 1; If yes, then obtain the second sub-word starting with the Lth word of the target word; Otherwise, skip obtaining the second sub-word starting from the Lth word of the target word.

2. The method according to claim 1, characterized in that The initial values ​​of the second element to the last element in the target array are the same; The step of determining whether the target word is composed of two or more words preceding the target word according to the updated value of the element in the target array comprises: Get the value of the last element in the target array; According to the value of the last element, it is determined whether the target word is composed of two or more words preceding the target word.

3. The method according to claim 2, characterized in that The initial value of the first element in the target array is 0, and the initial value of each of the other elements is -1; the preset value is 1; The step of determining, according to the value of the last element, whether the target word is composed of two or more words preceding the target word comprises: Determine whether the value of the last element is greater than or equal to 2; If so, it is determined that the target word is composed of two or more words preceding the target word.

4. A language data processing device, characterized in that: The device comprises: An acquisition module, used for acquiring an array of words to be recognized, wherein the elements in the array of words are mutually different words; A sorting module, used to sort the words in ascending order of length; A judgment module, used for judging, for each of the words, whether it is composed of two or more words preceding the word; A determination module, configured to determine that the word is a compound word when the word is composed of two or more words preceding the word; The judging module comprises: An establishing unit, used for establishing a target array for a target word currently being judged, wherein the length of the target array is 1 more than the length of the target word; An initialization unit, used for setting each element in the target array to an initial value; an updating unit, configured to determine in order whether each sub-word of the target word is a word arranged before the target word; if so, updating the value of the corresponding first target element in the target array; A first determining unit, configured to determine whether the target word is composed of two or more words preceding the target word according to the updated value of the element in the target array; The updating unit comprises: A first sequential acquisition unit is used to sequentially acquire the first sub-word starting from the first character of the target word in order from short to long; A first judging unit, configured to judge whether the first sub-word obtained is a word that precedes the target word; a first updating unit, configured to update a value of the first target element corresponding to the first sub-word in the target array when the first sub-word obtained is a word that precedes the target word; The second sequential acquisition unit is used to sequentially acquire, in order from short to long, a second sub-word starting with the second character of the target word and a second sub-word starting with other characters of the target word; other characters are characters other than the first character and the second character of the target word; A second judgment unit, configured to judge whether the acquired second sub-word is a word that precedes the target word; a second updating unit, configured to update a value of the first target element corresponding to the second sub-word in the target array when the acquired second sub-word is a word that precedes the target word; The second updating unit comprises: a first target element determining unit, configured to determine that the first target element corresponding to the second sub-word in the target array is the N+1th element when the starting character of the second sub-word is the Mth character in the target word and the ending character of the second sub-word is the Nth character in the target word; A value updating unit, used for updating the value of the N+1th element in the target array to the value of the Mth element plus a preset value; The second sequential acquisition unit includes: a second determination unit, for determining whether to obtain the second sub-word starting from the Lth word of the target word according to the latest value of the second target element in the target array; the second target element is an element corresponding to the L-1th word of the target word, and the latest value of the second target element can indicate whether the sub-word composed of the first word to the L-1th word of the target word is the word preceding the target word, or is composed of two or more words preceding the target word; L is an integer greater than 1; The second sub-word acquiring unit is configured to acquire the second sub-word starting from the Lth character of the target word if it is determined to acquire the second sub-word starting from the Lth character of the target word; otherwise, skip acquiring the second sub-word starting from the Lth character of the target word.

5. A computer device, characterized in that: include: A memory and a processor, wherein the memory and the processor are communicatively connected to each other, the memory stores computer instructions, and the processor executes the language data processing method according to any one of claims 1 to 3 by executing the computer instructions.

6. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a computer to execute the language data processing method according to any one of claims 1 to 3.