Input method multi-word discovery method and device

Abstract

The present application provides an input method multi-word discovery method and device, which are used to solve the technical problem of low efficiency of input method discovery of multi-words. Among them, an input method multi-word discovery method includes the following steps: obtaining corpus data; segmenting the corpus data through a word segmentation model to generate a number of word segmentation units; storing the word segmentation units in a WFP data structure; comparing the association between word segmentation units and word segmentation units by reading the WFP data structure to determine the word segmentation unit combination; adding the word segmentation unit combination to the word library. Compared with the prior art, the input method multi-word discovery method can realize the discovery of multi-word relationships and improve the accuracy and efficiency of discovering multi-words.

Description

Technical Field

[0001] The present application relates to the technical field of input methods, and in particular to an input method multi-word discovery method and device. Background Art

[0002] With the continuous development and improvement of social economy, culture and science and technology, the vocabulary used in daily life and professional fields is also constantly developing, which leads to the rapid emergence and rapid iterative development of new words.

[0003] Quickly discovering new words that people use in daily life will greatly improve the accuracy and timeliness of pinyin input methods.

[0004] In the prior art, new word discovery methods can be divided into two categories: one is a method based on language rules, and the other is a method based on statistical learning.

[0005] In the process of implementing the prior art, the inventor discovered that:

[0006] Traditional language rule-based methods mainly rely on linguists to build new word discovery rules in related fields, and linguists are also required to maintain these rules in related fields. This method requires a lot of time and money, and also makes it difficult to transfer these rules to other fields, limiting the versatility of this type of method.

[0007] Methods based on statistical learning. This method is to discover new words by counting the probability of word formation, word frequency and information entropy around words from a large amount of corpus data. However, this method requires a large amount of corpus data as support. Since this method performs statistical analysis based on words, its analysis speed is relatively slow. In addition, many methods based on statistical learning can only be used to count the relationship between two-words, but cannot count the relationship between polywords.

[0008] Therefore, it is necessary to provide an input method multi-word discovery method and device to solve the technical problem of low efficiency of input method discovery of multi-words. Summary of the invention

[0009] The embodiments of the present application provide an input method multi-word discovery method and device, which are used to solve the technical problem of low efficiency of input method discovery of multi-words.

[0010] Specifically, an input method multi-word discovery method includes the following steps:

[0011] Get corpus data;

[0012] Segmenting the corpus data using a word segmentation model to generate a number of word segmentation units;

[0013] Store word segmentation units in WFP data structure;

[0014] By reading the WFP data structure, comparing the correlation between word segmentation units and word segmentation units, the word segmentation unit combination is determined;

[0015] Add the word segmentation unit combination to the word library.

[0016] Furthermore, the storing of word segmentation units in the WFP data structure specifically includes:

[0017] Construct character index table and frequency feature tree to generate initial WFP data structure;

[0018] Importing the sentence logic body consisting of several word segmentation units into the initial WFP data structure to generate the final WFP data structure;

[0019] The frequency feature tree is composed of several nodes, and the frequency feature tree is used to describe the word frequency information of the word segmentation unit in the corpus data; the node has a preceding index set, a succeeding index set, a word segmentation unit of the current node, and the word frequency of the current node;

[0020] The character index table is composed of several character indexes, and a single character index represents a unique corresponding word segmentation unit; the character index is used to record the position information of the node in the frequency feature tree.

[0021] Furthermore, the step of importing the sentence logic body composed of a plurality of word segmentation units into the initial WFP data structure to generate the final WFP data structure specifically includes:

[0022] The initial WFP data structure obtains the first word segmentation unit in the sentence logical body;

[0023] Find the node node corresponding to the previous word segmentation unit of the first word segmentation unit pre ;

[0024] Determine whether the first word segmentation unit is in the previous word segmentation unit node pre In the post-item index set postset;

[0025] When the first word segmentation unit is in the previous word segmentation unit node pre In the post-item index set postset, the previous word segmentation unit node node pre The node where the first word segmentation unit is located in the post-index set postset pre-＞p The frequency of the node increases by 1;

[0026] Get the subsequent word segmentation unit of the first word segmentation unit in the sentence logic body, and repeatedly determine whether the subsequent word segmentation unit is in the subsequent index set of the first word segmentation unit node, until the initial WFP data structure stores the sentence logic body composed of several word segmentation units, and generate the final WFP data structure.

[0027] Furthermore, the step of importing the sentence logic body composed of a plurality of word segmentation units into the initial WFP data structure to generate the final WFP data structure specifically includes:

[0028] The initial WFP data structure stores the logical body of a sentence consisting of several word segmentation units;

[0029] The initial WFP data structure obtains the first word segmentation unit in the sentence logical body;

[0030] Find the node node corresponding to the previous word segmentation unit of the first word segmentation unit pre ;

[0031] Determine whether the first word segmentation unit is in the previous word segmentation unit node pre In the post-item index set postset;

[0032] When the first word segmentation unit is not in the previous word segmentation unit node pre In the post-item index set postset, determine whether the first word segmentation unit is in the character index table WTable;

[0033] When the first word segmentation unit is in the character index table WTable, add a record w to the character index windex corresponding to the first word segmentation unit in the character index table WTable. k POS p ;

[0034] Get the next word segmentation unit of the first word segmentation unit in the sentence logic body, and repeatedly determine whether the next word segmentation unit is in the character index table WTable, until the initial WFP data structure stores the sentence logic body composed of several word segmentation units, and generates the final WFP data structure.

[0035] Furthermore, the step of importing the sentence logic body composed of a plurality of word segmentation units into the initial WFP data structure to generate the final WFP data structure specifically includes:

[0036] The initial WFP data structure stores the logical body of a sentence consisting of several word segmentation units;

[0037] The initial WFP data structure obtains the first word segmentation unit in the sentence logical body;

[0038] Find the node node corresponding to the previous word segmentation unit of the first word segmentation unit pre ;

[0039] Determine whether the first word segmentation unit is in the previous word segmentation unit node pre In the post-item index set postset;

[0040] When the first word segmentation unit is not in the previous word segmentation unit node pre In the post-item index set postset, determine whether the first word segmentation unit is in the character index table WTable;

[0041] When the first word segmentation unit is not in the character index table WTable, a new character index windex is added to the character index table WTable based on the first word segmentation unit;

[0042] Add a record w to the corresponding character index windex in the character index table WTable for the first word segmentation unit k POS p ;

[0043] Get the next word segmentation unit of the first word segmentation unit in the sentence logic body, and repeatedly determine whether the next word segmentation unit is in the character index table WTable, until the initial WFP data structure stores the sentence logic body composed of several word segmentation units, and generates the final WFP data structure.

[0044] Further, the method of reading the WFP data structure, comparing the relevance between word segmentation units, and determining the word segmentation unit combination specifically includes:

[0045] According to the WFP data structure, construct a multi-word relationship table;

[0046] By scanning the multi-word relationship table, the expanded information entropy between word segmentation units is calculated;

[0047] When the expanded information entropy of the word segmentation units and the word segmentation units is less than or equal to a first preset threshold, determining a word segmentation unit combination;

[0048] The multi-word relationship table describes the association between word segmentation unit combinations.

[0049] Furthermore, the multi-word relationship table is constructed according to the WFP data structure, specifically including:

[0050] By scanning the character index table, the frequency of the frequency feature tree node where the first word segmentation unit is located and the sum of the position information pointed to by the frequency feature tree where the first word segmentation unit is located are calculated;

[0051] When the sum of the position information pointed to by the frequency feature tree where the first word segmentation unit is located is greater than a second preset threshold, the first word segmentation unit is stored in the unigram relationship table;

[0052] Based on the uni-gram relationship table, the word segmentation unit sequence information and word frequency information in the uni-gram relationship table are expanded to the K-gram relationship table; it is expressed as follows:

[0053]

[0054] In the formula, To expand to K-word relationships, node i and node j To try to expand the new words that add relationships; RTable1 is a unigram word relationship table; node r，p，k-1，1 is the node corresponding to the pth combination in the rth relationship, post_node r，p，k-1，k-1 For its node r，p，k-1，k-1 A collection of indexes of subsequent items in the node.

[0055] Furthermore, the expanded information entropy of the word segmentation unit combination is calculated by scanning the multi-word relationship table, which is specifically manifested as:

[0056]

[0057]

[0058] Among them, AE is the expansion entropy; c(nodei) is the number of times the word segmentation unit represented by the statistical node nodei appears in the corpus data; c(pre_node r，p，k-1，1 ) is the statistical node r，p，k-1，1 The number of times the word segmentation unit represented by the node in the previous index set appears in the corpus data; To count nodes r，p，k-1，k-1 The number of times the word segmentation unit represented by the node in the post-index set of appears in the corpus data; for The multi-word expansion information entropy of for The multi-word expansion information entropy;

[0059] When the calculated If it is greater than the third preset threshold, the expansion is abandoned.

[0060] The embodiment of the present application also provides an input method new word discovery device.

[0061] Specifically, an input method multi-word discovery device includes:

[0062] The acquisition module is used to obtain corpus data;

[0063] A word segmentation module, used to segment the corpus data through a word segmentation model to generate a plurality of word segmentation units;

[0064] A storage module, used for storing word segmentation units in a WFP data structure;

[0065] A comparison module is used to compare the relevance between word segmentation units and word segmentation units by reading the WFP data structure, and determine the word segmentation unit combination;

[0066] The output module is used to add the word segmentation unit combination to the word library.

[0067] Further, the storage module is used to store word segmentation units in a WFP data structure, specifically for:

[0068] Construct character index table and frequency feature tree to generate initial WFP data structure;

[0069] Importing the sentence logic body consisting of several word segmentation units into the initial WFP data structure to generate the final WFP data structure;

[0070] The frequency feature tree is composed of several nodes, and the frequency feature tree is used to describe the word frequency information of the word segmentation unit in the corpus data; the node has a preceding index set, a succeeding index set, a word segmentation unit of the current node, and the word frequency of the current node;

[0071] The character index table is composed of several character indexes, and a single character index represents a unique corresponding word segmentation unit; the character index is used to record the position information of the node in the frequency feature tree.

[0072] The technical solution provided in the embodiments of the present application has at least the following beneficial effects:

[0073] By driving the deep learning model, after the acquired corpus data is segmented, the segmentation unit combination judgment is performed on the generated several segmentation units, which improves the accuracy and efficiency of discovering polyglot words. BRIEF DESCRIPTION OF THE DRAWINGS

[0074] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0075] Figure 1 A flowchart of an input method multi-word discovery method provided in an embodiment of the present application.

[0076] Figure 2 A flowchart of constructing a WFP data structure is provided in an embodiment of the present application.

[0077] Figure 3 A schematic diagram of the structure of an input method multi-word discovery device provided in an embodiment of the present application.

[0078] 100 Input method multi-word discovery device

[0079] 11 Get Module

[0080] 12 Word segmentation module

[0081] 13 Storage Module

[0082] 14 Comparison Module

[0083] 15 Output Module DETAILED DESCRIPTION

[0084] In order to make the purpose, technical solution and advantages of the present application clearer, the technical solution of the present application will be clearly and completely described below in combination with the specific embodiments of the present application and the corresponding drawings. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present application.

[0085] With the continuous development and improvement of social economy, culture and science and technology, the vocabulary used in daily life and professional fields is also constantly developing, which leads to the rapid emergence and rapid iterative development of new words.

[0086] Rapidly discovering new words used in daily life will greatly improve the accuracy and timeliness of Pinyin input methods. However, the statistical learning-based methods in the prior art usually perform statistical analysis on a character basis, and the analysis speed is relatively slow. Therefore, a multi-word discovery scheme for input methods is needed to solve the technical problem of low efficiency in discovering multi-words by input methods.

[0087] Please refer to Figure 1 The present application discloses an input method multi-word discovery method, comprising the following steps:

[0088] S110: Acquire corpus data.

[0089] It is understandable that the corpus data can be understood as at least one text combination including characters, words, sentences, and documents. Video titles in video websites, comments in video websites, documents in information websites, etc. can all be the corpus data described in this application document.

[0090] S120: Segment the corpus data using a word segmentation model to generate a number of word segmentation units.

[0091] It can be understood that the word segmentation model can be a neural network algorithm model based on language grammatical rules, which is used to identify the grammatical components of corpus data or the word groups of corpus data, and segment the corpus data according to the grammatical components or the word groups of corpus data to generate a number of word segmentation units. Among them, the grammatical components of the corpus data can be expressed as grammatical components such as the subject, predicate, object, verb, attributive, adverbial, complement, predicative, appositive, etc. in the sentence. The word groups of the corpus data can be expressed as word groups such as nouns, pronouns, verbs, prepositions, and numerals.

[0092] Therefore, step S120 can be expressed as follows: inputting corpus data into the word segmentation model; then, the word segmentation model segments the corpus data according to the word groups such as nouns, pronouns, verbs, prepositions, and numerals in the sentence; finally, the word segmentation model splits the corpus data into several word segmentation units.

[0093] Alternatively, step S120 may also be performed as follows: inputting corpus data into a word segmentation model; then, the word segmentation model performs word segmentation on the corpus data according to grammatical components such as subject, predicate, object, verb, attributive, adverbial, complement, predicative, appositive, etc. in the sentence; finally, the word segmentation model splits the corpus data into a number of word segmentation units.

[0094] S130: Storing word segmentation units in a WFP data structure.

[0095] It should be pointed out that in the process of implementing the storage of word segmentation units, the inventors found that using different data structures to store word segmentation units resulted in different system reading, scanning, and processing speeds. In order to optimize system processing efficiency and operating space consumption, this application preferably uses the WFP data structure to store word segmentation units, which facilitates rapid system reading and thus speeds up data processing.

[0096] Specifically, in a specific implementation provided in the present application, storing word segmentation units in a WFP data structure can be specifically expressed as follows:

[0097] The system constructs a character index table and a frequency feature tree to generate an initial WFP data structure. Then the system imports the sentence logic body consisting of several word segmentation units into the initial WFP data structure to generate a final WFP data structure.

[0098] Among them, the frequency feature tree is composed of several nodes, and the frequency feature tree is used to describe the word frequency information of the word segmentation unit in the corpus data; the node has a preceding index set, a succeeding index set, the word segmentation unit of the current node, and the word frequency of the current node.

[0099] The character index table is composed of a number of character indexes, and a single character index represents a unique corresponding word segmentation unit; the character index is used to record the position information of the node in the frequency feature tree.

[0100] It should be noted that the system can form a sentence logic body with several word segmentation units. The sentence logic body can be understood as a sentence formed by several word segmentation units in language logic. Therefore, the sentence logic body can also be understood as the word segmentation units are arranged according to their position information in the corpus data to form a sentence logic body.

[0101] It can be understood that storing the sentence logic body in the WFP data structure focuses more on the connection between word segmentation units than directly storing disordered word segmentation units in the WFP data structure. It is also faster in terms of calculation and has closer connections between word segmentation units than directly storing corpus data in the WFP data structure.

[0102] The following reveals the construction process of the initial WFP data structure.

[0103] When the corpus data is segmented through the word segmentation model, several word segmentation units are generated. In which, s is a corpus data set, which consists of n logical sentences. i It is represented as the logical body of the i-th sentence, which consists of m word segmentation units. It represents the jth word segmentation unit in the logical body of the i-th sentence.

[0104] For s i (i∈n), with root as the start initial point set, construct the WFP data structure. The WFP data structure mainly consists of two parts, namely the character index table WTable and the frequency feature tree FPTree. Among them, the frequency feature tree is composed of several nodes node. The node node has a preceding index set preset, a subsequent index set postset, a phrase w of the current node, and a current node word frequency freq. The character index table WTable is composed of several character indexes windecode, and a single character index windecode represents a unique corresponding word segmentation unit uniq_w k The character index windex records the corresponding word segmentation unit uniq_w k The location information w of the node in the frequency feature tree FPTre k pos,w k pos={w k pos1,w k pos2,...,w k POS p}.

[0105] Please refer to Figure 2In a specific implementation manner provided in the present application, a sentence logic body consisting of a plurality of word segmentation units is imported into an initial WFP data structure to generate a final WFP data structure, which can be expressed as follows:

[0106] Initial WFP data structure to obtain the first word segmentation unit in the sentence logical body Then, find the preceding word segmentation unit corresponding to the first word segmentation unit node pre Then determine the first word segmentation unit Whether it is in the post-item index set postset of the previous word segmentation unit node nodep.

[0107] When the first participle unit In the previous word segmentation unit node pre In the post-item index set postset, the previous word segmentation unit node node pre The first word unit in the post-index set postset The node where pre-＞p The frequency of the node is increased by 1. Then the next word segmentation unit of the first word segmentation unit in the logical body of the sentence is obtained, that is, Repeat judgment post-term participle unit Whether it is in the subsequent index set of the first word segmentation unit node, until the initial WFP data structure stores the sentence logic body composed of several word segmentation units, and generates the final WFP data structure.

[0108] In another specific implementation provided by the present application, a sentence logic body consisting of a plurality of word segmentation units is imported into an initial WFP data structure to generate a final WFP data structure, which can also be expressed as:

[0109] Initial WFP data structure to obtain the first word segmentation unit in the sentence logical body Then, find the preceding word segmentation unit corresponding to the first word segmentation unit node pre Then determine the first word segmentation unit Is it in the previous word segmentation unit node node pre The post-item index set postset.

[0110] When the first participle unit Not in the preceding word segmentation unit node pre In the post-item index set postset, determine Is it in the character index table WTable? In the character index table WTable, the first word segmentation unit Add a record w to the corresponding character index windex in the character index table WTable k POS p . Then get the next word segmentation unit of the first word segmentation unit in the sentence logic body, that is, Repeat judgment post-term participle unit Whether it is in the subsequent index set of the first word segmentation unit node, until the initial WFP data structure stores the sentence logic body composed of several word segmentation units, and generates the final WFP data structure.

[0111] In another specific implementation provided by the present application, a sentence logic body consisting of a plurality of word segmentation units is imported into an initial WFP data structure to generate a final WFP data structure, which can also be expressed as:

[0112] Initial WFP data structure to obtain the first word segmentation unit in the sentence logical body Then, find the preceding word segmentation unit corresponding to the first word segmentation unit node pre Then determine the first word segmentation unit Whether it is in the post-item index set postset of the previous word segmentation unit node nodep.

[0113] When the first participle unit Not in the preceding word segmentation unit node pre In the post-item index set postset, determine Is it in the character index table WTable? If it is not in the character index table WTable, add a new character index windex to the character index table WTable based on the first word segmentation unit. Then add the first word segmentation unit Add a record w to the corresponding character index windex in the character index table WTable k POS p . Then get the next word segmentation unit of the first word segmentation unit in the sentence logic body, that is, Repeat judgment post-term participle unit Whether it is in the subsequent index set of the first word segmentation unit node, until the initial WFP data structure stores the sentence logic body composed of several word segmentation units, and generates the final WFP data structure.

[0114] It should be emphasized that by storing word segmentation units in the WFP data structure, the system can quickly read the word segmentation unit information, thereby speeding up data processing. In addition, the character index table WTable and the frequency feature tree FPTree also allow the system to not be limited to determining unigrams and bigrams. The character index table WTable and the frequency feature tree FPTree can analyze the relationship between multiple words, making the input method more accurate.

[0115] S140: By reading the WFP data structure, comparing the relevance between word segmentation units, and determining the word segmentation unit combination.

[0116] It should be noted that the reading of the WFP data structure, comparing the relevance between word segmentation units, and determining the word segmentation unit combination can be specifically performed as follows:

[0117] The system constructs a polymorphic word relationship table according to the WFP data structure. Afterwards, the system calculates the expanded information entropy of the word segmentation units by scanning the polymorphic word relationship table. When the expanded information entropy of the word segmentation units is less than or equal to the first preset threshold, the word segmentation unit combination is determined.

[0118] It should be noted that after completing the construction of the WFP data structure in step S130, the system also needs to construct a multi-word relationship table Multi-RTable based on the WFP data structure. Specifically, the multi-word relationship table Multi-RTable is composed of several sub-tables, which are used to describe the association between word segmentation units. They are the unigram relationship sub-table (RTable1), the K-1 unigram relationship sub-table (RTable2), the K-1 unigram relationship sub-table (RTable3), the K-1 unigram relationship sub-table (RTable4), and the K-1 unigram relationship sub-table (RTable5). k-1 ), K-gram relation subtable (RTable k )constitute.

[0119] The K-gram relation subtable can be expressed as

[0120]

[0121] exist

[0122] in

[0123] To construct the multi-word relation table Multi-RTable, at least a uni-word relation sub-table (RTable1) needs to be constructed, which is expressed as follows:

[0124] The system scans the character index table WTable, and for any uniq_w k , calculate its w k The frequency and sum(w k pos).

[0125] If sum(w k pos) is greater than the second preset threshold Freq_Threshold, then uniq_w k Put it into the unigram relationship table.

[0126] Then, it is necessary to expand the K-gram word list based on the K-1-gram word list, and the K value of the K-gram word list is at least 2. For example, when K is 2, the bigram word relationship subtable (RTable2) can be constructed based on the unigram word relationship subtable (RTable1).

[0127] It should be pointed out that the K-gram word list is expanded based on the K-1-gram word list, which can be expressed as follows:

[0128] make is the p-th combination in the r-th relation in the k-1-tuple, pre_node r，p，k-1，1 For its node r，p，k-1，1 The index collection of the previous item in the node, post_node r，p，k-1，k-1 For its node r，p，k-1，k-1 A collection of indexes of subsequent items in the node.

[0129] Then, according to the following formula, the relationship in the K-gram word list is inferred.

[0130]

[0131] In the formula, To expand to the relationship of K-grams, node i and node j To try to expand the new words to add relations; RTable1 is the unigram relation table; node r，p，k-1，1 is the node corresponding to the pth combination in the rth relationship, post_node r，p，k-1，k-1 For its node r，p，k-1，k-1 A collection of indexes of subsequent items in the node.

[0132] Then calculate Multi-word expansion information entropy The calculation formula is as follows:

[0133]

[0134]

[0135] Where AE is the expanded entropy; c(nodei) is the statistical node node i The number of times the word segmentation unit represented appears in the corpus data; c(pre_node r，p，k-1，1 ) is the statistical noder，p，k-1，1 The number of times the word segmentation unit represented by the node in the previous index set appears in the corpus data; To count nodes r，p，k-1，k-1 The number of times the word segmentation unit represented by the node in the post-index set of appears in the corpus data; for The multi-word expansion information entropy of for The multi-word expanded information entropy of If it is greater than the third preset threshold, the expansion is abandoned.

[0136] Finally, repeat the above steps until the K-gram word relationship table is obtained.

[0137] By constructing the WFP data structure and the K-gram word relationship table, a multi-word relationship table from 1 to K can be obtained. The multi-word relationship in the table can be considered as the multi-word relationship between word segmentation units in the corpus data. After hashing the multi-word relationship between word segmentation units, the word segmentation unit combination relationship can be determined.

[0138] It should be emphasized that due to the establishment of the K-gram relationship table, the word segmentation unit combinations discovered by the system are not limited to unigrams and bigrams. The system can analyze multi-word relationships by scanning the K-gram relationship table. Combining word segmentation units corresponding to the determined multi-word relationships and saving them to the word library can enrich the word library of the input method and make the input more accurate.

[0139] S150: Add the word segmentation unit combination to the word library.

[0140] It is understandable that when the polyword is determined to be a combination of word segmentation units, it can be regarded as a new word group, so that the word segmentation unit combination is added to the word library.

[0141] In a specific implementation provided by the present application, new words are discovered by the input method multi-word discovery method. After the system obtains the corpus data "due to the Covid19 virus pandemic", it can obtain the word segmentation unit through the word segmentation model based on deep learning.

[0142] The word segmentation units are expressed as "cause", "Covid", "19", "virus", and "pandemic". At the same time, the part of speech of the word segmentation unit can also be obtained. For example, the part of speech of "cause" is a preposition, "Covid" is English, "19" is a number, and "virus" is a noun. The part of speech of the word segmentation unit and the connection frequency between the word segmentation units are used as word segmentation unit attributes and stored in the WFP data structure. By scanning the WFP tree, the association relationship between the word segmentation units that meet the preset threshold requirements can be determined.

[0143] For example, it can be determined through calculation that the information entropy of "Covid" followed by "19" and the information entropy of "19" followed by "Covid" are both less than the first preset threshold. Therefore, the judgment result is "Covid19" as a new phrase. Then "Covid19" is added to the vocabulary as a new phrase, completing the discovery and recording of new words by the input method.

[0144] The input method multi-word discovery method drives the deep learning model to segment the acquired corpus data, and then generates several segmentation units to make segmentation unit combination judgments, thereby improving the accuracy and efficiency of discovering new words.

[0145] Please refer to Figure 3 To support the input method multi-word discovery method, the present application also provides an input method multi-word discovery device 100, comprising:

[0146] An acquisition module 11 is used to acquire corpus data;

[0147] A word segmentation module 12, used to segment the corpus data through a word segmentation model to generate a plurality of word segmentation units;

[0148] The storage module 13 is used to store word segmentation units in a WFP data structure;

[0149] A comparison module 14 is used to compare the relevance of word segmentation units and word segmentation units by reading the WFP data structure, and determine the word segmentation unit combination;

[0150] The output module 15 is used to add the word segmentation unit combination to the word library.

[0151] It is understandable that the corpus data acquired by the acquisition module 11 can be understood as at least one text combination of characters, words, sentences, and documents. For example, the corpus data can be video summaries on video websites, comments on photo websites, documents on information websites, etc.

[0152] The word segmentation model called by the word segmentation module 12 can be a neural network algorithm model based on language grammar rules, which is used to identify the grammatical components of the corpus data or the word groups of the corpus data, and segment the corpus data according to the grammatical components or the word groups of the corpus data to generate a number of word segmentation units. For example, the word segmentation model can segment the corpus data according to the word groups such as nouns, pronouns, verbs, prepositions, and numerals in the corpus data, and split the corpus data into a number of word segmentation units.

[0153] Alternatively, the word segmentation model may also perform word segmentation on the corpus data according to grammatical components such as subject, predicate, object, verb, attributive, adverbial, complement, predicative, appositive, etc. in the corpus data, and split the corpus data into several word segmentation units.

[0154] It should be noted that, in order to optimize the system processing efficiency and running space consumption, the storage module 13 uses the WFP data structure to store the word segmentation units, so as to facilitate the input method multi-word discovery device 100 to quickly read, thereby speeding up the data processing speed. Specifically, in a specific implementation provided by the present application, the storage module 13 is used to store the word segmentation units in the WFP data structure, which can be specifically expressed as follows:

[0155] The storage module 13 constructs a character index table and a frequency feature tree to generate an initial WFP data structure. Then the system imports the sentence logic body composed of several word segmentation units into the initial WFP data structure to generate a final WFP data structure.

[0156] Among them, the frequency feature tree is composed of several nodes, and the frequency feature tree is used to describe the word frequency information of the word segmentation unit in the corpus data; the node has a preceding index set, a succeeding index set, the word segmentation unit of the current node, and the word frequency of the current node.

[0157] The character index table is composed of a number of character indexes, and a single character index represents a unique corresponding word segmentation unit; the character index is used to record the position information of the node in the frequency feature tree.

[0158] It should be noted that the storage module 13 can form a sentence logic body with several word segmentation units. The sentence logic body can be understood as a sentence formed by several word segmentation units in language logic. Therefore, the sentence logic body can also be understood as the word segmentation units are arranged according to their position information in the corpus data to form a sentence logic body.

[0159] It can be understood that storing the logical body of a sentence in a WFP data structure focuses more on the connection between word segmentation units than directly storing disordered word segmentation units in a WFP data structure. It is also faster in terms of computation and the connection between word segmentation units is closer than directly storing corpus data in a WFP data structure.

[0160] The following discloses the process of the storage module 13 constructing the initial WFP data structure.

[0161] When the corpus data is segmented through the word segmentation model, several word segmentation units are generated. In which, s is a corpus data set, which consists of n logical sentences. i It is represented as the logical body of the i-th sentence, which consists of m word segmentation units. It represents the jth word segmentation unit in the logical body of the i-th sentence.

[0162] For s i(i∈n), with root as the start initial point set, construct the WFP data structure. The WFP data structure mainly consists of two parts, namely the character index table WTable and the frequency feature tree FPTree. Among them, the frequency feature tree is composed of several nodes node. The node node has a preceding index set preset, a subsequent index set postset, a phrase w of the current node, and a current node word frequency freq. The character index table WTable is composed of several character indexes windecode, and a single character index windecode represents a unique corresponding word segmentation unit uniq_w k The character index windex records the corresponding word segmentation unit uniq_w k The location information w of the node in the frequency feature tree FPTre k pos,w k pos={w k pos1,w k pos2,...,w k POS p}.

[0163] The storage module 13 imports the sentence logic body composed of several word segmentation units into the initial WFP data structure to generate the final WFP data structure, which can be expressed as follows:

[0164] The storage module 13 obtains the first word segmentation unit in the sentence logic body Then, find the preceding word segmentation unit corresponding to the first word segmentation unit node pre Then determine the first word segmentation unit Is it in the previous word segmentation unit node node pre The post-item index set postset.

[0165] When the first participle unit In the previous word segmentation unit node pre In the post-item index set postset, the previous word segmentation unit node node pre The first word unit in the post-index set postset The node where pre-＞p The frequency of the node is increased by 1. Then the next word segmentation unit of the first word segmentation unit in the logical body of the sentence is obtained, that is, Repeat judgment post-term participle unit Whether it is in the subsequent index set of the first word segmentation unit node, until the initial WFP data structure stores the sentence logic body composed of several word segmentation units, and generates the final WFP data structure.

[0166] In another specific implementation provided by the present application, the storage module 13 imports the sentence logic body composed of several word segmentation units into the initial WFP data structure to generate the final WFP data structure, which can also be expressed as:

[0167] The storage module 13 obtains the first word segmentation unit in the sentence logic body Then, find the preceding word segmentation unit corresponding to the first word segmentation unit node pre Then determine the first word segmentation unit Is it in the previous word segmentation unit node node pre The post-item index set postset.

[0168] When the first participle unit Not in the preceding word segmentation unit node pre In the post-item index set postset, determine Is it in the character index table WTable? In the character index table WTable, the first word segmentation unit Add a record w to the corresponding character index windex in the character index table WTable k POS p . Then get the next word segmentation unit of the first word segmentation unit in the sentence logic body, that is, Repeat judgment post-term participle unit Whether it is in the subsequent index set of the first word segmentation unit node, until the initial WFP data structure stores the sentence logic body composed of several word segmentation units, and generates the final WFP data structure.

[0169] In another specific implementation manner provided by the present application, the storage module 13 imports the sentence logic body composed of several word segmentation units into the initial WFP data structure to generate the final WFP data structure, which can also be expressed as:

[0170] The storage module 13 obtains the first word segmentation unit in the sentence logic body Then, find the preceding word segmentation unit corresponding to the first word segmentation unit node pre Then determine the first word segmentation unit Is it in the previous word segmentation unit node node pre The post-item index set postset.

[0171] When the first participle unit Not in the preceding word segmentation unit node pre In the post-item index set postset, determine Is it in the character index table WTable? If it is not in the character index table WTable, add a new character index windex to the character index table WTable based on the first word segmentation unit. Then add the first word segmentation unit Add a record w to the corresponding character index windex in the character index table WTable k POS p . Then get the next word segmentation unit of the first word segmentation unit in the sentence logic body, that is, Repeat judgment post-term participle unit Whether it is in the subsequent index set of the first word segmentation unit node, until the initial WFP data structure stores the sentence logic body composed of several word segmentation units, and generates the final WFP data structure.

[0172] It should be emphasized that the storage module 13 stores word segmentation units through the WFP data structure, which enables the input method multi-word discovery device 100 to quickly read the word segmentation unit information, thereby speeding up the data processing speed. In addition, the character index table WTable and the frequency feature tree FPTree also allow the input method multi-word discovery device 100 to not be limited to determining unigrams and bigrams. The input method multi-word discovery device 100 can analyze the relationship between multi-words through the character index table WTable and the frequency feature tree FPTree, thereby making the input method more accurate.

[0173] It should be noted that after the storage module 13 completes the construction of the WFP data structure, the comparison module 14 still needs to construct a multi-word relationship table Multi-RTable according to the WFP data structure in order to compare the relevance between word segmentation units.

[0174] Specifically, the comparison module 14 can construct a polyword relationship table according to the WFP data structure. Afterwards, the comparison module 14 can calculate the expanded information entropy of the word segmentation unit and the word segmentation unit by scanning the polyword relationship table. When the expanded information entropy of the word segmentation unit and the word segmentation unit is less than or equal to the first preset threshold, the word segmentation unit combination is determined.

[0175] The multi-word relation table Multi-RTable is composed of several sub-tables, which are used to describe the association between word segmentation units. The multi-word relation table Multi-RTable can be composed of a unigram relation sub-table (RTable1), a K-1 unigram relation sub-table (RTable2), a K-1 unigram relation sub-table (RTable3), a K-1 unigram relation sub-table (RTable4), a K-1 unigram relation sub-table (RTable5), a K-1 unigram relation sub-table (RTable6), a K-1 unigram relation sub-table (RTable7), a K-1 unigram relation sub-table (RTable8), a K-1 unigram relation sub-table (RTable9), a K-1 unigram relation sub-table (RTable11), a K-1 unigram relation sub-table (RTable12), a K-1 unigram relation sub-table (RTable13), a K-1 unigram relation sub-table (RTable14), k-1 ), K-gram relation subtable (RTable K )constitute.

[0176] The K-gram relation subtable can be expressed as

[0177]

[0178] exist

[0179] in

[0180] To construct the multi-word relation table Multi-RTable, at least a uni-word relation sub-table (RTable1) needs to be constructed, which is expressed as follows:

[0181] The comparison module 14 scans the character index table WTable, and for any uniq_w k , calculate its w k The frequency and sum(w k pos). If sum(w k pos) is greater than the second preset threshold Freq_Threshold, then uniq_w k Put it into the unigram relationship table.

[0182] Afterwards, the comparison module 14 needs to expand the K-gram word list based on the K-1-gram word list, and the K value of the K-gram word list is at least 2. For example, when K is 2, the bigram relation subtable (RTable2) can be constructed based on the unigram relation subtable (RTable1).

[0183] It should be pointed out that the K-gram word list is expanded based on the K-1-gram word list, which can be expressed as follows:

[0184] make is the p-th combination in the r-th relation in the k-1-tuple, pre_node r，p，k-1，1 For its node r，p，k-1，1 The index collection of the previous item in the node, post_node r，p，k-1，k-1 For its node r，p，k-1，k-1 A collection of indexes of subsequent items in the node.

[0185] Afterwards, the comparison module 14 infers the relationship in the K-gram word list according to the following formula.

[0186]

[0187] In the formula, To expand to the relationship between K-grams, nodei and node j To try to expand the new words to add relations; RTable1 is the unigram relation table; node r，p，k-1，1 is the node corresponding to the pth combination in the rth relationship, post_node r，p，k-1，k-1 For its node r，p，k-1，k-1 A collection of indexes of subsequent items in the node.

[0188] Then the comparison module 14 calculates Multi-word expansion information entropy The calculation formula is as follows:

[0189]

[0190]

[0191] Where AE is the expanded entropy; c(nodei) is the number of times the word segmentation unit represented by the statistical node nodei appears in the corpus data; c(pre_node r，p，k-1，1 ) is the statistical node r，p，k-1，1 The number of times the word segmentation unit represented by the node in the previous index set appears in the corpus data; To count nodes r，p，k-1，k-1 The number of times the word segmentation unit represented by the node in the post-index set of appears in the corpus data; for The multi-word expansion information entropy of for The multi-word expanded information entropy of If it is greater than the third preset threshold, the expansion is abandoned.

[0192] Finally, repeat the above steps until the K-gram word relationship table is obtained.

[0193] By constructing the WFP data structure and the K-gram word relationship table, the comparison module 14 will obtain a multi-word relationship table from 1 to K. The multi-word relationship in the table can be considered as the multi-word relationship between word segmentation units and word segmentation units in the corpus data. After the multi-word relationship between word segmentation units and word segmentation units is hashed and judged, the word segmentation unit combination relationship can be determined.

[0194] It should be emphasized that, due to the establishment of the K-gram relationship table, the word segmentation unit combination found by the comparison module 14 is not limited to unigrams and bigrams. The comparison module 14 can analyze the multi-word relationship by scanning the K-gram relationship table. The comparison module 14 combines the word segmentation units corresponding to the determined multi-word relationship with the word segmentation units and saves them to the word library, which can make the word library of the input method richer and make the input more accurate.

[0195] When the word segmentation unit combination relationship is determined, the word segmentation unit combination can be regarded as a new phrase, and the word segmentation unit combination is added to the word library by the output module 15.

[0196] In a specific implementation provided by this application, the input method multi-word discovery device 100 is used to discover new words. After the acquisition module 11 acquires the corpus data "due to the Covid19 virus pandemic", the word segmentation module 12 performs word segmentation on the corpus data through a deep learning-based word segmentation model to obtain word segmentation units.

[0197] The word segmentation units are shown as "due to", "Covid", "19", "virus", "pandemic". At the same time, the part-of-speech of the word segmentation units can also be obtained. For example, the part-of-speech of "due to" is a preposition, "Covid" is an English word, "19" is a number, and "virus" is a noun.

[0198] After that, the storage module 13 stores the part-of-speech of the word segmentation units, the connection frequency between the word segmentation units and the word segmentation units as the word segmentation unit attributes in the WFP data structure.

[0199] The comparison module 14 can determine the association relationship between the word segmentation units that meet the preset threshold requirements by scanning the WFP tree.

[0200] For example, through calculation, it can be determined that the information entropy of "Covid" followed by "19" and the information entropy of "19" preceded by "Covid" are both less than the first preset threshold. Therefore, the determination result of the comparison module 14 is that "Covid19" is a new phrase. Furthermore, the output module 15 adds "Covid19" as a new phrase to the thesaurus, completing the discovery and recording of new words by the input method.

[0201] After the input method multi-word discovery device 100 performs word segmentation on the acquired corpus data by driving a deep learning model, and makes a judgment on the combination of the generated word segmentation units, the accuracy and efficiency of discovering new words are improved.

[0202] It should be noted that the term "including", "comprising" or any other variant thereof is intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements not only includes those elements, but also includes other elements not expressly listed, or also includes elements inherent to such process, method, commodity or device. Without more limitations, an element defined by the statement "including one..." does not exclude the existence of another identical element in the process, method, commodity or device including the said element.

[0203] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that contain computer-usable program code.

[0204] The above is only an embodiment of the present application and is not intended to limit the present application. For those skilled in the art, the present application may have various changes and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application should be included in the scope of the claims of the present application.

Claims

1. A method for discovering multiple words in an input method, characterized in that: The following steps are involved: Get corpus data; Segmenting the corpus data using a word segmentation model to generate a number of word segmentation units; Store word segmentation units in WFP data structure; By reading the WFP data structure, comparing the correlation between word segmentation units and word segmentation units, the word segmentation unit combination is determined; Adding the word segmentation unit combination to the word library; The storing of word segmentation units in the WFP data structure specifically includes: Construct character index table and frequency feature tree to generate initial WFP data structure; Importing the sentence logic body consisting of several word segmentation units into the initial WFP data structure to generate the final WFP data structure; The frequency feature tree is composed of several nodes, and the frequency feature tree is used to describe the word frequency information of the word segmentation unit in the corpus data; the node has a preceding index set, a succeeding index set, a word segmentation unit of the current node, and the word frequency of the current node; The character index table consists of several character indexes, and a single character index represents a unique corresponding word segmentation unit; the character index is used to record the position information of the node in the frequency feature tree; The method of reading the WFP data structure, comparing the correlation between word segmentation units, and determining the word segmentation unit combination specifically includes: According to the WFP data structure, construct a multi-word relationship table; By scanning the multi-word relationship table, the expanded information entropy between word segmentation units is calculated; When the expanded information entropy of the word segmentation units and the word segmentation units is less than or equal to a first preset threshold, determining a word segmentation unit combination; The multi-word relationship table describes the association between word segmentation unit combinations.

2. The input method multi-word discovery method as claimed in claim 1, characterized in that: The step of importing the sentence logic body composed of a plurality of word segmentation units into the initial WFP data structure to generate the final WFP data structure specifically includes: The initial WFP data structure obtains the first word segmentation unit in the sentence logical body; Find the node node corresponding to the previous word segmentation unit of the first word segmentation unit pre ; Determine whether the first word segmentation unit is in the previous word segmentation unit node pre In the post-item index set postset; When the first word segmentation unit is in the previous word segmentation unit node pre In the post-item index set postset, the previous word segmentation unit node node pre The node where the first word segmentation unit is located in the post-index set postset pre->p The frequency of the node increases by 1; Get the subsequent word segmentation unit of the first word segmentation unit in the sentence logic body, and repeatedly determine whether the subsequent word segmentation unit is in the subsequent index set of the first word segmentation unit node, until the initial WFP data structure stores the sentence logic body composed of several word segmentation units, and generate the final WFP data structure.

3. The input method multi-word discovery method as claimed in claim 2, characterized in that: The step of importing the sentence logic body composed of a plurality of word segmentation units into the initial WFP data structure to generate the final WFP data structure specifically includes: The initial WFP data structure stores the logical body of a sentence consisting of several word segmentation units; The initial WFP data structure obtains the first word segmentation unit in the sentence logical body; Find the node node corresponding to the previous word segmentation unit of the first word segmentation unit pre ; Determine whether the first word segmentation unit is in the previous word segmentation unit node pre In the post-item index set postset; When the first word segmentation unit is not in the previous word segmentation unit node pre In the post-item index set postset, determine whether the first word segmentation unit is in the character index table WTable; When the first word segmentation unit is in the character index table WTable, add a record w to the character index windex corresponding to the first word segmentation unit in the character index table WTable. k POS p ; Get the next word segmentation unit of the first word segmentation unit in the sentence logic body, and repeatedly determine whether the next word segmentation unit is in the character index table WTable, until the initial WFP data structure stores the sentence logic body composed of several word segmentation units, and generate the final WFP data structure.

4. The input method multi-word discovery method as claimed in claim 1, characterized in that: The step of importing the sentence logic body composed of a plurality of word segmentation units into the initial WFP data structure to generate the final WFP data structure specifically includes: The initial WFP data structure stores the logical body of a sentence consisting of several word segmentation units; The initial WFP data structure obtains the first word segmentation unit in the sentence logical body; Find the node node corresponding to the previous word segmentation unit of the first word segmentation unit pre ; Determine whether the first word segmentation unit is in the previous word segmentation unit node pre In the post-item index set postset; When the first word segmentation unit is not in the previous word segmentation unit node pre In the post-item index set postset, determine whether the first word segmentation unit is in the character index table WTable; When the first word segmentation unit is not in the character index table WTable, a new character index windex is added to the character index table WTable based on the first word segmentation unit; Add a record w to the corresponding character index windex in the character index table WTable for the first word segmentation unit k POS p ; Get the next word segmentation unit of the first word segmentation unit in the sentence logic body, and repeatedly determine whether the next word segmentation unit is in the character index table WTable, until the initial WFP data structure stores the sentence logic body composed of several word segmentation units, and generate the final WFP data structure.

5. The input method multi-word discovery method as claimed in claim 1, characterized in that: The method of constructing a multi-word relationship table according to the WFP data structure specifically includes: By scanning the character index table, the frequency of the frequency feature tree node where the first word segmentation unit is located and the sum of the position information pointed to by the frequency feature tree where the first word segmentation unit is located are calculated; When the sum of the position information pointed to by the frequency feature tree where the first word segmentation unit is located is greater than a second preset threshold, the first word segmentation unit is stored in the unigram relationship table; Based on the uni-gram relationship table, the word segmentation unit sequence information and word frequency information in the uni-gram relationship table are expanded to the K-gram relationship table; it is expressed as follows: In the formula, To expand to K-word relationships, node i and node j To try to expand the new words that add relationships; RTable1 is a unigram word relationship table; node r,p,k-1,1 is the node corresponding to the pth combination in the rth relationship, post_node r,p,k-1,k-1 For its node r,p,k-1,k-1 A collection of indexes of subsequent items in the node.

6. The input method multi-word discovery method as claimed in claim 5, characterized in that: The expanded information entropy of word segmentation unit combinations is calculated by scanning the multi-word relationship table, which is specifically manifested as: Among them, AE is the expansion entropy; c(node i ) is the statistical node node i The number of times the word segmentation unit represented appears in the corpus data; c(pre_node r,p,k-1,1 ) is the statistical node r,p,k-1,1 The number of times the word segmentation unit represented by the node in the previous index set appears in the corpus data; To count nodes r,p,k-1,k-1 The number of times the word segmentation unit represented by the node in the post-index set of appears in the corpus data; for The multi-word expansion information entropy of for The multi-word expansion information entropy; When the calculated If it is greater than the third preset threshold, the expansion is abandoned.

7. An input method multi-word discovery device, characterized in that: include: Acquisition module, used to obtain corpus data; A word segmentation module, used to segment the corpus data through a word segmentation model to generate a number of word segmentation units; A storage module, used for storing word segmentation units in a WFP data structure; A comparison module, used for comparing the relevance between word segmentation units and word segmentation units by reading the WFP data structure, and determining the word segmentation unit combination; An output module, used for adding the word segmentation unit combination to the word library; Wherein, the storage module is used to store word segmentation units in a WFP data structure, specifically for: Construct character index table and frequency feature tree to generate initial WFP data structure; Importing the sentence logic body consisting of several word segmentation units into the initial WFP data structure to generate the final WFP data structure; The frequency feature tree is composed of several nodes, and the frequency feature tree is used to describe the word frequency information of the word segmentation unit in the corpus data; the node has a preceding index set, a succeeding index set, a word segmentation unit of the current node, and the word frequency of the current node; The character index table consists of several character indexes, and a single character index represents a unique corresponding word segmentation unit; the character index is used to record the position information of the node in the frequency feature tree; Wherein, the comparison module is specifically used for: According to the WFP data structure, construct a multi-word relationship table; By scanning the multi-word relationship table, the expanded information entropy between word segmentation units is calculated; When the expanded information entropy of the word segmentation units is less than or equal to a first preset threshold, determining a word segmentation unit combination; The multi-word relationship table describes the association between word segmentation unit combinations.

Images