Entity identification method, equipment and device

Abstract

The invention provides an entity identification method, equipment and device. The entity identification method comprises the following steps: acquiring a statement to be identified; based on a preset dictionary, constructing a word graph according to the to-be-recognized statement, and determining a path matching probability of each word segmentation path in one or more word segmentation paths contained in the word graph; and determining the word segmentation path with the maximum path matching probability in the one or more word segmentation paths as a target word segmentation path, and determining one or more words corresponding to the to-be-identified statement in a preset dictionary according to the target word segmentation path.

Description

Technical Field

[0001] The present disclosure relates to the field of language processing technology, and more specifically, to an entity recognition method, device and apparatus. Background Art

[0002] With the development of information technology, computers are increasingly used to automatically process language and perform corresponding functions. At present, unsupervised or supervised methods can be used to achieve understanding of natural language. However, in some cases, users may mistype, miss or overtype certain words in the sentence, making it difficult to effectively identify specific words (also known as entities) contained in the sentence using an unsupervised method. Using a supervised method to recognize sentences can solve the above difficulties to a certain extent. However, the premise of using a supervised method for recognition is that a large amount of manual annotation of the corpus is required in advance, which will result in high costs and cannot solve the problem of cold start. Summary of the invention

[0003] One of the purposes of the present disclosure is to provide an entity recognition method, device and apparatus.

[0004] According to a first aspect of the present disclosure, there is provided an entity recognition method, comprising:

[0005] Get the sentence to be recognized;

[0006] Based on a preset dictionary, a word graph is constructed according to the sentence to be recognized, and a path matching probability of each word segmentation path in one or more word segmentation paths included in the word graph is determined, wherein the one or more word segmentation paths include at least one fuzzy word segmentation path, in which at least one word to be recognized is distinguished from a corresponding expected word in the preset dictionary; and

[0007] A segmentation path with the largest path matching probability among the one or more segmentation paths is determined as a target segmentation path, and one or more words corresponding to the sentence to be recognized in a preset dictionary are determined according to the target segmentation path.

[0008] In some embodiments, based on a preset dictionary, constructing a word graph according to the sentence to be recognized, and determining the path matching probability of each word segmentation path in one or more word segmentation paths included in the word graph includes:

[0009] Based on the preset dictionary, a word graph including one or more precise word segmentation paths is constructed according to the sentence to be recognized, wherein in the precise word segmentation path, each of all the words to be recognized is respectively consistent with the corresponding expected word in the preset dictionary;

[0010] Determine a path matching probability of each precise word segmentation path in the one or more precise word segmentation paths;

[0011] Compare the path matching probability of each precise word segmentation path with the first preset probability threshold respectively;

[0012] When the path matching probabilities of all precise word segmentation paths are less than the first preset probability threshold, one or more fuzzy word segmentation paths are added to the word graph.

[0013] In some embodiments, the path matching probability of the word segmentation path is determined according to the word matching probability of one or more to-be-recognized words contained in the word segmentation path.

[0014] In some embodiments, the path matching probability P(w1, w2, …, w n )satisfy:

[0015] ,

[0016] Among them, P(w i ) is the i-th word to be recognized w in the word segmentation path i The probability of a word match; or

[0017] The path matching probability P(w1, w2, …, w n )satisfy:

[0018] ,

[0019] Among them, P(w i |w i-j+1 , …, w i-1 ) is the i-th word to be recognized in the word segmentation path w i The previous j-1 words from front to back are w i-j+1 ,…,w i-1 In the case of the i-th word to be recognized w i The word matching probability of , j is an integer greater than 1.

[0020] In some embodiments, when the word to be recognized in the word segmentation path is consistent with the corresponding expected word in the preset dictionary, the word occurrence probability of the expected word in the preset dictionary is used as the word matching probability of the word to be recognized.

[0021] In some embodiments, when there is a difference between a word to be recognized in the word segmentation path and a corresponding expected word in a preset dictionary, the word matching probability of the word to be recognized is determined based on the character similarity between the word to be recognized and the expected word, the phonetic similarity between the word to be recognized and the expected word, and the word occurrence probability of the expected word in the preset dictionary.

[0022] In some embodiments, the word matching probability P(w e|w)Satisfy:

[0023] ,

[0024] Among them, k is the preset granularity coefficient, charsim(w,w e ) is the word similarity between the word to be recognized and the expected word, spellsim(w,w e ) is the pinyin similarity between the word to be recognized and the expected word, P(w e ) is the probability of occurrence of the expected word in the preset dictionary.

[0025] In some embodiments, the character similarity charsim(w,w e )satisfy:

[0026] ,

[0027] Among them, D(w,w e ) is the word edit distance between the word to be recognized and the expected word, len(w) is the word length of the word to be recognized, len(w e ) is the word length of the expected word; and / or

[0028] The pinyin similarity between the word to be recognized and the expected word is spellsim(w,w e )satisfy:

[0029] ,

[0030] where D[spell(w),spell(w e )] is the phonetic edit distance between the word to be recognized and the expected word, len[spell(w)]_ is the phonetic length of the word to be recognized, len[spell(w e )] is the pinyin length of the expected word.

[0031] In some embodiments, based on a preset dictionary, constructing a word graph according to the sentence to be recognized, and determining the path matching probability of each word segmentation path in one or more word segmentation paths included in the word graph includes:

[0032] According to the word order of the sentence to be recognized, determine the current word to be recognized in the current word segmentation path;

[0033] Determine the word matching probability of the current word to be recognized, and determine the segment matching probability of the segment from the first word to be recognized in the current word segmentation path to the current word to be recognized based on the word matching probability;

[0034] Compare the segment matching probability with a second preset probability threshold;

[0035] When the segment matching probability is less than the second preset probability threshold, the current word segmentation path is removed from the word graph.

[0036] In some embodiments, the entity identification method further includes:

[0037] Access to historical data; and

[0038] Determine the word occurrence probability of words contained in the historical corpus based on the shortest path word segmentation method to construct a preset dictionary;

[0039] Among them, the word w in the preset dictionary i The word occurrence probability P(w i )satisfy:

[0040] exist In the case of ,

[0041] exist In the case of ,

[0042] Among them, c(w i ) is the word w i The number of times it appears in the historical corpus, P0 is the first preset probability.

[0043] In some embodiments, the entity identification method further includes:

[0044] Obtain one or more words corresponding to the newly added business scenario;

[0045] Determining a type corresponding to each of the one or more words; and

[0046] The word occurrence probability of the word is determined according to the word type.

[0047] In some embodiments, the entity identification method further includes:

[0048] The word occurrence probabilities of the one or more words in the preset dictionary are updated according to the one or more words in the preset dictionary corresponding to the sentence to be recognized.

[0049] In some embodiments, the preset dictionary includes a general dictionary and an entity dictionary.

[0050] According to a second aspect of the present disclosure, there is provided an entity recognition device, comprising:

[0051] An acquisition unit, wherein the acquisition unit is configured to acquire a sentence to be recognized;

[0052] a probability calculation unit, the probability calculation unit being configured to construct a word graph according to the sentence to be recognized based on a preset dictionary, and determine a path matching probability of each word segmentation path in one or more word segmentation paths included in the word graph, wherein the one or more word segmentation paths include at least one fuzzy word segmentation path, in which at least one word to be recognized is distinguished from a corresponding expected word in the preset dictionary; and

[0053] A determination unit is configured to determine a segmentation path with the largest path matching probability among the one or more segmentation paths as a target segmentation path, and determine one or more words in a preset dictionary corresponding to the sentence to be recognized according to the target segmentation path.

[0054] According to a third aspect of the present disclosure, an entity recognition device is provided, including a memory and a processor, wherein instructions are stored in the memory, and when the instructions are executed by the processor, the operations of the entity recognition method described above are implemented.

[0055] According to a fourth aspect of the present disclosure, a non-transitory computer-readable storage medium is provided, on which instructions are stored. When the instructions are executed by a processor, the operations of the entity recognition method as described above are implemented.

[0056] According to a fifth aspect of the present disclosure, a computer program product is provided, the computer program product comprising instructions, which, when executed by a processor, implement the operations of the entity recognition method as described above.

[0057] Other features and advantages of the present disclosure will become more apparent from the following detailed description of exemplary embodiments of the present disclosure with reference to the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0058] The accompanying drawings, which constitute a part of the specification, illustrate embodiments of the present disclosure and, together with the description, serve to explain the principles of the present disclosure.

[0059] The present disclosure may be more clearly understood from the following detailed description with reference to the accompanying drawings, in which:

[0060] Figure 1 A schematic diagram showing a flow chart of an entity recognition method according to an exemplary embodiment of the present disclosure is shown;

[0061] Figure 2 A word graph based on the shortest path word segmentation method in a specific example is shown;

[0062] Figure 3 A word graph of a probability maximization word segmentation method based on a bigram model in a specific example is shown;

[0063] Figure 4 A word graph including a fuzzy matching path of a sentence to be recognized in a specific example is shown;

[0064] Figure 5 A block diagram of an entity identification device according to an exemplary embodiment of the present disclosure is shown;

[0065] Figure 6 A block diagram of an entity identification device according to an exemplary embodiment of the present disclosure is shown.

[0066] Note that in the embodiments described below, the same reference numerals are sometimes used in common between different drawings to represent the same parts or parts with the same functions, and their repeated descriptions are omitted. In this specification, similar reference numerals and letters are used to represent similar items, so once an item is defined in one drawing, it does not need to be further discussed in subsequent drawings.

[0067] For ease of understanding, the positions, sizes, and ranges of various structures shown in the drawings and the like sometimes do not represent actual positions, sizes, and ranges, etc. Therefore, the disclosed invention is not limited to the positions, sizes, and ranges, etc. disclosed in the drawings and the like. In addition, the drawings are not necessarily drawn to scale, and some features may be exaggerated to show details of specific components. DETAILED DESCRIPTION

[0068] Various exemplary embodiments of the present disclosure will now be described in detail with reference to the accompanying drawings. It should be noted that the relative arrangement of components and steps, numerical expressions and numerical values ​​set forth in these embodiments do not limit the scope of the present disclosure unless otherwise specifically stated.

[0069] The following description of at least one exemplary embodiment is merely illustrative in nature and is not intended to limit the present disclosure and its application or use. It will be appreciated by those skilled in the art that they are merely exemplary, rather than exhaustive, ways in which the present disclosure may be implemented.

[0070] Technologies, methods, and equipment known to ordinary technicians in the relevant art may not be discussed in detail, but where appropriate, the technologies, methods, and equipment should be considered as part of the specification.

[0071] In computer science, sentences from users can be identified in an unsupervised or supervised manner to determine the entities (specific words) contained in the sentences, and then it is possible to perform corresponding operations based on the determined entities, such as automatically querying the indicators corresponding to the determined entities and displaying them to the user. The unsupervised method mainly segmented the sentences based on dictionaries, which requires adding the entities to the dictionaries in advance, and the specific segmentation algorithms can include the shortest path segmentation method, the probability maximization segmentation method, etc., as will be explained in detail later. The supervised method requires a large amount of corpus annotation in advance for model training. Specific models can include conditional random field (CRF) models, Transformer-based bidirectional encoder representation-conditional random field (BERT-CRF) models, and large language models (LLMs). However, with the development of business scenarios, there may be more and more long entities (containing more words), such as "abnormal flight information passenger-level accessibility rate". For such entities, users are likely to enter them incorrectly, miss them, or enter them more than once. For example, the user may enter "abnormal flight information passenger-level accessibility rate" as "abnormal flight passenger-level accessibility rate", omitting the word "information". Alternatively, the user input may also contain some typos, such as writing "passenger load factor" as "passenger seat rate". In this case, based on the traditional unsupervised method, it will be difficult to identify the user input as the corresponding entity, and it will not be possible to correct it. If a supervised method is used to identify sentences, although it may be possible to correct errors to a certain extent, a large amount of entity corpus needs to be manually annotated in advance, especially when there are a large number of newly added business scenarios. This annotation often requires a lot of manpower and time costs. Moreover, the supervised method cannot solve the problem of cold start.

[0072] In order to solve at least one of the above problems, the present disclosure proposes an entity recognition method, which is based on a preset dictionary and improves the recognition effect of a sentence by adding a fuzzy segmentation path to the segmentation path. In an exemplary embodiment of the present disclosure, Figure 1 As shown, the entity recognition method may include:

[0073] Step S110, obtaining a sentence to be recognized;

[0074] Step S120, constructing a word graph according to the sentence to be recognized based on a preset dictionary, and determining a path matching probability of each word segmentation path in one or more word segmentation paths included in the word graph; and

[0075] Step S130 , determining a segmentation path with the largest path matching probability among the one or more segmentation paths as a target segmentation path, and determining one or more words (or entities) in a preset dictionary corresponding to the sentence to be recognized according to the target segmentation path.

[0076] Here, the preset dictionary can be provided based on various methods, which is not limited herein. Additionally, the words described in this article can be single-character words or multi-character words. According to specific application requirements, the preset dictionary can include a general dictionary and an entity dictionary. Among them, the general dictionary can contain common words in general scenarios, such as words representing time, place, person, etc., and can also contain auxiliary words and function words such as "de" (的), which is not limited herein. The entity dictionary can be provided according to specific business scenarios to include specific words (or entities) involved in specific business scenarios. For example, an entity dictionary corresponding to the business scenario of the aviation industry can be provided and incorporated into the preset dictionary. Such an entity dictionary can contain entities with aviation industry characteristics such as "passenger-level accessibility rate of abnormal flight information" and "load factor". It can be understood that entity dictionaries can also be provided according to other business scenarios, which is not limited herein.

[0077] In some embodiments, the preset dictionary can be constructed based on historical corpus. Specifically, the entity recognition method can further include: obtaining historical corpus; and determining the word occurrence probability of the words included in the historical corpus based on the shortest path word segmentation method to construct the preset dictionary. Among them, for the word w i in the preset dictionary, the word occurrence probability P(w i ) can satisfy:

[0078] In the case of ,

[0079] In the case of ,

[0080] wherein, c(w i ) is the number of times the word w i appears in the historical corpus, and P0 is the first preset probability.

[0081] In the shortest path word segmentation method, for each statement included in the historical corpus, all the words in the statement can be matched to form a word graph, and then the shortest path from the starting point to the ending point is found as the best combination method, and then the number of times the corresponding word appears c(w i ) is determined to determine the word occurrence probability P(w i ). The word graph mentioned in this article can be a directed acyclic graph (DAG). Among them, each character in the statement can be used as a vertex in the word graph, and each edge in the word graph can correspond to a word segmentation, and it can be considered that the weight of each word in the word graph is equal, that is, the weight of each edge is 1. In a specific example, as Figure 2 shown, the statement "What he said is really reasonable" may be segmented into Figure 2The three segmentation paths shown in the figure can be combined to determine the best combination by calculating the length of each segmentation path, and then determine the number of occurrences of the corresponding segmentation. In addition, considering that in the probability maximization segmentation method, the path matching probability may be determined by the product of the word matching probabilities of the segmentations contained therein (as will be explained in detail later), and considering the limitations of the historical corpus (it is generally impossible to cover all possible words), in order to avoid the occurrence of a certain segmentation in the historical corpus being zero, which will lead to the path matching probability of the entire path being zero and the recognition effect being poor, for words w that do not appear in the historical corpus i , that is, in c(w i )=0, we can assign a smaller word occurrence probability to it, for example, we can let P(w i )=P0, wherein the first preset probability P0 can be set as needed, for example, P0 can be set to be similar to or equal to the word occurrence probability of the least occurring word in the historical corpus.

[0082] Further, in some embodiments, the preset dictionary can be updated according to the changes in the business scenario to adapt it to the needs of actual applications. Generally speaking, one or more words corresponding to the newly added business scenario may not have appeared in the historical corpus or the frequency of occurrence is very low. In this case, if the preset dictionary is still constructed based on the historical corpus, it may lead to a large recognition error. To this end, the words or entities corresponding to the newly added business scenario can be assigned corresponding word occurrence probabilities based on experience. Specifically, the entity recognition method may also include: obtaining one or more words corresponding to the newly added business scenario; determining the type corresponding to each of the one or more words; and determining the word occurrence probability of the word according to the type of the word. Here, the specific value of the word occurrence probability of a word in the newly added business scenario can be determined according to its type, and can also be determined according to the word occurrence probability of related words in existing business scenarios (for example, existing business scenarios similar to the newly added business scenarios), for example, it can be set to be similar or equal to the word occurrence probability of the related words. In a specific example, for example, in the business scenario of the aviation industry, the word occurrence probability P1 of the indicator type word can be greater than the word occurrence probability P2 of the conditional type word, where the indicator type word can be a word that represents a concept such as an overall quantitative feature, such as "occupancy rate", "punctuality rate", etc., and the conditional type word can be a word that represents the status of things, such as "the route from Beijing to Shanghai", "during the Spring Festival", etc., without limitation here. It can be understood that, if necessary, the newly added words or entities can also be given appropriate, non-zero word occurrence probabilities based on other factors, without limitation here.

[0083] In the case where a preset dictionary is obtained, the sentence can be recognized based on the preset dictionary. In some embodiments, the sentence to be recognized can be input by the user, for example, the user can input the sentence to be recognized through a pinyin input method, or the user can directly input the sentence to be recognized through voice input. In the case where the pinyin of the words in the sentence to be recognized is required for recognition (as will be explained in detail later), the corresponding pinyin can be directly obtained from the pinyin input, or the pinyin can be extracted according to the user's voice, or the corresponding pinyin can be determined according to the glyphs of the words in the sentence to be recognized, etc., without limitation here.

[0084] After obtaining the sentence to be recognized, a word graph can be constructed based on the preset dictionary according to the sentence to be recognized, and the path matching probability of each word segmentation path in one or more word segmentation paths contained in the word graph can be determined. In some embodiments, the path matching probability of the word segmentation path can be determined based on the probability maximization word segmentation method (also known as the word segmentation method of the n-gram model). Specifically, the path matching probability of the word segmentation path can be determined based on the word matching probability of one or more words to be recognized contained in the word segmentation path. Assuming that the sentence to be recognized consists of n words in total, P(w1, w2, …, w n ) is used to represent the path matching probability of the word segmentation path. According to the chain rule in probability theory, the path matching probability P(w1, w2, …, w n ) can be expressed as:

[0085] P(w1, w2, …, w n )=P(w1)*P(w2|w1)*P(w3|w1, w2)*…*P(w n |w1, w2, …, w n-1 ),

[0086] Among them, P(w i ) is the i-th word to be recognized w in the word segmentation path i The word matching probability, P(w i |w i-j+1 , …,w i-1 ) is the i-th word to be recognized in the word segmentation path w i The previous j-1 words from front to back are w i-j+1 ,…,w i-1 In the case of the i-th word to be recognized w i The word matching probability.

[0087] However, it is generally difficult to directly calculate the above probabilities. For this reason, we can adopt the n-gram assumption that the current word is only related to the previous words. Accordingly:

[0088] P(wi |w1, w2, …, w i-1 )=P(w i |w i-j+1 ,…,w i-1 ),

[0089] From this we can conclude that:

[0090] .

[0091] Specifically, when j = 1, that is, corresponding to a unigram model, ;

[0092] In the case of j=2, corresponding to the binary model, ,

[0093] In the case of j=3, corresponding to the ternary model, .

[0094] You can choose a suitable value of j as needed to calculate the path matching probability based on the word matching probability. In a specific example, for simplicity, you can choose a unigram model (i.e., j=1) to calculate the path matching probability of the word segmentation path. In another specific example, Figure 3 As shown in Figure 1, a word graph based on a bigram model is shown. It can be seen that in multivariate models such as bigram models, there may be data sparseness. For example, P(w i |w i-1 ) may be zero. In order to avoid the word matching probability of a certain word being zero, which will lead to the path matching probability containing it being directly zero, it may be necessary to use smoothing technology to process the data, such as assigning some P(w i |w i-1 )A smaller preset probability is used to improve the recognition effect.

[0095] In this article, one or more word segmentation paths in the word graph may include at least one fuzzy word segmentation path, in which at least one word to be recognized may be different from a corresponding expected word in a preset dictionary. In addition, the word segmentation path may also include an accurate word segmentation path, in which each of all words to be recognized is consistent with a corresponding expected word in a preset dictionary. For example, in Figure 4In a specific example shown, the second word segmentation path from top to bottom (word segmentation path 2) is an exact word segmentation path, where each word to be recognized, namely "客 (customer)", "坐 (sit)", and "率 (rate)", is respectively consistent with the corresponding expected words "客 (customer)", "坐 (sit)", and "率 (rate)" in the preset dictionary; while the first word segmentation path and the fourth word segmentation path from top to bottom are fuzzy word segmentation paths, where there are differences between the words to be recognized "客坐 (customer sitting)" or "客坐率 (customer sitting rate)" and the corresponding expected word "客座率 (occupancy rate)" in the preset dictionary.

[0096] In some embodiments, the sentence to be recognized can be segmented based on the precise segmentation path first. If the segmentation effect is not ideal, a fuzzy segmentation path can be added. Specifically, based on a preset dictionary, a word graph is constructed according to the sentence to be recognized, and the path matching probability of each segmentation path in one or more segmentation paths contained in the word graph is determined. It can include: based on a preset dictionary, a word graph containing one or more precise segmentation paths is constructed according to the sentence to be recognized; Determine the path matching probability of each precise segmentation path in the one or more precise segmentation paths; And compare the path matching probability of each precise segmentation path and the first preset probability threshold value respectively. Here, the first preset probability threshold value can be set according to the desired recognition effect. The higher the first preset probability threshold value is set, the better the recognition accuracy is generally, but correspondingly, the recognition difficulty will also increase. In some specific examples, the first preset probability threshold value can be set to 80%~85%, or 85%~90%, or 90%~95%. When the path matching probability of at least one precise word segmentation path is greater than or equal to the first preset probability threshold, it indicates that a relatively accurate recognition effect can be obtained based on the precise word segmentation path. At this time, the precise word segmentation path with the largest path matching probability can be determined as the target word segmentation path, and one or more words (entities) corresponding to the sentence to be recognized in the preset dictionary are determined according to the target word segmentation path. In addition, when the path matching probability of all precise word segmentation paths is less than the first preset probability threshold, it indicates that it is difficult to obtain a relatively accurate recognition effect based only on the precise word segmentation path. At this time, there may be mistyped, missing or over-typed words in the sentence to be recognized, resulting in recognition failure. Accordingly, one or more fuzzy word segmentation paths can be added to the word graph to further determine the path matching probability of the fuzzy word segmentation path, and the word segmentation path with the largest path matching probability is determined as the target word segmentation path, and one or more words (entities) corresponding to the sentence to be recognized in the preset dictionary are determined according to the target word segmentation path. In this case, the word segmentation path with the largest path matching probability may be a fuzzy word segmentation path. Alternatively, in some embodiments, one or more precise word segmentation paths and one or more fuzzy word segmentation paths may be directly constructed based on the sentence to be recognized, and the path matching probability of each word segmentation path may be calculated, and then the word segmentation path with the largest path matching probability among the one or more word segmentation paths may be determined as the target word segmentation path, and one or more words (entities) corresponding to the sentence to be recognized in the preset dictionary may be determined based on the target word segmentation path.

[0097] In some embodiments, especially when the word to be recognized in the word segmentation path (the word segmentation path can be a precise word segmentation path or a fuzzy word segmentation path) is consistent with the corresponding expected word in the preset dictionary, the word occurrence probability of the expected word in the preset dictionary can be directly used as the word matching probability of the word to be recognized to simplify the calculation.

[0098] In addition, in some embodiments, especially when there is a difference between the word to be recognized in the word segmentation path and the corresponding expected word in the preset dictionary, the word matching probability of the word to be recognized can be determined according to the character similarity between the word to be recognized and the expected word, the pinyin similarity between the word to be recognized and the expected word, and the word occurrence probability of the expected word in the preset dictionary. Here, the word matching probability of the word to be recognized can be determined only for the word to be recognized that is different from the expected word according to the character similarity, pinyin similarity and the word occurrence probability of the expected word, and for the word to be recognized that is the same as the expected word, the word occurrence probability of the expected word in the preset dictionary can be directly used as the word matching probability of the word to be recognized as described above. Alternatively, the word matching probability of the word to be recognized can also be determined for all words to be recognized in the fuzzy word segmentation path according to the character similarity, pinyin similarity and the word occurrence probability of the expected word, and for all words to be recognized in the precise word segmentation path, the word occurrence probability of the expected word in the preset dictionary is directly used as the word matching probability of the word to be recognized. Alternatively, for all the words to be recognized in all the word segmentation paths, the word matching probability of the words to be recognized can be determined according to the character similarity, the pinyin similarity and the word occurrence probability of the expected word.

[0099] In a specific example, the word matching probability P(w e |w) can satisfy:

[0100] ,

[0101] Among them, k is the preset granularity coefficient, charsim(w,w e ) is the word similarity between the word to be recognized and the corresponding expected word, spellsim(w,w e ) is the phonetic similarity between the word to be recognized and the corresponding expected word, P(w e ) is the probability of occurrence of the corresponding expected word in the preset dictionary.

[0102] The preset granularity coefficient k can be set according to experience or needs. Generally speaking, the larger the value of k, the larger the granularity of the word segmentation and the shorter the word segmentation path; and the smaller the value of k, the closer the fuzzy word segmentation path will be to the precise word segmentation path, and the difficulty of identifying erroneous input, missing input or multiple input may increase.

[0103] In addition, the character similarity and phonetic similarity between the word to be recognized and the corresponding expected word can be specifically defined as needed. Generally, the character similarity and phonetic similarity can be defined as numbers with values ​​between 0 and 1, and the larger the value, the higher the similarity.

[0104] In a specific example, the character similarity charsim(w,w e ) can be expressed as:

[0105] ,

[0106] where D(w, w e ) is the character edit distance between the word to be recognized and the corresponding expected word, len(w) is the character length of the word to be recognized, and len(w e ) is the character length of the expected word. Here, the edit distance is a method for measuring the similarity between two strings, and the edit distance can measure the minimum number of single-character edit operations (insertion, deletion, or replacement) required to convert one string into another. For example, the edit distance between the two words "客坐率" and "客座率" can be 1 (corresponding to the operation of replacing "坐" with "座"). Additionally, the lengths of "客坐率" and "客座率" are each 3. According to the calculation method described above, the character similarity between "客坐率" and "客座率" is 2 / 3.

[0107] In a specific example, the pinyin similarity spellsim(w, w e ) between the word to be recognized and the corresponding expected word can be expressed as:

[0108] ,

[0109] where D[spell(w), spell(w e )] is the pinyin edit distance between the word to be recognized and the corresponding expected word, len[spell(w)] is the pinyin length of the word to be recognized, and len[spell(w e )] is the pinyin length of the expected word. It should be noted that considering the characteristics of the Chinese language, here the initials, medial vowels (if any), and finals in the pinyin can be regarded as a whole respectively to calculate the pinyin edit distance and pinyin length. For example, the pinyins of "客坐率" and "客座率" are both "ke / z u o / l v", their pinyin edit distance is zero, and their pinyin lengths are each 2 + 3 + 2 = 7. Correspondingly, the pinyin similarity between "客坐率" and "客座率" is 1. In another example, the pinyins "chang" and "chi" only differ in the finals, their pinyin edit distance is 1 (replacing "ang" with "i"), and their pinyin lengths are each 2, so the pinyin similarity between "chang" and "chi" is 1 / 2.

[0110] According to the above description, in the Figure 4 specific example shown, the path matching probability P(客座率|客坐率) corresponding to the segmentation path 4 will satisfy:

[0111] P(occupancy rate|occupancy rate)=k*(2 / 3+1)*P(occupancy rate)=5*k*P(occupancy rate) / 3,

[0112] Among them, P (occupancy rate) is the probability of the word "occupancy rate" appearing in the preset dictionary. Figure 4 Among the four segmentation paths shown, segmentation path 4 has the largest path matching probability, so segmentation path 4 can be determined as the target segmentation path, and thus "occupancy rate" can be determined as the word or entity corresponding to "occupancy rate" according to the target segmentation path. It can also be understood that in some embodiments, the sentence to be identified may contain multiple words or multiple entities, and the corresponding multiple words or entities can also be identified according to the scheme described in this article, which is not limited here.

[0113] Further, in some embodiments, in order to reduce the amount of calculation and improve the efficiency of the algorithm, some segmentation paths with too small path matching probabilities can be clipped or removed according to the segment matching probability of a part of the segmentation path. Specifically, based on a preset dictionary, a word graph is constructed according to the sentence to be recognized, and the path matching probability of each segmentation path in one or more segmentation paths contained in the word graph is determined, which may include: determining the current word to be recognized in the current segmentation path according to the word order of the sentence to be recognized; determining the word matching probability of the current word to be recognized, and determining the segment matching probability of the segment from the first word to be recognized to the current word to be recognized in the current segmentation path according to the word matching probability; and comparing the segment matching probability with the second preset probability threshold. Here, the second preset probability threshold can be set according to experience or needs. The smaller the second preset probability threshold is set, the less likely the segmentation path is removed or clipped in advance, but accordingly, the amount of calculation of the entire recognition algorithm may increase and the calculation efficiency decreases. In some specific examples, the second preset probability threshold may be set to 40% to 45%, or 45% to 50%, or 50% to 55%, or 55% to 60%, or 60% to 65%, or 65% to 70%. When the segment matching probability is greater than or equal to the second preset probability threshold, the next segmentation may be determined for the segmentation path and the corresponding word matching probability may be calculated. Figure 4 In the word segmentation path 1, if P(occupancy rate|passenger seat) is greater than or equal to the second preset probability threshold, the word segmentation path can be continued, and its next word segmentation can be determined to be "rate", and P(rate) can be continued. In addition, when the segment matching probability is less than the second preset probability threshold, the current word segmentation path can be removed from the word graph, that is, the calculation for the word segmentation path is no longer continued. For example, for Figure 2For the word segmentation path 3 shown in , if P (occupancy rate|customer) is less than the second preset probability threshold, then according to the relationship between the path matching probability and the word matching probability described above, it can be determined that the entire path matching probability will also be very small, and the word segmentation path 3 can be trimmed or removed in time to reduce the amount of calculation and improve the efficiency of the algorithm.

[0114] In some embodiments, in order to improve the sentence recognition effect, the preset dictionary can also be adjusted or updated in a timely manner according to the results of the previous recognition. Specifically, the entity recognition method can also include: updating the word occurrence probability of one or more words in the preset dictionary according to one or more words in the preset dictionary corresponding to the sentence to be recognized. For example, the preset dictionary can be updated regularly according to the collected recognition results, or the preset dictionary can be updated when a certain amount of sentence recognition is accumulated, etc., which is not limited here.

[0115] In addition, the present disclosure also proposes an entity recognition device, such as Figure 5 As shown, the entity recognition device may include: an acquisition unit 501, a probability calculation unit 502 and a determination unit 503. Among them, the acquisition unit 501 can be configured to acquire a sentence to be recognized. The probability calculation unit 502 can be configured to build a word graph according to the sentence to be recognized based on a preset dictionary, and determine the path matching probability of each word segmentation path in one or more word segmentation paths contained in the word graph. The determination unit 503 can be configured to determine the word segmentation path with the largest path matching probability among the one or more word segmentation paths as the target word segmentation path, and determine one or more words in the preset dictionary corresponding to the sentence to be recognized according to the target word segmentation path. The specific operations of the various units in the entity recognition device 500 can refer to the detailed description of the entity recognition method above, and will not be repeated here.

[0116] According to another aspect of the present disclosure, Figure 6 As shown, an entity recognition device 600 is also provided. The entity recognition device 600 may include a memory 601 and a processor 602. The memory 601 may store instructions, and when the instructions are executed by the processor 602, the operation of the entity recognition method described above may be implemented.

[0117] Specifically, the processor 602 can perform various actions and processes according to the instructions stored in the memory 601. The processor 602 can be an integrated circuit chip with signal processing capabilities. The above processor can be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. The various methods, steps and logic block diagrams disclosed in the embodiments of the present disclosure can be implemented or executed. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor, etc., which can be an X86 architecture or an ARM architecture, etc.

[0118] The memory 601 stores executable instructions, which implement the entity recognition method described above when executed by the processor 602. The memory 601 can be a volatile memory or a non-volatile memory, or can include both volatile and non-volatile memories. The non-volatile memory can be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM) or a flash memory. The volatile memory can be a random access memory (RAM), which is used as an external cache. By way of exemplary but not limiting explanation, many forms of RAM are available, such as static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDRSDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous connection dynamic random access memory (SLDRAM) and direct memory bus random access memory (DR RAM). It should be noted that the memory of the method described herein is intended to include but is not limited to these and any other suitable types of memory.

[0119] The present disclosure also proposes a non-transitory computer-readable storage medium, on which instructions are stored. When the instructions are executed by a processor, the operations of the entity recognition method described above can be implemented.

[0120] Similarly, non-transitory computer-readable storage media in embodiments of the present disclosure are intended to include, but are not limited to, the above and any other suitable types of memory.

[0121] The present disclosure also proposes a computer program product, which may include instructions. When the instructions are executed by a processor, the operations of the entity recognition method described above can be implemented.

[0122] Instructions may be any set of instructions to be executed directly by one or more processors, such as machine code, or any set of instructions to be executed indirectly, such as a script. The terms "instructions," "applications," "processes," "steps," and "programs" herein may be used interchangeably herein. Instructions may be stored in an object code format for direct processing by one or more processors, or in any other computer language, including scripts or collections of independent source code modules that are interpreted on demand or compiled in advance. Instructions may include instructions that cause one or more processors, such as to act as various neural networks that may exist. The functions, methods, and routines of instructions are explained in more detail elsewhere herein.

[0123] In the technical solution disclosed in the present invention, by adding a fuzzy segmentation path to the word graph based on the unsupervised segmentation method, it is possible to approximately find the words to be recognized that match the words or entities in the sentence to be recognized, so that the words or entities that are wrongly input, missed or over-entered by the user can be recognized and corrected to a certain extent. Further, in some embodiments, the word matching probability in the segmentation path is determined based on both the word similarity and the pinyin similarity, and then the path matching probability of the segmentation path is determined, which can be combined with the currently widely used pinyin input method or voice input method to better solve the problem of typos of homophonic characters. In addition, the use of an unsupervised method to segment the sentence to be recognized can flexibly expand more business scenarios by updating the preset dictionary without the need for manual annotation of the entity corpus, which helps to greatly save time and manpower, reduce costs, and effectively solve the cold start problem.

[0124] The words "left", "right", "front", "back", "top", "bottom", "up", "down", "high", "low", etc., in the specification and claims, if present, are used for descriptive purposes and are not necessarily used to describe an unchanging relative position. It should be understood that the words so used are interchangeable under appropriate circumstances, so that the embodiments of the present disclosure described herein, for example, can operate in other orientations than those shown or otherwise described herein. For example, when the device in the drawings is turned over, features originally described as "above" other features may now be described as "below" the other features. The device may also be oriented in other ways (rotated 90 degrees or in other orientations), and the relative spatial relationships will be interpreted accordingly.

[0125] In the specification and claims, when an element is referred to as being "on", "attached", "connected", "coupled", or "contacting" another element, the element may be directly on, directly attached, directly connected, directly coupled, or directly contacting another element, or one or more intermediate elements may be present. In contrast, when an element is referred to as being "directly" "on", "directly attached", "directly connected", "directly coupled", or "directly contacting" another element, there will be no intermediate elements. In the specification and claims, a feature being arranged "adjacent" to another feature may refer to a feature having a portion that overlaps with an adjacent feature or a portion that is located above or below an adjacent feature.

[0126] As used herein, the word "exemplary" means "serving as an example, instance, or illustration" rather than as a "model" to be exactly copied. Any implementation described as an example herein is not necessarily to be construed as preferred or advantageous over other implementations. Furthermore, the present disclosure is not to be limited by any expressed or implied theory given in the technical field, background, summary, or detailed description.

[0127] As used herein, the term "substantially" is intended to include any minor variations due to design or manufacturing imperfections, device or component tolerances, environmental influences, and / or other factors. The term "substantially" also allows for deviations from a perfect or ideal situation due to parasitic effects, noise, and other practical considerations that may exist in actual implementations.

[0128] In addition, the terms "first", "second" and the like may also be used herein for reference purposes only and are not intended to be limiting. For example, the terms "first", "second" and other such numerical terms referring to structures or elements do not imply a sequence or order unless the context clearly indicates otherwise.

[0129] It should also be understood that when the term “include / comprises” is used in this document, it indicates the presence of the specified features, integers, steps, operations, units and / or components, but does not exclude the presence or addition of one or more other features, integers, steps, operations, units and / or components and / or their combinations.

[0130] In this disclosure, the term "provide" is used in a broad sense to cover all ways of obtaining an object, and thus "providing an object" includes but is not limited to "purchasing", "preparing / manufacturing", "arranging / setting up", "installing / assembling", and / or "ordering" an object, etc.

[0131] As used herein, the term "and / or" includes any and all combinations of one or more of the associated listed items. The terms used herein are for the purpose of describing specific embodiments only and are not intended to limit the present disclosure. As used herein, the singular forms "a", "an", and "the" are also intended to include the plural forms, unless the context clearly indicates otherwise.

[0132] Those skilled in the art will appreciate that the boundaries between the above operations are merely illustrative. Multiple operations can be combined into a single operation, a single operation can be distributed in additional operations, and operations can be performed at least partially overlapping in time. Moreover, alternative embodiments may include multiple instances of a specific operation, and the order of operations may be changed in various other embodiments. However, other modifications, variations, and replacements are equally possible. Aspects and elements of all embodiments disclosed above may be combined in any manner and / or in combination with aspects or elements of other embodiments to provide multiple additional embodiments. Therefore, this specification and accompanying drawings should be considered illustrative, not restrictive.

[0133] Although some specific embodiments of the present disclosure have been described in detail by way of example, it should be understood by those skilled in the art that the above examples are for illustration only and are not intended to limit the scope of the present disclosure. The various embodiments disclosed herein may be combined in any manner without departing from the spirit and scope of the present disclosure. It should also be understood by those skilled in the art that various modifications may be made to the embodiments without departing from the scope and spirit of the present disclosure. The scope of the present disclosure is defined by the appended claims.

Claims

1. An entity recognition method, characterized in that: The entity recognition method comprises: Get the sentence to be recognized; Based on a preset dictionary, a word graph is constructed according to the sentence to be recognized, and a path matching probability of each word segmentation path in one or more word segmentation paths included in the word graph is determined, wherein the one or more word segmentation paths include at least one fuzzy word segmentation path, in which at least one word to be recognized is distinguished from a corresponding expected word in the preset dictionary; and A segmentation path with the largest path matching probability among the one or more segmentation paths is determined as a target segmentation path, and one or more words corresponding to the sentence to be recognized in a preset dictionary are determined according to the target segmentation path.

2. The entity recognition method according to claim 1, characterized in that: Based on the preset dictionary, a word graph is constructed according to the sentence to be recognized, and the path matching probability of each word segmentation path in one or more word segmentation paths included in the word graph is determined, including: Based on the preset dictionary, a word graph including one or more precise word segmentation paths is constructed according to the sentence to be recognized, wherein in the precise word segmentation path, each of all the words to be recognized is respectively consistent with the corresponding expected word in the preset dictionary; Determine a path matching probability of each precise word segmentation path in the one or more precise word segmentation paths; Compare the path matching probability of each precise word segmentation path with the first preset probability threshold respectively; When the path matching probabilities of all precise word segmentation paths are less than the first preset probability threshold, one or more fuzzy word segmentation paths are added to the word graph.

3. The entity recognition method according to claim 1, characterized in that: The path matching probability of the word segmentation path is determined according to the word matching probability of one or more words to be recognized contained in the word segmentation path.

4. The entity recognition method according to claim 3, characterized in that: The path matching probability P(w1,w2, …, w n )satisfy: ， Among them, P(w i ) is the i-th word to be recognized w in the word segmentation path i The probability of a word match; or The path matching probability P(w1, w2, …, w n )satisfy: ， Among them, P(w i |w i-j+1 , …, w i-1 ) is the i-th word to be recognized in the word segmentation path w i The previous j-1 words from front to back are w i-j+1 ,…,w i-1 In the case of the i-th word to be recognized w i The word matching probability of , j is an integer greater than 1.

5. The entity recognition method according to claim 3, characterized in that: When the word to be recognized in the word segmentation path is consistent with the corresponding expected word in the preset dictionary, the word occurrence probability of the expected word in the preset dictionary is used as the word matching probability of the word to be recognized.

6. The entity recognition method according to claim 3, characterized in that: When there is a difference between the word to be recognized in the word segmentation path and the corresponding expected word in the preset dictionary, the word matching probability of the word to be recognized is determined based on the character similarity between the word to be recognized and the expected word, the pinyin similarity between the word to be recognized and the expected word, and the word occurrence probability of the expected word in the preset dictionary.

7. The entity recognition method according to claim 6, characterized in that: The word matching probability P(w e |w)Satisfy: ， Among them, k is the preset granularity coefficient, charsim(w,w e ) is the word similarity between the word to be recognized and the expected word, spellsim(w,w e ) is the pinyin similarity between the word to be recognized and the expected word, P(w e ) is the probability of occurrence of the expected word in the preset dictionary.

8. The entity recognition method according to claim 7, characterized in that: The character similarity between the word to be recognized and the expected word is charsim(w,w e )satisfy: ， Among them, D(w,w e ) is the word edit distance between the word to be recognized and the expected word, len(w) is the word length of the word to be recognized, len(w e ) is the word length of the expected word; and / or The pinyin similarity between the word to be recognized and the expected word is spellsim(w,w e )satisfy: ， where D[spell(w),spell(w e )] is the phonetic edit distance between the word to be recognized and the expected word, len[spell(w)]_ is the phonetic length of the word to be recognized, len[spell(w e )] is the pinyin length of the expected word.

9. The entity recognition method according to claim 1, characterized in that: Based on the preset dictionary, a word graph is constructed according to the sentence to be recognized, and the path matching probability of each word segmentation path in one or more word segmentation paths included in the word graph is determined, including: According to the word order of the sentence to be recognized, determine the current word to be recognized in the current word segmentation path; Determine the word matching probability of the current word to be recognized, and determine the segment matching probability of the segment from the first word to be recognized in the current word segmentation path to the current word to be recognized based on the word matching probability; Compare the segment matching probability with a second preset probability threshold; When the segment matching probability is less than the second preset probability threshold, the current word segmentation path is removed from the word graph.

10. The entity recognition method according to any one of claims 1 to 9, characterized in that: The entity recognition method further includes: Access to historical data; and Determine the word occurrence probability of words contained in the historical corpus based on the shortest path word segmentation method to construct a preset dictionary; Among them, the word w in the preset dictionary i The word occurrence probability P(w i )satisfy: exist In the case of , exist In the case of , Among them, c(w i ) is the word w i The number of times it appears in the historical corpus, P0 is the first preset probability.

11. The entity recognition method according to any one of claims 1 to 9, characterized in that: The entity recognition method further includes: Obtain one or more words corresponding to the newly added business scenario; Determining a type corresponding to each of the one or more words; and The word occurrence probability of the word is determined according to the word type.

12. The entity recognition method according to any one of claims 1 to 9, characterized in that: The entity recognition method further includes: The word occurrence probabilities of the one or more words in the preset dictionary are updated according to the one or more words in the preset dictionary corresponding to the sentence to be recognized.

13. The entity recognition method according to any one of claims 1 to 9, characterized in that: The preset dictionaries include general dictionaries and entity dictionaries.

14. An entity recognition device, characterized in that: The entity recognition device comprises: An acquisition unit, wherein the acquisition unit is configured to acquire a sentence to be recognized; a probability calculation unit, the probability calculation unit being configured to construct a word graph according to the sentence to be recognized based on a preset dictionary, and determine a path matching probability of each word segmentation path in one or more word segmentation paths included in the word graph, wherein the one or more word segmentation paths include at least one fuzzy word segmentation path, in which at least one word to be recognized is distinguished from a corresponding expected word in the preset dictionary; and A determination unit is configured to determine a segmentation path with the largest path matching probability among the one or more segmentation paths as a target segmentation path, and determine one or more words in a preset dictionary corresponding to the sentence to be recognized according to the target segmentation path.

15. An entity recognition device, characterized in that: The entity recognition device includes a memory and a processor, wherein instructions are stored in the memory, and when the instructions are executed by the processor, operations of the entity recognition method according to any one of claims 1 to 13 are implemented.

16. A non-transitory computer-readable storage medium, characterized in that: The non-transitory computer-readable storage medium stores instructions, and when the instructions are executed by the processor, the operation of the entity recognition method according to any one of claims 1 to 13 is implemented.

17. A computer program product, characterized in that The computer program product comprises instructions, which, when executed by a processor, implement the operation of the entity identification method according to any one of claims 1 to 13.

Images