Sentence recognition method, device and computer readable medium

Abstract

The present application discloses a sentence recognition method, device and computer-readable medium, which obtains an original combination of multiple words arranged in word order; inputs the original combination into a target N-gram grammar model, and if the original combination contains a name in a list of names of people and places, the target N-gram grammar model replaces the name contained in the original combination with a corresponding category label in the list of names of people and places, obtains the replaced combination, and calculates the probability associated with the replaced category label, and obtains the probability of the original combination as a sentence according to the probability associated with each word in the replaced combination, the probability associated with the category label, and the proportion of all names under the corresponding category label in the list of names of people and places. Compared with the existing N-gram grammar model, the target N-gram grammar model of the present application makes it easier to correctly recognize sentences containing names of people or places.

Description

Technical Field

[0001] The present invention relates to the field of language recognition technology, and in particular to a sentence recognition method, device and computer-readable medium. Background Art

[0002] Language models are used to calculate the probability of a combination of multiple words appearing as a sentence. They are important modules in applications such as speech recognition, handwriting recognition, and input methods. N-gram grammar is a common modeling method in language models. It assumes that the probability of the current word appearing depends only on the previous N-1 words, and thus calculates the probability of a combination of multiple words appearing as a sentence. For example, in the case of bigram grammar, the probability of the sentence "I am Xiaoming" P (I am Xiaoming) is equal to the product of P (I), P (I am / I), and P (Xiaoming / I am).

[0003] However, since words with specific identifiers such as names of people and places are relatively rare, even if a large number of names of people and places are collected so that the language model can recognize the semantics of names of people and places, it is impossible for the sentences used to train the model to include all the usage forms of names of people and places, resulting in sentences containing names of people or places being difficult to correctly recognize, that is, the sparsity problem of language models related to names of people and places is relatively serious. For example, although the name "Xiao Ming" is collected in the language model, sentences such as "I am Xiao Ming" are not collected in the sentence set of the training model. Therefore, the probability of P(Xiao Ming / is), that is, the probability of "is" followed by "Xiao Ming" is 0, and the probability of "I am Xiao Ming" as a sentence is also 0, which cannot be recognized by the language model. Summary of the invention

[0004] Based on the above-mentioned deficiencies of the prior art, the present application proposes a sentence recognition method, device and computer-readable medium to solve the sparse problem of language models related to names of people and places.

[0005] The first aspect of the present invention discloses a sentence recognition method, comprising:

[0006] Obtaining the original combination of multiple words arranged in word order;

[0007] Input the original combination into the target N-gram grammar model to obtain the probability of the original combination as a sentence; wherein, if the original combination contains a name in the list of names of people and places, the target N-gram grammar model replaces the name contained in the original combination with the corresponding category label in the list of names of people and places to obtain a replaced combination, and calculates the probability associated with the category label in the replaced combination, and obtains the probability of the original combination as a sentence based on the word-related probability of each word in the replaced combination, the probability associated with the category label, and the proportion of all names under the corresponding category label in the list of names of people and places;

[0008] The names in the list of names of persons and places include: names of persons and places; the category labels corresponding to the names in the list of names of persons and places are set according to preset classification rules.

[0009] Optionally, in the above sentence recognition method, the method for constructing the target N-gram grammar model includes:

[0010] Creating a first corpus; wherein each sentence in the first corpus contains a name; the name includes a person's name and / or a place name;

[0011] According to the preset classification rule, the names in the first corpus are replaced with corresponding category labels to obtain a replaced first corpus;

[0012] According to the replaced first corpus, an initial N-gram grammar model is trained; wherein the initial N-gram grammar model is used to calculate the probability of each word in the combination containing the replaced category label and the probability of the replaced category label;

[0013] The target N-gram grammar model is obtained based on the list of names and places and the initial N-gram grammar model; wherein the target N-gram grammar model is used to calculate the probability of a combination of multiple words containing the names in the list of names and places as a sentence; the list of names and places includes at least the names of people and places included in each sentence in the first corpus.

[0014] Optionally, in the sentence recognition method, the probability of the original combination being a sentence is obtained according to the word-related probability of each word in the replaced combination, the probability of the category label, and the proportion of all names in the original combination under the category label corresponding to the name in the list of personal names and place names, including:

[0015] Obtaining the probability of the names in the original combination according to the probability of the category label correlation and the proportion of all names under the category label corresponding to the name in the list of personal names and place names;

[0016] The probability of the original combination being a sentence is obtained based on the probability of the name in the original combination and the probability of the word in each word in the replaced combination.

[0017] Optionally, in the sentence recognition method, the probability of the original combination being a sentence is obtained according to the word-related probability of each word in the replaced combination, the probability of the category label, and the proportion of all names in the original combination under the category label corresponding to the name in the list of personal names and place names, including:

[0018] According to the number of names corresponding to the category labels in the replaced combination in the list of names and places, the proportion of all names under the category labels corresponding to the names in the original combination in the list of names and places is obtained;

[0019] The probability associated with the category label is multiplied by the proportion of all names in the original combination under the category label corresponding to the name in the list of personal names and place names to obtain the probability associated with the name in the original combination;

[0020] The probability of the name in the original combination is multiplied by the probability of the word in each word in the replaced combination to obtain the probability of the original combination being a sentence.

[0021] Optionally, in the above sentence recognition method, the list of names and places also includes: a weight value corresponding to each name in the list of names and places;

[0022] The method of obtaining the probability of the original combination being a sentence according to the probability of word correlation of each word in the replaced combination, the probability of category label correlation, and the proportion of all names under the category label corresponding to the name in the original combination in the list of names and places includes:

[0023] According to the weight values ​​corresponding to all the names under the category labels corresponding to the names in the original combination in the list of names and places, the proportions of all the names under the category labels corresponding to the names in the original combination in the list of names and places are obtained;

[0024] The probability associated with the category label is multiplied by the proportion of all names in the original combination under the category label corresponding to the name in the list of personal names and place names to obtain the probability associated with the name in the original combination;

[0025] The probability of the name in the original combination is multiplied by the probability of the word in each word in the replaced combination to obtain the probability of the original combination being a sentence.

[0026] Optionally, in the above sentence recognition method, if the sentences in the first corpus contain name agglutinative words, then according to the preset classification rule, the names in the corpus are replaced with corresponding category labels to obtain the replaced first corpus, including:

[0027] The agglutinative words containing names in the sentences of the first corpus are divided into root parts and affix parts; wherein the root part is a name;

[0028] According to the preset classification rule, the root part of the agglutinative language is replaced with a corresponding category label to obtain a replaced first corpus.

[0029] Optionally, in the sentence recognition method, inputting the original combination into a target N-gram model to obtain a probability of the original combination being a sentence includes:

[0030] Inputting the original combination into a target N-gram model to obtain a first probability of the original combination being a sentence;

[0031] Inputting the original combination into a universal N-gram model to obtain a second probability that the original combination is a sentence; wherein the universal N-gram model is trained by a universal corpus; the universal corpus includes sentences containing names and sentences not containing names;

[0032] The probability that the original combination is a sentence is obtained according to a preset weight coefficient, a first probability that the original combination is a sentence, and a second probability that the original combination is a sentence.

[0033] Optionally, in the sentence recognition method, obtaining the probability that the original combination is a sentence according to a preset weight coefficient, a first probability that the original combination is a sentence, and a second probability that the original combination is a sentence includes:

[0034] Substituting the preset weight coefficient, the first probability of the original combination being a sentence, and the second probability of the original combination being a sentence into the first formula to obtain the probability of the original combination being a sentence;

[0035] Among them, the first formula is: P=a×P1+(1-a)×P2; P is the probability that the original combination is a sentence, a is the preset weight coefficient, and a is a number less than 1; P1 is the first probability that the original combination is a sentence; P2 is the second probability that the original combination is a sentence.

[0036] A second aspect of the present invention discloses a sentence recognition device, comprising:

[0037] An acquisition unit, used for acquiring an original combination of multiple words arranged in word order;

[0038] A calculation unit, used for inputting the original combination into a target N-gram grammar model to obtain the probability of the original combination being a sentence; wherein, if the original combination contains a name in a list of names of persons and places, the target N-gram grammar model replaces the name contained in the original combination with a corresponding category label in the list of names of persons and places to obtain a replaced combination, and calculates the probability associated with the category label in the replaced combination, and obtains the probability of the original combination being a sentence based on the word-related probability of each word in the replaced combination, the probability associated with the category label, and the proportion of all names under the category label corresponding to the name in the original combination in the list of names of persons and places;

[0039] The names in the list of names of persons and places include: names of persons and places; the category labels corresponding to the names in the list of names of persons and places are set according to preset classification rules.

[0040] Optionally, in the above sentence recognition device, it also includes:

[0041] A creating unit, configured to create a first corpus; wherein each sentence in the first corpus contains a name; the name includes a person's name and / or a place name;

[0042] A replacement unit, configured to replace the names in the first corpus with corresponding category labels according to the preset classification rule, so as to obtain a replaced first corpus;

[0043] A first modeling unit is used to train an initial N-gram grammar model according to the replaced first corpus; wherein the initial N-gram grammar model is used to calculate the probability of each word in the combination containing the replaced category label and the probability of the replaced category label;

[0044] The second modeling unit is used to obtain the target N-gram grammar model based on the list of names and place names and the initial N-gram grammar model; wherein the target N-gram grammar model is used to calculate the probability of a combination of multiple words containing the names in the list of names and place names as a sentence; the list of names and place names includes at least the names of people and places included in each sentence in the first corpus.

[0045] Optionally, in the above-mentioned sentence recognition device, when the calculation unit performs the word-related probability of each word in the replaced combination, the probability of the category label, and the proportion of all names under the category label corresponding to the name in the original combination in the list of names and places, to obtain the probability that the original combination is a sentence, it is used to:

[0046] Obtaining the probability of the names in the original combination according to the probability of the category label correlation and the proportion of all names under the category label corresponding to the name in the list of personal names and place names;

[0047] The probability of the original combination being a sentence is obtained based on the probability of the name in the original combination and the probability of the word in each word in the replaced combination.

[0048] Optionally, in the above-mentioned sentence recognition device, when the calculation unit performs the word-related probability of each word in the replaced combination, the probability of the category label, and the proportion of all names under the category label corresponding to the name in the original combination in the list of names and places, to obtain the probability that the original combination is a sentence, it is used to:

[0049] According to the number of names corresponding to the category labels in the replaced combination in the list of names and places, the proportion of all names under the category labels corresponding to the names in the original combination in the list of names and places is obtained;

[0050] The probability associated with the category label is multiplied by the proportion of all names in the original combination under the category label corresponding to the name in the list of personal names and place names to obtain the probability associated with the name in the original combination;

[0051] The probability of the name in the original combination is multiplied by the probability of the word in each word in the replaced combination to obtain the probability of the original combination being a sentence.

[0052] Optionally, in the above-mentioned sentence recognition device, the list of names and places also includes: a weight value corresponding to each name in the list of names and places;

[0053] The calculation unit performs the following steps when obtaining the probability of the original combination as a sentence based on the probability of word correlation of each word in the replaced combination, the probability of the category label correlation, and the proportion of all names in the original combination under the category label corresponding to the name in the list of names and places:

[0054] According to the weight values ​​corresponding to all the names under the category labels corresponding to the names in the original combination in the list of names and places, the proportions of all the names under the category labels corresponding to the names in the original combination in the list of names and places are obtained;

[0055] The probability associated with the category label is multiplied by the proportion of all names in the original combination under the category label corresponding to the name in the list of personal names and place names to obtain the probability associated with the name in the original combination;

[0056] The probability of the name in the original combination is multiplied by the probability of the word in each word in the replaced combination to obtain the probability of the original combination being a sentence.

[0057] Optionally, in the above sentence recognition device, if the sentence of the first corpus contains an agglutinative word of a name, the replacement unit includes:

[0058] a segmentation unit, configured to segment the agglutinative language containing a name in the sentence of the first corpus into a root part and an affix part; wherein the root part is a name;

[0059] The replacement subunit is used to replace the root part of the agglutinative language with the corresponding category label according to the preset classification rule to obtain the replaced first corpus.

[0060] Optionally, in the above sentence recognition device, the computing unit includes:

[0061] A first calculation subunit, configured to input the original combination into a target N-gram model to obtain a first probability of the original combination being a sentence;

[0062] A second calculation subunit is used to input the original combination into a universal N-gram grammar model to obtain a second probability that the original combination is a sentence; wherein the universal N-gram grammar model is trained by a universal corpus; the universal corpus includes sentences containing names and sentences not containing names;

[0063] The third calculation subunit is used to obtain the probability that the original combination is a sentence according to a preset weight coefficient, a first probability that the original combination is a sentence, and a second probability that the original combination is a sentence.

[0064] Optionally, in the above sentence recognition device, the third calculation subunit includes:

[0065] a fourth calculation subunit, configured to substitute the preset weight coefficient, the first probability that the original combination is a sentence, and the second probability that the original combination is a sentence into the first formula to obtain the probability that the original combination is a sentence;

[0066] Among them, the first formula is: P=a×P1+(1-a)×P2; P is the probability that the original combination is a sentence, a is the preset weight coefficient, and a is a number less than 1; P1 is the first probability that the original combination is a sentence; P2 is the second probability that the original combination is a sentence.

[0067] A third aspect of the present invention discloses a computer-readable medium having a computer program stored thereon, wherein when the program is executed by a processor, the method described in any one of the first aspects above is implemented.

[0068] It can be seen from the above technical solution that in the sentence recognition method proposed in the present application, by obtaining the original combination of multiple words arranged in word order, the original combination is input into the target N-gram grammar model to obtain the probability of the original combination as a sentence. Since the target N-gram grammar model in the present application will replace the names in the original combination with the corresponding category labels in the list of names and place names in the original combination, the replaced combination is obtained, and the probability of the category label in the replaced combination is calculated. Then, according to the probability of the word correlation of each word in the replaced combination, the probability of the category label correlation, and the proportion of all names under the corresponding category label in the list of names and place names in the original combination, the probability of the original combination as a sentence is obtained. The names in the list of names and place names include: names and place names; the category labels corresponding to the names in the list of names and place names are set according to the preset classification rules. Compared with the N-gram grammar model in the prior art which directly calculates the name-related probabilities in the original combination to obtain the probability of the original combination as a sentence, the target N-gram grammar model of the present application obtains the sentence probability of the original combination by calculating the probabilities related to the category labels corresponding to the names, thereby solving the sparsity problem of the language model related to names of people and places, making sentences containing names of people or places easier to be correctly identified. BRIEF DESCRIPTION OF THE DRAWINGS

[0069] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying creative work.

[0070] Figure 1 A flowchart of a sentence recognition method disclosed in an embodiment of the present invention;

[0071] Figure 2 A schematic diagram of an input method application disclosed in an embodiment of the present invention;

[0072] Figure 3 A flowchart of a method for obtaining the probability of an original combination being a sentence disclosed in an embodiment of the present invention;

[0073] Figure 4 A flowchart of a method for obtaining the probability of name correlation in an original combination disclosed in an embodiment of the present invention;

[0074] Figure 5 A flow chart of another method for obtaining the probability of name correlation in an original combination disclosed in an embodiment of the present invention;

[0075] Figure 6 A schematic diagram of a process for constructing a target N-gram model disclosed in an embodiment of the present invention;

[0076] Figure 7 A flowchart of a method for obtaining a replaced first corpus disclosed in an embodiment of the present invention;

[0077] Figure 8 A schematic diagram of a process of replacing a category label of a sentence containing an agglutinative language disclosed in an embodiment of the present invention;

[0078] Fig. 9 A flow chart of another method for obtaining the probability of an original combination being a sentence disclosed in an embodiment of the present invention;

[0079] Fig.10 The present invention is a schematic diagram of the structure of a sentence recognition device disclosed in an embodiment of the present invention. DETAILED DESCRIPTION

[0080] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0081] Natural language processing (NLP) is an important direction in the fields of computer science and artificial intelligence. It studies various theories and methods that can achieve effective communication between people and computers using natural language. Natural language processing is a science that integrates linguistics, computer science, and mathematics. Therefore, research in this field will involve natural language, that is, the language people use in daily life, so it is closely related to the study of linguistics. Natural language processing technology usually includes text processing, semantic understanding, machine translation, robot question answering, knowledge graph and other technologies.

[0082] Natural language processing technology usually requires the use of language models to process natural language. Language models are mainly used to calculate the probability of a combination of multiple words appearing as a sentence. They are important modules in applications involving natural language processing technology, such as speech recognition, handwriting recognition, and input methods.

[0083] N-gram grammar is a common modeling method in language models. It assumes that the probability of the current word occurring depends only on the previous N-1 words, and thus calculates the probability of a combination of multiple words occurring as a sentence.

[0084] However, since words with specific identifiers such as names of people and places are relatively rare, even if a large number of names of people and places are collected so that the language model can recognize the semantics of names of people and places, it is impossible for the sentences used to train the model to contain all the usage forms of names of people and places, resulting in sentences containing names of people or places being difficult to be correctly identified, that is, the sparsity problem of the N-gram grammar model related to names of people and places is relatively serious.

[0085] In response to the above problems, the present application proposes a sentence recognition method, device and computer-readable medium to solve the sparsity problem of language models related to names of people and places.

[0086] The solution provided in the embodiments of the present application involves technologies such as natural language processing of artificial intelligence, which is specifically described by the following embodiments:

[0087] See also Figure 1 The present application embodiment discloses a sentence recognition method, which specifically includes the following steps:

[0088] S101. Acquire an original combination of multiple words arranged in word order.

[0089] In various sentence recognition applications such as speech recognition, input method, and handwriting recognition, the sentence recognition system obtains the original combination of multiple words arranged in word order based on the received sentence information. For example, see Figure 2 When a user uses an input method application, he inputs the pinyin of "what are you doing now" to the sentence recognition system in the order of the sentence. Since the pinyin of each character corresponds to multiple characters, for example, the pinyin "ni" corresponds to multiple characters such as "ni", "ni", "ni", etc., the sentence recognition system arranges each pinyin-recognized character in the order of words, and finally obtains the original combination of multiple words. For another example, when a user uses a voice recognition application, the pronunciation of a character corresponds to multiple characters, so the sentence recognition system will also obtain multiple original combinations of multiple words arranged in the order of words.

[0090] Therefore, when a user inputs sentence information of a sentence into the sentence recognition system, the sentence recognition system will recognize each character or word therein, and then arrange and combine the recognized possible characters or words in a specific order to obtain the original combination of multiple words arranged in multiple word orders corresponding to the input sentence.

[0091] It should be noted that the sentence information input by the user can be in different languages ​​such as Chinese, English, etc., or can be a sentence composed of multiple different languages. The difference in language types does not affect the implementation of the embodiments of the present application.

[0092] S102, input the original combination into the target N-gram model to obtain the probability of the original combination as a sentence. If the original combination contains names in the list of names and places, the target N-gram model replaces the names contained in the original combination with corresponding category labels in the list of names and places to obtain the replaced combination, and calculates the probability of the category label in the replaced combination. According to the word-related probability of each word in the replaced combination, the probability of the category label, and the proportion of all names under the corresponding category label in the list of names and places, the probability of the original combination as a sentence is obtained. The names in the list of names and places include: names and place names; the category labels corresponding to the names in the list of names and places are set according to the preset classification rules.

[0093] When executing step S102, each original combination corresponding to a sentence obtained in step S101 is input into the target N-gram model, and the probability of each original combination being a sentence is calculated, and then the original combination with the highest probability can be selected as the final recognized sentence combination by comparing the probabilities of multiple original combinations corresponding to the sentence. There are many ways to select the final recognized sentence combination based on the calculated sentence probability of each original combination corresponding to a sentence, including but not limited to the contents proposed in the embodiments of the present application.

[0094] Among them, if the original combination contains a name in the list of names and place names, the target N-gram grammar model first replaces the name contained in the original combination with the corresponding category label in the list of names and place names to obtain the replaced combination. The list of names and place names includes multiple names and the category label corresponding to each name. The name includes at least a name and a place name. The category label corresponding to the name in the list of names and place names is set according to the preset classification rules. For example, the names in the list of names and place names can be directly divided into two categories: names and place names. Those belonging to the name category are assigned the "name" label in the list, and those belonging to the place name category are assigned the "location" label in the list. For the original combination of "I", "I am", and "Xiao Ming", if the name "Xiao Ming" exists in the list of names and place names, and the corresponding category label in the list of names and place names is "name", then "Xiao Ming" is replaced with "name" to obtain the replaced combination of "I am name". Among them, there are many rules for classifying the names in the list of names and place names, which can be set manually. In addition to being able to divide the names in the list of names and place names into two categories, the names and place names can also be classified more finely, and the last two letters of the names and place names can be divided into multiple categories such as vowels, unvoiced consonants, and voiced consonants. The preset classification rules for the names in the list of names and place names include but are not limited to the contents proposed in the embodiments of the present application.

[0095] According to the replaced combination, the probability of the category label in the replaced combination can be calculated. Among them, the replaced combination is composed of multiple words and the replaced category label, and the multiple words are non-name words. The probability of the category label in the replaced combination refers to the probability used to represent the correlation between the category label and the words before and after the category label. For example, if the target N-gram grammar model is a bigram grammar model, the probability of the category label in the replaced combination "I am name" is P(name / is), that is, the probability that the word "is" is followed by the label "name". According to the replaced combination, the word correlation probability of each word can also be calculated. Among them, the word correlation probability of each word is used to represent the correlation between the word and the words before and after the word. For example, for the replaced combination "I am name", there are a total of two words "I" and "is", and the label "name". If the target N-gram grammar model is a bigram grammar model, the word correlation probability of the word "I" is P(I), and the word correlation probability of the word "is" is P(is / I).

[0096] Then, according to the word-related probability of each word in the replaced combination, the probability related to the category label, and the proportion of all names under the corresponding category label in the list of names and places in the original combination, the probability of the original combination as a sentence is obtained. For example, for the original combination "I am Xiao Ming", the probability of P(I am Xiao Ming) can be calculated based on P(I), P(I am / I), P(name / I am), and the proportion of the name "Xiao Ming" in the list of names and places under the label "name". Among them, P(I) and P(I am / I) are word-related probabilities, and P(name / I am) is the probability related to the category label. The proportion of the name "Xiao Ming" in the list of names and places under the label "name" can be set manually. For example, the proportion of all names in the name and place category under the label name can be made the same. That is, if there are 100 names under the label "name", the proportion of each name is 1 / 100, and the proportion of "Xiao Ming" is 1 / 100.

[0097] Specifically, see Figure 3 In a specific embodiment of the present application, the probability of the original combination being a sentence is obtained based on the probability of word correlation of each word in the replaced combination, the probability of category label correlation, and the proportion of all names under the category label corresponding to the name in the original combination in the list of names and place names, including:

[0098] S301, obtaining the probability of the names in the original combination being related according to the probability of the category labels and the proportion of all the names in the original combination under the category labels corresponding to the names in the list of personal names and place names.

[0099] The probability associated with the category label is actually obtained by summing the probabilities associated with all names under the category label in the list of names and places. For example, in the replaced combination "I am name", P(name / is) is the probability associated with the category label. If there are three names in the name label, Xiao Ming, Xiao Hong and Xiao Hua, then P(name / is) is equal to the sum of P(Xiao Ming / is), P(Xiao Hong / is) and P(Xiao Hua / is). That is, all names under the category label in the list of names and places share the relevant probability of the category label. Therefore, if the proportion of the names in the original combination in the list of names and places is specified, then the probability value associated with the names in the original combination can be obtained.

[0100] Optionally, see Figure 4 In a specific embodiment of the present application, an implementation of step S301 includes:

[0101] S401. According to the number of names corresponding to the category labels in the replaced combination in the list of names and places, the proportion of all names under the category labels corresponding to the names in the original combination in the list of names and places is obtained.

[0102] That is, the proportion of each name corresponding to the category label in the replaced combination in the list of names and places is set to be equal. Therefore, you only need to know the number of names corresponding to the category label in the replaced combination in the list of names and places to know the proportion of all names under the category label corresponding to the name in the original combination in the list of names and places. For example, the original combination is "I am Xiao Ming", and the replaced combination is "I am name" according to the list of names and places. If there are 50 names under the "name" label in the list of names and places, then the proportion of "Xiao Ming" in all names under the "name" label in the list of names and places is 1 / 50.

[0103] S402: multiply the probability associated with the category label by the proportion of all names in the original combination under the category label corresponding to the name in the list of personal names and place names to obtain the probability associated with the name in the original combination.

[0104] It should be noted that the probability value associated with the category label in the replaced combination is not necessarily only one, and may also contain multiple. When executing step S402, it means that the probability associated with each category label is multiplied by the proportion of all names under the category label corresponding to the name in the original combination in the list of names and place names, and finally the probability associated with the names in multiple original combinations is obtained. For example, the original combination is "This is Xiao Ming's book", and the replaced combination is "This is name's book". If the target N-gram grammar model is a binary grammar model, then the relevant probabilities of the category labels are P(name / is) and P(de / name). Multiply P(name / is) by the proportion of all names of "Xiao Ming" under the name label to obtain P(Xiao Ming / is). Multiply P(de / name) by the proportion of all names of Xiao Ming under the "name" label to obtain P(de / Xiao Ming).

[0105] Optionally, see Figure 5 In a specific embodiment of the present application, if the list of names and place names also includes a weight value corresponding to each name in the list of names and place names, then an implementation method of executing step S301 includes:

[0106] S501. Obtain the proportion of all names in the original combination under the category labels corresponding to the names in the list of names and place names according to the weight values ​​corresponding to all names in the category labels corresponding to the names in the original combination in the list of names and place names.

[0107] In general, the weight value corresponding to a name under a label in a list of names and places is 1, that is, each name under a category label in the list of names and places has the same proportion to all the names under the category label. However, in actual applications, if it is found that the sentence containing an individual name is still difficult to be correctly recognized after entering the target N-gram grammar model, the weight value corresponding to the name in the list of names and places can be increased, thereby increasing the probability that the sentence can be correctly recognized. Or if it is found that an individual name appears more frequently in daily language, the weight value corresponding to the name in the list of names and places can also be increased. That is, the weight value corresponding to each name can be adjusted according to the actual situation, thereby adjusting the proportion of each name in all the names under the corresponding category label.

[0108] For example, in the list of names and places shown in Table 1, the weight value of the name mAmAt is set to 3. If mAmAt has a total of 3 names under the corresponding tag "name4" in Table 1, and the weight values ​​of other names are 1, then the sum of the weight values ​​of all names under the "name4" tag is: 3+1+1=5. The proportion of mAmAt in all names under the "name4" tag is 3 / 5.

[0109] Table 1:

[0110]

[0111]

[0112] S502: multiply the probability associated with the category label by the proportion of all names in the original combination under the category label corresponding to the name in the list of personal names and place names to obtain the probability associated with the name in the original combination.

[0113] The principle and process of executing step S502 are the same as Figure 4 The step S402 shown is the same as that shown in FIG. 4 , and can be referred to, and will not be described again here.

[0114] S302: According to the probability of the name in the original combination and the probability of the word in the replaced combination, the probability of the original combination being a sentence is obtained.

[0115] Among them, the word relevance probability of a word is used to illustrate the relevance between the word and the words before and after it. The word relevance probability of each word in the replaced combination is equivalent to the word relevance probability of the words other than the name in the original combination. The relevance probability of the original combination as a sentence is obtained based on the name relevance probability in the original combination and the word relevance probability of the words other than the name in the original combination (i.e., the word relevance probability of each word in the replaced combination).

[0116] Specifically, in a specific embodiment of the present application, an implementation of step S302 includes:

[0117] Multiply the probability of the name in the original combination by the probability of each word in the replaced combination to get the probability of the original combination as a sentence.

[0118] For example, if the original combination is "This is Xiao Ming's book", and the target N-gram model is a bigram model, then the name-related probabilities in the original combination are P(Xiao Ming / is) and P(de / Xiao Ming), and the word-related probabilities of each word in the replaced combination are P(this), P(is / this), and P(book / de). Therefore, the probability of "This is Xiao Ming's book" as a sentence is P(This is Xiao Ming's book) equal to the product of P(this), P(is / this), P(Xiao Ming / is), P(de / Xiao Ming), and P(book / de).

[0119] In the prior art, for the original combination "This is Xiao Ming's book", the bigram model directly calculates P(is / this), P(Xiao Ming / is), P(of / Xiao Ming) and P(book / of), and then calculates their product to obtain the probability of the original combination as a sentence. However, since the frequency of names of people and places in actual sentences is relatively low, and it is impossible for actual sentences to contain all the usages of the name "Xiao Ming", the probability of P(Xiao Ming / is) or P(of / Xiao Ming) is likely to be 0, so the bigram model is likely to obtain the probability of 0 for the original combination "This is Xiao Ming's book" as a sentence. Even if we then fall back to using a unigram model to calculate the probability of the original combination as a sentence, the probability value obtained will be close to 0, that is, there is a sparsity problem.

[0120] When the target N-gram grammar model in the embodiment of the present application is a binary grammar model, P(Xiao Ming / is) and P(de / Xiao Ming) are not directly obtained, but the original combination is replaced with the combination of "this is name's book", and P(Xiao Ming / is) and P(de / Xiao Ming) are obtained according to the proportion of all names under the corresponding category labels in the list of names and places of P(name / is), P(de / name) and the names in the original combination. The probability of directly calculating P(Xiao Ming / is) or P(de / Xiao Ming) may be very low, or even 0, that is, it has sparsity, and the probability of P(name / is) or P(de / name) being 0 is very low. The probability of P(name / is) is equivalent to the sum of the probabilities of the names of all name tags in the list of names and places, and the same is true for P(de / name). Therefore, P(Xiao Ming / is) can be obtained according to P(name / is), and P(de / Xiao Ming) can be obtained according to P(de / name), and then the probability of the original combination of "this is Xiao Ming's book" as a sentence can be calculated. The calculation method of the target N-gram grammar model of the present application obtains the relevant probabilities of the category labels in the replaced combination, and then obtains the probability of the original combination as a sentence, thereby solving the sparsity problem of the language model related to names of people and places, making sentences containing names of people or places easier to be correctly identified.

[0121] See also Figure 6 In a specific embodiment of the present application, a method for constructing a target N-gram model includes the following steps:

[0122] S601: Create a first corpus.

[0123] Each sentence in the first corpus contains a name; the name includes a person's name and / or a place's name. Specifically, sentences containing names can be collected from multiple websites, newspapers, books and other places containing more language information, that is, the first corpus is created for sentences containing names. In practical applications, in order to improve the accuracy of the calculation of the target N-gram model, the corpus in the first corpus can be updated periodically, so that the first corpus can include some sentences containing person's names and / or place's names that are more common at present.

[0124] S602: According to a preset classification rule, replace the names in the first corpus with corresponding category labels to obtain a replaced first corpus.

[0125] The preset classification rules mentioned in step S602 are Figure 1 The classification rules for the category labels corresponding to the names in the list of names and place names mentioned in step S102 are the same. For example, the names in the list of names and place names are divided into two categories, namely, names and place names, and the category labels corresponding to names are "name" and "location". Then, when executing step S602, the names in the first corpus are replaced with "name" and the place names are replaced with "location", thereby obtaining the first corpus after replacement.

[0126] Optionally, see Figure 7 In a specific embodiment of the present application, if the sentence of the first corpus contains an agglutinative word of a name, an implementation of step S602 is executed, including:

[0127] S701. Split the agglutinative words containing names in the sentences of the first corpus into root parts and affix parts.

[0128] Among them, the root part is the name, and the name includes names of people and places. Agglutinative language is composed of roots and affixes, and is a grammatical type of language. Agglutinative language realizes grammatical functions by pasting different affixes before, during and after the root. Grammatical meaning is mainly expressed by the affixes added to the root. For example, "give Alimu" is expressed as "valimGa" in the language, where "valim" is the root, that is, the name of "Alimu", and "Ga" is the affix, which means "give". Due to the existence of agglutinative language, the root and affix are connected together to form a word, even if a large number of roots, that is, names of people and places, are collected, because the names of people and places are followed by affixes, it will cause the sentence recognition system to be unable to recognize. In the prior art, the sparse problem of language models related to names of people and places is itself serious, and the special word formation method of agglutinative language has aggravated the difficulty of recognizing names of people and places, and has aggravated the sparse problem of language models related to names of people and places. Therefore, for agglutinative languages ​​containing names, the roots and affixes need to be segmented first so that the sentence recognition system can recognize the names.

[0129] For example, see Figure 8 If the first corpus contains sentences such as "You bring this book to Alim", "You bring this book to Mailhaba", etc., then it is necessary to segment the agglutinative language containing the name "Bring to Alim (valimGa)" into the form of "valim+Ga".

[0130] S702: According to a preset classification rule, the root part of the agglutinative language is replaced with a corresponding category label to obtain a first corpus after replacement.

[0131] After the roots and affixes are segmented, the names of the root parts are replaced with corresponding category labels according to the preset classification rules to obtain the replaced corpus.

[0132] For example, see Figure 8 , due to the different vowels, voiceless consonants, and voiced consonants at the end of the name, the added suffixes will also be different. Therefore, the name part of the name is divided into four category labels: "name1", "name2", "name3", and "name4". For names belonging to the "name1" category label, the suffix added when expressing the meaning of "give" is "Ga", for names belonging to the "name2" category label, the suffix added when expressing the meaning of "give" is "qa", for names belonging to the "name3" category label, the suffix added when expressing the meaning of "give" is "gA", and for names belonging to the "name4" category label, the suffix added when expressing the meaning of "give" is "kA".

[0133] It should be noted that the classification rules can take many forms, including but not limited to those proposed in the embodiments of the present application.

[0134] S603: According to the replaced first corpus, train and obtain an initial N-gram grammar model.

[0135] The initial N-gram model is used to calculate the probability associated with each word in the combination containing the replaced category label, as well as the probability associated with the replaced category label. For example, for a replaced sentence "I am name" input into the initial N-gram model, if the initial N-gram model is a bigram model, P(I), P(is / I), and P(name / is) are calculated. The process of training the initial N-gram model is to continuously calculate the conditional probabilities of words such as P(I), P(is / I), and P(name / is) based on the replaced sentences in the first corpus.

[0136] Optionally, after obtaining the initial N-gram model, the initial N-gram model can be further optimized. In order to further strengthen the probability of the part related to the name, the word-related probability of each word in the combination containing the replaced category label can be weakened, that is, the probability score of the word unrelated to the category label is weakened, and the probability score related to the replaced category label is strengthened, and finally an optimized initial N-gram model is obtained. For example, in the replaced combination "I am name", P (I) and P (is / I) are both word-related probabilities of words unrelated to the category label "name", so these probabilities can be weakened, so that the accuracy of the optimized initial N-gram model in calculating the probability related to the name is further improved.

[0137] S604: Obtain a target N-gram model according to the list of names and places and the initial N-gram model.

[0138] The target N-gram model is used to calculate the probability of a combination of multiple words containing names in the list of names as a sentence. The list of names at least includes the names and places included in each sentence in the first corpus. The category labels corresponding to the names in the list of names are obtained according to the preset classification rules in step S602. The names collected in the list of names may be more than the names appearing in the first corpus.

[0139] When executing step S604, the initial N-gram grammar model is expanded according to the list of names and places to obtain the final target N-gram grammar model. Specifically, the names in the list of names and places can be used to share the probability associated with the category label corresponding to the name, thereby obtaining the conditional probability associated with the name. For example, if you want to obtain the probability associated with the name "Alimu", you can obtain it according to the probability of the category label corresponding to "Alimu". If the category label corresponding to "Alimu" is "name", and the "name" label corresponds to 100 names in the list of names and places, therefore, "Alimu" can share the probability of "name" with the other 99 names. For the sentence "I am Alimu", if P (name / is) in "I am name" is 0.01, then P (Alimu / is) in "I am Alimu" can be P (name / is) divided by 100.

[0140] It should be noted that N in the target N-gram model is a positive integer, and the target N-gram model can be any N-gram model. However, the larger N is, the more accurate the calculated probability of the original combination as a sentence will be. In actual use, multiple different target N-gram models can be used, and the multiple probability results obtained can be added in a certain proportion to obtain the final probability result. The target N-gram model with a larger N value can be given a larger proportion, and the model with a smaller N value can be given a smaller proportion. For example, the target tri-gram model is first used to calculate the probability P3 of the original combination as a sentence, and then the target bigram model is used to calculate the probability P2 of the original combination as a sentence, and then the target uni-gram model is used to calculate the probability P3 of the original combination as a sentence, and then the final probability of the original combination as a sentence is obtained P=a×P1+b×P2+c×P3, where a+b+c=1, and a is greater than b, and b is greater than c.

[0141] You can also use a target N-gram model with a larger N to calculate the probability of the original combination as a sentence. If the probability of a sentence is 0, that is, multiple original combinations corresponding to a sentence are all 0, and the sentence cannot be recognized, then use a target N-gram model with a smaller value than the original N until the sentence can be recognized. For example, first use the target tri-gram model to calculate the sentence probabilities of multiple original combinations corresponding to a sentence. If the sentence probabilities calculated for multiple original combinations are all 0, and the correct sentence cannot be recognized, then use the target bigram model for calculation, and finally select the original combination with the largest sentence probability as the final recognized sentence. There are many ways to use the target N-gram model to obtain the probability of an original combination as a sentence, including but not limited to the contents proposed in the embodiments of the present application.

[0142] It should also be noted that if the original combination obtained in step S101 does not contain any name, then when executing step S102, the target N-gram model does not need to replace the name in the original combination with the corresponding category label, but directly calculates the probability of the original combination as a sentence. For example, the original combination "what are you doing now" is input into the target N-gram model. If the target N-gram model is a bigram model, the values ​​of P(you), P(now / you), P(do / now) and P(what / do) are directly calculated, and then the product of P(you), P(now / you), P(do / now) and P(what / do) is used as the sentence probability of the original combination.

[0143] Optionally, since the target N-gram grammar model in the embodiment of the present application is constructed based on a corpus containing personal names or place names, the accuracy of the probability value calculated for the original combination containing personal names and place names as a sentence is higher than the probability value of the original combination not containing personal names and place names as a sentence. Therefore, the original combination in step S101 can also be judged first. If it contains personal names or place names, it is input into the target N-gram grammar model in the embodiment of the present application. If it does not contain personal names or place names, it is input into the general N-gram grammar model, wherein the general N-gram grammar model is trained by a general corpus; the general corpus includes sentences containing names and sentences without names.

[0144] Optionally, see Fig. 9 In a specific embodiment of the present application, an implementation of step S102 includes:

[0145] S901. Input the original combination into the target N-gram model to obtain a first probability of the original combination being a sentence.

[0146] Among them, if the original combination contains a name in the list of personal names and place names, the target N-gram grammar model replaces the name contained in the original combination with the corresponding category label in the list of personal names and place names to obtain a replaced combination, and obtains the first probability of the original combination as a sentence based on the word-related probability of each word in the replaced combination, the probability related to the category label in the replaced combination, and the proportion of all names in the original combination under the corresponding category label in the list of personal names and place names.

[0147] If the original combination does not contain any name in the list of names of persons or places, the related probabilities of the words in the original combination are directly calculated to directly obtain the first probability of the original combination as a sentence.

[0148] S902: Input the original combination into a universal N-gram model to obtain a second probability of the original combination being a sentence.

[0149] The general N-gram model is trained by a general corpus; the general corpus includes sentences containing names and sentences without names, which can generally be obtained through resources such as web crawlers and news newspapers.

[0150] The general N-gram model is different from the target N-gram model. Whether it is for the original combination containing a name or for the original combination without a name, the word-related probabilities of the words in the original combination are directly calculated to directly obtain the probability of the original combination as a sentence. For example, for the original combination "I am Alimu", the values ​​of P(I), P(I am / I) and P(Alimu / I am) are directly calculated, and then the second probability of the original combination as a sentence is obtained based on the product of P(I), P(I am / I) and P(Alimu / I am).

[0151] S903. Obtain the probability that the original combination is a sentence according to a preset weight coefficient, a first probability that the original combination is a sentence, and a second probability that the original combination is a sentence.

[0152] The target N-gram grammar model has a higher accuracy in calculating the probability that the original combination containing names of people and places is a sentence, while for the original combination without names of people and places, the probability value calculated by the commonly used grammar model is more accurate. Therefore, the advantages of the two grammar models can be combined, and the probability of the original combination as a sentence can be obtained based on the probability values ​​obtained by the two grammar models and a preset weight coefficient. Among them, the preset weight coefficient is a number less than 1, which is used to set the weights of the first probability and the second probability.

[0153] Optionally, in a specific embodiment of the present application, an implementation of step S903 includes:

[0154] Substitute the preset weight coefficient, the first probability of the original combination being a sentence, and the second probability of the original combination being a sentence into the first formula to obtain the probability of the original combination being a sentence.

[0155] The first formula is: P = a × P1 + (1-a) × P2. P is the probability of the original combination as a sentence, a is a preset weight coefficient, and a is a number less than 1; P1 is the first probability of the original combination as a sentence; P2 is the second probability of the original combination as a sentence.

[0156] In practical applications, since the corpus in the general corpus used to construct the general N-gram model is more comprehensive and rich, the weight of the probability value obtained using the general N-gram model can be increased, that is, the value of (1-a) can be increased to obtain a higher probability of the original combination as a sentence.

[0157] In the sentence recognition method proposed in the present application, by obtaining the original combination of multiple words arranged in word order, the original combination is input into the target N-gram grammar model to obtain the probability of the original combination as a sentence. Since the target N-gram grammar model in the present application will replace the names in the original combination with the corresponding category labels in the list of names and places in the original combination, the replaced combination is obtained, and the probability of the category label in the replaced combination is calculated. Then, according to the probability of the word correlation of each word in the replaced combination, the probability of the category label correlation, and the proportion of all names under the corresponding category label in the list of names and places, the probability of the original combination as a sentence is obtained. The names in the list of names and places include: names and places; the category labels corresponding to the names in the list of names and places are set according to the preset classification rules. Compared with the N-gram grammar model in the prior art which directly calculates the name-related probabilities in the original combination to obtain the probability of the original combination as a sentence, the target N-gram grammar model of the present application obtains the sentence probability of the original combination by calculating the probabilities related to the category labels corresponding to the names, thereby solving the sparsity problem of the language model related to names of people and places, making sentences containing names of people or places easier to be correctly identified.

[0158] See also Fig.10 Based on the sentence recognition method proposed in the above-mentioned embodiment of the present application, the embodiment of the present application correspondingly discloses a sentence recognition device 1000, including: an acquisition unit 1001 and a calculation unit 1002.

[0159] The acquisition unit 1001 is used to acquire an original combination of multiple words arranged in word order.

[0160] The calculation unit 1002 is used to input the original combination into the target N-gram grammar model to obtain the probability of the original combination as a sentence. If the original combination contains a name in the list of names and places, the target N-gram grammar model replaces the name contained in the original combination with the corresponding category label in the list of names and places to obtain the replaced combination, and calculates the probability related to the category label in the replaced combination, and obtains the probability of the original combination as a sentence according to the word-related probability of each word in the replaced combination, the probability related to the category label, and the proportion of all names under the corresponding category label in the list of names and places. The names in the list of names and places include: names and places; the category labels corresponding to the names in the list of names and places are set according to the preset classification rules.

[0161] Optionally, in a specific embodiment of the present application, the calculation unit 1002 performs the following steps to obtain the probability of the original combination being a sentence according to the probability of word correlation of each word in the replaced combination, the probability of category label correlation, and the proportion of all names under the category label corresponding to the name in the original combination in the list of names and places:

[0162] According to the probability associated with the category label and the proportion of all names in the original combination under the corresponding category label in the list of names and places, the probability associated with the names in the original combination is obtained.

[0163] According to the probability of the name in the original combination and the probability of each word in the replaced combination, the probability of the original combination as a sentence is obtained.

[0164] Optionally, in a specific embodiment of the present application, the calculation unit 1002 performs the following steps to obtain the probability of the original combination being a sentence according to the probability of word correlation of each word in the replaced combination, the probability of category label correlation, and the proportion of all names under the category label corresponding to the name in the original combination in the list of names and places:

[0165] According to the number of names corresponding to the category labels in the replaced combination in the list of names and place names, the proportion of all names under the category labels corresponding to the names in the original combination in the list of names and place names is obtained.

[0166] The probability associated with the category label is multiplied by the proportion of all names in the original combination under the category label corresponding to the name in the list of names and places to obtain the probability associated with the name in the original combination.

[0167] Multiply the probability of the name in the original combination by the probability of each word in the replaced combination to get the probability of the original combination as a sentence.

[0168] Optionally, in a specific embodiment of the present application, the list of names and places also includes: a weight value corresponding to each name in the list of names and places.

[0169] The calculation unit 1002 performs the following operations when obtaining the probability of the original combination being a sentence based on the probability of each word in the replaced combination, the probability of the category label, and the proportion of all names under the category label corresponding to the name in the original combination in the list of names and places:

[0170] According to the weight values ​​corresponding to all the names under the category labels corresponding to the names in the original combination in the list of names and place names, the proportion of all the names under the category labels corresponding to the names in the original combination in the list of names and place names is obtained.

[0171] The probability associated with the category label is multiplied by the proportion of all names in the original combination under the category label corresponding to the name in the list of names and places to obtain the probability associated with the name in the original combination.

[0172] Multiply the probability of the name in the original combination by the probability of each word in the replaced combination to get the probability of the original combination as a sentence.

[0173] Optionally, in the above sentence recognition device, if the sentence of the first corpus contains an agglutinative word of a name, the replacement unit includes:

[0174] a segmentation unit, configured to segment the agglutinative language containing a name in the sentence of the first corpus into a root part and an affix part; wherein the root part is a name;

[0175] The replacement subunit is used to replace the root part of the agglutinative language with the corresponding category label according to the preset classification rule to obtain the replaced first corpus.

[0176] Optionally, in a specific embodiment of the present application, the calculation unit 1002 includes:

[0177] The first calculation subunit is used to input the original combination into the target N-gram model to obtain a first probability of the original combination being a sentence.

[0178] The second calculation subunit is used to input the original combination into a universal N-gram model to obtain a second probability that the original combination is a sentence. The universal N-gram model is trained by a universal corpus. The universal corpus includes sentences containing names and sentences without names.

[0179] The third calculation subunit is used to obtain the probability that the original combination is a sentence according to a preset weight coefficient, a first probability that the original combination is a sentence, and a second probability that the original combination is a sentence.

[0180] Optionally, in a specific embodiment of the present application, the third computing subunit includes:

[0181] The fourth calculation subunit is used to substitute the preset weight coefficient, the first probability that the original combination is a sentence, and the second probability that the original combination is a sentence into the first formula to obtain the probability that the original combination is a sentence.

[0182] The first formula is: P = a × P1 + (1-a) × P2; P is the probability of the original combination as a sentence, a is the preset weight coefficient, and a is a number less than 1. P1 is the first probability of the original combination as a sentence; P2 is the second probability of the original combination as a sentence.

[0183] Optionally, in a specific embodiment of the present application, the sentence recognition device 1000 further includes: a creation unit, a replacement unit, a first modeling unit and a second modeling unit.

[0184] The creation unit is used to create a first corpus, wherein each sentence in the first corpus contains a name; the name includes a person's name and / or a place name.

[0185] The replacement unit is used to replace the names in the first corpus with corresponding category labels according to a preset classification rule to obtain a replaced first corpus.

[0186] The first modeling unit is used to train an initial N-gram grammar model based on the replaced first corpus; wherein the initial N-gram grammar model is used to calculate the probability of each word in the combination containing the replaced category label and the probability of the replaced category label.

[0187] The second modeling unit is used to obtain a target N-gram model according to the list of names and places and the initial N-gram model. The target N-gram model is used to calculate the probability of obtaining a combination of multiple words containing names in the list of names and places as a sentence. The list of names and places at least includes the names and places included in each sentence in the first corpus.

[0188] The specific principles and execution process of the sentence recognition device 1000 disclosed in the above embodiment of the present application are the same as those of the sentence recognition method disclosed in the above embodiment of the present application. Please refer to the corresponding parts of the sentence recognition method disclosed in the above embodiment of the present application, and will not be repeated here.

[0189] In the sentence recognition device proposed in the present application, the original combination of multiple words arranged in word order is obtained by the acquisition unit 1001, and then the original combination is input into the target N-gram grammar model by the calculation unit 1002 to obtain the probability of the original combination as a sentence. Since the target N-gram grammar model in the present application will replace the name in the original combination with the corresponding category label in the list of names and places in the original combination, obtain the replaced combination, and calculate the probability of the category label in the replaced combination. Then, according to the probability of the word correlation of each word in the replaced combination, the probability of the category label correlation, and the proportion of all names under the category label corresponding to the name in the original combination in the list of names and places, the probability of the original combination as a sentence is obtained. The names in the list of names and places include: names and places; the category labels corresponding to the names in the list of names and places are set according to the preset classification rules. Compared with the N-gram grammar model in the prior art which directly calculates the name-related probabilities in the original combination to obtain the probability of the original combination as a sentence, the target N-gram grammar model of the present application obtains the sentence probability of the original combination by calculating the probabilities related to the category labels corresponding to the names, thereby solving the sparsity problem of the language model related to names of people and places, making sentences containing names of people or places easier to be correctly identified.

[0190] An embodiment of the present application provides a computer-readable medium having a computer program stored thereon, wherein the program, when executed by a processor, implements the sentence recognition method provided by the above method embodiments.

[0191] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0192] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0193] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0194] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0195] The memory may include non-permanent memory in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. The memory is an example of a computer-readable medium.

[0196] Computer readable media include permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. Information can be computer readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic tape magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer readable media does not include temporary computer readable media (transitory media), such as modulated data signals and carrier waves.

[0197] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that contain computer-usable program code.

[0198] The above description is only a preferred embodiment of the present application and an explanation of the technical principles used, and is not intended to limit the present application. For those skilled in the art, the present application may have various modifications and variations. The scope of the invention involved in the present application is not limited to the technical solution formed by a specific combination of the above-mentioned technical features, but should also cover other technical solutions formed by any combination of the above-mentioned technical features or their equivalent features without departing from the above-mentioned inventive concept. For example, a technical solution formed by replacing the above-mentioned features with the technical features with similar functions disclosed in this application (but not limited to).

Claims

1. A sentence recognition method, characterized in that: include: Obtaining the original combination of multiple words arranged in word order; The original combination is input into the target N-gram grammar model to obtain the probability of the original combination as a sentence; wherein, for the names in the list of names of people and places contained in the original combination, the target N-gram grammar model replaces the names contained in the original combination with the corresponding category labels in the list of names of people and places to obtain the replaced combination, and calculates the probability associated with the category label in the replaced combination, and obtains the probability of the original combination as a sentence based on the word-related probability of each word in the replaced combination, the probability associated with the category label, and the proportion of all names in the original combination under the corresponding category label in the list of names of people and places; The names in the list of names and places include: names and places; the category labels corresponding to the names in the list of names and places are set according to a preset classification rule; The method for constructing the target N-gram model includes: Creating a first corpus; wherein each sentence in the first corpus contains a name; the name includes a person's name and / or a place name; According to the preset classification rule, the names in the first corpus are replaced with corresponding category labels to obtain a replaced first corpus; According to the replaced first corpus, an initial N-gram grammar model is trained; wherein the initial N-gram grammar model is used to calculate the probability of each word in the combination containing the replaced category label and the probability of the replaced category label; The target N-gram grammar model is obtained according to the list of names and places and the initial N-gram grammar model; wherein the target N-gram grammar model is used to calculate the probability of a combination of multiple words containing names in the list of names and places as a sentence; the list of names and places at least includes the names and places included in each sentence in the first corpus; The method of obtaining the probability of the original combination being a sentence according to the probability of word correlation of each word in the replaced combination, the probability of category label correlation, and the proportion of all names under the category label corresponding to the name in the original combination in the list of names and places includes: Obtaining the probability of the names in the original combination according to the probability of the category label correlation and the proportion of all names under the category label corresponding to the name in the list of personal names and place names; The probability of the original combination being a sentence is obtained based on the probability of the name in the original combination and the probability of the word in each word in the replaced combination.

2. The method according to claim 1, characterized in that The obtaining of the probability of the names in the original combination according to the probability of the category label and the proportion of all names under the category label corresponding to the name in the list of names and places, comprises: According to the number of names corresponding to the category labels in the replaced combination in the list of names and places, the proportion of all names under the category labels corresponding to the names in the original combination in the list of names and places is obtained; The probability associated with the category label is multiplied by the proportion of all names in the original combination under the category label corresponding to the name in the list of personal names and place names to obtain the probability associated with the name in the original combination.

3. The method according to claim 1, characterized in that: The list of names and places also includes: a weight value corresponding to each name in the list of names and places; The method of obtaining the probability of the names in the original combination according to the probability of the category label and the proportion of all names under the category label corresponding to the name in the list of names and places includes: According to the weight values ​​corresponding to all the names under the category labels corresponding to the names in the original combination in the list of names and places, the proportions of all the names under the category labels corresponding to the names in the original combination in the list of names and places are obtained; The probability associated with the category label is multiplied by the proportion of all names in the original combination under the category label corresponding to the name in the list of personal names and place names to obtain the probability associated with the name in the original combination.

4. The method according to claim 1, characterized in that: The step of obtaining the probability of the original combination being a sentence based on the probability of the name in the original combination and the probability of the word in each word in the replaced combination comprises: The probability of the name in the original combination is multiplied by the probability of the word in each word in the replaced combination to obtain the probability of the original combination being a sentence.

5. The method according to claim 1, characterized in that If the sentences in the first corpus contain name agglutinants, then according to the preset classification rule, the names in the first corpus are replaced with corresponding category labels to obtain the replaced first corpus, including: The agglutinative words containing names in the sentences of the first corpus are divided into root parts and affix parts; wherein the root part is a name; According to the preset classification rule, the root part of the agglutinative language is replaced with a corresponding category label to obtain a replaced first corpus.

6. The method according to claim 1, characterized in that The inputting the original combination into the target N-gram model to obtain the probability of the original combination being a sentence includes: Inputting the original combination into a target N-gram model to obtain a first probability of the original combination being a sentence; Inputting the original combination into a universal N-gram model to obtain a second probability that the original combination is a sentence; wherein the universal N-gram model is trained by a universal corpus; the universal corpus includes sentences containing names and sentences not containing names; The probability that the original combination is a sentence is obtained according to a preset weight coefficient, a first probability that the original combination is a sentence, and a second probability that the original combination is a sentence.

7. The method according to claim 6, characterized in that The obtaining the probability of the original combination being a sentence according to the preset weight coefficient, the first probability of the original combination being a sentence, and the second probability of the original combination being a sentence includes: Substituting the preset weight coefficient, the first probability of the original combination being a sentence, and the second probability of the original combination being a sentence into the first formula to obtain the probability of the original combination being a sentence; Among them, the first formula is: P=a×P1+(1-a)×P2; P is the probability that the original combination is a sentence, a is the preset weight coefficient, and a is a number less than 1; P1 is the first probability that the original combination is a sentence; P2 is the second probability that the original combination is a sentence.

8. A sentence recognition device, characterized in that: include: An acquisition unit, used for acquiring an original combination of multiple words arranged in word order; A calculation unit is used to input the original combination into a target N-gram grammar model to obtain the probability of the original combination being a sentence; wherein, for the names in the list of names of people and places contained in the original combination, the target N-gram grammar model replaces the names contained in the original combination with the corresponding category labels in the list of names of people and places to obtain a replaced combination, and calculates the probability associated with the category label in the replaced combination, and obtains the probability of the original combination being a sentence based on the word-related probability of each word in the replaced combination, the probability associated with the category label, and the proportion of all names in the original combination under the corresponding category label in the list of names of people and places; The names in the list of names and places include: names and places; the category labels corresponding to the names in the list of names and places are set according to a preset classification rule; A creating unit, configured to create a first corpus; wherein each sentence in the first corpus contains a name; the name includes a person's name and / or a place name; A replacement unit, configured to replace the names in the first corpus with corresponding category labels according to the preset classification rule, so as to obtain a replaced first corpus; A first modeling unit is used to train an initial N-gram grammar model according to the replaced first corpus; wherein the initial N-gram grammar model is used to calculate the probability of each word in the combination containing the replaced category label and the probability of the replaced category label; a second modeling unit, for obtaining the target N-gram model according to the list of names and places and the initial N-gram model; wherein the target N-gram model is used to calculate the probability of obtaining a combination of multiple words containing names in the list of names and places as a sentence; the list of names and places at least includes the names and places included in each sentence in the first corpus; The calculation unit is used to obtain the probability of the original combination being a sentence according to the probability of word correlation of each word in the replaced combination, the probability of category label correlation, and the proportion of all names under the category label corresponding to the name in the original combination in the list of names and places, including: Obtaining the probability of the names in the original combination according to the probability of the category label correlation and the proportion of all names under the category label corresponding to the name in the list of personal names and place names; The probability of the original combination being a sentence is obtained based on the probability of the name in the original combination and the probability of the word in each word in the replaced combination.

9. The device according to claim 8, characterized in that The calculation unit is used to obtain the probability of the name in the original combination according to the probability of the category label and the proportion of all names under the category label corresponding to the name in the list of personal names and place names, including: According to the number of names corresponding to the category labels in the replaced combination in the list of names and places, the proportion of all names under the category labels corresponding to the names in the original combination in the list of names and places is obtained; The probability associated with the category label is multiplied by the proportion of all names in the original combination under the category label corresponding to the name in the list of personal names and place names to obtain the probability associated with the name in the original combination.

10. The device according to claim 8, characterized in that The list of names and places also includes: a weight value corresponding to each name in the list of names and places; The calculation unit is used to obtain the probability of the names in the original combination according to the probability of the category label and the proportion of all names under the category label corresponding to the name in the list of personal names and place names, including: According to the weight values ​​corresponding to all the names under the category labels corresponding to the names in the original combination in the list of names and places, the proportions of all the names under the category labels corresponding to the names in the original combination in the list of names and places are obtained; The probability associated with the category label is multiplied by the proportion of all names in the original combination under the category label corresponding to the name in the list of personal names and place names to obtain the probability associated with the name in the original combination.

11. The device according to claim 8, characterized in that The calculation unit is used to obtain the probability of the original combination being a sentence according to the probability of the name in the original combination and the probability of the word in each word in the replaced combination, including: The probability of the name in the original combination is multiplied by the probability of the word in each word in the replaced combination to obtain the probability of the original combination being a sentence.

12. The device according to claim 8, characterized in that If the sentence of the first corpus contains an agglutinative word of a name, the replacement unit includes: a segmentation unit, configured to segment the agglutinative language containing a name in the sentence of the first corpus into a root part and an affix part; wherein the root part is a name; The replacement subunit is used to replace the root part of the agglutinative language with the corresponding category label according to the preset classification rule to obtain the replaced first corpus.

13. The device according to claim 8, characterized in that The computing unit comprises: A first calculation subunit, configured to input the original combination into a target N-gram model to obtain a first probability of the original combination being a sentence; A second calculation subunit is used to input the original combination into a universal N-gram grammar model to obtain a second probability that the original combination is a sentence; wherein the universal N-gram grammar model is trained by a universal corpus; the universal corpus includes sentences containing names and sentences not containing names; The third calculation subunit is used to obtain the probability that the original combination is a sentence according to a preset weight coefficient, a first probability that the original combination is a sentence, and a second probability that the original combination is a sentence.

14. The device according to claim 13, characterized in that The third computing subunit comprises: a fourth calculation subunit, configured to substitute the preset weight coefficient, the first probability that the original combination is a sentence, and the second probability that the original combination is a sentence into the first formula to obtain the probability that the original combination is a sentence; Among them, the first formula is: P=a×P1+(1-a)×P2; P is the probability that the original combination is a sentence, a is the preset weight coefficient, and a is a number less than 1; P1 is the first probability that the original combination is a sentence; P2 is the second probability that the original combination is a sentence.

15. A computer readable medium, characterized in that A computer program is stored thereon, wherein when the program is executed by a processor, the method according to any one of claims 1 to 7 is implemented.

Images