Chinese named entity recognition and classification method and device

Abstract

The present application discloses a method and device for Chinese named entity recognition and classification, the method comprising: S1, identifying professional words in an input text and combining them with a pre-installed word list to form a first dictionary; S2, if the number of words is greater than a preset threshold, executing S3, otherwise using a first entity recognition algorithm to obtain a list of significant words from the input text, selecting words to obtain a selected word list, and then executing S5; step S3, using the input text and the first dictionary as inputs to a second entity recognition algorithm, outputting a first classification word list, selecting words to obtain a selected word list, and then executing S5; S4, using the input text, the first dictionary and the selected word list as inputs to a second entity recognition algorithm, outputting a second classification word list, selecting words to obtain a selected word list, and then executing S5; S5, if the selected word list covers core words, outputting a named entity list, otherwise executing S4. The present application belongs to weakly supervised learning, does not rely on text training data labeled with named entities, and greatly saves human resources.

Description

Technical Field

[0001] The present invention relates to the technical field of text recognition, and in particular to a method and device for Chinese named entity recognition and classification. Background Art

[0002] The recognition and classification of open domain Chinese named entities is the basis for understanding open domain texts. Open domain refers to text fields with little or no annotated data, such as medical diagnosis texts, legal documents, ancient books, poems, intangible cultural heritage documents, etc. Its extraction effect and efficiency affect the comprehensive analysis of texts. However, the named entities in open domain texts are complicated and confusing, and the wording is not uniform, making it difficult to determine the core vocabulary system. General named entity recognition methods require a large amount of text annotated data for training, which is difficult to obtain in the open domain.

[0003] At present, there are some word segmentation tools at the text analysis algorithm level that can discover and identify named entities, such as Jieba, THULAC, PKUSEG, StanfordNLP, etc., but these methods cannot identify named entities of customized categories. In addition, these methods are trained through a large amount of word segmentation data (such as the Sighan 2005Bakeoff dataset), and have good performance on the test corpus, but they rely on training data, that is, texts with manual word segmentation and annotation. In the open domain, due to the differences in idiomatic vocabulary and writing style, word segmentation data in general fields is not suitable, and the annotation of open domain text data consumes a lot of manpower and financial resources. These methods cannot efficiently identify and classify named entities. Summary of the invention

[0004] To solve the above problems, this application provides a Chinese named entity recognition and classification method, including:

[0005] Step S1, identifying professional terms in the input text by regular matching, and merging them into a pre-installed vocabulary to form a first dictionary, wherein the pre-installed vocabulary contains classified professional terms;

[0006] Step S2: if the number of each type of words in the first dictionary is greater than a preset threshold, then execute step S3; otherwise, use the first entity recognition algorithm to obtain a significant word list sorted by word significance from the input text, select words from the significant word list to obtain a selected word list, and then execute step S5;

[0007] Step S3, using the input text and the first dictionary as inputs of a second entity recognition algorithm, outputting a first classification vocabulary, selecting words from the first classification vocabulary to obtain a selected word vocabulary, and then executing step S5;

[0008] Step S4, using the input text, the first dictionary and the word selection word list as inputs of a second entity recognition algorithm, outputting a second classification word list, selecting words for the second classification word list to obtain a word selection word list, and then executing step S5;

[0009] Step S5: if the word selection vocabulary covers the core words, output the named entity list, otherwise execute step S4.

[0010] The present invention also provides a computer device, which includes a memory and at least one processor, wherein the memory stores at least one instruction, and when the at least one instruction is executed by the at least one processor, the above-mentioned Chinese named entity recognition and classification method is implemented.

[0011] This application determines whether to use the first entity recognition algorithm or the second entity recognition algorithm to perform entity recognition and classification through a preset threshold. The first entity recognition algorithm can enrich the vocabulary of the pre-installed word list, and the second entity recognition algorithm can realize entity recognition and classification. Both the first entity recognition algorithm and the second entity recognition algorithm belong to weakly supervised learning, and the training data they need are mainly the first dictionary and the results of expert word selection, and do not rely on text training data labeled with named entities, which greatly saves human resources. And categories can be customized according to the field, such as processes, materials, etc., which are different from general named entity categories of people and places. In addition, this application pioneered the efficient combination of named entity recognition and manual auxiliary judgment, and used the first entity recognition algorithm and the second entity recognition algorithm to accurately perform entity recognition and classification, so as to achieve fast and accurate entity recognition and classification. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] The above-mentioned features and technical advantages of the present invention will become clearer and easier to understand by describing its embodiments in conjunction with the following drawings.

[0013] Figure 1 is a flow chart showing a Chinese named entity recognition and classification method according to an embodiment of the present invention;

[0014] Figure 2 is a flow chart representing a first entity recognition algorithm of an embodiment of the present invention;

[0015] Figure 3 is a flow chart representing a second entity recognition algorithm of an embodiment of the present invention;

[0016] Figure 4A This is an example diagram of segmenting words in a target text provided by an embodiment of the present application;

[0017] Figure 4B is an example diagram of using different tags to distinguish named entities in a target text provided by an embodiment of the present application;

[0018] Figure 4C is an example diagram of a super pattern and its frequency provided by an embodiment of the present application, the length of which does not exceed the maximum length threshold of a shape symbol in the super pattern;

[0019] Figure 4D is an example diagram of a super mode provided by an embodiment of the present application, where the frequency is lower than the minimum frequency threshold for the occurrence of the super mode;

[0020] Figure 4E is an example diagram of a super mode provided by an embodiment of the present application, wherein the frequency is not less than the minimum frequency threshold for the occurrence of the super mode;

[0021] Figure 5 is an example diagram of the generation probability of the hidden state of the unsegmented text segment "Zhao Pu is the Grand Master" provided in an embodiment of the present application;

[0022] Figure 6 is an example diagram of the conditional probability of the hidden state of the unsegmented text segment "Zhao Pu is the Grand Master" provided in an embodiment of the present application;

[0023] Figure 7 The embodiment of this application provides the basis Figure 6 The n calculated by the parameters in p (T) and n cw (T) Example diagram;

[0024] Figure 8 It is an architectural diagram of a computer device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0025] The embodiments of the present invention will be described below with reference to the accompanying drawings. It will be appreciated by those skilled in the art that the described embodiments may be modified in various ways or combinations thereof without departing from the spirit and scope of the present invention. Therefore, the drawings and descriptions are illustrative in nature and are not intended to limit the scope of the claims. In addition, in this specification, the drawings are not drawn to scale and the same reference numerals represent the same parts.

[0026] In this embodiment, the Chinese named entity recognition and classification method can be applied to a computer device (for example, Figure 8 In the computer device 3) shown in the figure, named entity recognition and classification in the input text are realized. Specifically, for a computer device that needs to perform named entity recognition and classification, the functions for named entity recognition and classification provided by the method of the present application can be directly integrated on the computer device, or run on the computer device in the form of a software development kit.

[0027] like Figure 1 As shown, the Chinese named entity recognition and classification method of this embodiment includes the following steps:

[0028] Step S1, identifying professional words in the input text through preprocessing, as a supplement to the preinstalled word list, merging them into the preinstalled word list to form a first dictionary, wherein the preinstalled word list contains classified professional words. In addition, the input text can also be quality controlled, such as identifying repeated paragraphs and deleting them to avoid interference with subsequent analysis. The preprocessing can be to use regular matching (such as book title marks) to identify professional words in the input text.

[0029] Step S2, if the number of each category of words in the first dictionary is greater than the preset threshold, then execute step S3, otherwise use the first entity recognition algorithm to extract the significant words in the input text, and obtain a significant word list sorted by word significance, select words from the significant word list to obtain a selected word list, and then execute step S5. Among them, the preset threshold of each category of words can be 200, that is, when the preset threshold of each category of words is greater than 200, then execute step S3. The word selection can be performed by experts to further select the correct named entity from them.

[0030] Step S3, taking the input text and the first dictionary as inputs of a second entity recognition algorithm, outputting a first classification vocabulary, selecting words from the first classification vocabulary to obtain a selected word vocabulary, and then executing step S5.

[0031] Step S4, taking the input text, the first dictionary and the word selection word list as inputs of the second entity recognition algorithm, outputting a second classification word list, selecting words for the second classification word list, obtaining the word selection word list, and then executing step S5. The word selection can be performed by an expert to further select the correct named entity and classification therefrom.

[0032] Step S5, if the word selection list covers the core words, then output the named entity list, otherwise execute step S4, the core vocabulary is covered according to a certain coverage rate, for example, 99%, then it is considered that the core vocabulary has been covered, then the above iterative steps can be terminated and the named entity list can be output.

[0033] Further, such as Figure 2 As shown, the first entity recognition algorithm includes:

[0034] Step S11, using a maximum expectation algorithm to filter the first dictionary to obtain a second dictionary;

[0035] Among them, using the maximum expectation algorithm to screen the first dictionary to obtain the second dictionary is a key step of the present invention. Because the first dictionary contains a large number of character strings obtained by exhaustive enumeration from the input text as candidate words, the scale of the first dictionary is very large. The first dictionary contains both words with practical significance and a large number of character strings without practical significance. The EM algorithm can reset the parameters of most character strings without practical significance to zero by continuously optimizing system parameters, thereby realizing the screening of candidate words in the first dictionary.

[0036] A systematic method for selecting candidate words is given by applying the EM algorithm and model selection technology. The EM algorithm is an iterative algorithm that includes two basic steps, the E step and the M step. The E step is closely related to word segmentation, and the M step is closely related to unknown word recognition. Based on the first dictionary, the EM algorithm is used to unify the two problems of unknown word recognition and word segmentation and transform them into a statistical inference problem. The second dictionary gives the basic results of unknown word recognition.

[0037] The EM algorithm is a general statistical method for obtaining the maximum likelihood estimate of model parameters in the context of missing data. The method starts with an initial estimate of the unknown parameters and gradually optimizes the parameter estimate by repeatedly iterating between the two basic steps of the E step (Expectation) and the M step (Maximization). Statistical theory ensures that the EM algorithm converges to at least a local maximum point of the likelihood function. In the present invention, the unknown word separators in the input text are taken as missing data, and the input text is taken as incomplete data, and the EM algorithm is used to estimate the parameter θ in the statistical dictionary model. A complete EM algorithm includes four steps: selection of initial values, E step, M step and stopping criterion. In this embodiment, the following steps are specifically included:

[0038] (a) Selection of initial value: for the substring W in the first dictionary D i , whose parameter θ i The initial value is set to

[0039]

[0040] in,

[0041] I(·) is the indicator function,

[0042] Among them, the integer value refers to the string T j Neutron string W i The number of occurrences, L j It is T j The length of the string T j The tth character in k iIt is W i Length, T j [a,b] is T j The substring from the character at position a to the character at position b (including the characters at position a and position b);

[0043] For example, for

[0044] T j =ABCABCD,W i =ABC,

[0045] Then there is

[0046] Wherein, the first dictionary D={W1,W2,…,W N ,τ} includes N words and a special termination symbol τ;

[0047] Character set A = {A1, A2, ... A p} is the totality of all basic language units of the target language;

[0048] vocabulary is a sequence of l elements in A;

[0049] Word parameter θ={θ1,θ2,…θ N ,θ τ} gives the usage probability of each element in the first dictionary D, where 0≤θ i <1 for i=1,2,...,N,0<θ τ <1, and (D,θ) is called the first dictionary model;

[0050] sentence is a sequence of k words, where the symbol “|” represents a word separator;

[0051] Unsegmented text It is the concatenation of k words. The unsegmented text does not contain word separators.

[0052] The input text T = {T1, T2, ..., T n} is a set of n unsegmented texts;

[0053] The following basic assumptions are adopted: It is generated by performing k independent extractions with replacement on the words in the first dictionary D, and each time a word W is extracted i The probability is θ i , until the terminal symbol τ is drawn, and the sentence is generated The likelihood function is:

[0054]

[0055] Generate unsegmented text T j The likelihood function is:

[0056]

[0057] in, Indicates that under the first dictionary D, the unsegmented text T j All possible ways of segmenting .

[0058] For example, given the following dictionary (D,θ):

[0059] D Vice-minister Wang Anshi for king install stone τ θ 0.15 0.15 0.15 0.15 0.15 0.15 0.10

[0060] Then, for unsegmented text

[0061] T j = "Wang Anshi is the Vice Minister of State Affairs",

[0062] There are two possible word segmentations:

[0063] S1 = "Wang Anshi is the Vice Minister of State Affairs",

[0064] S2 = "Wang Anshi was the Vice-Chancellor of State Affairs".

[0065] Thus, we have and

[0066] P(S1|D,θ)=0.15×0.15×0.15×0.15×0.15×0.10≈7.59×10 -6 ,

[0067] P(S2|D,θ)=0.15×0.15×0.15×0.10≈3.38×10 -4 ;

[0068] P(T j |D,θ)=P(S1|D,θ)+P(S2|D,θ)≈3.45×10 -4 .

[0069] Thus, we can also get a given T j When , the conditional probabilities of the two possible word segmentation methods are:

[0070]

[0071]

[0072] Obviously, in this example, the word segmentation method S2 with a larger conditional probability corresponds to the correct word segmentation.

[0073] (b) Let θ (r) For the parameter estimates obtained in the rth iteration of the maximum expectation algorithm, calculate the following statistics:

[0074]

[0075] Among them, P(S|T j ; D, θ) means that under the first dictionary model (D, θ), T j The conditional probability generated by the word segmentation method S is defined as follows:

[0076]

[0077] Indicates that under the dictionary model (D, θ), T j Conditional probability generated by the word segmentation method S; symbol n i (S) means the word W in sentence S i For example, for

[0078] S=ABC|ABCD|AB|C,W i =ABC,

[0079] There are i (S) = 1. In actual calculations, the statistic The value of can be obtained in linear time through dynamic programming. Its computational complexity is O(L j K), where L j It is T j The length of D, K is the maximum length of the vocabulary in D.

[0080] (c) Update the estimate of parameter θ according to the following formula

[0081]

[0082] (d) Repeat steps b and c until θ (r) and θ (r+1) The Euclidean distance d(θ (r) ,θ (r+1) ) is less than a predetermined threshold, the first entity recognition algorithm is stopped. In this embodiment, the default threshold is δ=1×10 -6 .

[0083] (e) All words in the first dictionary D whose parameter θ is not zero form the second dictionary.

[0084] Step S12, using the log-likelihood ratio statistic to calculate the statistical significance of the words in the second dictionary to obtain the significance scores of the words in the second dictionary, and sorting the words in the second dictionary according to the significance scores to obtain a significant word list.

[0085] The significance of the words in the second dictionary is calculated by the log-likelihood ratio statistic. In this embodiment, the log-likelihood ratio statistic is used to calculate the statistical significance of the words in the second dictionary, and this is used as an indicator to judge the importance of the words. The higher the importance score of an identified unregistered word (i.e., higher than a preset threshold), the identified unregistered word is valid; if the importance score of an identified unregistered word is too low (i.e., lower than a preset threshold), the unregistered word will be determined to be invalid and will be removed from the system. This step can be used to further filter vocabulary based on the first dictionary.

[0086] Given a first dictionary D and its parameter θ, for a set of input texts, the statistical significance of different elements in D is different. Understanding the statistical significance of vocabulary is important for improving the accuracy of unregistered word recognition. This is because the elements in D include both vocabulary imported from the pre-installed vocabulary and regular character strings extracted from the input text as candidate vocabulary. If an element in the second dictionary has a high importance score and is not included in the known vocabulary, then it can be confidently identified as an unregistered word; conversely, if the importance score of this element is very low, then the recognition result is likely to be invalid and should be handled with caution (for example, confirmed by manual verification). In this embodiment, the log-likelihood ratio statistic defined as follows can be used to measure the vocabulary W i The statistical significance of the word W i The statistical significance of the log-likelihood ratio statistic is defined as follows:

[0087]

[0088] in,

[0089]

[0090] P(T|D,θ) represents the likelihood function of the first dictionary model (D,θ) fitting the input text T. i This means that once the word W i The parameter θ i If it is forcibly set to 0, the fit to the input data will be significantly worse, which means that W i Very important for input text.

[0091] Furthermore, the first dictionary D and its parameters θ, for a given unsegmented text T∈T, implements word segmentation by calculating and analyzing the conditional probabilities of different word segmentation methods of T; assuming is all possible word segmentation methods of T under the first dictionary D. According to the basic assumption, under the condition of given T, the word segmentation method The conditional probability is:

[0092]

[0093] According to the maximum likelihood principle, the maximum likelihood estimate of the unknown segmentation method is the segmentation method that maximizes P(S|T; D,θ), that is,

[0094]

[0095] Further, the conditional probability of a word separator appearing after each character of the unsegmented text T is calculated, and word segmentation is performed according to the conditional probability; if the word segmentation method S has a word separator after the tth character of T, then let the indicative function I t (S) = 1, otherwise, let the indicative function I t (S) = 0; therefore, the conditional probability of a separator appearing after the tth character is

[0096]

[0097] A natural choice is: if the conditional probability γ t >0.5, a word separator is placed after the t-th character in T. This embodiment uses the word segmentation method based on conditional probability by default.

[0098] Further, such as Figure 3 As shown, the second entity recognition algorithm includes (it should be noted that the second entity recognition algorithms used in steps S3 and S4 are the same, but there are some differences in input. In step S3, the input text and the first dictionary are used as the input text of the second entity recognition algorithm, and in step S4, the input text, the first dictionary, and the word selection table are used as the input text of the second entity recognition algorithm):

[0099] Step S21, constructing a compound dictionary model, the compound dictionary model includes a starting word dictionary obtained from the input text Starting super-schema dictionary and the starting parameter system (γ,θ w∣c ), the input text can be Chinese input text in a specific field, such as classical Chinese, medical electronic medical records, prose, poetry, etc.

[0100] The input text is segmented character by character to obtain a character set consisting of all characters in the input text Where 1≤l≤L;

[0101] According to the prior information of the words in the input text, the word set is set The prior information of the words in the input text may be a known word library pre-stored in a computer device, wherein the known word library contains the words in the input text, and the words include a character set. For example, Wang Anshi, Wei, Canzhi Zhengshi, Zhao Pu, Wei, Taishi, etc. It should be noted that the above “Zhao”, “Wei”, and “Shi” are traditional Chinese characters, which are used to illustrate the analysis performed when the input text includes ancient Chinese.

[0102] In one embodiment, a word set is obtained according to a preset first rule. The preset first rule includes: setting is the maximum length threshold of characters in a word, is the minimum frequency threshold of the occurrence of a word, and the input text is mined using the first entity recognition algorithm to extract the input text whose length does not exceed and the frequency is not less than the words, and obtain the word set Although the first entity recognition algorithm can achieve efficient word discovery while segmenting the input text, it still has certain limitations: (1) it only uses character sequence count information during the learning process; (2) it cannot automatically distinguish word categories; (3) it omits low-frequency words. For example, Then, after extracting the words from the input text "AABBABCCABCD" using the first entity recognition algorithm, the word set obtained is The numbers in brackets represent the frequency of occurrence of the word.

[0103] In one embodiment, according to the prior information of the word category in the input text, the word category set in the input text is set to where c k represents the category of the named entity, c0 represents the category of the background word that is different from the named entity, Each word in the input text belongs to a specific word category, which includes a category of a named entity and a category of a background word that is different from the category of the named entity, where c krepresents the category of the named entity, and c0 represents the category of the background words that are different from the named entity. The prior information of the word category in the input text can be a known word category library pre-stored in a computer device, and the known word category library contains all the word categories in the input text. The named entities include names of people, official titles, place names, and all other entities identified by names, such as Zhao Pu, Wang Anshi, Taishi, Canzhi Zhengshi, etc.; the background words include all words that are different from the named entities, such as Zhao, Pu, Wei, Tai, Shi, Wei, Wang, An, Shi, Wei, Can, Zhi, Zheng, Shi, etc.

[0104] In one embodiment, a word set is obtained according to a preset second rule. The preset second rule includes: using a pre-trained weak classifier to classify the input text that is not longer than and the frequency is not less than the Extract words; determine the prior classification probability π c∣w , where c represents the word category in the input text, w represents the words extracted by the weak classifier, and π c∣w Represents the prior classification probability of word category c given word w, the prior classification probability π c∣w represents a conditional probability (for example, the probability that the word "Zhao Pu" is a "name"); and selects the words extracted by the weak classifier, the prior classification probability π c∣w Words that exceed a preset classification threshold (e.g., 0.5) are used to obtain the word set The pre-trained weak classifier is a word classifier (or input text classifier), which can be obtained by using a word training model in a known word library pre-stored in a computer device. The classification accuracy of the weak classifier is between 60% and 80%.

[0105] Integrate the character set Word collection Word collection Word collection Get the starting word dictionary The starting word dictionary in, , 1≤n≤N,i, represents a positive integer; w n is composed of the Words composed of Chinese characters,

[0106] Wherein, the starting super-mode dictionary Among them, e represents the terminator, 1≤m≤M,p mDenote the super-pattern composed of tokens t j where j represents a positive integer. For example, the token t can be a person's name (Name, N), an official title (Office title, O), a background word (Background, B), "為" (regarded as j the characters in), "為" (regarded as the characters in), etc. Then the super-pattern p can be N, O, B, N is O, N為O, etc. The terminator e represents the stop criterion for extracting the super-patterns in the input text. When the terminator e is extracted, the extraction of super-patterns stops. For example, when considering the generation process of the input text fragment "趙普為太師" based on the composite dictionary model, first extract the super-pattern "N為O" and the terminator to obtain the super-pattern sequence, then extract words according to the extracted super-pattern sequence to obtain the word sequence, and finally obtain the input text fragment "趙普為太師" corresponding to the word sequence. m In one embodiment, according to the prior information of the super-patterns in the input text, set the super-pattern set

[0107] The prior information of the super-patterns in the input text can be a known super-pattern library pre-stored in a computer device, and the known super-pattern library contains all the super-patterns in the input text.

[0108] In one embodiment, obtain the super-pattern set according to a preset third rule The preset third rule includes: for example FIG. 4A to FIG. 4E As shown, based on the starting word dictionary identify and segment the words in the input text, and use different marks to distinguish the named entities in the input text whose word confidence meets (for example Figure 4B in, distinguish the person's name and official title with bold and italic: N, O, and can also be distinguished with different colors), where is a preset confidence threshold (for example, 0.85); set as the maximum length threshold of the tokens in the super-pattern (for example, 3), as the minimum frequency threshold for the appearance of the super-pattern (for example, 2), and scan in order (for example, use an optical character recognition algorithm or an image recognition algorithm to sequentially scan and recognize the characters with the same mark feature) and extract the adjacent marked named entities in the input text whose length does not exceed the and whose frequency is not lower than the For example Figure 4DThe frequency of the "Wang N Wei" in the super pattern is 1, which is lower than the lowest frequency threshold 2 for the appearance of the super pattern, so it is not extracted; for example Figure 4E The frequency of the "N Wei O" in the super pattern is 2, which is not lower than the lowest frequency threshold 2 for the appearance of the super pattern, so it is extracted), integrate the super patterns extracted in this process to obtain the super pattern set

[0109] Integrate the terminator e and the word category set Super pattern set Super pattern set Obtain the starting super pattern dictionary The starting super pattern dictionary

[0110] Among them, the construction of the starting parameter system (γ, θ w∣c ) includes: obtaining the word count frequency θ of the words in the starting word dictionary in the input text w ; according to the count word frequency θ w and the prior classification probability π c∣w , determine the starting value of the sampling probability distribution of the words in the starting word dictionary when the given word category is c where represents the sampling probability distribution of words when the given word category is c, θ w∣c ∝θ w ·π c∣w , θ w∣c represents the sampling probability of the word w when the given word category is c (for example, the sampling probability of the word "Zhao Pu" when the given word category is "person's name"); the sampling probability of the super patterns in in the input text to obtain the starting value γ of the sampling probability distribution of the super patterns in γ p represents the sampling probability of the super pattern p (for example, the sampling probability of the super pattern "N Wei O" is 0.01).

[0111] Step S22, calculate the updated parameter system η = (γ, θ w∣c ) based on the starting parameter system (γ, θ w , θ c∣w ); set the prior distribution of the updated parameter system (γ, θ w , θ c∣w ), and the updated parameter system (γ, θ w , θ c∣w)’s posterior distribution.

[0112] In one embodiment, the starting parameter system (γ, θ w∣c ) calculates the updated parameter system (γ,θ w ,θ c∣w ) includes: for a word category and a super mode definition is the number of times word category c appears in super-pattern p. Then, in the composite dictionary model, the frequency of word category c is in, The frequency of word w is Given a word w, the frequency of word category c is make Represents the distribution of words, let Represents the sampling probability distribution of a set of word categories, where represents the conditional distribution of word categories given word w, θ c∣w Represents the sampling probability of word category c given word w.

[0113] It should be noted that when the updated parameter system (γ,θ w ,θ c∣w ) satisfies the constraints: When the two parameter systems can be converted to each other, there is a one-to-one mapping. In one embodiment, the computer device can update the parameter system (γ, θ w ,θ c∣w ),calculate The starting parameter system (γ,θ w∣c ).

[0114] In one embodiment, the parameter system (γ, θ) of setting the update w ,θ c∣w ), and the updated parameter system (γ,θ w ,θ c∣w ) includes: a computer device performs Bayesian modeling on the composite dictionary model, and sets the updated parameter system (γ, θ) according to the Dirichlet distribution Dir (α, F) w ,θ c∣w ), the expression of the Dirichlet distribution is Among them, α>0 represents the intensity (Magnitude), F=(F1,…,F d) represents the base distribution, and F is also the mode of the Dir(α,F) distribution. Indicates an indicator function. When the conditions in "{}" are met, The value is 1, otherwise The value is 0. In the updated parameter system (γ,θ w ,θ c∣w ), the prior distribution of the parameter is set to γ~Dir(α γ ,π γ ), θ w ~Dir(α w ,π w ), θ ·∣w ~Dir(α ∣w ,π ·∣w ). According to the definition of Dir(α,F), the computer device uses (π γ ,π w ,π ·∣w ) represents the updated parameter system parameters (γ,θ w ,θ c∣w )’s prior propensity value, (α γ ,α w ,α ·∣w ) represents the strength of prior information. Let π w For uniform distribution, π ·∣w Given by the word prior classification distribution, the word prior classification distribution is predefined in the computer device. According to the constraint condition of the updated parameter system set up Set the intensity hyperparameter to (α γ ,α w ,α ·∣w )=(αN,αN,α), get (γ,θ w ,θ c∣w ) set up is the B unsegmented text fragments in the input text, then the updated parameter system (γ,θ w ,θ c∣w ) is:

[0115] Step S23, based on the compound dictionary model, obtain the unsegmented text segment T of the input text, determine the hidden state H of the unsegmented text segment T, and set missing data

[0116] In one embodiment, the set is the B unsegmented text segments T in the input text b A set of , b and B represent positive integers and 1≤b≤B; Representing a collection The unsegmented text fragment T in b It should be noted that when the unsegmented text segment When the hidden state H of the unsegmented text segment T can be expressed as hidden state H T .

[0117] In one embodiment, from the starting super-pattern dictionary Extract the super pattern p from the super pattern sequence P to obtain the super pattern sequence P; extract the word w according to the super pattern sequence P to obtain the word sequence W i and the word sequence W i A sequence W, where Based on the sequence W, an unsegmented text segment T is obtained, and a hidden state H of the unsegmented text segment T is determined.

[0118] In one embodiment, the starting super-pattern dictionary Extracting super patterns from the sample to obtain a super pattern sequence P includes: the computer device performs sampling according to the sampling probability γ of each super pattern. p , starting from the super-pattern dictionary The superpattern P is randomly drawn with replacement i , until the termination symbol e is extracted; set the super pattern sequence P = P1 | P2 | ... | P n |e, where super mode

[0119] In one embodiment, the computer device calculates the sampling probability of P where γ e Represents the sampling probability of the terminal symbol e.

[0120] In one embodiment, the word w is extracted according to the super pattern sequence P to obtain the word sequence W i and the word sequence W i The sequence W comprises: the computer device determines the super mode P i Each t in ij Category; when the shape character When w ij =t ij ; When the shape character When, according to the sampling probability From the starting word dictionary Extract words w ij ; Obtain the sequence and the sequence W = W1|W2|…|Wn 。

[0121] In one embodiment, the computer device calculates the sampling probability of the W given the P i of the sampling probability wherein, represents the Indicator Function, when the condition in "{}" is satisfied, the value is 1, otherwise the value is 0, and when when when when represents the sampling probability of w when ij of the sampling probability of w when w ij = t ij when, w ij the sampling probability of w is 1; when w ij ≠ t ij when, w ij the sampling probability of w is 0; Calculate the sampling probability of the W given the P

[0122] For example, assume the composite dictionary model given in Table 1 below,

[0123] Table 1

[0124]

[0125]

[0126] Then, for example Figure 5 as shown, the extracted super-pattern sequence P3: N is O|e, where the super-pattern "N is O" contains 3 morphemes, namely "N", "is", and "O", and the morpheme "N" belongs to the word category set According to θ w∣(c＝N) the corresponding extracted word is "Zhao Pu"; the morpheme "is" belongs to the character set the corresponding extracted word is "is"; the morpheme "O" belongs to the word category set According to θ w∣(c＝O) the corresponding extracted word is "Grand Tutor"; so the word sequence W3 extracted according to the super-pattern sequence P3 is: Zhao Pu|is|Grand Tutor; Similarly, it can be obtained Figure 5 in

[0127] In one embodiment, based on the sequence W, an untokenized text segment T is obtained, multiple hidden states H of the untokenized text segment T are determined, and the generation probability of each hidden state H is calculated

[0128] In one embodiment, the delimiters in the sequence W are removed to obtain an untokenized text segment T = T1T2…T n , where For example, when the sequence W = word sequence W3: "Zhao Pu|is|Grand Tutor", the delimiters are removed to obtain the untokenized text segment: Zhao Pu is Grand Tutor.

[0129] In one embodiment, let H = (P, W) be the hidden state of the untokenized text segment T, and the generation probability (i.e., sampling probability) of the hidden state H is calculated by the formula: where represents the number of occurrences of the super-pattern p in the hidden state H, represents the number of occurrences of the category-word pair (c, w) (e.g., "person name", "Zhao Pu") in the hidden state H. For example Figure 5 as shown, the generation probability of the hidden state H1 = (P1, W1) the generation probability of the hidden state H3 = (P3, W3)

[0130] It should be noted that in practical applications, the hidden state H is unobservable and is missing data, and only the untokenized text segment T is observable data. Different untokenized text segments T can be generated by different hidden states H. Let be denoted as the set of all hidden states that can generate the untokenized text segment T. For example Figure 5 as shown, the set of all hidden states of the untokenized text segment "Zhao Pu is Grand Tutor" Then, in the framework of the composite dictionary model, the generation probability of the untokenized text segment T is For example Figure 5 as shown, the generation probability of the untokenized text segment "Zhao Pu is Grand Tutor" is

[0131] In one embodiment, given the untokenized text segment T, the conditional distribution of the hidden state H is where, when the hidden state H belongs to the set , takes the value of 1; when the hidden state H does not belong to the set , takes the value of 0.

[0132] In one embodiment, the hidden state H of each unsegmented text segment T is T is unobservable, the observed data likelihood function contains the hidden state H T The sum of , resulting in the updated parameter system η has no explicitly expressible posterior distribution, where represents the set of all hidden states H of any unsegmented text segment T. We can use the missing data analysis framework in statistics to set It should be noted that for missing data The hidden state H of any unsegmented text segment T in , its value space is

[0133] Step S24, performing parameter inference on the updated parameter system η, comprising:

[0134] The parameter inference of the updated parameter system η includes method 1 and method 2. It should be noted that method 1 is more practical among the two methods.

[0135] In one embodiment, the method includes: maximizing the posterior distribution of the updated parameter system η using an expectation maximization (EM) algorithm, wherein the expectation maximization algorithm includes an E step and an M step, wherein the expectation maximization algorithm converts the updated parameter system η = (γ, θ w ,θ c∣w )’s starting value η (0) As the starting value of the first iteration, the parameter η obtained after the previous iteration is converged through the iterations performed in the E step and the M step. old Get the parameter η obtained in this iteration new , in the next iteration, the parameter η new As the parameter η obtained after the previous iteration old Repeat this process until the point estimation result of the updated parameter system η is obtained. (0) It can be obtained from the prior distribution of the updated parameter system η.

[0136] In one embodiment, the step E in the method 1 includes: calculating the Q function: Wherein, T represents an unsegmented text segment of the input text obtained based on the compound dictionary model, and the hidden state of the unsegmented text segment T is a hidden state H; Indicates missing data, set is the B unsegmented text segments T in the input text bThe set of, b and B represent positive integers, 1≤b≤B; H T Representing a collection The hidden state of the unsegmented text segment T in η; old represents the parameters obtained after the previous iteration, represents the number of times the super pattern p appears in the hidden state H, represents the set of all hidden states H of the unsegmented text segment T, represents the number of occurrences of the category word pair (c,w) in the hidden state H. For example, according to Figure 5 The parameter pairs in For example, the calculation Figure 6 shown.

[0137] In one embodiment, the unsegmented text segment T is set to A1A2…A L , where the characters definition T [＞r] =A r+1 A r+2 …A L , , is a sub-string of T; let represents a set of hidden states containing only one super-pattern, where Len(P) represents the number of super-patterns in the super-pattern sequence P; let represents the forward recursive hidden state set, let Represents the set of backward recursive hidden states; It should be noted that the length of the unsegmented text segment generated by any super-pattern (i.e., the number of characters contained) does not exceed in, is the maximum length threshold of the glyphs in the preset super-pattern (e.g., 3), is the maximum length threshold of characters in a word (e.g. 4), then when any unsegmented text segment T contains more than L P characters, the unsegmented text segment T is an empty set. It should be noted that, from this, it can be concluded that no matter how many characters are contained in the unsegmented text segment T, and contain only a few elements, and and The size of the set will not grow infinitely in direct proportion to the length of the unsegmented text segment T.

[0138] In one embodiment, for n p (T) and n cw The calculation of (T) includes The computer device uses the framework of the Baum-Welch algorithm and uses dynamic programming to calculate n p (T) and n cw (T) Solve, including: in, Indicates that T is not considered when the terminal character e is not considered [＜l] The observed likelihood of Indicates that T is not considered when the terminal character e is not considered [＞r] The observed likelihood of Recursive calculation using the forward-backward algorithm and include: The starting state of the recursion is Len(H) represents the number of characters in the unsegmented text segment corresponding to the hidden state H. For example, according to Figure 6 The n calculated by the parameters in p (T) and n cw (T) Figure 7 shown.

[0139] In one embodiment, the M steps in the method 1 include: setting a restriction condition θ c (γ) is a fixed value; based on the constraints, the Expectation Conditional Maximization (ECM) algorithm is used to calculate The maximum values ​​of include: Q(γ||η old )+Q(θ w ∣∣η old )+Q(θ c∣w ∣∣η old )+Constant, where π c∣w represents the prior classification probability of word category c given word w, Constant represents an arbitrary constant; using the maximum Q(θ w ∣η old ) and Q(θ c∣w ∣η old ) Calculate θ w The extreme value and θ c∣w The extreme value include: In a given and Under the condition of It can be known that θ c (γ) is a fixed value, so we maximize Q(γ|η old ) is equivalent to maximizing Its maximum value is It should be noted that the point estimation result of the updated parameter system η obtained by solving the EM algorithm is Satisfy the constraints The stopping criterion for iterative convergence between the E step and the M step can be: the parameter η obtained in this iteration new and the parameter η obtained in the previous iteration old The Euclidean distance between them is less than a pre-defined convergence threshold (e.g., 10 -6 ).

[0140] In one embodiment, the second method includes: using a data augmentation (DA) algorithm to perform posterior distribution sampling on the updated parameter system η to obtain a target posterior distribution of the updated parameter system η, wherein the target posterior distribution is: in, represents the augmented updated sampling space obtained by expanding the sampling space η; wherein the posterior distribution sampling of the updated parameter system η includes: sampling from the conditional distribution and conditional distribution Iterative sampling is performed until the target posterior distribution is obtained, where Represents the set of all hidden states H of the unsegmented text segment T.

[0141] Step S26, based on the result obtained by the parameter inference, obtain the input text segmentation result of the unsegmented text segment T, the input text segmentation result includes the optimal hidden state And based on the input text segmentation result, semantic understanding and relationship extraction are performed on the unsegmented text segment T.

[0142] Wherein, the step of obtaining the input text segmentation result of the unsegmented text segment T based on the result obtained by the parameter inference includes: using the maximum likelihood method to give the prediction result of the optimal hidden state of the unsegmented text segment T Since it is not possible to enumerate All hidden states in the optimal hidden state are obtained The computer device is based on the idea of ​​dynamic programming and uses the framework of the Viterbi algorithm. For the optimal hidden state Solve, where represents the optimal hidden state The number of characters in the corresponding unsegmented text segment.

[0143] Among them, the optimal hidden state of the unsegmented text segment T can be The unsegmented text segment T is semantically understood and relations extracted to help users understand semantics. For example, the optimal hidden state of the unsegmented text segment "Zhao Pu is the Grand Master" is For H3 = (P3, W3) = ("N is O|e", "Zhao Pu| is| Tai Shi"), then through semantic understanding and relationship extraction, we can know that "Zhao Pu" is a person's name and "Tai Shi" is an official title, which means that Zhao Pu's official title is Tai Shi.

[0144] In an optional embodiment, before step S26, the method further includes step S25 of performing a posteriori analysis on the composite dictionary model based on the result obtained by the parameter inference;

[0145] In one embodiment, the posterior analysis of the composite dictionary model based on the results obtained by the parameter inference includes: constructing a significance score of the super-pattern p, word w and category word pair (c, w); and constructing a posterior classification distribution of word w.

[0146] In one embodiment, constructing a significance score of a super-pattern p, a word w, and a class word pair (c, w) comprises: setting is the point estimation result of the updated parameter system η obtained by the parameter inference, let and is a set of updated parameter values; from Set the parameters γ p ,θ w and θ w∣c Set to zero and keep other parameters unchanged; define the significance scores of super pattern p, word w and category word pair (c, w) as follows: All are baseline models The logarithm of the ratio of the observed likelihood of the reduced model. Under the condition that the simplified model is established, the likelihood ratio test shows that 2ψ asymptotically obeys the chi-square distribution, and the model selection task can be performed through hypothesis testing. It should be noted that in actual cases, multiple tests require the use of Bonferroni to correct the results or control the false discovery rate.

[0147] In an optional embodiment, constructing the posterior classification distribution of word w includes: defining the posterior classification distribution of the word as It uses the word prior classification information π c∣w As well as the syntactic and semantic structure information of the input text (i.e., super-pattern information), the optimization result of the prior classification distribution of words is given by using the Bayesian statistical inference method. It should be noted that through the parameter inference and posterior analysis of the composite dictionary model, many elements (e.g., super-patterns, words, etc.) with low significance scores (e.g., less than 10) in the composite dictionary model will be eliminated, and the remaining elements will constitute the optimized composite dictionary model.

[0148] The above describes in detail the Chinese named entity recognition and classification method of the present application. The following describes the hardware device architecture for implementing the Chinese named entity recognition and classification method. Figure 8 As shown, it is a schematic diagram of the structure of a computer device provided in a preferred embodiment of the present application. In a preferred embodiment of the present application, the computer device 3 includes a memory 31 and at least one processor 32. Those skilled in the art should understand that Figure 8 The structure of the computer device shown does not constitute a limitation of the embodiments of the present application, and can be either a bus structure or a star structure. The computer device 3 can also include more or less other hardware or software than shown in the figure, or a different component arrangement.

[0149] In some embodiments, the memory 31 is used to store program codes and various data, such as the first dictionary, the second dictionary, the compound dictionary model, the Chinese named entity recognition and classification system 30 installed in the computer device 3, etc., and to achieve high-speed and automatic access to programs or data during the operation of the computer device 3. The memory 31 includes a read-only memory, a programmable read-only memory, an erasable programmable read-only memory, a one-time programmable read-only memory, an electronically erasable rewritable read-only memory, a read-only optical disk or other optical disk memory, a magnetic disk memory, a magnetic tape memory, or any other computer-readable storage medium capable of carrying or storing data.

[0150] In some embodiments, the at least one processor 32 may be composed of an integrated circuit, for example, a single packaged integrated circuit, or a plurality of packaged integrated circuits with the same or different functions, including a combination of one or more central processing units, microprocessors, digital processing chips, graphics processors, and various control chips. The at least one processor 32 is the control core of the computer device 3, and uses various interfaces and lines to connect the various components of the entire computer device 3, and executes or executes the programs or modules stored in the memory 31, and calls the data stored in the memory 31 to execute various functions of the computer device 3 and process data, such as executing the functions of Chinese named entity recognition and classification.

[0151] In some embodiments, the Chinese named entity recognition and classification system 30 runs in a computer device 3. The Chinese named entity recognition and classification system 30 may include a plurality of functional modules composed of program code segments. Each functional module is a different program code corresponding to the division of the above-mentioned Chinese named entity recognition and classification method. The program code of each program segment in the Chinese named entity recognition and classification system 30 may be stored in a memory 31 of the computer device 3 and executed by at least one processor 32 to implement the Chinese named entity recognition and classification function (see Figures 1 to 3 describe).

[0152] Although not shown, the computer device 3 may also include a power source (such as a battery) for supplying power to each component. Preferably, the power source may be logically connected to the at least one processor 32 through a power management device, so that the power management device can manage charging, discharging, and power consumption management. The power source may also include one or more DC or AC power sources, recharging devices, power failure detection circuits, power converters or inverters, power status indicators, and other arbitrary components. The computer device 3 may also include a variety of sensors, Bluetooth modules, Wi-Fi modules, etc., which will not be repeated here.

[0153] The above-mentioned integrated unit implemented in the form of a software function module can be stored in a computer-readable storage medium. The above-mentioned software function module is stored in a storage medium, including a number of instructions for enabling a computer device (which can be a server, a personal computer, etc.) or a processor to execute a part of the method described in each embodiment of the present application.

[0154] In a further embodiment, in combination Figure 8 The at least one processor 32 can execute the operating device of the computer device 3 and various installed application programs (such as the Chinese named entity recognition and classification system 30), program codes, etc., for example, the above-mentioned modules.

[0155] In one embodiment of the present application, the memory 31 stores one or more instructions (ie, at least one instruction), and the at least one instruction is executed by the at least one processor 32 to implement Figure 1 The purpose of entity recognition and classification is shown.

[0156] In the several embodiments provided in this application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only illustrative, for example, the division of the modules is only a logical function division, and there may be other division methods in actual implementation.

[0157] The modules described as separate components may or may not be physically separated, and the components shown as modules may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0158] In addition, each functional module in each embodiment of the present application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The above integrated unit can be implemented in the form of hardware or in the form of hardware plus software functional modules.

[0159] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A Chinese named entity recognition and classification method, characterized in that: include: Step S1, identifying professional terms in the input text by regular matching, and merging them into a pre-installed vocabulary to form a first dictionary, wherein the pre-installed vocabulary contains classified professional terms; Step S2: if the number of each type of words in the first dictionary is greater than a preset threshold, then execute step S3; otherwise, use the first entity recognition algorithm to obtain a significant word list sorted by word significance from the input text, select words from the significant word list to obtain a selected word list, and then execute step S5; Step S3, using the input text and the first dictionary as inputs of a second entity recognition algorithm, outputting a first classification vocabulary, selecting words from the first classification vocabulary to obtain a selected word vocabulary, and then executing step S5; Step S4, using the input text, the first dictionary and the word selection word list as inputs of a second entity recognition algorithm, outputting a second classification word list, selecting words for the second classification word list to obtain a word selection word list, and then executing step S5; Step S5: if the word list covers the core words, output the named entity list; otherwise, execute step S4. The first entity recognition algorithm comprises: Using the maximum expectation algorithm to filter the first dictionary to obtain a second dictionary; using a log-likelihood ratio statistic to calculate the statistical significance of the words in the second dictionary to obtain a significance score for the words in the second dictionary, and sorting the words in the second dictionary according to the significance score, The step of using the maximum expectation algorithm to filter the first dictionary to obtain the second dictionary includes: (a) Selection of initial value: For the substring W in the first dictionary i , whose parameter θ i The initial value is set to in, I(·) is the indicator function, Among them, the integer value refers to the string T j Neutron string W i The number of occurrences, L j It is T j The length of the string T j The tth character in k i It is W i Length, T j [a,b] is T j The substring from the character at position a to the character at position b in ; Wherein, the first dictionary D={W1,W2,…,W N ,τ} includes N words and a special termination symbol τ; Word parameter θ={θ1,θ2,…θ N ,θ τ } gives the usage probability of each element in the first dictionary D, where 0≤θ i <1 for i=1,2,...,N,0<θ τ <1, and (D,θ) is called the first dictionary model; sentence is a sequence of k words, where the symbol "|" represents a word separator; Unsegmented text It is the concatenation of k words. The unsegmented text does not contain word separators. The text T = {T1, T2, ..., T n } is a set of n unsegmented texts; The following basic assumptions are adopted: It is generated by performing k independent extractions with replacement on the words in the first dictionary D, and each time a word W is extracted i The probability is θ i , until the terminal symbol τ is drawn, and the sentence is generated The likelihood function is: Generate unsegmented text T j The likelihood function is: in, Indicates that under the first dictionary D, the unsegmented text T j All possible participles of (b) Let θ (r) For the parameter estimates obtained in the rth iteration of the maximum expectation algorithm, calculate the following statistics: Among them, P(S|T j ; D, θ) means that under the first dictionary model (D, θ), T j The conditional probability generated by the word segmentation method S is defined as follows: n i (S) means word W in sentence S i Number of occurrences; (c) Update the estimate of parameter θ according to the following formula (d) Repeat steps b and c until θ (r) and θ (r+1) The Euclidean distance d(θ (r) ,θ (r+1) ) is less than a predetermined threshold; (e) forming the second dictionary from all words in the first dictionary D whose parameter θ is not zero, The second entity recognition algorithm comprises: Step S21, constructing a compound dictionary model, the compound dictionary model includes a dictionary of starting words of the input text Starting super-schema dictionary and the starting parameter system (γ,θ w∣c ), The starting word dictionary in, i, represents a positive integer; w n It is composed of Words composed of Chinese characters, The starting super-mode dictionary Expressed as Among them, e represents the terminator, p m Indicated by the pictogram t j The super-pattern composed of the pictogram Wherein, j represents a positive integer; The word category set in the input text where c k represents the category of the named entity, c0 represents the category of the background word that is different from the named entity, The starting parameter system (γ, θ w∣c ),in γ represents the sampling probability distribution of the super mode, where γ p represents the sampling probability of super mode p; represents a set of sampling probability distributions, where represents the sampling probability distribution of words given word category c, θ w∣c represents the conditional sampling probability of word w given word category c; Step S22, based on the initial parameter system (γ, θ w∣c ) Calculate the updated parameter system η=(γ,θ w ,θ c∣w ); Set the updated parameter system (γ, θ w ,θ c∣w ), Wherein, the starting parameter system (γ, θ w∣c ) calculates the updated parameter system (γ,θ w ,θ c∣w ) includes: for a word category and a super mode definition is the number of times word category c appears in super pattern p, and the frequency of word category c is in, The frequency of word w is Given a word w, the frequency of word category c is make Representing the distribution of words, let Represents the sampling probability distribution of a set of word categories, where represents the conditional distribution of word categories given word w, θ c∣w represents the sampling probability of word category c given word w; Wherein, the updated parameter system (γ, θ w ,θ c∣w ), and the updated parameter system (γ,θ w ,θ c∣w ) includes: performing Bayesian modeling on the composite dictionary model, setting the updated parameter system (γ,θ) according to the Dirichlet distribution Dir(α,F) w ,θ c∣w ), the expression of the Dirichlet distribution is Among them, α>0 represents intensity, F=(F1,…,F d ) represents the base distribution, and F is also the mode of the Dir(α,F) distribution. Represents an indicative function. When the conditions in "{}" are met, The value is 1, otherwise The value is 0, in the updated parameter system (γ,θ w ,θ c∣w ), the prior distribution of the parameter is set to γ~Dir(α γ ,π γ ), θ w ~Dir(α w ,π w ), θ ∣w ~Dir(α ∣w ,π ∣w ), according to the definition of Dir(α,F), using (π γ ,π w ,π ∣w ) represents the updated parameter system parameters (γ,θ w ,θ c∣w )’s prior propensity value, (α γ ,α w ,α ·∣w ) represents the strength of prior information, let π w For uniform distribution, π ·∣w Given by the word prior classification distribution, the word prior classification distribution is predefined, according to the constraints of the updated parameter system set up Set the intensity hyperparameter to (α γ ,α w ,α ·∣w )=(αN,αN,α), get (γ,θ w ,θ c∣w ) set up is the B unsegmented text fragments in the input text, then the updated parameter system (γ,θ w ,θ c∣w ) is: Step S23, based on the compound dictionary model, obtaining an unsegmented text segment T of the input text, and determining a hidden state H of the unsegmented text segment T, including: Assuming missing data Among them, the collection is the B unsegmented text segments T in the input text b A set of , b and B represent positive integers and 1≤b≤B; Representing a collection The unsegmented text fragment T in b The hidden state of Starting from the super-schema dictionary Extract the super pattern p from the super pattern sequence P to obtain the super pattern sequence P; extract the word w according to the super pattern sequence P to obtain the word sequence W i and the word sequence W i A sequence W, where Based on the sequence W, an unsegmented text segment T is obtained, and a hidden state H of the unsegmented text segment T is determined; Step S24, performing parameter inference on the updated parameter system η, comprising: The posterior distribution of the updated parameter system η is maximized using the maximum expectation algorithm, the maximum expectation algorithm includes an E step and an M step, and the maximum expectation algorithm converts the updated parameter system η = (γ, θ w ,θ c∣w )’s starting value η (0) As the starting value of the first iteration, through the iterative convergence in the E step and the M step, repeat this process until the point estimation result of the updated parameter system η is obtained, where the starting value η (0) Obtained from the prior distribution of the updated parameter system η; Step S26, based on the result obtained by the parameter inference, obtain the input text segmentation result of the unsegmented text segment T, the input text segmentation result includes the optimal hidden state And based on the input text segmentation result, semantic understanding and relationship extraction are performed on the unsegmented text segment T.

2. The Chinese named entity recognition and classification method as described in claim 1, characterized in that: Measuring word W i The statistical significance of the log-likelihood ratio statistic is defined as follows: in, P(T|D, θ) represents the likelihood function of the first dictionary model (D, θ) fitting the input text T.

3. The Chinese named entity recognition and classification method as described in claim 1, characterized in that: The first dictionary D and its parameters θ, for a given unsegmented text T∈T, implements word segmentation by calculating and analyzing the conditional probabilities of different word segmentation methods of T; assuming is all possible word segmentation methods of T under the first dictionary D. According to the basic assumption, under the condition of given T, the word segmentation method The conditional probability is: According to the maximum likelihood principle, the maximum likelihood estimate of the unknown segmentation method is the segmentation method that maximizes P(S|T; D,θ), that is, 4. The Chinese named entity recognition and classification method as described in claim 3, characterized in that: Calculate the conditional probability that a word separator appears after each character of the unsegmented text T, and perform word segmentation based on the conditional probability; if the word segmentation method S has a word separator after the tth character of T, let the indicative function I t (S) = 1, otherwise, let the indicative function I t (S) = 0; therefore, the conditional probability of a separator appearing after the tth character is 5. The Chinese named entity recognition and classification method as described in claim 1, characterized in that: The step E in step S24 includes: calculating the Q function: where η old represents the parameters obtained after the previous iteration, represents the number of times the super pattern p appears in the hidden state H, represents the set of all hidden states H of the unsegmented text segment T, Indicates the number of times the category word pair (c, w) appears in the hidden state H; And, using the framework of the Baum-Welch algorithm, dynamic programming is used to solve n p (T) and n cw (t) Solve, including: in, Indicates that T is not considered when the terminal character e is not considered [＜l] The observed likelihood of Indicates that T is not considered when the terminal character e is not considered [>r] The observed likelihood of Recursive calculation using forward-backward algorithm and The M steps include: setting constraints θ c (γ) is a fixed value; based on the constraints, calculate The maximum value of .

6. The Chinese named entity recognition and classification method as claimed in claim 1, It is characterized in that in, Get the starting word dictionary include: The input text is segmented character by character to obtain a character set consisting of all characters in the input text Where 1≤l≤L; According to the prior information of the words in the input text, the word set is set The prior information of the words in the input text is a known word library containing the words in the input text, Obtain the word set according to the first rule The first rule includes: setting is the maximum length threshold of all characters in a word, is the minimum frequency threshold of the word occurrence, and the length of the significant words in the list does not exceed and the frequency is not less than the words, and obtain the word set According to the second rule, we can get the word set The second rule includes: using a pre-trained weak classifier to classify the input text that is not longer than and the frequency is not less than the Extract words; determine the prior classification probability π c∣w , where c represents the word category in the input text, w represents the words extracted by the weak classifier, and π c∣w Indicates the prior classification probability of word category c for a given word w, and selects the words extracted by the weak classifier, the prior classification probability π c∣w Words that exceed the preset classification threshold, obtain the word set Integrate the character set Word collection Word collection Word collection Get the starting word dictionary Wherein, the starting super-mode dictionary The build includes: According to the prior information of the super pattern in the input text, set the super pattern set The prior information of the super pattern in the input text is a known super pattern library, and the known super pattern library contains all the super patterns in the input text; Obtain the super-pattern set according to the third rule The preset third rule includes: based on the starting word dictionary Identify and segment the words in the input text, and use different tags to distinguish the words in the input text that meet the confidence level. , where is a pre-set confidence threshold; set is the maximum length threshold of the glyphs in the super pattern, is the minimum frequency threshold of the super pattern, and sequentially scans and extracts the input text whose length does not exceed and the frequency is not less than the The adjacent labeled named entities are integrated with the super-patterns extracted in the process to obtain the super-pattern set Integrate the terminator e and word category set Hypermode Collection Hypermode Collection Get the starting super-pattern dictionary The starting hypermode dictionary 7. A computer device, characterized in that: The computer device includes a memory and at least one processor, wherein the memory stores at least one instruction, and when the at least one instruction is executed by the at least one processor, the Chinese named entity recognition and classification method as claimed in any one of claims 1 to 6 is implemented.

Images