A method and device for mining and displaying representations of domain information in short texts

Abstract

The present invention discloses a method and device for mining and displaying domain information in short texts. The method includes: dividing the encyclopedic knowledge into domains, collecting domain keywords for each domain according to the division result and web crawlers, constructing a domain dictionary, and performing deduplication processing on each domain dictionary; calculating and annotating the dictionary frequency of each keyword in the domain dictionary; segmenting the short text to be processed, and removing the stop words in the segmentation result according to the stop word list to obtain the feature words to be calculated; regarding each feature word as a local part, calculating the local domain contribution degree according to its dictionary frequency value; combining the local domain contribution degrees of each feature word, comprehensively obtaining the domain score of the entire short text, and performing sorting and screening on the alignment, and returning the domain corresponding to the short text. The present invention can improve the overall performance of the task model based on the deep learning framework without modifying the task model.

Description

Technical Field

[0001] The present invention relates to the technical field of natural language processing, and in particular, to a method and device for mining and displaying domain information in short texts. Background Art

[0002] In tasks centered around short texts such as intelligent question answering, due to the lack of information in short text questions, most research focuses on the model, aiming to obtain as much information as possible from the context around the mention, the relationships around the entity, and the entity, and then use this information to be attributed as features to better represent the mention and the entity, and use this high-quality representation to process various downstream tasks. For the exploration and selection of features, at the current stage, it can be divided into two main methods. One is centered around the model, that is, by changing the structure, parameters, etc. of the model to improve the performance of the model, and further enhance the ability of the model to discover and mine features in the original data. Most of this method focuses on specific datasets, and continuously achieves better and better results by improving the performance of the model. However, this method often has poor transferability and may have bad effects when applied to different datasets. The other method is centered around data. This method draws on feature engineering in machine learning and places the selection and extraction of features in the preprocessing stage, that is, a method of using relevant knowledge in the data domain to create features that can make the model reach the best performance. Most of the work in this area does not modify the model, but focuses on the data, and improves the performance of the model through high-quality data, and further enhances the overall effect of the task.

[0003] Recently, various super-large models starting with the BERT large-scale pre-trained model, from the hundreds of billions of parameters of GPT-3 to the trillions of parameters of Google switch transformer, the model size is getting larger and larger, but there are still problems such as low-level errors in real data and various biases exposed in fairness. In addition, through the training of a large amount of data, the deep neural network model captures features by itself. This method of feature capture is blind and has poor effects on different datasets. These also reflect that the improvement of the model has gradually reached a bottleneck. Moreover, the method centered around the model has also tended to mature, and many current models are already very standard, and a mature model and parameters can be called with just a few lines of code. Summary of the Invention

[0004] In view of the various biases and other problems existing in the existing method of text information mining based on changing the model architecture in the task of intelligent question answering centered on short texts, the present invention proposes a method and device for mining and displaying the representation of domain information in short texts. By means of feature engineering, the data is converted into features that better represent potential problems, and changes are made centered on the data, so as to improve the overall performance of the model without changing the task model based on the deep learning framework.

[0005] To achieve the above object, the present invention adopts the following technical solutions:

[0006] On the one hand, the present invention proposes a method for mining and displaying the representation of domain information in short texts, including:

[0007] Step 1: Divide the encyclopedic knowledge into domains, collect domain keywords for each domain according to the division result and the web crawler, construct a domain dictionary, and perform duplicate removal processing on each domain dictionary;

[0008] Step 2: Calculate and label the dictionary frequency of each keyword in the domain dictionary, where the dictionary frequency refers to the number of dictionaries corresponding to each keyword;

[0009] Step 3: Segment the short text to be processed, and remove the stop words in the segmentation result according to the stop word list to obtain the feature words to be calculated;

[0010] Step 4: Treat each feature word as a local part, and calculate the local domain contribution degree according to its dictionary frequency value;

[0011] Step 5: Combine the local domain contribution degrees of each feature word, comprehensively obtain the domain score of the entire short text, and perform sorting and screening on it, and return the domain corresponding to the short text.

[0012] Further, the step 4 includes:

[0013] Treat each feature word as a local part, and model the local domain contribution degree P(B i |a j ) according to the prior information and sample information of the Naive Bayes:

[0014]

[0015] where P(a j ) is a constant representing the marginal probability, so there is:

[0016]

[0017] where P(B i) is the prior probability, representing the probability of randomly selecting a field, which is calculated using formula (3):

[0018]

[0019] where Count(B i ) represents the number of feature words related to field B i in the text;

[0020] Finally, use C(a j , B i ) to replace P(a j | B i ) in formula (1). C(a j , B i ) represents the contribution degree of feature word a j to field B i , and C(a

[0021]

[0022] where df(a j ) represents the lexical frequency of feature word a j ;

[0023] Furthermore, step 5 includes:

[0024] Calculating the popularity of each feature word based on the lexical frequency of the feature word:

[0025]

[0026] Based on and the local field contribution degree of the feature word, calculate the score of each field, and sort based on the scores of each field to obtain the following overall modeling:

[0027]

[0028]

[0029] where represents the score of field B i ; Select() is a method used to screen fields; P(B i | a j ) is the local field contribution degree of feature word a j to field B i ; α is a constant used for smoothing; n represents the number of feature words in the short text; σ is the standard deviation used to measure the sorted scores ; x i represents the i-th sample, that is μ represents all of the average value; N represents the number of;

[0030] Finally, according to the overall modeling, the local contribution degrees of each feature word are integrated and selected, and finally the domain of F(A) is returned.

[0031] On the other hand, the present invention proposes a device for mining and displaying the domain information in short texts, including:

[0032] A domain dictionary construction module, which is used to divide the encyclopedia knowledge into domains, collect domain keywords for each domain according to the division result and the web crawler, construct a domain dictionary, and perform duplicate removal processing on each domain dictionary;

[0033] A dictionary frequency calculation module, which is used to calculate and label the dictionary frequency of each keyword in the domain dictionary, where the dictionary frequency refers to the number of dictionaries corresponding to each keyword;

[0034] A short text processing module, which is used to segment the short text to be processed, and remove the stop words in the segmentation result according to the stop word list to obtain the feature words to be calculated;

[0035] A local domain contribution degree calculation module, which is used to regard each feature word as a local, and calculate the local domain contribution degree according to its dictionary frequency value;

[0036] A domain derivation module, which is used to combine the local domain contribution degrees of each feature word, comprehensively obtain the domain score of the entire short text, and perform sorting and screening on it, and return the domain corresponding to the short text.

[0037] Further, the local domain contribution degree calculation module is specifically used for:

[0038] Regarding each feature word as a local, and modeling the local domain contribution degree P(B i |a j ) according to the prior information and sample information by Naive Bayes:

[0039]

[0040] where P(a j ) is a constant representing the marginal probability, so there is:

[0041]

[0042] where P(B i ) is the prior probability, representing the probability of randomly selecting a domain, and is calculated by formula (3):

[0043]

[0044] Among them, Count(B i ) represents the number of feature words related to field B in the text i ;

[0045] Finally, use C(a j , B i ) to replace P(a j |B i ) in formula (1). C(a j , B i ) represents the contribution degree of feature word a j to field B i , and C(a

[0046]

[0047] ) is calculated using formula (4): j Among them, df(a j ) represents the lexical frequency of feature word a

[0048] Furthermore, the field obtaining module is specifically configured to:

[0049] Calculate the popularity of each feature word based on the lexical frequency of each feature word:

[0050]

[0051] Based on and the local field contribution degree of this feature word, calculate the score of each field, and sort based on the scores of each field to obtain the following overall modeling:

[0052]

[0053]

[0054] Among them represents the score of field B i ; Select() is a method used to screen fields; P(B i |a j ) is the local field contribution degree of feature word a j to field B i ; α is a constant used for smoothing; n represents the number of feature words in the short text; σ is the standard deviation used to measure the sorted scores ; x i represents the i-th sample, that is μ represents the average value of all ; N represents the number of ;

[0055] Finally, based on the overall modeling, the local contribution degrees of each feature word are integrated and selected, and finally the domain of F(A) is returned.

[0056] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0057] Starting from the feature engineering in preprocessing, the present invention uses a lightweight and highly migratable method to explicitly display the domain information hidden in the data, so as to improve the overall performance without changing the task model architecture based on the deep learning framework. BRIEF DESCRIPTION OF THE DRAWINGS

[0058] Figure 1 FIG. is a basic flowchart of a method for mining and explicitly representing domain information in short texts according to an embodiment of the present invention;

[0059] Figure 2 FIG. is a structural schematic diagram of a device for mining and explicitly representing domain information in short texts according to an embodiment of the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0060] The following further explains the present invention with reference to the drawings and specific embodiments:

[0061] Starting from the feature engineering in preprocessing, the present invention uses a lightweight and highly migratable method to explicitly display the domain information hidden in the data, so as to improve the overall performance without changing the model architecture.

[0062] Taking the data in knowledge base question answering as an example, it is found in the research that a large amount of domain information is contained in sentence patterns and entity description information. For example, in the question "Who is the director of Game of Thrones?", the word "director" means that this question may be related to film and television works, and among the relationships within one hop around the entity "Game of Thrones", "release time", "producer", "screenwriter", etc. also mean that this entity may be related to film and television works. Therefore, in order to explicitly represent this domain information, a method for mining and explicitly representing domain information in short texts is proposed based on the Naive Bayes theorem.

[0063] As shown in FIG. 1, a method for mining and explicitly representing domain information in short texts mainly includes the following steps:

[0064] Step S101: Divide the encyclopedic knowledge into domains, collect domain keywords for each domain according to the division results and web crawlers, construct a domain dictionary, and perform duplicate removal processing on each domain dictionary;

[0065] Step S102: Calculate and label the dictionary frequency of each keyword in the domain dictionary, where the dictionary frequency refers to the number of dictionaries corresponding to each keyword;

[0066] Step S103: Segment the short text to be processed, and remove the stop words in the segmentation result according to the stop word list to obtain the feature words to be calculated;

[0067] Step S104: Treat each feature word as a local part, and calculate the domain contribution degree of the local part according to its lexical frequency value;

[0068] Step S105: Combine the local domain contribution degrees of each feature word, comprehensively obtain the domain score of the entire short text, and perform sorting and screening on it, and return the domain corresponding to the short text.

[0069] Further, the step S101 includes:

[0070] Divide the knowledge of encyclopedias into nine fields. Table 1 gives examples of these fields and their representative sub-fields. According to such a division standard, millions of domain keywords are obtained through web crawlers, and based on this, a domain dictionary is constructed and duplicate removal is performed on each domain dictionary.

[0071] Table 1

[0072]

[0073]

[0074] Further, the step S102 includes:

[0075] Introduce the lexical frequency df(a j ) to represent how many domain dictionaries contain the feature word a j . Calculate and annotate the lexical frequency of each word in the domain dictionary. For example, "Notre-Dame de Paris" is both the name of a game, the name of a landmark building, and the name of a movie. Therefore, it appears in the three domain dictionaries of sports, culture, and art. Therefore, its df(a j ) is 3.

[0076] Further, the step S103 includes:

[0077] Segment the short text to be processed, and remove the stop words in the segmentation result according to the stop word list to obtain the feature words to be calculated. For example, for the question sentence "What is the architectural style of Notre-Dame de Paris?", the feature words obtained after segmenting it and removing the stop words are two: "Notre-Dame de Paris" and "architectural style".

[0078] Further, the step S104 includes:

[0079] Let A = {a1, a2, …, a nDescribe the text composed of the feature words or entity nodes in the question and their surrounding relationships, using B i To represent different domains and obtain the local domain contribution P(B i |a j ), the present invention models the prior information and sample information based on Naive Bayes:

[0080]

[0081] Since P(a j ) is a constant and can be regarded as the marginal probability, we have:

[0082]

[0083] Among them, P(B i ) is used as the prior probability, representing the probability of randomly selecting a domain, and is calculated using formula (3).

[0084]

[0085] Among them, Count(B i ) represents the number of feature words related to domain B i in the text;

[0086] Finally, use C(a j , B i ) to replace P(a j |B i ) in formula (1). C(a j to domain B i 's contribution (Contribution), C(a j , B i ) is calculated using formula (4).

[0087]

[0088] Furthermore, the step S105 includes: ]>

[0089] To assign higher weights to common words, the present invention introduces popularity to model the usage frequency of feature words in daily life. Popularity can be calculated through formula (5):

[0090]

[0091] Combined with the feature word a in the text obtained in step S104 j 's local contribution to the text in domain B i P(B i|a j ), so the whole can be modeled as:

[0092]

[0093] where represents the score of each domain, and Select is a method used to screen the domains. Since the questions and entities in real life may belong to multiple domains simultaneously, the standard deviation σ is introduced to measure the sorted scores Specifically, the difference between the scores of the top two domains and can be calculated first If s1 > σ, only return domain B1 as the result of F(A); if s1 < σ, return domains B1 and B2 as the result of F(A), and then compare and And so on to obtain the final result of F(A). α is a constant, which is a smoothing process to avoid the zero-probability problem caused by all feature words not matching successfully in the dictionary.

[0094]

[0095] where x i represents the i-th sample, μ represents the average value of all samples, and N represents the number of samples. In this embodiment, x i represents the score of each i-th domain while μ represents the average value of all and N represents the number of

[0096] Finally, based on the overall modeling, the local contribution degrees of each feature word are integrated and selected, and finally the domain of F(A) is returned.

[0097] As an implementable manner, further, after the step S105, it further includes:

[0098] Step S106: Use the selected domain information as the display representation of external information, splice it with the entity or question to form a new data set, and use this new data set to lead the existing deep neural network to learn, so as to achieve the purpose of improving the performance of downstream tasks without changing the architecture of the deep neural network model.

[0099] To verify the effectiveness of the present invention, the similarity task is converted into a binary classification task for training and evaluation, and the BERT model trained on a large-scale Chinese corpus is fine-tuned using the original data and the data processed by the method of this patent respectively. The experiment uses accuracy and F1 value as evaluation indicators. The experimental results are shown in Table 2. The experimental results show that the method of the present invention is effective, and there are significant improvements in both accuracy and F1 value.

[0100] Table 2

[0101]

[0102] Based on the above embodiments, as Figure 2 shown, on the other hand, the present invention proposes a device for mining and displaying the domain information in short texts, including:

[0103] A domain dictionary construction module, which is used to divide the encyclopedic knowledge into domains, collect domain keywords for each domain according to the division results and web crawlers, construct a domain dictionary, and perform duplicate removal processing on each domain dictionary;

[0104] A dictionary frequency calculation module, which is used to calculate and label the dictionary frequency of each keyword in the domain dictionary, where the dictionary frequency refers to the number of dictionaries corresponding to each keyword;

[0105] A short text processing module, which is used to segment the short text to be processed and remove the stop words in the segmentation results according to the stop word list to obtain the feature words to be calculated;

[0106] A local domain contribution calculation module, which is used to regard each feature word as a local, and calculate the local domain contribution according to its dictionary frequency value;

[0107] A domain derivation module, which is used to combine the local domain contributions of each feature word, comprehensively obtain the domain score of the entire short text, and perform sorting and screening on it, and return the domain corresponding to the short text.

[0108] Further, the local domain contribution calculation module is specifically used for:

[0109] Regarding each feature word as a local, and modeling the local domain contribution P(B i |a j ) according to the prior information and sample information of Naive Bayes:

[0110]

[0111] where P(a j ) is a constant representing the marginal probability, so there is:

[0112]

[0113] Among them, P(B i ) is the prior probability, representing the probability of randomly selecting a field, and is calculated using formula (3):

[0114]

[0115] Among them, Count(B i ) represents the number of feature words related to field B i in the text;

[0116] Finally, use C(a j , B i ) to replace P(a j |B i ) in formula (1), indicating the contribution degree of feature word a j to field B i , and C(a j , B i ) is calculated using formula (4):

[0117]

[0118] Among them, df(a j ) represents the dictionary frequency of feature word a j .

[0119] Furthermore, the field obtaining module is specifically used for:

[0120] Calculating the popularity of each feature word based on the dictionary frequency of each feature word:

[0121]

[0122] Based on and the local field contribution degree of this feature word, calculating the score of each field, and sorting based on the scores of each field to obtain the following overall modeling:

[0123]

[0124]

[0125] Among them represents the score of field B i ; Select() is a method used to screen fields; P(B i |a j ) is the contribution degree of feature word a j to field B iThe contribution degree of the local domain; α is a constant used for smoothing; n represents the number of feature words in the short text; σ is the standard deviation used to measure the sorted scores for measurement; x i represents the i-th sample, that is μ represents all the average value; N represents the number of;

[0126] Finally, based on the overall modeling, the local contribution degrees of each feature word are integrated and selected, and finally the domain of F(A) is returned.

[0127] In summary, starting from the feature engineering in the preprocessing, the present invention uses a lightweight and highly migratable method to display the implicit domain information in the data, so as to improve the overall performance without changing the task model architecture based on the deep learning framework.

[0128] The above are only the preferred embodiments of the present invention. It should be noted that for those of ordinary skill in the art, without departing from the principle of the present invention, several improvements and refinements can be made, and these improvements and refinements should also be regarded as the protection scope of the present invention.

Claims

1. A method for mining and displaying the domain information in short texts, characterized in that, Including: Step 1: Divide the encyclopedic knowledge into domains, collect domain keywords for each domain based on the division results and web crawlers, construct a domain dictionary, and perform deduplication processing on each domain dictionary; Step 2: Calculate and annotate the dictionary frequency of each keyword in the domain dictionary, where the dictionary frequency refers to the number of dictionaries corresponding to each keyword; Step 3: Segment the short text to be processed, and remove the stop words in the segmentation result according to the stop word list to obtain the feature words to be calculated; Step 4: Treat each feature word as a local part, and calculate the local domain contribution degree according to its dictionary frequency value; Step 5: Combine the local domain contribution degrees of each feature word, comprehensively obtain the domain score of the entire short text, and perform sorting and screening on it, and return the domain corresponding to the short text; The said Step 4 includes: Regarding each feature word as a local part, a model is built based on the prior information and sample information of Naive Bayes for the contribution degree P(B i |a j ) of the local part to the domain: where P(a j ) is a constant representing the marginal probability, so we have: Among them, P(B i ) is the prior probability, representing the probability of randomly selecting a field, which is calculated using formula (3): Among them, Count(B i ) represents the number of characteristic words related to field B i in the text; Finally, use C(a j , B i ) to replace P(a j | B i ) in formula (1). Here, C(a j to domain B i represents the contribution degree of feature word a j , B i ), and C(a j , B i ) is calculated using formula (4): Among them, df(a j ) represents the dictionary frequency of the feature word a j .

2. A method for mining and displaying representations of domain information in short texts according to claim 1, characterized in that The said Step 5 includes: Calculate the popularity of each feature word based on its dictionary frequency: Based on and the local domain contribution degree of the feature word, calculate the scores of each domain, and sort based on the scores of each domain to obtain the following overall modeling: Among them represents the score of field B i ; Select( ) is a method used to screen fields; P(B i |a j ) is the local field contribution degree of the feature word a j to field B i ; α is a constant used for smoothing; n represents the number of feature words in the short text; σ is the standard deviation used to measure the sorted scores ; x i represents the i-th sample, that is μ represents all averages; N represents the number of Finally, integrate and select the local contribution degrees of each feature word according to the overall modeling, and finally return the domain of F(A).

3. An apparatus for mining and displaying the domain information in short texts, characterized in that, Including: Domain dictionary construction module, used to divide the encyclopedic knowledge into domains, collect domain keywords for each domain based on the division results and web crawlers, construct a domain dictionary, and perform deduplication processing on each domain dictionary; Dictionary frequency calculation module, used to calculate and annotate the dictionary frequency of each keyword in the domain dictionary, where the dictionary frequency refers to the number of dictionaries corresponding to each keyword; Short text processing module, used to segment the short text to be processed, and remove the stop words in the segmentation result according to the stop word list to obtain the feature words to be calculated; Local domain contribution degree calculation module, used to treat each feature word as a local part, and calculate the local domain contribution degree according to its dictionary frequency value; Domain obtaining module, used to combine the local domain contribution degrees of each feature word, comprehensively obtain the domain score of the entire short text, and perform sorting and screening on it, and return the domain corresponding to the short text; The said local domain contribution degree calculation module is specifically used for: Regarding each feature word as a local part, a model is built based on the prior information and sample information of Naive Bayes for the contribution degree P(B i |a j ) of the local part to the domain: where P(a j ) is a constant representing the marginal probability, so we have: where P(B i ) is the prior probability, representing the probability of randomly selecting a field, and is calculated using formula (3): Among them, Count(B i ) represents the number of characteristic words related to field B i in the text; Finally, replace P(a j ,B i ) in formula (1) with C(a j |B i ), where C(a j |B i ) represents the contribution degree of the feature word a j to the domain B i . C(a j ,B i ) is calculated using formula (4): Among them, df(a j ) represents the dictionary frequency of the feature word a j .

4. A device for mining and displaying the representation of domain information in short texts according to claim 3, characterized in that, The said domain obtaining module is specifically used for: Calculate the popularity of each feature word based on its dictionary frequency: Based on and the local domain contribution degree of the feature word, calculate the scores of each domain, and sort based on the scores of each domain to obtain the following overall modeling: Among them represents the score of domain B i ; Select( ) is a method used to screen domains; P(B i |a j ) is the local domain contribution degree of the feature word a j to domain B i ; α is a constant used for smoothing; n represents the number of feature words in the short text; σ is the standard deviation used to measure the sorted scores ; x i represents the i-th sample, that is μ represents all the average value; N represents the number of Finally, integrate and select the local contribution degrees of each feature word according to the overall modeling, and finally return the domain of F(A).

Images