Domain entity and associated item mining method and system

Abstract

The invention relates to a mining method and system for domain entities and associated items of the domain entities, belongs to the technical field of data mining, and solves the problems of high dependency on annotated data and low accuracy of mined entities in the prior art. Comprising the steps of obtaining corpora of a target field, preprocessing the corpora to obtain a field text set, and mining entities and associated items thereof from the field text set; each entity and each associated item form a retrieval keyword, a search result of a search engine is captured by utilizing a web crawler, and a plurality of search texts corresponding to each retrieval keyword are extracted from the search result; calculating the degree of association between each retrieval keyword and the corresponding plurality of search texts, and the degree of aggregation among the plurality of search texts; calculating the confidence coefficient according to the weights of the entities and the associated items in each retrieval keyword, the association degree and the aggregation degree; and obtaining a field entity set and a field associated item set according to the entities and the associated items of the retrieval keywords of which the confidence is greater than a confidence threshold. The efficient and accurate mining of the entities and the associated items thereof is realized.

Description

Technical Field

[0001] The present invention relates to the technical field of data mining, and in particular to a method and system for mining domain entities and related items thereof. Background Art

[0002] With the development of the Internet, the amount of data generated is increasing, and the traditional industries involved are becoming more and more extensive. How to establish automated business knowledge mining for each traditional industry is more important for the next step of building a domain knowledge graph and establishing a system to serve the traditional field.

[0003] At present, for domain business item knowledge mining, data collection, text clustering, unsupervised key phrase mining are usually used to build domain relevance models based on domain data. Common practices include word frequency part-of-speech mining, dependency syntactic analysis mining, CRF tagging and named entity recognition.

[0004] However, for business item mining in certain small corpora or complex and confusing fields, traditional methods, on the one hand, require a large amount of labeled data, require different degrees of professional knowledge, consume a lot of manpower and material resources, and lack flexibility and cannot adapt well to changes in user needs; they do not utilize unlabeled big data, resulting in a small scope of data analysis and incomplete identification of domain entities; on the other hand, they do not further measure the correlation between the extracted business entities and the domain, resulting in the inability to characterize the intrinsic information of the data, and the extracted entities are not strongly correlated and inappropriate. Summary of the invention

[0005] In view of the above analysis, an embodiment of the present invention aims to provide a method and system for mining domain entities and their related items, so as to solve the existing problems of strong dependence on labeled data and low accuracy of mined entities.

[0006] On the one hand, an embodiment of the present invention provides a method for mining domain entities and their associated items, comprising the following steps:

[0007] Obtain the corpus of the target domain, obtain the domain text set after preprocessing, mine entities and their associated items from the domain text set, and put them into the entity candidate set and the associated item candidate set;

[0008] Based on the entity candidate set and the associated item candidate set, each entity and each of its associated items are combined into search keywords, and the search results of the search engine are captured by a web crawler; multiple search texts corresponding to each search keyword are extracted from the search results;

[0009] Calculate the relevance between each search keyword and the corresponding multiple search texts, as well as the aggregation between the multiple search texts; calculate the confidence of each search keyword based on the weights of the entities and associated items in each search keyword, as well as the corresponding relevance and aggregation;

[0010] A domain entity set and a domain associated item set are obtained according to entities and associated items of search keywords whose confidence is greater than a confidence threshold.

[0011] Based on the further improvement of the above method, the correlation between each search keyword and the corresponding multiple search texts is calculated, including: obtaining a sentence set of each text in the domain text set, putting the sentence containing each search keyword in the domain text set into a first set, putting the multiple search texts corresponding to each search keyword into a second set, and calculating the similarity between the first set and the second set as the correlation.

[0012] Based on a further improvement of the above method, the similarity between the first set and the second set is calculated by using the LDA model to obtain the first topic vector of the first set and the second topic vector of the second set, and then calculating the cosine similarity between the first topic vector and the second topic vector.

[0013] Based on a further improvement of the above method, the similarity between the first set and the second set is calculated by using the word2vec model, as well as the word frequency of the first set and the word frequency of the second set, to obtain the first sentence vector matrix of the first set and the second sentence vector matrix of the second set respectively, and calculate the maximum cosine similarity between the first sentence vector matrix and the second sentence vector matrix.

[0014] Based on the further improvement of the above method, the first sentence vector matrix of the first set and the second sentence vector matrix of the second set are obtained by the following steps:

[0015] Get the vocabulary in the first set and the second set, and use the trained word2vec model to get the word vector of each word;

[0016] According to the word frequency of each word in the first set and the second set, the word vectors in each sentence in the first set and the second set are weighted averaged to obtain the initial sentence vector of each sentence;

[0017] The initial sentence vector of each sentence is used as a column vector to construct the initial sentence matrices of the first set and the second set, and the first principal components of the initial sentence matrices of the first set and the second set are obtained respectively;

[0018] After removing the projection vector of the corresponding first principal component from each initial sentence vector in the initial sentence matrices of the first set and the second set, a first sentence vector matrix and a second sentence vector matrix are obtained.

[0019] Based on a further improvement of the above method, the aggregation degree between multiple search texts is obtained by calculating the inverse of the variance of the second sentence vector matrix.

[0020] Based on the further improvement of the above method, the weights of the entities and related items in each search keyword are obtained by filling multiple search texts corresponding to each search keyword into the domain text set, and calculating the TF-IDF value of each entity and related item in the search keyword in the filled domain text set to obtain their respective weights.

[0021] Based on the further improvement of the above method, the confidence of each search keyword is obtained by normalizing the weights of the entities and related items in each search keyword, as well as the corresponding relevance and aggregation degree and then multiplying them respectively.

[0022] Based on the further improvement of the above method, entities and their associated items are mined from the domain text set and put into the entity candidate set and the associated item candidate set, including:

[0023] Obtain a sentence set of each text in the domain text set, and perform word segmentation and part-of-speech tagging on each sentence in the sentence set;

[0024] Using unsupervised knowledge mining algorithm to obtain keywords in the sentence set, taking keywords with noun as the main words, and obtaining other main words in a window with each main word in the sentence set by sliding preset windows as the associated items of the current main word;

[0025] After deleting the main words without associated items, the remaining main words are taken as entities, and each entity and its associated items are filtered and put into the entity candidate set and the associated item candidate set.

[0026] On the other hand, an embodiment of the present invention provides a system for mining domain entities and their associated items, including:

[0027] The candidate set acquisition module is used to acquire the corpus of the target domain, obtain the domain text set after preprocessing, mine entities and their associated items from the domain text set, and put them into the entity candidate set and the associated item candidate set;

[0028] The search text extraction module is used to form search keywords for each entity and each of its associated items based on the entity candidate set and the associated item candidate set, and use a web crawler to crawl the search results of the search engine; extract multiple search texts corresponding to each search keyword from the search results;

[0029] The confidence calculation module is used to calculate the relevance between each search keyword and the corresponding multiple search texts, as well as the aggregation between the multiple search texts; the confidence of each search keyword is calculated according to the weights of the entities and associated items in each search keyword, as well as the corresponding relevance and aggregation;

[0030] The domain set acquisition module is used to obtain a domain entity set and a domain associated item set according to entities and associated items of search keywords whose confidence is greater than a confidence threshold.

[0031] Compared with the prior art, the present invention can achieve at least one of the following beneficial effects: using web crawler technology to mine rich information on the Internet, covering a wide range of and the latest information sources, without the need for a large amount of manual annotation, thus saving a large amount of labor costs; by analyzing the correlation between retrieval keywords and search texts, the semantic expression ability of entities is enhanced, so that the mined entities can better reflect the user's search intent; by calculating the aggregation degree between search texts, the correlation between entities and fields is reflected; finally, the confidence level is calculated by integrating multi-dimensional information such as correlation, aggregation and weight, so as to more accurately evaluate the importance of entities and their relevance to fields, thereby improving the accuracy of entity mining.

[0032] In the present invention, the above-mentioned technical solutions can also be combined with each other to achieve more preferred combination solutions. Other features and advantages of the present invention will be described in the subsequent description, and some advantages can become obvious from the description, or can be understood by practicing the present invention. The purpose and other advantages of the present invention can be realized and obtained through the contents particularly pointed out in the description and the drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] The accompanying drawings are only used for the purpose of illustrating specific embodiments and are not to be considered as limiting the present invention. In the entire drawings, the same reference symbols represent the same components;

[0034] Figure 1 This is a flow chart of a method for mining domain entities and their associated items in Embodiment 1 of the present invention;

[0035] Figure 2 This is a schematic diagram of the structure of a system for mining domain entities and their associated items in Example 2 of the present invention. DETAILED DESCRIPTION

[0036] The preferred embodiments of the present invention are described in detail below in conjunction with the accompanying drawings, wherein the accompanying drawings constitute a part of this application and are used together with the embodiments of the present invention to illustrate the principles of the present invention, but are not used to limit the scope of the present invention.

[0037] Example 1

[0038] A specific embodiment of the present invention discloses a method for mining domain entities and their associated items, such as Figure 1 As shown, the following steps are included:

[0039] S1. Obtain the corpus of the target domain, obtain the domain text set after preprocessing, mine entities and their associated items from the domain text set, and put them into the entity candidate set and the associated item candidate set.

[0040] It should be noted that the target domain can be one or more, and the corpus includes text corpus, audio corpus and video corpus. The audio corpus and video corpus are extracted using speech recognition technology to obtain text, which is then used together with the text corpus as domain text.

[0041] Mining entities and their associated items from domain text sets, including:

[0042] ① Obtain the sentence set of each text in the domain text set, and perform word segmentation and part-of-speech tagging on each sentence in the sentence set.

[0043] It should be noted that, considering that the domain dictionary contains the commonly used professional terms in a specific field, the domain dictionary and domain word frequency are used to improve the accuracy of word segmentation during word segmentation. Part-of-speech tagging refers to assigning a part-of-speech label to each word after word segmentation, such as noun, verb, adjective, etc.

[0044] ② Use unsupervised knowledge mining algorithm to obtain keywords in the sentence set, take keywords with noun as the main words, and obtain other main words in a window with each main word in the sentence set by sliding the preset window as the associated items of the current main word.

[0045] Exemplarily, the unsupervised knowledge mining algorithm adopts the TF-IDF (Term Frequency-Inverse Document Frequency) method or the TextRank algorithm.

[0046] Preferably, a feature representation of each sentence in the sentence set is obtained, such as a sentence vector or an N-gram feature, and a clustering algorithm, such as an AP clustering algorithm or a K-means clustering algorithm, is used to cluster the sentence set into multiple classes; the key phrases in each class are obtained according to step ②, and the main words and their associated items of each class are obtained according to the part of speech of the key phrases and a preset window.

[0047] It should be noted that based on the extracted subject words, the WAM (Word Alignment Model) model is used to introduce the translation method, and the subject word and the sentence in which it is located are formed into a "subject word-text word" translation pair, and then the IBM-Model 1 model is used to train to obtain the "subject word-text word" translation probability table, which contains the translation probability of each text word becoming a subject word. Combining the translation probability table and the TF-IDF method, a comprehensive score is calculated for each text word, and all words are sorted according to the comprehensive score. The text words with scores exceeding the threshold and not subject words are selected as new keywords, and the new keywords with the part of speech of nouns are used as new subject words.

[0048] The WAM model takes into account the contextual information of words when constructing translation pairs, which helps to identify keywords that do not appear frequently in the text but are more important in the context.

[0049] ③ After deleting the main words without associated items, the remaining main words are taken as entities, and each entity and its associated items are filtered and put into the entity candidate set and the associated item candidate set.

[0050] It should be noted that filtering each entity and its associated items includes removing stop words, using a preset dictionary to remove entities that are not business-related, and using regular expressions to remove pure numeric items and special characters. The entities are sorted from large to small according to the TF-IDF value and put into the entity candidate set, and the associated items of each entity are put into the associated item candidate set, and an association relationship is established between the entity and the associated items.

[0051] S2. Based on the entity candidate set and the associated item candidate set, each entity and each of its associated items are respectively combined into search keywords, and the search results of the search engine are captured by a web crawler, and multiple search texts corresponding to each search keyword are extracted from the search results.

[0052] It should be noted that when crawling data using a web crawler, links are obtained from the queue of links to be crawled and distributed to various download threads. In order to avoid excessive consumption of resources, the number of threads is set in advance; in order to avoid wasting resources caused by long waiting times during crawling and affecting the performance of the crawler system, a timeout is set in advance, and web pages with too long waiting times are discarded, saving system resources and improving crawler performance. In addition, the access time interval for each site, the recursive crawling depth, etc. can also be set.

[0053] When crawling web pages, construct the search engine's query URL based on the search keywords, simulate browser access based on the query URL, send HTTP requests to the search engine, receive the search engine's search results, usually HTML pages, extract links from the HTML pages, use asynchronous HTTP requests to access the links, further crawl the web page content pointed to by the links, and store the source code files of the web pages locally.

[0054] Furthermore, the stored source code files of the webpages are analyzed offline. Considering that the source code files are in HTML format, an Xpath plug-in is used to extract corresponding text content from the source code files of the webpages.

[0055] Preferably, the extracted text content includes but is not limited to: web page title, web page metadata and web page main content. Among them, the web page title includes: <title>、< / title> <h1>,< / h1> <h2>,< / h2> <h3>Tags such as ; web page metadata includes: <meta> The content attribute content in the tag; the main content of the web page includes: Content attribute content in the tag, paragraph Tag content, list Tab content, table and tab content, etc.

[0056] For example, the XPath expression " / / title" means obtaining the text content of all title tags from the entire web page, and " / / div / text()" means obtaining the text content of all title tags from the source code file of the entire web page. The text content of the label.

[0057] After filtering each extracted text content, the search text corresponding to each search keyword is obtained, including: using regular expressions to remove HTML tags, URL links, special characters, non-Chinese characters and pure numbers in the text content; according to the preset minimum character threshold, removing text content less than the minimum character threshold; identifying whether the coverage of entities in the entity candidate set in the text content is greater than the minimum coverage threshold. If greater, the current text content is used as a search text for the retrieval keyword.

[0058] Preferably, a maximum number K of search texts is set, and the top K search texts are obtained according to the coverage of the search texts as the search texts corresponding to the search keywords.

[0059] S3. Calculate the association between each search keyword and the corresponding multiple search texts, as well as the aggregation between the multiple search texts; calculate the confidence of each search keyword based on the weights of the entities and related items in each search keyword, as well as the corresponding association and aggregation.

[0060] It should be noted that the sentences containing each search keyword in the domain text set are put into the first set, and the multiple search texts corresponding to each search keyword are put into the second set, and the similarity between the first set and the second set is calculated as the correlation degree.

[0061] Specifically, the similarity is calculated using any of the following methods.

[0062] (1) The LDA model is used to obtain the first topic vector of the first set and the second topic vector of the second set, and the cosine similarity between the first topic vector and the second topic vector is calculated.

[0063] It should be noted that the training process of the LDA (Latent Dirichlet Allocation) model is an unsupervised learning process. Before training the LDA model, the number of topics and the hyperparameters of the Dirichlet distribution are first set, and then a document-term matrix (DTM) is constructed based on the domain text set in step S1 as input to train the LDA model. In the document-term matrix, rows represent texts, columns represent words, and element values ​​represent the frequency of words in the texts. During the training process, algorithms such as Gibbs sampling or variational Bayesian inference are used to iteratively update the topic distribution of documents and the word distribution of topics until the model converges to obtain a trained LDA model.

[0064] For each text input, the LDA model outputs a topic distribution, which is a probability vector that indicates the degree of association between the text and each topic. Each element in the vector corresponds to a topic, and its value indicates the probability that the text belongs to that topic. For each discovered topic, the LDA model outputs a word distribution, which is also a probability vector that indicates the probability of occurrence of each word under that topic. Each element in the vector corresponds to a word in the vocabulary, and its value indicates the probability of the word appearing under a given topic.

[0065] In this step, each sentence in the first set is input into the trained LDA model in turn, and the first topic vector of each search keyword is obtained according to the output topic probability distribution of each sentence; each search text in the second set is input into the trained LDA model in turn, and the second topic vector corresponding to each search keyword is obtained according to the output topic probability distribution of each search text; the cosine similarity between the first topic vector and the second topic vector of each search keyword is calculated to obtain the association degree of each search keyword.

[0066] It should be noted that the first topic vector of each search keyword is obtained according to the topic probability distribution of each sentence output, which can be obtained by taking the average or by taking the weighted average according to the weight of the text in which the sentence is located; similarly, the second topic vector corresponding to each search keyword is obtained according to the topic probability distribution of each search text output, which can be obtained by taking the average or by taking the weighted average after assigning different weights according to the entity coverage in the search text.

[0067] (2) Using the word2vec model and the word frequencies of the first set and the second set, respectively obtain the first sentence vector matrix of the first set and the second sentence vector matrix of the second set, and calculate the maximum cosine similarity between the first sentence vector matrix and the second sentence vector matrix.

[0068] It should be noted that the word2vec model is first pre-trained using a large amount of Chinese Wikipedia, and then incrementally trained based on the corpus of the target field to improve the coverage and similarity of word vectors in the field.

[0069] Specifically, the sentence vector matrix is ​​obtained using the trained word2vec model and word frequency, including:

[0070] Get the vocabulary in the collection and use the trained word2vec model to get the word vector of each word;

[0071] According to the frequency of each word in the set, the word vectors in each sentence are weighted averaged to calculate the initial sentence vector of each sentence. The formula is as follows:

[0072]

[0073] in, represents the initial sentence vector of the i-th sentence, n i represents the number of words in the i-th sentence, p(w j ) represents the frequency of the jth word in the i-th sentence, represents the word vector of the jth word in the i-th sentence, and a represents the preset hyperparameter.

[0074] The initial sentence vector of each sentence is used as a column vector to construct the initial sentence matrix of the set. The eigenvector with the largest eigenvalue, i.e., the first principal component, is obtained by eigendecomposing the covariance matrix of the initial sentence matrix. The projection vector of the first principal component is removed from each initial sentence vector in turn according to the following formula to obtain the updated sentence vector and construct the final sentence vector matrix:

[0075]

[0076] in, represents the updated sentence vector of the i-th sentence, u represents the first principal component, u T Indicates transposing the first principal component.

[0077] It should be noted that, in this embodiment, the projection vector of the first principal component is subtracted from the initial sentence vector to update the sentence vector, so as to remove the main change direction in the sentence vector and reduce the influence of data deviation or noise.

[0078] The first sentence vector matrix of the first set and the second sentence vector matrix of the second set are obtained respectively according to the above method, the cosine similarity between any two vectors in the first sentence vector matrix and the second sentence vector matrix is ​​calculated, and the maximum cosine similarity is taken as the similarity of the two sentence vector matrices.

[0079] Furthermore, the variance of the second sentence vector matrix is ​​calculated, and the inverse of the variance is used as the degree of aggregation between multiple search texts. The smaller the variance, the smaller the difference between the sentence vectors of the search texts, and the higher the degree of aggregation.

[0080] It should be noted that multiple search texts corresponding to each search keyword are filled into the domain text set, and the TF-IDF value of each entity and associated item in the search keyword in the filled domain text set is calculated to obtain their respective weights.

[0081] Furthermore, the weights of the entities and related items in each search keyword, as well as the corresponding relevance and aggregation degree are normalized and then multiplied to obtain the weights.

[0082] S4. Obtain a domain entity set and a domain associated item set based on entities and associated items of the search keyword whose confidence is greater than a confidence threshold.

[0083] The calculated confidences are sorted from high to low, and search keywords with confidences greater than a confidence threshold are obtained. The entities therein are merged to obtain a domain entity set, and the associated items therein are merged to obtain a domain associated item set.

[0084] Preferably, the LDA model trained in step S3 is used to obtain the topic with the maximum probability corresponding to each entity in the domain entity set, as well as the vocabulary in the topic whose probability value is greater than the probability threshold; or the entity is linked to the corresponding entity in the knowledge base, which helps to provide more contextual information and semantic understanding.

[0085] The domain entity set and domain-related item set can be further applied in more scenarios, such as better understanding user questions in intelligent question-answering systems and providing accurate answers; helping to identify key information in texts in automatic abstract generation and generating more accurate and coherent abstracts, etc.

[0086] Compared with the prior art, the method for mining domain entities and their related items provided in this embodiment uses web crawler technology to mine rich information on the Internet, covering a wide range of and the latest information sources, does not require a large amount of manual annotation, and saves a lot of labor costs; by analyzing the correlation between retrieval keywords and search texts, the semantic expression ability of entities is enhanced, so that the mined entities can better reflect the user's search intent; by calculating the aggregation degree between search texts, the correlation between entities and domains is reflected; finally, the multi-dimensional information such as correlation, aggregation and weight is integrated to calculate the confidence, which more accurately evaluates the importance of entities and their relevance to the domain, thereby improving the accuracy of entity mining.

[0087] Example 2

[0088] Another embodiment of the present invention discloses a system for mining domain entities and their associated items, thereby realizing a method for mining domain entities and their associated items in embodiment 1. The specific implementation of each module refers to the corresponding description in embodiment 1. The system includes:

[0089] The candidate set acquisition module 101 is used to acquire the corpus of the target domain, obtain the domain text set after preprocessing, mine entities and their associated items from the domain text set, and put them into the entity candidate set and the associated item candidate set;

[0090] The search text extraction module 102 is used to form a search keyword for each entity and each of its associated items based on the entity candidate set and the associated item candidate set, and crawl the search results of the search engine using a web crawler; and extract multiple search texts corresponding to each search keyword from the search results;

[0091] The confidence calculation module 103 is used to calculate the association between each search keyword and the corresponding multiple search texts, and the aggregation between the multiple search texts; the confidence of each search keyword is calculated according to the weights of the entities and related items in each search keyword, and the corresponding association and aggregation;

[0092] The domain set acquisition module 105 is used to obtain a domain entity set and a domain associated item set according to entities and associated items of search keywords whose confidence is greater than a confidence threshold.

[0093] Since the present embodiment and the aforementioned method for mining domain entities and their associated items are related and can be mutually referenced, the description here is repeated and will not be repeated here. Since the present system embodiment and the aforementioned method embodiment have the same principle, the present system embodiment also has the corresponding technical effects of the aforementioned method embodiment.

[0094] Those skilled in the art will appreciate that all or part of the processes of the above-mentioned embodiments can be implemented by instructing related hardware through a computer program, and the program can be stored in a computer-readable storage medium, wherein the computer-readable storage medium is a disk, an optical disk, a read-only storage memory, or a random access memory, etc.

[0095] The above description is only a preferred specific implementation manner of the present invention, but the protection scope of the present invention is not limited thereto. Any changes or substitutions that can be easily conceived by any technician familiar with the technical field within the technical scope disclosed by the present invention should be covered within the protection scope of the present invention. < / h3>

Claims

1. A method for mining domain entities and their associated items, characterized in that: The following steps are involved: Acquire the corpus of the target domain, obtain a domain text set after preprocessing, mine entities and their associated items from the domain text set, and put them into an entity candidate set and an associated item candidate set; Based on the entity candidate set and the associated item candidate set, each entity and each of its associated items are respectively combined into search keywords, and search results of the search engine are captured by a web crawler; multiple search texts corresponding to each search keyword are extracted from the search results; Calculate the relevance between each search keyword and the corresponding multiple search texts, as well as the aggregation between the multiple search texts; The confidence of each search keyword is calculated based on the weights of the entities and related items in each search keyword, as well as the corresponding relevance and aggregation degree; A domain entity set and a domain associated item set are obtained according to entities and associated items of search keywords whose confidence is greater than a confidence threshold.

2. The method for mining domain entities and their associated items according to claim 1, characterized in that: The calculation of the association between each search keyword and the corresponding multiple search texts includes: obtaining a sentence set of each text in the domain text set, putting the sentence containing each search keyword in the domain text set into a first set, putting the multiple search texts corresponding to each search keyword into a second set, and calculating the similarity between the first set and the second set as the association.

3. The method for mining domain entities and their associated items according to claim 2, characterized in that: The similarity between the first set and the second set is calculated by using an LDA model to obtain a first topic vector of the first set and a second topic vector of the second set, and then calculating the cosine similarity between the first topic vector and the second topic vector.

4. The method for mining domain entities and their associated items according to claim 2, characterized in that: The similarity between the first set and the second set is calculated by using the word2vec model, as well as the word frequency of the first set and the word frequency of the second set, respectively obtaining a first sentence vector matrix of the first set and a second sentence vector matrix of the second set, and calculating the maximum cosine similarity between the first sentence vector matrix and the second sentence vector matrix.

5. The method for mining domain entities and their associated items according to claim 4, characterized in that: The first sentence vector matrix of the first set and the second sentence vector matrix of the second set are obtained by the following steps: Get the vocabulary in the first set and the second set, and use the trained word2vec model to get the word vector of each word; According to the word frequency of each word in the first set and the second set, the word vectors in each sentence in the first set and the second set are weighted averaged to obtain the initial sentence vector of each sentence; The initial sentence vector of each sentence is used as a column vector to construct the initial sentence matrices of the first set and the second set, and the first principal components of the initial sentence matrices of the first set and the second set are obtained respectively; After removing the projection vector of the corresponding first principal component from each initial sentence vector in the initial sentence matrices of the first set and the second set, a first sentence vector matrix and a second sentence vector matrix are obtained.

6. The method for mining domain entities and their associated items according to claim 4, characterized in that: The degree of aggregation between the multiple search texts is obtained by calculating the inverse of the variance of the second sentence vector matrix.

7. The method for mining domain entities and their associated items according to claim 4, characterized in that: The weights of the entities and associated items in each search keyword are obtained by filling multiple search texts corresponding to each search keyword into the domain text set, and calculating the TF-IDF value of each entity and associated item in the search keyword in the filled domain text set to obtain their respective weights.

8. The method for mining domain entities and their associated items according to claim 1, characterized in that: The confidence of each search keyword is obtained by normalizing the weights of the entities and associated items in each search keyword, and the corresponding relevance and aggregation degrees and then multiplying them respectively.

9. The method for mining domain entities and their associated items according to claim 1, characterized in that: Mining entities and their associated items from the domain text set and putting them into an entity candidate set and an associated item candidate set, including: Obtain a sentence set of each text in the domain text set, and perform word segmentation and part-of-speech tagging on each sentence in the sentence set; Using unsupervised knowledge mining algorithm to obtain keywords in the sentence set, taking keywords with noun as the main words, and obtaining other main words in a window with each main word in the sentence set by sliding preset windows as the associated items of the current main word; After deleting the main words without associated items, the remaining main words are taken as entities, and each entity and its associated items are filtered and put into the entity candidate set and the associated item candidate set.

10. A system for mining domain entities and their associated items, characterized in that: include: A candidate set acquisition module is used to acquire the corpus of the target domain, obtain the domain text set after preprocessing, mine entities and their associated items from the domain text set, and put them into the entity candidate set and the associated item candidate set; A search text extraction module is used to form search keywords for each entity and each of its associated items based on the entity candidate set and the associated item candidate set, and to crawl the search results of the search engine using a web crawler; and to extract multiple search texts corresponding to each search keyword from the search results; A confidence calculation module, used to calculate the relevance between each search keyword and the corresponding multiple search texts, and the aggregation between the multiple search texts; The confidence of each search keyword is calculated based on the weights of the entities and related items in each search keyword, as well as the corresponding relevance and aggregation degree; The domain set acquisition module is used to obtain a domain entity set and a domain associated item set according to entities and associated items of search keywords whose confidence is greater than a confidence threshold.

Images