A news entity analysis method and device based on unsupervised learning

Abstract

The present invention relates to a news entity analysis method and device based on unsupervised learning. The method comprises: performing word segmentation processing on each of the multiple news data to be processed, and labeling the multiple entities contained in each news after the word segmentation processing to obtain labeling results; constructing a distributed representation model based on the labeling results to obtain distributed representation information of the multiple entities, and the distributed representation information is identified as an entity vector; and performing cluster analysis on the multiple entities according to the distributed representation information of the multiple entities to obtain clustering results. The present application introduces the idea of ​​distribution into the processing of news entities, obtains the distributed representation of entities through the context of the location of the news entities, and obtains the clustering results of entities through cluster analysis of the entities.

Description

Technical Field

[0001] The present application relates to the field of text information mining, and more specifically, to a news entity analysis method and device based on unsupervised learning. Background Art

[0002] News is an important source of open source information. Since news is easy to obtain, has a wide range of dissemination and is very timely, the analysis of news has always been a hot topic in text analysis and mining, and there are many studies on news text analysis.

[0003] Entities in news can be obtained through related tools for Chinese named entity recognition. Related work is already relatively mature. However, it is difficult to perform large-scale annotation because the corresponding entities are of many types, cover a wide range, and related entries do not necessarily exist. Most of the existing work on news analysis requires annotation information. According to the survey, there is no unsupervised method that is directly applied to the modeling and analysis of entities in news. Summary of the invention

[0004] In order to solve the above problems, this application proposes a news entity analysis method and device based on unsupervised learning.

[0005] In a first aspect, the present application provides a news entity analysis method based on unsupervised learning, comprising:

[0006] Performing word segmentation processing on each of the multiple news data to be processed, and annotating multiple entities contained in each news after the word segmentation processing to obtain an annotation result;

[0007] Building a distributed representation model based on the annotation results to obtain distributed representation information of the multiple entities, wherein the distributed representation information is identified as an entity vector;

[0008] According to the distributed representation information of the multiple entities, cluster analysis is performed on the multiple entities to obtain a clustering result.

[0009] Preferably, the news entity method based on unsupervised learning also includes:

[0010] Perform topic clustering according to the annotation results corresponding to each piece of news data, obtain a topic clustering result, and obtain the topic to which each piece of news data belongs;

[0011] According to the topic to which each piece of news data belongs, the probability of each entity in the multiple entities appearing in the topic to which the multiple pieces of news data belong is counted to obtain the topic distribution of each entity in the multiple entities in the multiple pieces of news data.

[0012] Preferably, the news entity method based on unsupervised learning also includes:

[0013] The clustering effect of the clustering result is determined by the topic distribution of each entity in the multiple entities and the clustering result, and the clustering effect is represented by the average distance of the topic distribution of each entity.

[0014] Preferably, the news entity method based on unsupervised learning also includes:

[0015] Search is performed based on the seed information to obtain implicit information related to the multiple entities in the multiple news data.

[0016] Preferably, the news entity method based on unsupervised learning also includes:

[0017] The relationships between the multiple entities are constructed according to the distributed representations of the multiple entities and the number of co-occurrences of the multiple entities in the multiple news data; and the community structure existing in the relationships between the multiple entities is discovered using a community discovery algorithm.

[0018] In a second aspect, the present application provides a news entity analysis device based on unsupervised learning, comprising:

[0019] The tagging module is used to perform word segmentation processing on each of the multiple news data to be processed, and tag multiple entities contained in each news after the word segmentation processing to obtain a tagging result;

[0020] An acquisition module, configured to construct a distributed representation model based on the annotation results, and obtain distributed representation information of the multiple entities, wherein the distributed representation information is identified as an entity vector;

[0021] The clustering module is used to perform clustering analysis on the multiple entities according to the distributed representation information of the multiple entities to obtain a clustering result.

[0022] Preferably, the news entity analysis device based on unsupervised learning further includes:

[0023] A news topic acquisition module, used to cluster topics according to the annotation results corresponding to each piece of news data, obtain topic clustering results, and obtain the topic to which each piece of news data belongs;

[0024] The topic distribution acquisition module is used to count the probability of each entity in the multiple entities appearing in the topic to which the multiple news data belong, and obtain the topic distribution of each entity in the multiple entities in the multiple news data.

[0025] Preferably, the news entity analysis device based on unsupervised learning further includes:

[0026] The entity clustering analysis module is used to determine the clustering effect of the clustering result through the topic distribution of each entity in the multiple entities and the clustering result, and the clustering effect is represented by the average distance of the topic distribution of each entity.

[0027] Preferably, the news entity analysis device based on unsupervised learning further includes:

[0028] The seed information search module is used to search according to the seed information to obtain implicit information related to the multiple entities in the multiple news data.

[0029] Preferably, the news entity analysis device based on unsupervised learning further includes:

[0030] The community discovery module is used to construct the relationship between the multiple entities according to the distributed representation of the multiple entities and the number of co-occurrences of the multiple entities in the multiple news data; and to discover the community structure existing in the relationship between the multiple entities using a community discovery algorithm.

[0031] This application introduces the idea of ​​distribution into the processing of news entities, obtains the distributed representation of entities through the context of the location of news entities, and obtains the clustering results of entities through clustering analysis of entities. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] In order to more clearly illustrate the technical solutions of the embodiments of this specification, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0033] Figure 1 This is a schematic diagram of the application of the technical solution provided in the embodiments of the present application;

[0034] Figure 2 A schematic diagram of the method of the technical solution provided in the embodiments of the present application;

[0035] Figure 3 A method schematic diagram of a technical solution provided in another embodiment of the present application;

[0036] Figure 4 A method schematic diagram of a technical solution provided in another embodiment of the present application;

[0037] Figure 5 A schematic diagram of a device for the technical solution provided in an embodiment of the present application. DETAILED DESCRIPTION

[0038] The technical solution provided by the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments.

[0039] Figure 1 This is a schematic diagram of the application of the technical solution provided in the embodiment of this application. Please refer to Figure 1 After acquiring the news data, this technical solution can be used to discover the similarities between entities in the news data and the community relationships of entities. Seed information searches can also be performed to discover information related to the entities.

[0040] Figure 2 Schematic diagram of the method of the technical solution provided in the embodiment of this application. Figure 2 As shown, the unsupervised learning news entity analysis method provided by this application includes:

[0041] S201: Perform word segmentation processing on each of the multiple news data to be processed, and annotate multiple entities contained in each news after the word segmentation processing to obtain an annotation result.

[0042] In some possible embodiments, before performing word segmentation processing on each of the multiple news data to be processed, the multiple news data to be obtained must be pre-processed to obtain multiple news texts for word segmentation processing. Accordingly, the news data and the news texts correspond one to one; the word segmentation processing of the news data can specifically be the word segmentation processing of the news text corresponding to the news data. The acquisition of news data can rely on crawling from mainstream news websites by web crawlers.

[0043] The preprocessing process mainly includes: converting the Chinese encoding used in the text of multiple news data into UTF-8 format; removing illegal characters in the news text, using regular expressions to remove illegal characters in the news text, and only retaining Chinese characters, English words, numbers, and common punctuation marks; digital conversion, converting all Arabic numerals in the news text file into standardized simplified Chinese writing; punctuation conversion, converting half-width characters in the news text into corresponding full-width characters; news text size screening, deleting noise files in the news text whose number of words after removing punctuation is less than the predetermined number. It can be understood that the specific preprocessing rules should be determined according to different actual news texts. For example, the predetermined number can be 6 or other values. News data constitutes a corpus.

[0044] Specifically, you can download the corresponding pre-trained models for Chinese word segmentation and named entity recognition, load the models and perform word segmentation in the news text corresponding to each piece of news data in multiple news data. Check the word segmentation results, create a dictionary for words in the news text that are easily misclassified, and load it in the word segmentation stage to improve the accuracy of word segmentation. Perform named entity recognition on the news text after word segmentation, that is, perform entity annotation, and mark the corresponding entities in the text. Store the entity annotation results generated by the first named entity annotation. Then search and match in the news text after word segmentation and annotation. When a word is found to match the recognized entity but is not marked as an entity, re-annotate it in the text. In this way, you can get the annotation results.

[0045] S202: constructing a distributed representation model based on the annotation results to obtain distributed representation information of the multiple entities, where the distributed representation information is identified as entity vectors.

[0046] In a more specific example, based on the annotation results of Chinese word segmentation and entity annotation, the skip-gram model and the glove model are trained to obtain a distributed representation of each entity in multiple entities, that is, to obtain the entity vector of the entity.

[0047] The skip-gram model is a model that describes word2vec. It is a word vector representation model that uses a given central word w i To predict the possible context [w i-k ,…,w i-1 ],w i ,[w i+1 ,…,w i+k ] c The probability of occurrence is recorded as P(w c |w i ), in this scheme, the SoftMax function is used to calculate the given w i In the case of c The probability of occurrence is expressed as:

[0048]

[0049] Among them, P represents word w c The probability of occurrence, w j Represents the word vector of word j in the vocabulary, V represents the total vocabulary, Characterization word w c The transpose of the vector representation, w i Characterize the given central word.

[0050] The glove model is also a word vectorization representation model that can contain as much semantic and grammatical information as possible between vectors. In addition, the glove model introduces global information by obtaining the word co-occurrence matrix in the entire corpus. The optimized function after introducing the weighted function in this solution is expressed as follows:

[0051]

[0052] Among them, J represents the loss function to be solved, V represents the total vocabulary, and X ij represents the number of times word i and word j appear together, f(X ij ) represents the weight function, The transpose of the vector representing word i, w j The word vector representing word j, b i and b j Characterize word bias parameters. Word bias parameters need to be obtained through training.

[0053] In this way, the entity vector of the entity can be obtained through the skip-gram model and the glove model.

[0054] In some cases, the entity vector results can be used to find some entity results with incorrect recognition. These errors are caused by incorrect segmentation, but it can be found that the incorrect entities caused by these incorrect segmentations often have similar entity vectors due to similar contexts, and there is a certain possibility that they exist in the same text. The present invention uses the entity vector results of model training to judge whether different word segments are the same entity. When a character string of an entity recognition result represents a substring of another entity recognition result, the two results are considered to be the same entity if any of the following two rules are met:

[0055] Rule 1. The cosine similarity of two entities is > 0.80.

[0056] Rule 2: The cosine similarity of two entities is greater than 0.75 and they appear in at least one news text at the same time.

[0057] Among them, the cosine similarity is expressed as:

[0058]

[0059] w i and w j The entity vectors that represent the two entities respectively. The result of determining whether the entities are identical can be adjusted by changing the threshold of cosine similarity.

[0060] For example, if there is a person's name in the news text, and the name is the four characters "ABCD", then when performing word segmentation, the words may be divided into "ABCD", "ABC", "AB", etc., which will affect the generated entities. Or if there are two entities in the news text, namely a person's name and a place name, but the person's name and the place name happen to be called "AA", this will also affect the generated entities. Therefore, the above two rules are used to determine whether the entities are the same entity.

[0061] S203: Perform cluster analysis on the multiple entities according to the distributed representation information of the multiple entities to obtain a clustering result.

[0062] In a more specific example, based on the distributed representation information of the entity obtained through the skip-gram model, that is, the entity vector of the entity, k-means clustering is performed with the cosine similarity between the entity vectors as the distance measurement function to obtain the clustering result of the entity.

[0063] Figure 3 This is a schematic diagram of a method of a technical solution provided in another embodiment of the present application. Figure 3 As shown, in some possible embodiments, the news entity analysis method of unsupervised learning may further include the following steps:

[0064] S204: Perform topic clustering according to the annotation results corresponding to each piece of news data, obtain a topic clustering result, and obtain the topic to which each piece of news data belongs.

[0065] Specifically, for the annotation results corresponding to each piece of news data, the Latent Dirichlet Allocation (LDA) topic clustering model is used to cluster the annotation results, and then the Gibbs sampling method is used to obtain the topic distribution of each piece of news data, and then the topic with the highest probability is assigned to each piece of news data as the topic to which each piece of news data belongs.

[0066] The LDA model is a three-layer Bayesian network. Considering that each news data and each topic satisfies the Dirichlet distribution with parameters α and β as the prior distribution, the topic distribution of each word position in the news text is generated from the topic distribution of each news data, and then the word at each position is generated from the topic distribution of each position, and finally a combination of words that are not related to the order can be obtained. The generation probability of the position is expressed as:

[0067]

[0068] Among them, p represents the probability, θ represents the distribution of topics, N represents the number of words, z represents the topic, w represents the word, and α and β represent the hyperparameters in LDA.

[0069] Therefore, the probability of word generation in each news text is expressed as:

[0070]

[0071] Among them, p represents the probability, w represents the word, α and β represent the hyperparameters in LDA, θ represents the distribution of topics, N represents the number of words, and z represents the topic.

[0072] Next, Gibbs sampling is used to estimate the model parameters, and the optimization goal is to maximize the likelihood of generating the corpus in the corpus. After multiple iterations, the results of the burning period are discarded and the parameter estimation results are obtained.

[0073] The specific algorithm input is a text word sequence, for example w = {w1, w2, ..., w M The input parameters are hyperparameters α and β, and the number of topics K. The output is a text topic sequence, for example z = {z1, z2, …, z M}, and the single text The counts of the posterior probability distribution p(z|w,α,β) of , and the estimates of the parameters θ.

[0074] The algorithm is expressed as follows:

[0075] S2041: Let n be the element of all counting matrices mk and n kv , the element n of the counting vector k and n m The initial value is 0. Among them, n mk Represents the number of words belonging to the kth topic in the text corresponding to the mth news data, n kv Represents the number of times word v belongs to the kth topic.

[0076] S2042: For all words w in the mth text mn Perform topic sampling, add text, add topics, and calculate the n after adding text and adding topics. mk and n kv Count.

[0077] S2043: Repeat the following steps until the combustion phase begins:

[0078] S2043a: If the current word is the vth word and the topic is the kth topic, then the count is reduced;

[0079] S2043b: Sampling is performed according to the full conditional distribution, the formula is as follows;

[0080]

[0081] S2043c: Add a new count.

[0082] S2044: Using the obtained sample count, the parameter θ can be calculated, and the formula is as follows:

[0083]

[0084] In some possible embodiments, in the annotation results corresponding to each news data among multiple news data, there may be many interfering words unrelated to the topic. In this case, these invalid information needs to be removed first.

[0085] For example, in the annotation result corresponding to a news, there are many high-frequency common words such as "de" (of) and "ni hao" (hello). This solution uses the method of removing stop words to remove them.

[0086] S205: According to the topic to which each news data belongs, count the probability that each entity in the multiple entities appears in the topic to which the multiple news data belong, and obtain the topic distribution of each entity in the multiple news data.

[0087] Specifically, for the frequency of all entities appearing in the news of a certain topic, the distribution of the appearance of all entities in the news topic can be obtained, and the entity topic distribution is represented by a vector of length M.

[0088] For example, there is an entity A that appears in 4 news. Among them, one news is the news belonging to topic T0, and three news are the news belonging to topic T M-1 belonging to. Then, the topic distribution vector of this entity A can be represented as

[0089] Figure 4 The following is a schematic diagram of the method of the technical solution provided in another embodiment of the present application. Please refer to Figure 4 , in some possible embodiments, the unsupervised learning news entity analysis method may further include:

[0090] S206: Determine the clustering effect of the clustering result through the topic distribution of each entity in the multiple entities and the clustering result, and the clustering effect is represented by the average distance of the topic distribution of each entity.

[0091] Specifically, according to the obtained topic distribution of the entity and the clustering result of the entity, the clustering effect of the entity can be determined, and this clustering effect is used to represent the effectiveness of the distributed representation (i.e., entity vector) of the entity.

[0092] The entity topic distributions within the cluster are marked as [x1, x2,..., x M , and the average distance of the topics within the class is used to measure the topic similarity. The formula is as follows:

[0093]

[0094] Among them, dist represents the average distance, and M represents the number of topics.

[0095] The smaller the dist value, the higher the topic similarity. In addition, the topic with the largest proportion in the clustering result is taken as the topic of the entity clustering result. The larger the value, the more concentrated the topic of the clustering result.

[0096] For some possible embodiments, please refer to Figure 4 , the unsupervised learning news entity analysis method can also include:

[0097] S207: Search according to the seed information to obtain implicit information related to the multiple entities in the multiple news data.

[0098] Specifically, the seed information is the information used in the search. For example, if you want to obtain information related to entity A, entity A is the seed information. Using the entity vector generated by the skip-gram model, the vector representation of the input seed is screened out, and the top K entities with similarity in the vocabulary are screened out, and the parts with higher similarity are used as the input related content.

[0099] There may be some information hidden in the text that has not been discovered. In order to discover some hidden relevant information as much as possible, the specific process of the designed algorithm is as follows:

[0100] S2071: Search for all possible recognition results of the seed entity found in the entity merging phase, and use all possible recognition situations as candidates for input search.

[0101] For example, when searching for an entity named "ABCD", possible related information such as "ABCD", "ABC", "AB", etc. are searched as seeds to obtain search recognition results of all related entities. This recognition result is implicit information.

[0102] S2072: For each element in the search candidate, if it is an entity, extract the top 50 similar words in the model dictionary and the similarity is greater than the threshold value of 0.80 and add them to the keyword list.

[0103] S2073: Find all entities from the keyword list and search for entities in the vocabulary with the top 30 similarities for each entity.

[0104] S2074: Search for the intersection of the top 50 words of the new entity and the existing keyword list. If the number of overlaps is greater than 12, add them to the list. If the number of words in the keyword list is less than 24, the number of overlaps can be greater than half of the number of words in the list.

[0105] S2075: Repeat the operations of S2073 and S2074 until the elements in the list no longer change.

[0106] The specific value of the threshold can be adjusted according to the usage results of different corpora. If you want to increase the recall rate, reduce the threshold; if you want to increase the precision, increase the threshold.

[0107] For some possible embodiments, please refer to Figure 4 , the unsupervised learning news entity analysis method can also include:

[0108] S208: constructing relationships between the multiple entities based on the distributed representations of the multiple entities and the number of co-occurrences of the multiple entities in the multiple news data, and using a community discovery algorithm to discover community structures existing in the relationships between the multiple entities.

[0109] Specifically, the relationship between entities is constructed using the entity distributed representation (i.e., entity vector) generated by the glove model and the number of co-occurrences of the entities in the news text.

[0110] For example, the relationship between two entities can be constructed by the following formula:

[0111]

[0112] rel'(w1,w2)=min(log2(co_times+1)*sim(w1,w2) / 2,ξ2)

[0113]

[0114] Among them, w1 and w2 represent two entities, and co_times represents the number of co-occurrences of entities w1 and w2. The more co-occurrences between entities, the greater the similarity of entity vectors, and the stronger the association between entities. When the similarity is higher than ξ1, it is considered that there is an association between entities; ξ2 represents the maximum value of the entity relationship; ξ3 is used to remove some weak associations to reduce some unnecessary edges and facilitate the analysis of results.

[0115] Adjust the threshold according to specific needs. The smaller the threshold, the more entity relationships are considered. However, if the threshold is too small, a lot of noise will be introduced. If the threshold is too large, the number of relationships considered may be too small and information may be lost.

[0116] After the relationship between entities is constructed, the Louvain algorithm is used to perform community discovery tasks on the entity relationship graph. A community is a group of people with close connections that are mined from a graph structure consisting of entities and their relationships. It can be used to discover hidden groups in text. The Louvain algorithm optimizes the modularity of community division based on a greedy strategy. Modularity is an indicator of the level of community division without knowing the true division of the community. It can be used to measure the degree of association within the community and is expressed as:

[0117]

[0118] Among them, Q represents the modularity, m represents the sum of the network connection weights, v and w are any two points in the entity relationship construction graph, and A vw Represents the connection weight between two points, k v Characterize the degree of point v, k w Characterizes the degree of point w, δ(c v ,c w ) represents whether point v and point w belong to the same community. If so, it is 1, and if not, it is 0.

[0119] In some possible embodiments, the Louvain algorithm may also be optimized. The optimization may be divided into two steps:

[0120] 1) Divide each node into a community, calculate the modularity at this time, then merge a node into the community of its neighboring nodes to calculate the modularity gain, find the node with the largest modularity gain and merge it, repeat several times until the modularity no longer increases;

[0121] 2) Merge the new communities into points and repeat the operation in 1 until convergence.

[0122] By calculating the association between entity pairs extracted from the same community in the community division results and comparing them with the association between randomly extracted entities to form entity pairs, the effectiveness of the community discovery task can be demonstrated.

[0123] Figure 5 Schematic diagram of the device of the technical solution provided in the embodiment of this application. Figure 5 As shown, the news entity analysis device based on unsupervised learning may include:

[0124] The tagging module 501 is used to perform word segmentation processing on each of the multiple news data to be processed, and tag multiple entities contained in each news after the word segmentation processing to obtain a tagging result;

[0125] An acquisition module 502 is used to construct a distributed representation model based on the annotation results to obtain distributed representation information of the multiple entities, where the distributed representation information is identified as an entity vector;

[0126] The clustering module 503 is used to perform cluster analysis on the multiple entities according to the distributed representation information of the multiple entities to obtain a clustering result.

[0127] In some possible embodiments, the news entity analysis device based on unsupervised learning further includes:

[0128] The news topic acquisition module 504 is used to perform topic clustering according to the labeling results corresponding to each piece of news data including the plurality of news data, and obtain the topic to which each piece of news data belongs;

[0129] The topic distribution acquisition module 505 is used to count the probability of each entity in the multiple entities appearing in the topic to which the news data belongs, and obtain the topic distribution of each entity in the multiple entities in multiple news items.

[0130] In some possible embodiments, the news entity analysis device based on unsupervised learning further includes:

[0131] The entity clustering analysis module 506 is used to determine the clustering effect of the clustering result through the topic distribution of the multiple entities and the clustering result, and the clustering effect is represented by the average distance of the topic distribution in the topic clustering result.

[0132] In some possible embodiments, the news entity analysis device based on unsupervised learning further includes:

[0133] The seed information search module 507 is used to search according to the seed information to obtain implicit information related to the multiple entities in the multiple news data.

[0134] In some possible embodiments, the news entity analysis device based on unsupervised learning further includes:

[0135] The community discovery module 508 is used to construct the relationship between the multiple entities based on the distributed representations of the multiple entities and the number of co-occurrences of the multiple entities in the multiple news data; and to use the community discovery algorithm to discover the community structure existing in the relationship between the multiple entities.

[0136] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the presence of other identical elements in the process, method, article or device including the elements.

[0137] A person skilled in the art will understand that all or part of the steps to implement the above embodiments may be accomplished by hardware or by instructing related hardware through a program, and the program may be stored in a computer-readable storage medium, and the above-mentioned storage medium may be a read-only memory, a disk or an optical disk, etc.

[0138] The specific implementation methods described above further illustrate the objectives, technical solutions and beneficial effects of the present invention in detail. It should be understood that the above description is only a specific implementation method of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A news entity analysis method based on unsupervised learning, characterized in that: include: Perform word segmentation on each of the multiple news data to be processed, and annotate multiple entities contained in each news after the word segmentation to obtain an annotation result; A distributed representation model is constructed based on the annotation results to obtain distributed representation information of the multiple entities, wherein the distributed representation information is identified as an entity vector; and according to the entity vector, whether different word segments are the same entity is judged; wherein the judgment of whether different word segments are the same entity is specifically as follows: when a character string representation of an entity recognition result is a substring of another entity recognition result, the two results are considered to be the same entity if any one of the following two rules is met: Rule 1. The cosine similarity of two entities is > 0.80; Rule 2: The cosine similarity of two entities is greater than 0.75 and they appear in at least one news text at the same time. Wherein, the cosine similarity is expressed as: w i and w j Represent the entity vectors of the two entities respectively, and adjust the result of determining whether the entities are the same by changing the threshold of cosine similarity; performing cluster analysis on the multiple entities according to the distributed representation information of the multiple entities to obtain a clustering result; Perform topic clustering according to the annotation results corresponding to each of the news data to obtain a topic clustering result, and obtain the topic to which each of the news data belongs; based on the topic to which each of the news data belongs, count the probability of each of the multiple entities appearing in the topic to which the multiple news data belong, and obtain the topic distribution of each of the multiple entities in the multiple news data; determine the clustering effect of the clustering result through the topic distribution of each of the multiple entities and the clustering result, and the clustering effect is represented by the average distance of the topic distribution of each entity.

2. The method according to claim 1, characterized in that The method further comprises: Search is performed based on the seed information to obtain implicit information related to the multiple entities in the multiple news data.

3. The method according to claim 1, characterized in that The method further comprises: The relationships between the multiple entities are constructed according to the distributed representations of the multiple entities and the number of co-occurrences of the multiple entities in the multiple news data; and the community structure existing in the relationships between the multiple entities is discovered using a community discovery algorithm.

4. A news entity analysis device based on unsupervised learning, characterized in that: include: The tagging module is used to perform word segmentation processing on each of the multiple news data to be processed, and tag multiple entities contained in each news after the word segmentation processing to obtain a tagging result; An acquisition module is used to build a distributed representation model based on the annotation results to obtain distributed representation information of the multiple entities, wherein the distributed representation information is identified as an entity vector; based on the entity vector, whether different word segments are the same entity is judged; wherein the judgment of whether different word segments are the same entity is specifically as follows: when a character string representation of an entity recognition result is a substring of another entity recognition result, the two results are considered to be the same entity if any one of the following two rules is met: Rule 1. The cosine similarity of two entities is > 0.80; Rule 2: The cosine similarity of two entities is greater than 0.75 and they appear in at least one news text at the same time. Wherein, the cosine similarity is expressed as: w i and w j Represent the entity vectors of the two entities respectively, and adjust the result of determining whether the entities are the same by changing the threshold of cosine similarity; A clustering module, configured to perform cluster analysis on the multiple entities according to the distributed representation information of the multiple entities to obtain a clustering result; A news topic acquisition module, used to cluster topics according to the annotation results corresponding to each piece of news data, obtain topic clustering results, and obtain the topic to which each piece of news data belongs; A topic distribution acquisition module, for counting the probability of each entity in the multiple entities appearing in the topic to which each piece of news data belongs, and obtaining the topic distribution of each entity in the multiple entities in the multiple pieces of news data; The entity clustering analysis module is used to determine the clustering effect of the clustering result through the topic distribution of each entity in the multiple entities and the clustering result, and the clustering effect is represented by the average distance of the topic distribution of each entity.

5. The device according to claim 4, characterized in that The device also includes: The seed information search module is used to search according to the seed information to obtain implicit information related to the multiple entities in the multiple news data.

6. The device according to claim 4, characterized in that The device also includes: The community discovery module is used to construct the relationship between the multiple entities according to the distributed representation of the multiple entities and the number of co-occurrences of the multiple entities in the multiple news data; and to use the community discovery algorithm to discover the community structure existing in the relationship between the multiple entities.

Images