A text aggregation method and a text recommendation method

Abstract

The invention provides a text aggregation method and a text recommendation method. The text aggregation method comprises the following steps: obtaining a text to be processed; calculating fingerprint information of the text by using an improved SimHash algorithm, wherein the improved SimHash algorithm weights the hash values ​​of corresponding words in the text by using the weights of the words in the text and the domain weights of the words in the domain where the text is located; scoring the domain relevance of the text by using the domain weights to obtain the domain scores of the text; dividing the fingerprint information of the text into a plurality of fingerprint segments, and constructing an inverted index based on the values ​​of the fingerprint segments, wherein in the key-value pairs corresponding to the inverted index, the keys store the values ​​of the fingerprint segments, and the values ​​store the text related information, and the text related information includes the fingerprint information and the domain scores of the text; scoring the text by using the domain weights, and constructing the inverted index, so as to avoid repeated processing of the text in the database when recommending similar texts later, thereby greatly improving the processing efficiency.

Description

Technical Field

[0001] The present invention relates to the field of data processing, in particular to the field of text similarity analysis, and more particularly to a text aggregation method and a text recommendation method. Background Art

[0002] In the field of text recommendation, it is usually necessary to calculate text similarity. The main text similarity detection algorithms are K-shingle algorithm, MinHash algorithm and SimHash algorithm. The following introduces these three algorithms respectively:

[0003] Assumptions Documentation Is In the K-shingle algorithm, a document is represented as a phrase consisting of k (the value of k can be set as needed) consecutive words, that is, ,in, Compare the K-shingles of document A and document B, and record the phrases that appear in both document A and document B as , the phrases appearing in document A or document B are recorded as , the similarity between documents A and B is calculated by Jaccard similarity: The larger the k value in the K-shingle algorithm, the stronger the ability to express the semantics of the document, and the more reliable the calculated similarity; conversely, the less reliable the similarity. Assuming that the document collection contains T documents, in order to calculate the similarity between all two documents, it is necessary to calculate Second-rate.

[0004] The MinHash algorithm improves the K-shingle algorithm and uses the K-shingle phrases of all documents to build an inverted index. . The K-shingle phrase is converted to The m-dimensional vector represented by The K-shingle phrase exists in , then the first bit of the vector is 1, and so on). Then the m-dimensional vector is reordered by hashing. From the sorting result, the subscripts of n non-zero elements are randomly selected to form an n-dimensional vector (generally n is much smaller than the inverted index length m and the phrase length k). Although the MinHash algorithm still needs to compare times, but reduces the dimension of the vector to be compared, which improves the computational efficiency compared to the K-shingle algorithm.

[0005] The SimHash algorithm calculates the weighted words in the article to generate a binary vector as the article fingerprint data. ,in, Represents the Hamming distance between two article fingerprint data, The number of binary digits representing the fingerprint data. If it is a long text, the word with higher frequency can be selected words as the basis for calculation, further reducing the vector dimension to be calculated to a constant level. In actual engineering practice, the drawer principle can be used to optimize the number of comparisons of SimHash: for example, when the article fingerprint data is 64 bits, the Hamming distance of the fingerprint data is less than or equal to 3 to consider the articles similar; if the fingerprint data is divided into 4 segments of equal length, then two similar articles have at least one segment of their fingerprint data that is equal. Therefore, only when one segment is equal can the article be similar. By comparing the fingerprint segments, you can quickly locate a collection of potentially similar articles. ( , is the total number of documents in the document collection). Compared with the MinHash algorithm, the SimHash algorithm can significantly reduce the number of comparisons while only increasing the storage by a constant multiple. When the SimHash algorithm calculates the fingerprint data of an article, it is only related to the word weight data and has nothing to do with other documents. It is suitable for streaming computing scenarios.

[0006] The traditional method of sorting article collections uses word frequency and article length to calculate the score:

[0007]

[0008] in, Indicates The frequency of the feature words, Indicates the total number of words in the article. Indicates the total number of feature words. This method only considers feature words and article length for sorting. The higher the proportion of feature words in an article, the higher the score.

[0009] In streaming data processing scenarios, since the number of document collections is usually large, the time complexity of using the K-shingle algorithm is high. In addition, when comparing article similarities, the K-shingle algorithm needs to record the K-shingle phrases of all documents, which takes up storage space. The time complexity and space complexity are both high, and it cannot effectively complete the task in this scenario. The MinHash algorithm will cause the inverted index to change due to the appearance of new words in the K-shingle phrases of new articles, and it is necessary to recalculate the similarity of all articles with the current article, resulting in a large number of repeated calculations. The SimHash algorithm reduces the number of comparisons and vector dimensions, and can also reuse the last calculated article fingerprint data as the basis for subsequent similarity comparisons, but it performs poorly in scenarios where short text data is a subset of long text data (such as news briefs and subsequent detailed reports).

[0010] When sorting a collection of articles, traditional methods only consider surface factors such as feature words and article length, without considering semantic and business scenario factors, resulting in a large gap between the sorting results and actual expectations. Summary of the invention

[0011] Therefore, the purpose of the present invention is to overcome the defects of the above-mentioned prior art and provide a text aggregation method and a text recommendation method.

[0012] The objective of the present invention is achieved through the following technical solutions:

[0013] According to a first aspect of the present invention, a text aggregation method is provided, comprising: obtaining a text to be processed; calculating fingerprint information of the text using an improved SimHash algorithm, wherein the improved SimHash algorithm uses the weight of the word in the text and the domain weight of the word in the domain to which the text belongs to perform weighting when weighting the hash value of the corresponding word in the text; scoring the domain relevance of the text using the domain weight to obtain the domain score of the text; dividing the fingerprint information of the text into a plurality of fingerprint segments, and constructing an inverted index based on the values ​​of the fingerprint segments, wherein, in the key-value pair corresponding to the inverted index, the key stores the value of the fingerprint segment, and the value stores text-related information, and the text-related information includes the fingerprint information and the domain score of the text.

[0014] In some embodiments of the present invention, in each field, the domain weight of a word is pre-calculated in the following manner: obtain the domain corpus of the field, wherein the domain corpus includes a plurality of sample articles belonging to the field that are collected in advance; extract a plurality of key feature words in the domain corpus to form a domain feature phrase; for each word in the domain feature phrase, calculate the domain weight of the corresponding word based on the total number of words in the domain corpus, the total number of articles, the number of times the corresponding word appears in the domain corpus, and the number of articles in the domain corpus that contain the corresponding word.

[0015] In some embodiments of the present invention, the domain weight of a word is calculated in the following manner:

[0016] ;

[0017] in, Expressing words The domain weight of Expressing words The number of times it appears in the domain corpus, Represents the total number of words in the domain corpus, represents the total number of articles in the domain corpus, It represents the number of articles containing the corresponding words in the domain corpus, and k represents the domain weight adjustment coefficient.

[0018] In some embodiments of the present invention, the step of calculating the fingerprint information of the text using the improved SimHash algorithm includes: processing the text to calculate the hash values ​​of multiple words included therein; multiplying the hash value of each word by the weight of the word in the text and the domain weight of the word in the domain where the text is located, to obtain the weighted hash value of each word; accumulating the weighted hash value of each word at the same bit and then reducing the dimension to form the fingerprint information of the text.

[0019] In some embodiments of the present invention, the domain score of a text is determined in the following manner: according to the entity dictionary corresponding to the domain of the text, the person entity information, place entity information, and organization entity information in the text are obtained, wherein the entity dictionary for each domain can be set independently; according to the person entity information, place entity information, and organization entity information in the text, the number of different person entities, the number of different place entities, and the number of different organization entities in the text are determined; according to the number of different person entities, the number of different place entities, the number of different organization entities in the text, the word frequency of words in the text feature phrases, and the domain weights of the words in the text feature phrases, the domain score of the text is determined.

[0020] In some embodiments of the present invention, the domain score of a text is determined as follows:

[0021] ;

[0022] in, Indicates the number of different character entities in the text, express The weight coefficient of Represents the number of different location entities in the text, express The weight coefficient of Indicates the number of different institutional entities in the text, express The weight coefficient of Represents the words in the text feature phrase The number of times it appears in the text, Represents the words in the text feature phrase The domain weight of Indicates the total number of words in the text.

[0023] In some embodiments of the present invention, the method further comprises: searching for other texts similar to the text according to the inverted index to determine a similar cluster, wherein the similar cluster stores fingerprint information of each text belonging to the similar cluster.

[0024] According to a second aspect of the present invention, there is provided a text recommendation method, comprising: obtaining an input text from a data stream; processing the input text using the text aggregation method described in the first aspect to construct an inverted index; searching for other texts similar to the text according to the inverted index to determine a similar cluster, wherein the similar cluster stores fingerprint information of each text belonging to the similar cluster; determining a domain score of each text in the inverted index according to the fingerprint information of each text in the similar cluster corresponding to the input text; and recommending similar texts to the input text according to the domain scores of each text in the similar cluster.

[0025] According to a third aspect of the present invention, there is provided an electronic device comprising: one or more processors; and a memory, wherein the memory is used to store executable instructions; the one or more processors are configured to implement the steps of the method described in the first aspect or the second aspect by executing the executable instructions. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] The embodiments of the present invention are further described below with reference to the accompanying drawings, in which:

[0027] Figure 1 A schematic diagram of a flow chart of a text aggregation method according to an embodiment of the present invention;

[0028] Figure 2 A schematic diagram of the calculation principle of fingerprint information in a text aggregation method according to an embodiment of the present invention;

[0029] Figure 3 A schematic diagram of constructing an inverted index in a text aggregation method according to an embodiment of the present invention;

[0030] Figure 4 A schematic diagram of an inverted index established in a text aggregation method according to an embodiment of the present invention. DETAILED DESCRIPTION

[0031] In order to make the purpose, technical solutions and advantages of the present invention more clearly understood, the present invention is further described in detail below through specific embodiments in conjunction with the accompanying drawings. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0032] As mentioned in the background technology section, when sorting a collection of articles, traditional methods only consider surface factors such as feature words and article length, without considering semantics and business scenario factors, resulting in a large gap between the sorting results and actual expectations. When processing real-time streaming data content duplication detection and sorting, the inventors have found that many traditional algorithms are aimed at scenarios with fixed data sets. When the data set increases dynamically, the time complexity and space complexity of traditional algorithms will increase quadratically with the number of documents, and cannot effectively handle the problem of content duplication detection in this scenario. The present invention provides a text aggregation method, which uses an improved SimHash algorithm to weight the hash value of the corresponding word in the text. In addition to multiplying the weight of the word in the text, the hash value of the corresponding word is also multiplied by the domain weight of the word in the field where the text is located, which can better discover and aggregate similar texts in the field; in addition, the present invention also scores the text by the field weight and constructs an inverted index to avoid repeated processing of the text in the database when recommending similar texts later, greatly improving the processing efficiency.

[0033] Implementation 1:

[0034] According to an embodiment of the present invention, a text aggregation method is provided, including four steps S1, S2, S3, and S4. Specifically:

[0035] Step S1: Obtain the text to be processed and the text number;

[0036] According to one embodiment of the present invention, the text to be processed and / or the number of the text can be transmitted in the form of a data stream. The text to be processed is, for example, a news article, a technical article, a historical document, a health-related article, a game play article, etc. that the user has recently viewed.

[0037] S2. Calculate the fingerprint information of the text using the improved SimHash algorithm, wherein when the improved SimHash algorithm weights the hash value of the corresponding word in the text, the hash value of the corresponding word is multiplied by the weight of the word in the text and the domain weight of the word in the domain where the text is located.

[0038] According to one embodiment of the present invention, step S2 includes: processing the text, calculating the hash values ​​of the multiple words included therein; multiplying the hash value of each word by the weight of the word in the text and the domain weight of the word in the field where the text is located, to obtain the weighted hash value of each word; accumulating the weighted hash value of each word at the same bit value and then reducing the dimension to form the fingerprint information of the text. In this field, it should be understood that processing the text includes preprocessing to obtain text feature phrases. Calculating the hash values ​​of the multiple words included therein refers to calculating the hash values ​​of the multiple words in the text feature phrases extracted from the text. It should be understood that when the hash value of each word is multiplied by the weight of the word in the text, if a bit of the hash value of the word is 0, it is regarded as -1. According to one embodiment of the present invention, step S2 includes: preprocessing the text to obtain a text feature phrase, wherein the preprocessing includes word segmentation and stop word removal; vectorizing the words in the text feature phrase to obtain a feature vector of each word; calculating the hash value of each word according to the feature vector of each word; weighting the hash value of each word to obtain a weighted hash value of each word, wherein the weighting is performed using the weight of the word in the text and the domain weight of the word in the domain where the text is located; accumulating the weighted hash values ​​of each word at the same bit position and then reducing the dimension to form the fingerprint information of the text. This embodiment has at least the following beneficial effects: the present invention improves the SimHash algorithm, redefines the weighting method, and in addition to the original weighting using the weight of the word in the text, adds a process of weighting using the domain weight of the word, and uses domain semantic similarity to reduce the situation where some similar texts are classified as dissimilar. In addition, the inventors discovered during the research process that when the SimHash algorithm processes long text data and subset short text, it will mistakenly judge that the similarity between the long text and the subset short text does not exceed the threshold, resulting in detection errors; after weighting using the domain weight, in the scenario where the short text data is a subset of the long text data, the short text and similar long text in the field can be more accurately aggregated into similar texts, and review articles and sub-viewpoint chapters in the field can also be better aggregated into similar texts.

[0039] For example, accessing text from a data stream , preprocess the text including word segmentation and stop word removal to obtain text feature phrases ,in, Represents feature words, that is, words in the text feature phrase.

[0040] See also Figure 2 , assuming that the input text is "Central Bank: The RMB exchange rate will not appreciate unilaterally due to...", in order to simplify the explanation, the vector dimension of the fingerprint information is set to 5 dimensions (usually 64 dimensions).

[0041] Assume that after preprocessing, the text feature phrase "central bank / RMB / exchange rate / … / appreciation" is obtained;

[0042] Vectorize the words in the text feature phrase and calculate the hash value to obtain the hash value of each word "10101 / 10110 / 01101 / … / 00101";

[0043] Based on the weight of each word in the text "t1 / t2 / t3 / … / tm", weighting is performed, where the hash value of a word is regarded as -1 when it is 0, and the result is "t1,-t1,t1,-t1,t1 / t2,-t2,t2,t2,-t2 / -t3,t3,t3,-t3,t3 / … / -tm,-tm,tm,-tm,tm", and based on the domain weight "w1,w2,w3,…, wm" is weighted to obtain "t1*w1,-t1*w1,t1*w1,-t1*w1,t1*w1 / t2*w2,-t2*w2,t2*w2,t2*w2,-t2 *w2 / -t3*w3,t3*w3,t3*w3,-t3*w3,t3*w3 / … / -tm*wm,-tm*wm,tm*wm,-tm*wm,tm*wm”;

[0044] The weighted hash values ​​of each word are accumulated at the same bit. Taking the first bit as an example, the accumulation method is: t1*w1+ t2*w2+(-t3*w3)+…+(-tm*wm); Assume that after accumulation, a 1×5-dimensional vector "16, 8, -3, 6, 118" is obtained;

[0045] The accumulated vector is reduced in dimension, wherein the bits with values ​​greater than 0 are set to 1 and the remaining bits are set to 0. For example, the aforementioned vector "16, 8, -3, 6, 118" is reduced in dimension to obtain the fingerprint information of the text "11011".

[0046] According to one embodiment of the present invention, the domain weight of a word is the importance of the word in the field calculated based on the pre-collected domain corpus. Preferably, the domain corpus of different fields is set independently. It should be understood that the classification of the field can be customized according to the actual situation or the needs of the operator. For example, the classification of the field includes: economy, politics, games, health, history, life or a combination thereof. In addition, there are many ways to calculate the domain weight of a word. For example, the word frequency in the domain corpus is used as the domain weight of the word, or it is calculated according to a weight calculation algorithm, such as: TF-IDF algorithm or TF-IWF algorithm.

[0047] Taking the TF-IDF algorithm as an example, according to one embodiment of the present invention, in each field, the domain weight of a word is pre-calculated in the following manner: obtain the domain corpus of the field, wherein the domain corpus includes a plurality of sample articles belonging to the field that are pre-collected; extract a plurality of key feature words in the domain corpus to form a domain feature phrase; for each word in the domain feature phrase, calculate the domain weight of the corresponding word based on the total number of words in the domain corpus, the total number of articles, the number of times the corresponding word appears in the domain corpus, and the number of articles in the domain corpus that contain the corresponding word. According to one embodiment of the present invention, the domain weight of a word is pre-calculated in the following manner:

[0048] ;

[0049] in, Expressing words The domain weight of Expressing words The number of times it appears in the domain corpus, Represents the total number of words in the domain corpus, represents the total number of articles in the domain corpus, Indicates the number of articles containing the corresponding words in the domain corpus.

[0050] For example, suppose there are 100 articles in the domain corpus of a certain field, and the word "stock market" appears in 59 of them, with a total number of appearances (corresponding to the number of appearances in the domain corpus) of 600 times, and the total number of words in the domain corpus is 10,000. Another word "population" appears in 9 of them, with a total number of appearances of 15 times. According to the above calculation, in this field, the domain weight of "stock market" is 0.01331, and the domain weight of "population" is 0.0015. This also shows that "stock market" has a higher weight than "population" in this field, and contributes more to the field, or the field is more relevant and important.

[0051] In order to balance the relationship between the domain-related weight and the proportion of other factors, the domain weight adjustment coefficient can be increased. According to one embodiment of the present invention, the domain weight of a word is pre-calculated in the following manner:

[0052] ;

[0053] in, Expressing words The domain weight of Expressing words The number of times it appears in the domain corpus, Represents the total number of words in the domain corpus, represents the total number of articles in the domain corpus, Indicates the number of articles containing the corresponding words in the domain corpus, and k indicates the domain weight adjustment coefficient. k is a constant. In different domains, different domain corpora will lead to deviations in some domain weights. After setting the domain weight adjustment coefficient, it is convenient to adjust some domain weights according to needs or experience, so as to scale the original domain weights and balance the relationship between domain-related weights and the proportion of other factors.

[0054] Step S3: Use the domain weight to score the domain relevance of the text to obtain the domain score of the text.

[0055] According to one embodiment of the present invention, the domain score of a text is determined in the following manner: the domain score of the text is determined according to the frequency of words in the text feature phrases of the text and the domain weights of the words in the text feature phrases. According to one embodiment of the present invention, the domain score of a text is determined in the following manner: ;in, Represents the words in the text feature phrase The number of times it appears in the text, Represents the words in the text feature phrase The domain weight of Indicates the total number of words in the text.

[0056] In terms of sorting similar content, traditional methods only consider surface factors such as feature words and text length, which are suitable for text data with similar text length and similar language usage habits. However, in actual data scenarios, the length of text is relatively random, and various network terms and new expressions emerge in an endless stream, which cannot handle the problem well. The inventors have found that texts with specific entity names are more representative in similar sets. For example, when the correlation between the feature words of the text is similar, the text is more authoritative and representative when it contains certain entity words. For example, texts containing institution names such as "Institution A", "Institution B", and typical characters such as "Character C" are more authoritative and representative, and the field score of the text should be higher. Therefore, it is proposed to combine the field feature entity information when scoring the text, so as to select texts that are more in line with the characteristics of the field in the subsequent recommendation process. According to one embodiment of the present invention, the domain score of a text is determined in the following manner: according to the entity dictionary corresponding to the domain of the text, the person entity information, place entity information, and organization entity information in the text are obtained, wherein the entity dictionary for each domain can be set independently; according to the person entity information, place entity information, and organization entity information in the text, the number of different person entities, the number of different place entities, and the number of different organization entities in the text are determined; according to the number of different person entities, the number of different place entities, the number of different organization entities in the text, the word frequency of words in the text feature phrases, and the domain weights of the words in the text feature phrases, the domain score of the text is determined.

[0057] According to one embodiment of the present invention, in order to calculate the domain score, the number of times the words in the text feature phrase appear in the text is counted and entity information is identified. The result of the number of times the words appear in the text is expressed as: , ,in, Expressing words , Expressing words The number of times it appears in the text. The result of entity information recognition is expressed as: , Represents a single person entity, Represents a collection of character entities, Represents a single place entity, Represents a collection of place entities, Represents a single institutional entity, Represents a set of organization entities. Based on the results of entity information recognition, the number of different person entities, the number of different location entities, and the number of different organization entities in the text can be determined.

[0058] According to one embodiment of the present invention, the domain score of the text is determined in the following manner: ;in, Indicates the number of different character entities in the text, express The weight coefficient of Represents the number of different location entities in the text, express The weight coefficient of Indicates the number of different institutional entities in the text, express The weight coefficient of Represents the words in the text feature phrase The number of times it appears in the text, Represents the words in the text feature phrase The domain weight of Indicates the total number of words in the text. That is, the words in the text feature phrase The word frequency.

[0059] The above-mentioned embodiments of the present invention have at least the following beneficial effects: the domain score is only related to the text itself and the domain corpus, but has nothing to do with other texts that need to calculate similarity with the text. Therefore, after other texts are updated, the domain score does not need to be recalculated. It can be seen that in the present invention, the domain score of the text is an absolute score related to the text and the domain corpus of the field to which the text belongs. The domain score of the text is independent of other texts and is not a relative score related to other texts. The domain score only needs to be calculated once, and there is no need for additional repeated processing of the original text, which greatly improves the processing efficiency and subsequent text recommendation efficiency.

[0060] Step S4: Divide the fingerprint information of the text into multiple fingerprint segments, and construct an inverted index based on the values ​​of the fingerprint segments, wherein in the key-value pairs corresponding to the inverted index, the key stores the values ​​of the fingerprint segments, and the value stores text-related information, and the text-related information includes the fingerprint information of the text, the domain score, and the text number.

[0061] According to an embodiment of the present invention, the fingerprint information and the fingerprint segment settings can be set as needed. Usually, the fingerprint information is set to 64 bits, and each 16 bits is divided into a segment, that is, divided into 4 fingerprint segments. However, it should be understood that this is not necessary, and fingerprint information of other lengths can be set, and the fingerprint segment can be divided into 3 segments, 4 segments, 5 segments, 6 segments, 7 segments, 8 segments, etc.; the specific length of the fingerprint information and the number of segments can be set as needed.

[0062] According to one embodiment of the present invention, see Figure 2 , assuming that the fingerprint information is divided into four fingerprint segments, namely H1, H2, H3 and H4. When constructing an inverted index, for each fingerprint segment, it is necessary to find out whether there is a key with the same value as the fingerprint segment; assuming that there are currently multiple keys, namely Key1, Key2, ..., Key n; among them, key key1=H1, then the text-related information is stored in the value corresponding to key key1=H1, including fingerprint information SimHash, domain score Score, and text number Doc_Id. If there are multiple values, they are stored in the form of a linked list (hereinafter referred to as a value linked list) (next points to the text-related information of the next text). Assuming that there is currently no key equal to H1, a new key is created for it. The schematic diagram of the inverted index constructed with 4 fingerprint segments of fingerprint information S0 is shown in Figure 4As shown, for the first fingerprint segment, S0 (text fingerprint information), 0.89 (text domain score), ABCDE (text number) are stored in the value linked list corresponding to the key H1. This embodiment has at least the following beneficial effects: in the inverted index constructed by the present invention, the key stores the value of the fingerprint segment, and the value stores the fingerprint information (SimHash value) and text-related information (including the text number, the text domain score and other key information). The mapping relationship of this structure can be used to reversely check the text details and avoid repeated calculations, so as to facilitate efficient recommendation of texts based on this structure; this structural form of storing fingerprint information in segments belongs to the method of using SimHash combined with the drawer principle to calculate text similarity, and can calculate fingerprint data in the data stream processing task with a time complexity that is independent of the number of documents; in the subsequent text recommendation, the fingerprint information that needs to be compared can be quickly reduced after a constant number of comparisons, thereby improving the recommendation efficiency.

[0063] The inverted index established by the above text aggregation method and the fingerprint information, domain score and text number of the text stored therein can efficiently perform text recommendation, and the recommended text is more in line with the business scenario (more adaptable to the domain).

[0064] According to one embodiment of the present invention, the present invention also provides a text recommendation method, including: obtaining input text and text serial number from a data stream; processing the input text using the text aggregation method of the aforementioned embodiment to construct an inverted index; searching for other texts similar to the text according to the inverted index to determine a similar cluster, wherein the fingerprint information of each text belonging to the similar cluster is stored in the similar cluster; determining the domain score of each text from the inverted index according to the fingerprint information of each text in the similar cluster corresponding to the input text; and recommending similar texts of the input text according to the domain score of each text in the similar cluster. This embodiment has at least the following beneficial effects: thanks to the adoption of the above-mentioned text aggregation method, the present invention can quickly determine similar clusters for texts input by the data stream and select matching texts from the similar clusters in real time for recommendation; due to the introduction of domain weights and domain scores, the recommended text has stronger domain adaptability in addition to having a certain similarity with the input text, thereby better realizing the text recommendation task; with the dynamic input of the data stream, the similar cluster can also be continuously and dynamically adjusted to better perform real-time recommendation for subsequent texts.

[0065] According to one embodiment of the present invention, the determination of similar clusters can be set as needed. For example, a similar cluster can be set to: a set consisting of texts corresponding to a value list found in an inverted index using each fingerprint segment of the corresponding text as a key; that is, any text recorded in a value associated with a key corresponding to any one of the four fingerprint segments of the corresponding text constitutes an element of a similar cluster of the corresponding text. In order to more accurately compare the similarity, a similar cluster can be set to: calculate the similarity between the fingerprint information of the text corresponding to a value list found in an inverted index using each fingerprint segment of the corresponding text as a key and the fingerprint information of the corresponding text, and add texts whose similarity reaches a predetermined threshold to a similar cluster. For example, assuming that the fingerprint information is 64 bits and the predetermined threshold is set to 62, then if the values ​​of 62 bits and above are consistent, it is considered that the similarity reaches the predetermined threshold. Similar fingerprint information Join to similar cluster .

[0066] According to an embodiment of the present invention, after determining the similar clusters of texts, the fingerprint information of each text in the similar cluster is recorded. Based on the fingerprint information of each text, the domain score of each text and the text number can be found from the inverted index, and the number of the text with the highest or top domain score is recommended. Fingerprint information The text with the highest domain score will be recommended. If the text number is recorded in the fingerprint information, the text number will be output.

[0067] Implementation 2:

[0068] This embodiment differs from the aforementioned embodiment 1 in that the text number is not obtained in step S1', the text-related information in step S4' does not save the text number, step S2' is consistent with step S2, and step S3' is consistent with step S3. Apart from this, the specific implementation details are the same as those in embodiment 1 or can refer to embodiment 1, and will not be elaborated later.

[0069] According to one embodiment of the present invention, the present invention also provides a text aggregation method, comprising: four steps S1', S2', S3', and S4', wherein step S1': obtain the text to be processed; step S2': use the improved SimHash algorithm to calculate the fingerprint information of the text, wherein, when the improved SimHash algorithm weights the hash value of the corresponding word in the text, the hash value of the corresponding word is multiplied by the weight of the word in the text and the domain weight of the word in the domain where the text is located; step S3': use the domain weight to score the domain relevance of the text to obtain the domain score of the text; step S4': divide the fingerprint information of the text into multiple fingerprint segments, and construct an inverted index based on the values ​​of the fingerprint segments, wherein, in the key-value pair corresponding to the inverted index, the key stores the value of the fingerprint segment, and the value stores text-related information, and the text-related information includes the fingerprint information and the domain score of the text.

[0070] According to an embodiment of the present invention, the present invention also provides a text recommendation method, including: obtaining input text from a data stream; processing the input text using the text aggregation method of the aforementioned embodiment in Implementation Method 2 to construct an inverted index; searching for other texts similar to the text according to the inverted index to determine a similar cluster, wherein the similar cluster stores fingerprint information of each text belonging to the similar cluster; determining the domain score of each text from the inverted index according to the fingerprint information of each text in the similar cluster corresponding to the input text; and recommending similar texts of the input text according to the domain score of each text in the similar cluster. The recommended similar texts may be fingerprint information of similar texts, and the numbers of similar texts are obtained by querying a pre-constructed comparison table of fingerprint information of each text and text numbers.

[0071] In addition, there are other feasible implementations, for example, changing step S1 in implementation 1 to: obtaining the input text and its address from the data stream, and changing the text-related information in step S4 in implementation 1 to: the text-related information includes the fingerprint information, domain score and address of the text. In the text recommendation method, when recommending similar texts of the input text, the address of the similar text stored in the text-related information can be directly fed back.

[0072] It should be noted that although the above describes the various steps in a specific order, it does not mean that the various steps must be executed in the above specific order. In fact, some of these steps can be executed concurrently or even in a different order as long as the required functions can be achieved.

[0073] The present invention may be a system, a method and / or a computer program product. The computer program product may include a computer-readable storage medium carrying computer-readable program instructions for causing a processor to implement various aspects of the present invention.

[0074] A computer-readable storage medium may be a tangible device that holds and stores instructions used by an instruction execution device. Computer-readable storage media may include, for example, but are not limited to, an electrical storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination thereof. More specific examples (non-exhaustive list) of computer-readable storage media include: a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a static random access memory (SRAM), a portable compact disk read-only memory (CD-ROM), a digital versatile disk (DVD), a memory stick, a floppy disk, a mechanical encoding device, such as a punch card or a protruding structure in a groove on which instructions are stored, and any suitable combination thereof.

[0075] The embodiments of the present invention have been described above, and the above description is exemplary, not exhaustive, and is not limited to the disclosed embodiments. Many modifications and variations will be apparent to those of ordinary skill in the art without departing from the scope and spirit of the described embodiments. The selection of terms used herein is intended to best explain the principles of the embodiments, practical applications, or technical improvements in the market, or to enable other persons of ordinary skill in the art to understand the embodiments disclosed herein.

Claims

1. A text aggregation method, characterized in that: include: Get the text to be processed; The fingerprint information of the text is calculated using the improved SimHash algorithm, wherein the improved SimHash algorithm uses the weight of the word in the text and the domain weight of the word in the field where the text is located to perform weighting on the hash value of the corresponding word in the text, wherein the domain weight of the word in each field is pre-calculated in the following manner: Obtain the domain corpus of the field, which includes multiple sample articles collected in advance in the field. Extract multiple key feature words from the domain corpus to form domain feature phrases. For each word in the domain characteristic phrase, the domain weight of the corresponding word is calculated based on the total number of words in the domain corpus, the total number of articles, the number of times the corresponding word appears in the domain corpus, and the number of articles in the domain corpus that contain the corresponding word; Use the domain weight to score the domain relevance of the text and obtain the domain score of the text; The fingerprint information of the text is divided into multiple fingerprint segments, and an inverted index is constructed based on the values ​​of the fingerprint segments. In the key-value pairs corresponding to the inverted index, the key stores the value of the fingerprint segment, and the value stores the text-related information, which includes the fingerprint information and the domain score of the text.

2. The text aggregation method according to claim 1, characterized in that: The domain weight of a word is calculated as follows: ； in, Expressing words The domain weight of Expressing words The number of times it appears in the domain corpus, Represents the total number of words in the domain corpus, represents the total number of articles in the domain corpus, It represents the number of articles containing the corresponding words in the domain corpus, and k represents the domain weight adjustment coefficient.

3. The text aggregation method according to claim 1, characterized in that: The step of calculating the fingerprint information of the text using the improved SimHash algorithm includes: Process the text and calculate the hash values ​​of multiple words included in it; Multiply the hash value of each word by the weight of the word in the text and the domain weight of the word in the domain where the text is located, to obtain the weighted hash value of each word; The weighted hash values ​​of each word at the same bit are accumulated and then reduced in dimension to form the fingerprint information of the text.

4. The text aggregation method according to claim 3, characterized in that: The domain score of a text is determined as follows: According to the entity dictionary corresponding to the field of the text, obtain the person entity information, location entity information, and organization entity information in the text, where the entity dictionary for each field can be set independently; According to the person entity information, location entity information, and organization entity information in the text, determine the number of different person entities, the number of different location entities, and the number of different organization entities in the text; The domain score of the text is determined based on the number of different person entities, the number of different location entities, the number of different organization entities in the text, the frequency of words in the text feature phrases, and the domain weights of the words in the text feature phrases.

5. The text aggregation method according to claim 4, characterized in that: The domain score of a text is determined as follows: ； in, Indicates the number of different character entities in the text, express The weight coefficient of Represents the number of different location entities in the text, express The weight coefficient of Indicates the number of different institutional entities in the text, express The weight coefficient of Represents the words in the text feature phrase The number of times it appears in the text, Represents the words in the text feature phrase The domain weight of Indicates the total number of words in the text.

6. The text aggregation method according to any one of claims 1 to 5, characterized in that: The method further comprises: Other texts similar to the text are searched according to the inverted index to determine a similar cluster, wherein the similar cluster stores fingerprint information of each text belonging to the similar cluster.

7. A text recommendation method, characterized in that: include: Get the input text from the data stream; Processing the input text using the text aggregation method described in any one of claims 1 to 6 to construct an inverted index; Searching for other texts similar to the text according to the inverted index to determine a similar cluster, wherein the similar cluster stores fingerprint information of each text belonging to the similar cluster; According to the fingerprint information of each text in the similar cluster corresponding to the input text, the domain score of each text is determined from the inverted index; Recommend similar texts to the input text based on the domain scores of each text in the similar cluster.

8. A computer-readable storage medium, characterized in that: A computer program is stored thereon, and the computer program can be executed by a processor to implement the steps of the method according to any one of claims 1 to 7.

9. An electronic device, characterized in that: include: one or more processors; as well as A memory, wherein the memory is used to store executable instructions; The one or more processors are configured to implement the steps of the method of any one of claims 1 to 7 by executing the executable instructions.

Images