Method and electronic device for calculating text similarity

Abstract

The present application relates to the field of natural language processing technology and provides a method for calculating text similarity, the method comprising: obtaining a first segmentation set and a second segmentation set, wherein the first segmentation set is a segmentation set obtained after segmentation of a first text, and the second segmentation set is a segmentation set obtained after segmentation of a second text; determining a first weighted sum of similarities between each segmentation set in the first segmentation set and each segmentation set in the second segmentation set based on a synonym dictionary; determining a first number of segmentations with different meanings in the first segmentation set and the second segmentation set based on the synonym dictionary; and determining a first similarity between the first text and the second text based on a first ratio of the first weighted sum to the first number, wherein the first similarity is positively correlated with the first ratio. This method can improve the accuracy of text clustering.

Description

Technical Field

[0001] The present application relates to the field of natural language processing technology, and in particular to a method and electronic device for calculating text similarity. Background Art

[0002] As we all know, text clustering algorithms are one of the important algorithms in the field of natural language processing and are often used in industrial scenarios such as similar news mining and hot public opinion discovery. For example, the K-means Clustering Algorithm (K-means) and the Density-Based Spatial Clustering of Applications with Noise (DBSCAN) algorithm both require converting the original text into vector representations before clustering to calculate the similarity between the texts. However, since vector representations occupy a large amount of memory space and take a long time to calculate, clustering efficiency is low. Currently, to improve the efficiency of text clustering, the Jaccard index is often used to compare the similarities and differences between limited sample sets. Although the Jaccard index does not require converting the original text into vector representations when calculating text similarity, it only considers identical words when calculating the similarity between texts and does not consider words with very similar semantics in two texts, which reduces the accuracy of text clustering.

[0003] Therefore, how to improve the accuracy of text clustering is an urgent problem that needs to be solved. Summary of the Invention

[0004] The present application provides a method and electronic device for calculating text similarity, which can improve the accuracy of text clustering.

[0005] In a first aspect, a method for calculating text similarity is provided, comprising: obtaining a first segmentation set and a second segmentation set, wherein the first segmentation set is a segmentation set obtained after segmentation processing of a first text, and the second segmentation set is a segmentation set obtained after segmentation processing of a second text; determining a first weighted sum of similarities between each segmentation set in the first segmentation set and each segmentation set in the second segmentation set according to a synonym dictionary; determining a first number of segmentations with different meanings in the first segmentation set and the second segmentation set according to the synonym dictionary; and determining a first similarity between the first text and the second text according to a first ratio of the first weighted sum to the first number, wherein the first similarity is positively correlated with the first ratio.

[0006] The above method can be executed by an electronic device or a chip in an electronic device. Compared with the prior art that only considers the same segmentation but not the semantically identical or similar segmentation, the present application not only considers the same segmentation (for example, today and today are the same segmentation) when calculating the similarity between two segmentation sets, but also takes into account synonyms or near-synonyms (for example, today and today are synonyms). This similarity calculation method that takes synonyms or near-synonyms into account can avoid the electronic device from incorrectly clustering texts with the same semantics but different wording, thereby improving the accuracy of text clustering.

[0007] Optionally, determining the first weighted sum of the similarities between each segmentation in the first segmentation set and each segmentation in the second segmentation set according to the synonym dictionary includes: determining the initial similarity between the first segmentation and the second segmentation according to the synonym dictionary, the first segmentation being any segmentation in the first segmentation set, and the second segmentation being any segmentation in the second segmentation set; determining the final similarity between the first segmentation and the second segmentation according to a first threshold and the initial similarity, wherein when the initial similarity is greater than the first threshold, the final similarity is equal to the initial similarity, and when the initial similarity is less than or equal to the first threshold, the final similarity is equal to 0; and determining the first weighted sum according to the final similarity.

[0008] In this embodiment, the initial similarity between the first participle and the second participle is first determined based on the synonym dictionary, and then the degree of similarity between the first participle and the second participle is judged based on the size of the initial similarity. For example, when the initial similarity is greater than the first threshold, it means that the similarity between the first participle and the second participle is very high. At this time, the final similarity is equal to the initial similarity. When the initial similarity is less than or equal to the first threshold, it means that the similarity between the first participle and the second participle is very small. When calculating the first weighted sum, it is not necessary to consider it (that is, the final similarity is equal to 0). This method of screening out participles with a high degree of similarity (for example, the first participle and the second participle) based on the size of the first threshold and the initial similarity, without considering participles with very small similarity, can prevent participles with very small similarity from affecting the accuracy of the first weighted sum calculation result.

[0009] Optionally, multiple similarities between the second word segmentation set and multiple word segmentation sets are determined, the multiple word segmentation sets include the first word segmentation set, and the multiple similarities include the first similarity; when the first similarity is the maximum value among the multiple similarities, and when the first similarity is greater than a second threshold, the first word segmentation set is updated using the second word segmentation set; or, when the first similarity is the maximum value among the multiple similarities, and when the first similarity is less than or equal to a second threshold, a third word segmentation set is generated based on the second word segmentation set, and the third word segmentation set includes some or all of the words in the second word segmentation set.

[0010] In this embodiment, compared with the prior art that directly determines the segmentation set to which the second segmentation set belongs based on the maximum value, the present application sets a second threshold value to determine whether the second segmentation set belongs to multiple existing segmentation sets. For example, the second segmentation set is calculated for similarity with two segmentation sets (for example, segmentation set A and segmentation set B), wherein the similarity between the second segmentation set and segmentation set A is 0.12, and the similarity with segmentation set B is 0.23, wherein the similarity with segmentation set B is the maximum value of the two similarities (i.e., 0.23). Obviously, the similarity between the second segmentation set and segmentation set B is not high. If the second segmentation set is directly clustered into the text cluster corresponding to segmentation set B based on the maximum value of the similarity, it is obviously inaccurate. However, the present application determines whether clustering can be performed based on the maximum value by setting a second threshold value. For example, the second threshold value is 0.7. Obviously, the second segmentation set and segmentation set B do not belong to the same text cluster. Therefore, the second text corresponding to the second segmentation set should be regenerated into a new text cluster. It can be seen that the present application can improve the accuracy of text clustering by setting a second threshold to determine whether the second text corresponding to the second word segmentation set belongs to a text cluster corresponding to a word segmentation set in multiple word segmentation sets.

[0011] Optionally, using the second segmentation set to update the first segmentation set includes: adding M segmentations in the second segmentation set to the first segmentation set to generate a fourth segmentation set, the fourth segmentation set being used for clustering processing of the third text, the M segmentations being the M segmentations with the highest frequency of occurrence in the second text, and M being a positive integer.

[0012] In this embodiment, updating the M word segments in the second word segmentation set to the first word segmentation set can improve the accuracy of the main idea (or theme) of the text cluster represented by the updated first word segmentation set, and thus can improve the accuracy of the similarity calculation between the updated first word segmentation set and the newly added text to be clustered.

[0013] Optionally, before generating the fourth word segmentation set, the method further includes: deleting N word segmentations in the first word segmentation set, where the N word segmentations in the first word segmentation set are the N word segmentations with the lowest frequency of occurrence in the first text, and N is a positive integer.

[0014] In this embodiment, when the second segmentation set is used to update the first segmentation set, all the segmentations in the second segmentation set and the first segmentation set are counted, and then the segmentations with the highest frequency of occurrence are screened out, and the N segmentations with lower frequency of occurrence in the previous first segmentation set are deleted, thereby ensuring that the number of segmentations in the updated first segmentation set (i.e., the fourth segmentation set) will not be too large, thereby avoiding the situation where the clustering efficiency is affected by the similarity calculation between the newly added text to be clustered (for example, the third text) and too many segmentations in the updated first segmentation set.

[0015] Optionally, the segmentations in the third segmentation set are K segmentations with the highest frequency of occurrence in the second text, where K is a positive integer.

[0016] In this embodiment, since the K most frequently appearing words can represent the main idea of ​​the second text, the K most frequently appearing words in the second text are used as the third word set, which is beneficial to improving the accuracy of text clustering performed by the electronic device based on the similarity calculation results between the newly added text to be clustered and the third word set.

[0017] Optionally, the first segmentation set is a segmentation set consisting of P segmentations with the highest frequency of occurrence in the first text, where P is a positive integer.

[0018] In this embodiment, the P segmented words with the highest frequency of occurrence can best represent the textual theme (or textual subject) that the first text wants to express. Using the P segmented words with the highest frequency of occurrence to form the first segmented word set is conducive to improving the accuracy of text clustering of the newly added text to be clustered and the first segmented word set.

[0019] Optionally, before obtaining the first word segmentation set and the second word segmentation set, the process includes: obtaining the first text and the second text; performing word segmentation preprocessing on the first text to obtain the first word segmentation set, wherein the word segmentation preprocessing includes word segmentation processing, part-of-speech screening processing, and synonym expansion processing; and performing the word segmentation preprocessing on the second text to obtain the second word segmentation set.

[0020] In this embodiment, the electronic device can not only accurately extract the key words that need to be extracted from the first text and the second text by performing word segmentation preprocessing on the first text and the second text, but also expand the extracted words into synonyms or near-synonyms, thereby facilitating the subsequent rapid determination of the similarity between the first text and the second text based on the synonym dictionary.

[0021] In a second aspect, an electronic device is provided, comprising a processor and a memory, wherein the memory is used to store a computer program, and the processor is used to call and run the computer program from the memory, so that the electronic device executes any one of the methods in the first aspect.

[0022] According to a third aspect, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the processor executes any one of the methods according to the first aspect.

[0023] For the beneficial effects of the second and third aspects of this application, refer to the beneficial effects of the first aspect. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following briefly introduces the drawings required for use in the embodiments or descriptions of the prior art. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0025] Figure 1 Schematic diagram of a method for calculating text similarity in an embodiment of the present invention;

[0026] Figure 2 A schematic diagram of the steps of a method for calculating text similarity in an embodiment of the present invention;

[0027] Figure 3 Schematic diagram of the structure of an electronic device in an embodiment of the present invention. DETAILED DESCRIPTION

[0028] In the following description, specific details such as specific system structures and techniques are provided for purposes of illustration rather than limitation to facilitate a thorough understanding of the embodiments of the present application. However, it will be apparent to those skilled in the art that the present application may be implemented in other embodiments without these specific details. In other cases, detailed descriptions of well-known systems, devices, and methods are omitted to avoid obscuring the description of the present application with unnecessary detail.

[0029] It should be understood that when used in the present specification and the appended claims, the term "comprising" indicates the presence of described features, integers, steps, operations, elements and / or components, but does not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components and / or collections thereof.

[0030] It will also be understood that the term "and / or" used in this specification and the appended claims refers to and includes any and all possible combinations of one or more of the associated listed items.

[0031] In addition, in the description of the present application specification and the appended claims, the terms "first", "second", "third", etc. are only used to distinguish the descriptions and cannot be understood as indicating or implying relative importance.

[0032] References to "one embodiment" or "some embodiments" in this specification mean that a particular feature, structure, or characteristic described in conjunction with that embodiment is included in one or more embodiments of the present application. Therefore, phrases such as "in one embodiment," "in some embodiments," "in other embodiments," and "in other embodiments" appearing in various places in this specification do not necessarily refer to the same embodiment, but rather mean "one or more but not all embodiments," unless otherwise specifically emphasized. The terms "including," "comprising," "having," and variations thereof all mean "including but not limited to," unless otherwise specifically emphasized.

[0033] In order to improve the accuracy of text clustering, this application proposes a method for calculating text similarity. This application will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0034] Figure 1 This is a flowchart of a method for calculating text similarity proposed in this application. The method can be executed by an electronic device or a chip on an electronic device. The method includes:

[0035] S101, obtaining a first segmentation set and a second segmentation set, wherein the first segmentation set is a segmentation set obtained after segmentation processing of a first text, and the second segmentation set is a segmentation set obtained after segmentation processing of a second text.

[0036] Exemplarily, before obtaining the first segmentation set and the second segmentation set, the electronic device first obtains the first text and the second text; then, the electronic device performs segmentation preprocessing on the first text to obtain the first segmentation set; the segmentation preprocessing includes but is not limited to segmentation processing, part-of-speech screening processing and synonym expansion processing, wherein the segmentation processing is used to divide the first text into multiple segmentations, and the tool for segmentation processing includes but is not limited to using the cut() method in the jieba library in Python; the above-mentioned multiple segmentations include but are not limited to punctuation marks, numerals, conjunctions, prepositions, nouns, verbs, adjectives, adverbs, names of people, places and organizations and groups.

[0037] Part-of-speech filtering is used to filter the word segmentation results. For example, only meaningful words that represent the main idea of ​​the text are retained, while excluding punctuation, numerals, conjunctions, and prepositions. In practice, nouns, verbs, adjectives, adverbs, names of people, places, organizations, and other types of words are typically retained. Tools that can be used for part-of-speech filtering include, but are not limited to, the posseg() method in Jieba.

[0038] Synonym expansion processing is used to expand the results after part-of-speech filtering processing into synonyms or near-synonyms. For example, the segmentation words after segmentation filtering processing are today, see, and very; synonym expansion processing is to expand today, see, and very into synonyms. The synonyms of today include: today, today, this day, and this day; the synonyms of see include: see, see, look at, and see; the synonyms of very include: very, extremely, special, and exceptionally; segmentation expansion of the results after part-of-speech filtering processing is conducive to quickly determining which segmentations are synonyms when calculating the similarity between texts, thereby improving the efficiency of text clustering.

[0039] For example, the content of the first text is: Pineapple is native to South America and is one of the famous tropical fruits; after the first text is segmented, a first segmentation processing result is obtained; the first segmentation processing result is "Pineapple / origin / South America / , / is / one of / the / famous / tropical / fruits / ."; the part-of-speech filtering processing performs part-of-speech filtering on the first segmentation processing result to obtain a first part-of-speech filtering result; the first part-of-speech filtering result is "Pineapple / origin / South America / famous / tropical / fruit"; the first part-of-speech filtering result only retains the nouns (e.g., pineapple) and verbs (e.g., origin) in the first text, and removes punctuation marks (e.g., periods), prepositions (e.g., for), etc. The synonym expansion processing performs synonym expansion on the segmentations in the first part-of-speech filtering result, for example, synonyms for famous are famous, well-known, and renowned.

[0040] Similarly, similar to the way the first text is preprocessed for word segmentation, the electronic device preprocesses the second text for word segmentation to obtain a second word segmentation set. The specific processing process will not be repeated here, and reference may be made to the process of preprocessing the first text for word segmentation.

[0041] In this embodiment, the electronic device can not only accurately extract the key words that need to be extracted from the first text and the second text by performing word segmentation preprocessing on the first text and the second text, but also expand the extracted words into synonyms or near-synonyms, thereby facilitating the subsequent rapid determination of the similarity between the first text and the second text based on the synonym dictionary.

[0042] S102 : Determine a first weighted sum of similarities between each segmentation in the first segmentation set and each segmentation in the second segmentation set according to a synonym dictionary.

[0043] For example, the aforementioned synonym dictionary can be trained independently or from a publicly available synonym dictionary. For example, the synonym dictionary can use the synonyms library in Python. The synonyms library can, based on an input Chinese word A, return multiple semantically similar words corresponding to the Chinese word A and provide the similarity between the Chinese word A and each of the multiple semantically similar words. For example, if the electronic device receives the Chinese word "famous," after calling the synonyms library, the returned results are [the similarity between famous and famous is 1.0, the similarity between famous and famous is 0.90, the similarity between famous and famous is 0.87, the similarity between famous and famous is 0.76, ...].

[0044] After the electronic device determines the similarity between each segmentation word in the first segmentation set and each segmentation word in the second segmentation word set based on the synonym dictionary, it calculates a weighted sum of the similarities of the individual segmentation words. For example, if the first segmentation word set is {today, I, shopping} and the second segmentation word set is {today, I, off duty}, after the electronic device enters "today" into the synonym dictionary, the synonym dictionary outputs that the similarity between "today and today" is 0.95, the similarity between "today and I" is 0, and the similarity between "today and off duty" is 0. After the electronic device enters "I" into the synonym dictionary, the synonym dictionary outputs that the similarity between "I and today" is 0, the similarity between "I and I" is 1.0, and the similarity between "I and off duty" is 0. After the electronic device enters "shopping" into the synonym dictionary, the synonym dictionary outputs that the similarity between "shopping and today" is 0, the similarity between "shopping and I" is 0, and the similarity between "shopping and off duty" is 0.32. A first weighted sum of similarities between each word in the first word segmentation set and each word in the second word segmentation set is 0.95+0+0+0+1.0+0+0+0+0+0.32=2.27.

[0045] Exemplarily, the initial similarity between the first participle and the second participle is determined based on a synonym dictionary, where the first participle is any participle in the first participle set, and the second participle is any participle in the second participle set; the final similarity between the first participle and the second participle is determined based on a first threshold and the initial similarity, wherein when the initial similarity is greater than the first threshold, the final similarity is equal to the initial similarity, and when the initial similarity is less than or equal to the first threshold, the final similarity is equal to 0; and the first weighted sum is determined based on the final similarity.

[0046] In order to consider only words with the same or similar semantics when calculating the first weighted sum, and not words with completely different semantics, the electronic device sets a first threshold to screen the similarities between each word in the first word set and each word in the second word set.

[0047] The electronic device determines the initial similarity between each segmentation in the first segmentation set and each segmentation in the second segmentation set based on the synonym dictionary. When the initial similarity between the two segmentations in the synonym dictionary is greater than a first threshold, the initial similarity is equal to the final similarity; when the initial similarity between the two segmentations in the synonym dictionary is less than or equal to the first threshold, the final similarity is 0. The calculation formula of the above final similarity is shown in formula (1):

[0048]

[0049] Where, Sim′(A i ,B j ) is the final similarity, C(A i ,B j ) is the initial similarity, h c is the first threshold.

[0050] For example, the first threshold h c is 0.7, the first participle set is {scenery, very, beautiful}, the first participle is scenery, the third participle is very, the fourth participle is beautiful; the second participle set is {scenery, general, charming}, the second participle is scenery, the fifth participle is general, and the sixth participle is charming; after the electronic device inputs "scenery" into the synonym dictionary, the synonym dictionary outputs the initial similarity of "scenery and scenery" (that is, the initial similarity of the first participle and the second participle) as 0.95, the initial similarity of "scenery and general" is 0, and " The initial similarity between "scenery" and "charming" is 0; after the electronic device inputs "very" into the synonym dictionary, the synonym dictionary outputs that the initial similarity between "very" and "scenery" is 0, the initial similarity between "very" and "general" is 0.47, and the initial similarity between "very" and "charming" is 0; after the electronic device inputs "beautiful" into the synonym dictionary, the synonym dictionary outputs that the initial similarity between "beautiful" and "scenery" is 0, the initial similarity between "beautiful" and "general" is 0.25, and the initial similarity between "beautiful" and "charming" is 0.87.

[0051] Since the initial similarity of "scenery and scenery" (i.e. the initial similarity between the first participle and the second participle) is 0.95 (greater than the first threshold 0.7), the final similarity of "scenery and scenery" (i.e. the final similarity between the first participle and the second participle) is 0.95 (equal to the initial similarity); since the initial similarity of "scenery and general" is 0 (less than the first threshold 0.7), the final similarity of "scenery and general" is 0; similarly, the final similarity of "scenery and charming" is 0; the final similarity of "ten and scenery" is 0, because "ten" The initial similarity of "excellent and average" is 0.47 (less than the first threshold of 0.7), so the final similarity of "very good and average" is 0. Similarly, the final similarity of "very good and charming" is 0, and the final similarity of "beautiful and scenery" is 0; since the initial similarity of "beautiful and average" is 0.25 (less than the first threshold of 0.7), the final similarity of "beautiful and average" is 0; since the initial similarity of "beautiful and charming" is 0.87 (greater than the first threshold of 0.7), the final similarity of "beautiful and charming" is 0.87. The first weighted sum is determined based on the final similarities between each participle in the first participle set and each participle in the second participle set. The first weighted sum is 1.82 (i.e., 0.95+0.87).

[0052] Thus, in this embodiment, the initial similarity between the first participle and the second participle is first determined according to the synonym dictionary, and then the degree of similarity between the first participle and the second participle is judged according to the size of the initial similarity. For example, when the initial similarity is greater than the first threshold value, it means that the degree of similarity between the first participle and the second participle is very high. At this time, the final similarity is equal to the initial similarity. When the initial similarity is less than or equal to the first threshold value, it means that the degree of similarity between the first participle and the second participle is very small. When calculating the first weighted sum, it is not necessary to consider it (that is, the final similarity is equal to 0). This method of screening out participles with a high degree of similarity (for example, the first participle and the second participle) according to the size of the first threshold value and the initial similarity, without considering participles with very small similarity, can avoid the participles with very small similarity affecting the accuracy of the first weighted sum calculation result.

[0053] S103: Determine a first number of segmentations with different meanings in the first segmentation set and the second segmentation set according to a synonym dictionary.

[0054] Exemplarily, the segmentations with different meanings in the first segmentation set and the second segmentation set refer to the two segmentations having semantics that are neither the same nor similar, that is, when the similarity between the two segmentations is lower than the first threshold, it is determined that the semantics between the two segmentations are neither the same nor similar. For example, the first segmentation set is {scenery, very, beautiful}, and the second segmentation set is {scenery, general, charming}. From the above analysis, we can see that the initial similarity of "scenery and scenery" is 0.95 (greater than the first threshold of 0.7); the initial similarity of "scenery and general" is 0 (less than the first threshold of 0.7); similarly, the initial similarity of "scenery and charming" is 0; the initial similarity of "very and scenery" is 0, the initial similarity of "very and general" is 0.47 (less than the first threshold of 0.7), the initial similarity of "very and charming" is 0, the initial similarity of "beautiful and scenery" is 0; the initial similarity of "beautiful and general" is 0.25 (less than the first threshold of 0.7); the initial similarity of "beautiful and charming" is 0.87 (greater than the first threshold of 0.7). The first weighted sum is determined based on the final similarity between each segmentation in the first segmentation set and each segmentation in the second segmentation set. The first weighted sum is 1.82 (i.e., 0.95+0.87). According to the initial similarity between the two participles, it can be judged that the first number of participles with different meanings in the first participle set and the second participle set is 7, namely, scenery and general, scenery and charming, very and scenery, very and general, very and charming, beautiful and scenery, beautiful and general.

[0055] S104 : Determine a first similarity between the first text and the second text according to a first ratio of the first weighted sum to the first quantity, where the first similarity is positively correlated with the first ratio.

[0056] Exemplarily, the electronic device determines the similarity between each segmentation in the first segmentation set and each segmentation in the second segmentation set based on the synonym dictionary. For example, the first segmentation set corresponding to the first text is segmentation set A, and the first segmentation set corresponding to the second text is segmentation set B. The similarity between each segmentation in segmentation set A and segmentation set B is Sim(A i ,B j ), where A i is the i-th word in the word set A, B j is the jth segmentation in segmentation set B; the electronic device determines the similarity D(A, B) (i.e., the first similarity) between segmentation set A and segmentation set B according to the synonym dictionary as shown in formula (2):

[0057]

[0058] Where I is the total number of segmentations in the first segmentation set, J is the total number of segmentations in the first segmentation set, ∑ i∈[01I),j∈[0,J) Sim(A i ,Bj ) is the first weighted sum of the similarities between each segmentation in segmentation set A and each segmentation in segmentation set B, and A∪B is the total number of segmentations with different semantics in the first segmentation set and the second segmentation set (i.e., the first number). Optionally, the electronic device determines the similarity D(A, B) (i.e., the first similarity) between segmentation set A and segmentation set B based on a synonym dictionary as shown in formula (3):

[0059]

[0060] For example, the first weighted sum of the first word segmentation set and the second word segmentation set is 6.6, the first number of word segmentations with different meanings in the first word segmentation set and the second word segmentation set is 13, and the first ratio of the first weighted sum to the first number is 0.51 (i.e., 6.6 divided by 13). That is, the first similarity between the first text corresponding to the first word segmentation set and the second text corresponding to the second word segmentation set is 0.51. The above first similarity formula can also be expressed as formula (4):

[0061]

[0062] It can be seen from formula (4) that the first similarity D(A, B) is positively correlated with the first ratio, where k is a positive real number.

[0063] Exemplarily, the above method also includes: determining multiple similarities between the second word segmentation set and multiple word segmentation sets, the multiple word segmentation sets include the first word segmentation set, and the multiple similarities include the first similarity; when the first similarity is the maximum value among the multiple similarities, and when the first similarity is greater than a second threshold, using the second word segmentation set to update the first word segmentation set; or, when the first similarity is the maximum value among the multiple similarities, and when the first similarity is less than or equal to the second threshold, generating a third word segmentation set based on the second word segmentation set, the third word segmentation set containing some or all of the words in the second word segmentation set.

[0064] The above-mentioned second threshold refers to the minimum similarity value at which two word segmentation sets can be clustered. The above-mentioned multiple word segmentation sets include previously clustered word segmentation sets (i.e., multiple existing word segmentation sets). For example, the previous 100 texts were formed into 10 word segmentation sets (i.e., 10 cluster centers) after text clustering. Each word segmentation set has multiple key word segmentations, and these multiple key word segmentations constitute the cluster center of each word segmentation set; these multiple key word segmentations are also called cluster center words; each cluster center represents a text cluster, and each text cluster includes multiple texts with the same or similar text themes (or topics). For example, the first word segmentation set can be one of the 10 cluster centers. For the newly added text to be clustered (e.g., the third text), the similarity calculation will be performed with each of the 10 cluster centers, without the need to perform similarity calculation with each text in the text cluster corresponding to each cluster center. This not only reduces the amount of calculation but also improves the clustering speed.

[0065] Of course, the multiple word segmentation sets can also be multiple corresponding word segmentation sets generated after multiple texts undergo the above word segmentation preprocessing. The newly added text to be clustered (for example, the third text) will be similar to these multiple corresponding word segmentation sets. For example, the first word segmentation set mentioned above is one of the multiple word segmentation sets.

[0066] The second segmentation set is a newly added segmentation set to be clustered. Since the method for calculating the first similarity between the first segmentation set (i.e., the segmentation set generated after the first text undergoes the segmentation preprocessing described above) and the second segmentation set has been described in detail above, here, taking the multiple segmentation sets as previously clustered segmentation sets (i.e., the multiple existing cluster centers) as an example, the calculation process of clustering the newly added second segmentation set to be clustered with the multiple segmentation sets (i.e., the multiple cluster centers) is described.

[0067] For example, the second threshold is 0.6, and there are three previously clustered word sets (i.e., multiple cluster centers), namely, word set A, word set B, and word set C (i.e., the first word set), where word set A is {fruit, apple, nutritional value}, word set B is {tourism, attractions, travel guide}, and word set C is {food, characteristics, local customs}; the second word set to be clustered is {Cantonese cuisine, authentic, worth tasting}; the second word set and word set The similarity X1 of A is 0.09, the similarity X2 between the second word segmentation set and word segmentation set B is 0.34, and the similarity X3 (i.e., the first similarity) between the second word segmentation set and word segmentation set C is 0.83. Among the similarities X1, X2, and X3, the one with the greatest similarity is similarity X3. The electronic device compares the similarity X3 with the second threshold. Obviously, the similarity X3=0.83 is greater than the second threshold 0.6. Therefore, the second word segmentation set is finally clustered to word segmentation set C.

[0068] It can be seen that compared with the prior art, which directly determines the segmentation set to which the second segmentation set belongs based on the maximum value, the present application sets a second threshold value to determine whether the second segmentation set belongs to multiple existing segmentation sets. For example, the second segmentation set and two segmentation sets (for example, segmentation set A and segmentation set B) are respectively similar. The similarity between the second segmentation set and segmentation set A is 0.12, and the similarity with segmentation set B is 0.23. The similarity with segmentation set B is the maximum value of the two similarities (i.e. 0.23). Obviously, the similarity between the second segmentation set and segmentation set B is not high. If the second segmentation set is directly clustered into the text cluster corresponding to segmentation set B based on the maximum value of the similarity, it is obviously inaccurate. The present application determines whether clustering can be performed based on the maximum value by setting a second threshold value. For example, the second threshold value is 0.7. Obviously, the second segmentation set and segmentation set B do not belong to the same text cluster. Therefore, the second text corresponding to the second segmentation set should be regenerated into a new text cluster. It can be seen that the present application can improve the accuracy of text clustering by setting a second threshold to determine whether the second text corresponding to the second word segmentation set belongs to a text cluster corresponding to a word segmentation set in multiple word segmentation sets.

[0069] In order to facilitate accurate clustering of subsequently added texts, at this time, the electronic device will update the word segmentation set C (ie, the first word segmentation set) with the second word segmentation set, so that the first word segmentation set can maintain the latest status in real time.

[0070] Optionally, the electronic device uses the second word segmentation set to update the first word segmentation set, including: adding M word segmentations in the second word segmentation set to the first word segmentation set to generate a fourth word segmentation set, the fourth word segmentation set is used for clustering processing of the third text, the M word segmentations are the M word segmentations with the highest frequency of occurrence in the second text, and M is a positive integer.

[0071] For example, the first segmentation set is {N1, N2, N3}, the second segmentation set is {S1, S2, S3, S4, S5}, and the electronic device counts the frequency of each segmentation in the second segmentation set in the second text corresponding to the second segmentation set, where S1 appears 40 times, S2 appears 65 times, S3 appears 76 times, S4 appears 21 times, and S5 appears 18 times; the electronic device adds the M=2 segmentations (i.e., segmentation S2 and segmentation S3) with the highest frequency in the second text to the first segmentation set to obtain a fourth segmentation set; the fourth segmentation set is {N1, N2, N3, S2, S3}. When the electronic device receives the third text to be clustered, it clusters the segmentation set after processing the third text with the fourth segmentation set.

[0072] It can be seen that in this embodiment, updating the M word segmentations in the second word segmentation set to the first word segmentation set can improve the accuracy of the main idea (or theme) of the text cluster represented by the updated first word segmentation set, and thus can improve the accuracy of the similarity calculation between the updated first word segmentation set and the newly added text to be clustered.

[0073] Optionally, before generating the fourth word segmentation set, the electronic device further includes: deleting N word segmentations in the first word segmentation set, where the N word segmentations in the first word segmentation set are the N word segmentations with the lowest frequency of occurrence in the first text, and N is a positive integer.

[0074] For example, the first word segmentation set is {W1, W2, W3, W4, W5, W6}, and the second word segmentation set is {T1, T2, T3, T4}. The electronic device counts the frequency of each word in the second word segmentation set appearing in the second text corresponding to the second word segmentation set, among which the frequency of T1 appearing is 36 times, the frequency of T2 appearing is 19 times, the frequency of T3 appearing is 66 times, and the frequency of T4 appearing is 12 times. Before the electronic device adds the M=2 word segmentations (i.e., word segmentation T1 and word segmentation T3) with the highest frequency in the second text to the first word segmentation set to obtain the fourth word segmentation set, the electronic device will add the first word segmentation set to the second text. Some relatively low-frequency segmentations in a segmentation set are deleted. For example, the frequencies of W1 and W5 are much lower than those of the other four segmentations. In this case, the electronic device can delete W1 and W5 from the first segmentation set to obtain a first segmentation set {W2, W3, W4, W6} after the segmentations are deleted. At this time, the electronic device adds the M=2 segmentations (i.e., segmentation T1 and segmentation T3) with the highest frequency in the second text to the first segmentation set {W2, W3, W4, W6} after the segmentations are deleted to obtain a fourth segmentation set; the fourth segmentation set is {T1, T3, W2, W3, W4, W6}. When the electronic device receives a third text to be clustered, it clusters the segmentation set after the third text is processed with the fourth segmentation set.

[0075] Optionally, before the electronic device adds the M=2 most frequently appearing words (i.e., word segmentation T1 and word segmentation T3) in the second text to the first word segmentation set to obtain the fourth word segmentation set (i.e., before the electronic device updates the first word segmentation set with the second word segmentation set), it will count all the word segmentations in the second word segmentation set and the first word segmentation set, and then filter out the word segmentations with the highest frequency, and delete the N word segmentations with lower frequency in the previous first word segmentation set, so as to ensure that the number of word segmentations in the updated first word segmentation set (i.e., the fourth word segmentation set) will not be too large, thereby avoiding the situation where the clustering efficiency is affected by similarity calculation between the newly added text to be clustered (for example, the third text) and too many word segmentations in the updated first word segmentation set.

[0076] Optionally, after generating the fourth segmentation set, the electronic device can also delete the segmentations with lower frequency in the fourth segmentation set. For example, the electronic device adds the M=2 segmentations with the highest frequency in the second text (i.e., segmentation T1 and segmentation T3) to the first segmentation set {W1, W2, W3, W4, W5, W6} to obtain the fourth segmentation set; the fourth segmentation set is {W1, W2, W3, W4, W5, W6, T1, T3}; at this time, the electronic device sorts the frequency of occurrence of each segmentation in the fourth segmentation set, deletes the N segmentations with lower frequency, and only retains the segmentations with higher frequency in the fourth segmentation set. These segmentations with higher frequency (i.e., key segmentations) can represent the text theme represented by the text cluster. It can be seen that this method can not only avoid the situation where the number of segmentations in the fourth segmentation set increases as the number of clustering increases, resulting in an excessive number of segmentations, but also avoid the situation where too many segmentations are used for similarity calculation and affect the clustering efficiency.

[0077] Exemplarily, the first word segmentation set is a word segmentation set consisting of P word segmentations with the highest frequency of occurrence in the first text, where P is a positive integer.

[0078] The segmentations in the above-mentioned first segmentation set are a segmentation set consisting of the P segmentations that appear most frequently in the first text. Since the P segmentations that appear most frequently can represent the main idea to be expressed by the first text, the P segmentations that appear most frequently in the first text are used as the first segmentation set; at this time, the first segmentation set is called the cluster center of the first text, and the segmentations in the first segmentation set are the central words of the cluster center of the first text.

[0079] The second segmented word set corresponding to the newly added second text to be clustered can be similarly calculated with the cluster center of the first text to determine whether the second text can be clustered under the relevant text theme (or text theme) reflected by the first text. For example, the second segmented word set corresponding to the newly added second text to be clustered is {today, school, open, parent-teacher meeting}, and the cluster center of the first text (i.e., the first segmented word set) is {education, school, exam}; the electronic device uses the aforementioned method to calculate the first similarity H between the second segmented word set and the cluster center of the first text, and determines that the first similarity H is greater than a second threshold, and clusters the second segmented word set into the first segmented word set.

[0080] For another example, the second threshold is 0.6, and there are three previously clustered word sets, namely, word set A (i.e., the first word set), word set B, and word set C. Among them, word set A is {fruit, apple, nutritional value}, word set B is {tourism, attractions, travel guide}, and word set C is {food, characteristics, local customs}; the second word set to be clustered is {vegetables, stir-fried, delicious}; the similarity X4 between the second word set and word set A is 0.07, the similarity X5 between the second word set and word set B is 0.14, and the similarity X5 between the second word set and word set C is 0. Similarity X6 (i.e., the first similarity) is 0.46. Among similarities X4, X5, and X6, similarity X6 has the greatest similarity. The electronic device compares similarity X6 with the second threshold. Obviously, similarity X6=0.46 is less than the second threshold 0.6, indicating that the second word segmentation set and word segmentation set A (i.e., the first word segmentation set), the second word segmentation set and word segmentation set B, and the second word segmentation set and word segmentation set C are not texts of the same type or similar topics. Therefore, the electronic device will regenerate a new type (or new topic) of the second word segmentation set (i.e., the third word segmentation set) from the second word segmentation set.

[0081] In this embodiment, the P segmented words with the highest frequency of occurrence can best represent the textual purpose (or textual theme) that the first text wants to express. Using the P segmented words with the highest frequency of occurrence to form the first segmented word set is conducive to improving the accuracy of text clustering of the newly added text to be clustered and the first segmented word set.

[0082] The third word segmentation set is used for clustering processing of a third text to be clustered; the third word segmentation set includes some or all of the words in the second word segmentation set.

[0083] For example, when the third segmentation set includes all the segmentations in the second segmentation set, the electronic device calculates the similarity between the segmentation set corresponding to the third text and the third segmentation set based on the synonym dictionary, thereby obtaining the similarity between the segmentation set corresponding to the third text and the third segmentation set. This third segmentation set including all the segmentations in the second segmentation set is conducive to accurately clustering the third text to be clustered.

[0084] For another example, when the third segmentation set contains all the segmentations in the second segmentation set, the segmentation set corresponding to the third text and the segmentation set are used to calculate the similarity between the segmentations. This will result in slow similarity calculation due to the excessive number of segmentations in the third segmentation set, thereby affecting the clustering efficiency. At this time, without affecting the accuracy of the clustering results, the third segmentation set can select some segmentations in the second segmentation set as the segmentations in the third set, that is, some segmentations in the second segmentation set are used as the cluster centers of the second text corresponding to the second segmentation set. In this way, the amount of computation required by the electronic device in calculating the similarity between the segmentation set corresponding to the third text and the segmentations in the third segmentation set can be reduced, thereby improving the clustering efficiency.

[0085] Optionally, the segmentations in the third segmentation set are the K segmentations with the highest frequency in the second text, where K is a positive integer. The electronic device performs segmentation statistics on the segmentation set after segmentation preprocessing of the second text, and counts the K segmentations with the highest frequency in the second text. These K segmentations can reflect the text theme or text subject to be expressed by the second text; the electronic device selects the K segmentations with the highest frequency in the second text as the segmentations in the third segmentation set, that is, the cluster center segmentations of the text cluster corresponding to the third segmentation set, wherein the text cluster corresponding to the third segmentation set contains the second text. It can be seen that in this embodiment, since the K segmentations with the highest frequency can represent the theme to be expressed by the second text, using the K segmentations with the highest frequency in the second text as the third segmentation set is conducive to improving the accuracy of the electronic device in performing text clustering based on the similarity calculation results between the newly added text to be clustered and the third segmentation set.

[0086] In summary, the above method can be executed by an electronic device or a chip in an electronic device. Compared with the prior art that only considers the same segmentation without considering the semantically identical or similar segmentation, the present application not only considers the same segmentation (for example, today and today are the same segmentation) when calculating the similarity between two segmentation sets, but also takes into account synonyms or near-synonyms (for example, today and today are synonyms). This similarity calculation method that takes synonyms or near-synonyms into account can avoid the electronic device from incorrectly clustering texts with the same semantics but different wording, thereby improving the accuracy of text clustering.

[0087] For ease of understanding, the following Figure 2 The method for calculating text similarity proposed in this application is further explained.

[0088] (1) The electronic device performs word segmentation processing, part-of-speech screening, and synonym expansion on all texts in the existing text set (including the first text). For example, taking the first text in the existing text set as an example, the electronic device uses multiple keywords in the first text (i.e., the most frequently occurring words) as the cluster center of the first text, and the subsequent texts to be clustered (e.g., the second text) can be similarity calculated with the cluster center of the first text. After the electronic device clusters all texts in the existing text set, multiple cluster centers (e.g., multiple word sets) are formed.

[0089] (2) The electronic device receives the newly added second text to be clustered and the second threshold, and performs word segmentation preprocessing on the second text to obtain a set of the second text after word segmentation (for example, a second word segmentation set).

[0090] (3) The electronic device calculates the similarity between the set of the second text after word segmentation and the multiple cluster centers to obtain multiple similarities.

[0091] (4) The electronic device determines the maximum similarity between the second text segmented set and the multiple cluster centers based on the multiple similarities. For example, the second text segmented set has the maximum similarity with cluster center F among the multiple cluster centers.

[0092] (5) The electronic device determines whether the maximum value is greater than a second threshold value; when the maximum value is greater than the second threshold value, the second text is classified into the current cluster (i.e., the text cluster corresponding to cluster center F); and the current cluster center (i.e., cluster center F) is updated based on the second text. When the maximum value is less than or equal to the second threshold value, a new cluster (i.e., a new text cluster) is generated based on the second text; a new cluster center is generated using the new cluster, and the new cluster center is used to update the existing multiple cluster centers (i.e., an additional cluster center is added based on the existing multiple cluster centers).

[0093] (6) Because the electronic device often receives multiple texts to be clustered when clustering newly added texts (i.e., newly added texts), the clustering is considered complete only when all of these multiple clustered texts are clustered. Otherwise, the electronic device will continue to calculate the similarity between the texts that have not been clustered and the multiple cluster centers until all the texts to be clustered are clustered.

[0094] Figure 3 A schematic structural diagram of an electronic device provided by the present application is shown. Figure 3 The dashed line in the figure indicates that the unit or module is optional. The electronic device 300 can be used to implement the method described in the above method embodiment. The electronic device 300 can be a server or a chip.

[0095] The electronic device 300 includes one or more processors 301, which can support the electronic device 300 to implement Figure 1The method in the corresponding method embodiment. The processor 301 can be a general-purpose processor or a special-purpose processor. For example, the processor 301 can be a central processing unit (CPU). The CPU can be used to control the electronic device 300, execute software programs, and process data of the software programs. The electronic device 300 can also include a communication unit 305 to implement signal input (reception) and output (transmission).

[0096] For example, the electronic device 300 may be a chip, and the communication unit 305 may be an input and / or output circuit of the chip, or the communication unit 305 may be a communication interface of the chip, and the chip may be a component of the electronic device.

[0097] For another example, the communication unit 305 may be a transceiver of the electronic device 300 , or the communication unit 305 may be a transceiver circuit of the electronic device 300 .

[0098] The electronic device 300 may include one or more memories 302, on which a program 304 is stored. The program 304 can be executed by the processor 301 to generate instructions 303, so that the processor 301 performs the method described in the above method embodiment according to the instructions 303. Optionally, data can also be stored in the memory 302. Optionally, the processor 301 can also read data stored in the memory 302. The data can be stored at the same storage address as the program 304, or the data can be stored at a different storage address than the program 304.

[0099] The processor 301 and the memory 302 may be provided separately or integrated together, for example, integrated on a system on chip (SOC) of an electronic device.

[0100] The specific manner in which the processor 301 executes the method for calculating text similarity can refer to the relevant description in the method embodiment.

[0101] It should be understood that each step of the above method embodiment can be completed by hardware logic circuits or software instructions in the processor 301. The processor 301 can be a CPU, a digital signal processor (DSP), a field programmable gate array (FPGA), or other programmable logic devices, such as discrete gates, transistor logic devices, or discrete hardware components.

[0102] The present application also provides a computer program product, which, when executed by the processor 301, implements the method of any method embodiment in the present application.

[0103] The computer program product may be stored in the memory 302 , for example, a program 304 . The program 304 is converted into an executable target file that can be executed by the processor 301 after undergoing processes such as preprocessing, compilation, assembly, and linking.

[0104] The present application also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a computer, implements any method embodiment of the present application. The computer program may be a high-level language program or an executable target program.

[0105] The computer-readable storage medium is, for example, memory 302. Memory 302 may be a volatile memory or a non-volatile memory, or may include both volatile and non-volatile memory. The non-volatile memory may be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. The volatile memory may be a random access memory (RAM), which is used as an external cache. By way of example and not limitation, many forms of RAM are available, such as static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDR SDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link dynamic random access memory (SynchLink DRAM, SLDRAM), and direct RAM bus random access memory (DRRAM).

[0106] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the above-described devices and equipment and the technical effects produced can refer to the corresponding processes and technical effects in the aforementioned method embodiments, and will not be repeated here.

[0107] In the several embodiments provided in this application, the disclosed systems, devices, and methods can be implemented in other ways. For example, some features of the method embodiments described above can be ignored or not executed. The device embodiments described above are merely schematic, and the division of units is merely a logical function division. In actual implementation, there may be other division methods, and multiple units or components may be combined or integrated into another system. In addition, the coupling between the units or the coupling between the components may be direct coupling or indirect coupling, and the above coupling includes electrical, mechanical or other forms of connection.

[0108] The above embodiments are only used to illustrate the technical solutions of the present application, and are not intended to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some of the technical features therein with equivalents, and such modifications or replacements do not deviate from the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the various embodiments of the present application, and should be included in the scope of protection of the present application.

Claims

1. A method for calculating text similarity, characterized in that: The method comprises: Obtain a first segmentation set and a second segmentation set, wherein the first segmentation set is a segmentation set obtained after segmentation processing of the first text, and the second segmentation set is a segmentation set obtained after segmentation processing of the second text; Determining an initial similarity between a first segmentation and a second segmentation according to a synonym dictionary, wherein the first segmentation is any segmentation in the first segmentation set, and the second segmentation is any segmentation in the second segmentation set; Determining a final similarity between the first participle and the second participle based on a first threshold and the initial similarity, wherein when the initial similarity is greater than the first threshold, the final similarity is equal to the initial similarity; and when the initial similarity is less than or equal to the first threshold, the final similarity is equal to 0; determining a first weighted sum according to the final similarity; determining a first number of word pairs having different meanings in the first word set and the second word set according to the synonym dictionary, wherein the word pairs having different meanings in the first word set and the second word set refer to word pairs having a similarity between two word pairs lower than the first threshold; A first similarity between the first text and the second text is determined according to a first ratio of the first weighted sum to the first number, where the first similarity is positively correlated with the first ratio.

2. The method according to claim 1, characterized in that Also includes: Determining a plurality of similarities between the second word segmentation set and a plurality of word segmentation sets, the plurality of word segmentation sets including the first word segmentation set, the plurality of similarities including the first similarity; When the first similarity is a maximum value among the multiple similarities, and when the first similarity is greater than a second threshold, using the second word segmentation set to update the first word segmentation set; or, When the first similarity is the maximum value among the multiple similarities and when the first similarity is less than or equal to a second threshold, a third word segmentation set is generated according to the second word segmentation set, and the third word segmentation set includes some or all of the word segments in the second word segmentation set.

3. The method according to claim 2, characterized in that The updating of the first word segmentation set using the second word segmentation set includes: The M segmentations in the second segmentation set are added to the first segmentation set to generate a fourth segmentation set, and the fourth segmentation set is used for clustering processing of the third text. The M segmentations are the M segmentations with the highest frequency of occurrence in the second text, and M is a positive integer.

4. The method according to claim 3, characterized in that Before generating the fourth word segmentation set, the method further includes: Delete N segmentations in the first segmentation set, where the N segmentations in the first segmentation set are the N segmentations with the lowest occurrence frequency in the first text, and N is a positive integer.

5. The method according to claim 2, characterized in that The segmentations in the third segmentation set are the K segmentations with the highest frequency of occurrence in the second text, where K is a positive integer.

6. The method according to claim 2, characterized in that The first segmentation set is a segmentation set consisting of P segmentations with the highest frequency of occurrence in the first text, where P is a positive integer.

7. The method according to any one of claims 1, 3 to 6, characterized in that Before obtaining the first word segmentation set and the second word segmentation set, the method includes: Obtaining the first text and the second text; Performing word segmentation preprocessing on the first text to obtain the first word segmentation set, wherein the word segmentation preprocessing includes word segmentation processing, part-of-speech screening processing, and synonym expansion processing; The second text is subjected to the word segmentation preprocessing to obtain the second word segmentation set.

8. An electronic device, characterized in that: The electronic device includes a processor and a memory, the memory is used to store a computer program, and the processor is used to call and run the computer program from the memory, so that the electronic device executes the method according to any one of claims 1 to 7.

9. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the processor is caused to perform the method according to any one of claims 1 to 7.

Images