Text similarity determination method and apparatus, electronic device, and computer storage medium

Abstract

The application discloses a text similarity determination method and device, electronic equipment and a computer storage medium. In the method, instead of directly obtaining text combinations of two texts and then calculating the similarity between the text combinations, each short text is subjected to word segmentation processing to obtain real words corresponding to the short text and obtain approximate words of the real words. Subsequently, candidate short texts are selected from multiple short texts to determine associated short texts having an association relationship with the candidate short texts according to the candidate real words and the candidate approximate words of the candidate real words. The candidate short texts and the associated short texts are used as text combinations to be compared. Based on the text combinations, the text similarity between the candidate short texts and the associated short texts is calculated. The text similarity between only the text combinations that may be related in the multiple short texts is calculated, and the text similarity between unrelated text combinations is no longer calculated. The calculation amount of the text similarity is reduced, and the calculation efficiency of the text similarity is improved.

Description

Technical Field

[0001] This application relates to the field of information processing, specifically to methods for determining text similarity, devices for determining text similarity, electronic devices, and computer storage media. Background Technology

[0002] Using large AI models for queries is becoming increasingly popular, resulting in a large number of short query texts. These short query texts can be short texts used directly by users in the query scenario, or they can be short texts obtained by processing the query information used in the query scenario. For example, if a user uses long text in the query, the long text can be segmented to obtain short texts.

[0003] As the number of short texts increases, it becomes necessary to determine the similarity of these large amounts of text and then perform further processing on these similar texts, such as aggregating user query hotspots. Current methods for determining text similarity directly calculate the similarity between pairs of texts within a large number of short texts, and then identify similar short texts based on the calculated similarity. However, due to the large number of short texts, the number of text combinations for which similarity needs to be calculated is enormous, resulting in a huge computational burden. Therefore, how to reduce the computational burden and improve computational efficiency in the short text similarity calculation process has become an urgent technical problem to be solved. Summary of the Invention

[0004] This application provides a text similarity determination method to address the problem of reducing computational load and improving computational efficiency in the process of calculating the similarity of a large number of short texts. This application also provides a text similarity determination device, electronic device, and computer storage medium.

[0005] This application provides a method for determining text similarity, including: Obtain multiple short texts for comparison to assess text similarity; The short text is segmented to obtain the content words corresponding to the short text, and the approximate words of the content words are obtained; Candidate short texts are selected from multiple short texts. Based on the candidate content words and their candidate similar words corresponding to the candidate short texts, associated short texts that are related to the candidate short texts are determined from the multiple short texts. The candidate short texts and the associated short texts are used as a text combination to compare the text similarity. Based on the text combination, the text similarity between the candidate short text and the associated short text is calculated.

[0006] Optionally, it also includes: pre-constructing a reference correspondence between reference content words and reference similar words that are similar to the reference content words; The step of obtaining approximate words of the content words includes: using the content words as indexes, searching for reference approximate words corresponding to the content words in the reference correspondence relationship as the approximate words.

[0007] Optionally, the construction of the reference correspondence between the reference content word and the reference similar word includes: Obtain a short text sample set within a preset historical time period; The short text samples in the short text sample set are segmented and deduplicated to obtain a content word sample set; Each real word sample in the real word sample set is vectorized to obtain the real word sample vector for each real word sample; Based on the real word sample vector of each real word sample and a preset similarity threshold, real word samples similar to each real word sample are determined in the real word sample set; Each content word sample is used as a reference content word, and the content word samples similar to each content word sample are used as reference approximate words. The reference content words and the reference approximate words are mapped to each other to construct the reference mapping relationship.

[0008] Optionally, determining the real word samples similar to each real word sample in the real word sample set based on the real word sample vector of each real word sample and a preset similarity threshold includes: Based on the real word sample vector of each real word sample and the real word sample vector of other real word samples in the real word sample set, calculate the real word sample similarity between each real word sample and other real word samples in the real word sample set; Based on the similarity of the real word samples and the preset similarity threshold, real word samples similar to each real word sample are determined in the real word sample set.

[0009] Optionally, calculating the text similarity between the candidate short text and the associated short text based on the text combination includes: The candidate short text and the associated short text are vectorized respectively to obtain the candidate short text vector and the associated short text vector; A preset text similarity algorithm is used to calculate the text similarity between the candidate short text and the associated short text based on the candidate short text vector and the associated short text vector.

[0010] Optionally, the step of segmenting the short text to obtain content words corresponding to the short text includes: The short text is segmented using a pre-defined natural language processing toolkit to obtain the segmented words and their parts of speech. Based on the word segmentation part-of-speech, word segments that conform to the preset word segmentation part-of-speech are selected from the word segments after the word segmentation process as the content words.

[0011] Optional, also includes: Based on the word segmentation process and the word segmentation part of speech, determine whether the short text contains word segments that conform to the preset word segmentation part of speech; If the short text does not contain any words that match the preset word segmentation part of speech, then the short text will be deleted from the plurality of short texts.

[0012] Optionally, determining the associated short texts that are related to the candidate short texts from among the plurality of short texts based on the candidate content words and their candidate similar words corresponding to the candidate short texts includes: Using the candidate content words and the candidate similar words as an inverted index, the short text containing at least one of the candidate content words and the candidate similar words is searched among the multiple short texts and is used as the associated short text.

[0013] Optionally, the number of text combinations can be multiple; the method further includes: Determine whether there is any overlap in the combined text within each text combination; If the combined texts in different text combinations overlap, the text combinations are deduplicated to obtain deduplicated text combinations, and text similarity calculations are performed on the deduplicated text combinations.

[0014] This application provides a text similarity determination device, comprising: The short text acquisition unit is used to obtain multiple short texts for comparing the similarity of the texts to be compared. The unit for obtaining content words and similar words is used to perform word segmentation on the short text, obtain the content words corresponding to the short text, and obtain the similar words of the content words; A text combination determination unit is used to select candidate short texts from multiple short texts, and determine related short texts that are associated with the candidate short texts based on the candidate content words and their candidate similar words corresponding to the candidate short texts; and use the candidate short texts and the related short texts as a text combination to be compared for text similarity. A text similarity calculation unit is used to calculate the text similarity between the candidate short text and the associated short text based on the text combination.

[0015] This application provides an electronic device, including: a processor; and a memory for storing a computer program. After the electronic device is powered on and runs the computer program through the processor, it executes the above-described text similarity determination method.

[0016] This application provides a computer storage medium storing computer execution instructions, which are executed by a processor to perform the above-described text similarity determination method.

[0017] Compared with the prior art, this application has the following advantages: This application provides a method for determining text similarity, including: obtaining multiple short texts to be compared for text similarity; performing word segmentation on the short texts to obtain content words corresponding to the short texts, and obtaining approximate words of the content words; selecting candidate short texts from the multiple short texts, and determining associated short texts that have a relationship with the candidate short texts based on the candidate content words and their candidate approximate words; using the candidate short texts and associated short texts as a text combination to be compared for text similarity; and calculating the text similarity between the candidate short texts and associated short texts based on the text combination. Instead of directly obtaining pairwise text combinations and calculating the similarity between them, the process after obtaining multiple short texts for comparison does not begin with the initial text combination. Instead, each short text is segmented to obtain corresponding content words and their approximate words. Candidate short texts are then selected from the multiple texts to identify related short texts based on their content words and approximate words. These candidate and related short texts are then used as text combinations for comparison. Based on these text combinations, the text similarity between the candidate and related short texts is calculated. This approach only calculates the text similarity between potentially related text combinations, eliminating the need to calculate the similarity between unrelated text combinations. This significantly reduces the computational load and improves the efficiency of text similarity calculation. Attached Figure Description

[0018] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in this application. For those skilled in the art, other drawings can be obtained based on these drawings.

[0019] Figure 1 A flowchart of the text similarity determination method provided in the first embodiment of this application.

[0020] Figure 2 This is a schematic diagram of the framework of the text similarity determination method provided in the first embodiment of this application.

[0021] Figure 3 This is a schematic diagram of a text similarity determination device provided in the second embodiment of this application.

[0022] Figure 4A schematic diagram of an electronic device provided in the third embodiment of this application. Detailed Implementation

[0023] To enable those skilled in the art to better understand the technical solutions of this application, the application will be clearly and completely described below with reference to the accompanying drawings of the embodiments. However, this application can be implemented in many other ways different from those described below. Therefore, based on the embodiments provided in this application, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of this application.

[0024] It should be noted that the terms "first," "second," "third," etc., in the claims, specification, and drawings of this application are used to distinguish similar objects and are not used to describe a specific order or sequence. Such data are interchangeable where appropriate so that the embodiments of this application described herein can be implemented in a sequence other than that shown or described in this application. Furthermore, the terms "comprising," "having," and their variations are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that includes a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to these processes, methods, products, or apparatuses.

[0025] It should be understood that in the embodiments of this application, "at least one" means one or more, and "more than one" means two or more. "And / or" is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, and B existing alone. The character " / " generally indicates that the related objects before and after it are in an "or" relationship. "Contains A, B and / or C" means containing any one, two, or three of A, B, and C.

[0026] It should be understood that in the embodiments of this application, "B corresponding to A", "B corresponding to A", "A corresponds to B" or "B corresponds to A" means that B is associated with A, and B can be determined based on A. Determining B based on A does not mean that B is determined solely based on A; B can also be determined based on A and / or other information.

[0027] This application provides a text similarity determination method, a text similarity determination apparatus, an electronic device, and a computer storage medium. The following specific embodiments illustrate the text similarity determination method, the text similarity determination apparatus, the electronic device, and the computer storage medium.

[0028] First Embodiment This embodiment provides a method for determining text similarity; please refer to [link / reference] for details. Figure 1 This is a flowchart of the text similarity determination method provided in the first embodiment of this application.

[0029] The text similarity determination method of this application includes the following steps.

[0030] Step S101: Obtain multiple short texts for comparing text similarity.

[0031] Multiple short texts are short texts generated by users in query scenarios. As artificial intelligence technology continues to mature, more and more users are using large AI models to ask questions, resulting in a large number of short text queries; for example, a user might query the short text "What medicine should I take when I have a cold, sneezing and a runny nose?"

[0032] Large-scale AI models receive a massive amount of short text queries daily. They can calculate similarity based on queries previously asked by users, thus aggregating trending topics of discussion based on similar queries. This process involves a huge amount of text similarity calculation: for example, it requires pairwise Cartesian product calculations on a large number of queries. If 10,000 queries require pairwise Cartesian product calculations, it would require 100 million similarity calculations. 100,000 queries would require 10 billion similarity calculations, involving an enormous amount of computation.

[0033] Cartesian product: A fundamental concept in set theory, it describes the set of all possible ordered pairs between two sets. In a Cartesian product, each element forms a pair with every element in the other set, thus generating all possible combinations. Therefore, the Cartesian product can be used to roughly obtain pairwise text combinations between a large number of short texts, and then the text similarity between each pairwise text combination can be calculated.

[0034] In practice, when calculating the text similarity between pairwise text combinations, one can calculate the cosine of the angle between the text vectors in the combination, i.e., cosine similarity, also known as cosine similarity. This primarily assesses the similarity between two vectors by calculating the cosine of the angle between them. Cosine similarity involves plotting the vectors in a vector space based on their coordinate values, such as the most common two-dimensional space. Since each text vector is high-dimensional, such as 1024-dimensional, using the Cartesian product to obtain text combinations requires calculating the cosine similarity of a large number of text combinations. In reality, this process involves a huge amount of computation, resulting in low computational efficiency.

[0035] To address the aforementioned issues, this application, when calculating the similarity of multiple short texts, does not directly use Cartesian product to combine multiple short texts pairwise. Instead, it uses the content words contained in the short texts and similar content words as the association conditions for similarity, thereby obtaining text combinations that may have similarity among multiple short texts. Then, it only calculates the text similarity between text combinations that may have similarity, without calculating the text similarity between text combinations that have no similarity or very low similarity. As a result, the number of text combinations obtained by this method is far less than the number of text combinations obtained by using Cartesian product, greatly improving the efficiency of similarity calculation among a large number of short texts and significantly reducing the computational load. Similar content words are the approximate words mentioned later.

[0036] To better understand the logic behind the text similarity determination method in this application, please refer to [link / reference needed]. Figure 2 This is a schematic diagram illustrating the framework of the text similarity determination method provided in the first embodiment of this application. The method essentially uses the content words and similar content words contained in a short text as the association conditions for similarity, and performs text indexing across multiple short texts to find potentially similar text combinations. First, the short text is segmented into words; then, content words are filtered from the segmented texts to obtain content word groups. Approximate word groups are then obtained from the content words within these word groups. These content word groups and approximate word groups are used as text indexes to search across multiple short texts, subsequently selecting text combinations from these short texts. Only these text combinations are used to calculate text similarity among the multiple short texts; there is no need to combine all the short texts in pairs.

[0037] Step S102: Perform word segmentation on the short text to obtain the content words corresponding to the short text, and obtain the similar words of the content words.

[0038] After obtaining multiple short texts, the short texts are segmented to obtain one or more content words corresponding to the short texts, and the approximate words of each content word are obtained according to the pre-constructed reference correspondence between content words and approximate words.

[0039] For example, if the short text is: "A person has a cold, sneezes, and has a runny nose, what medicine should they take?", the corresponding content words after word segmentation include "person", "cold", "sneezes", "runny nose", and "what medicine".

[0040] In this embodiment, as a way to obtain approximate words of content words: a reference correspondence relationship between content words and approximate content words is preset. The content words are used as indexes to find the reference approximate words corresponding to the content words in the reference correspondence relationship as approximate words. The reference correspondence relationship refers to the correspondence relationship between the reference content words and the reference approximate words that are similar to the reference content words.

[0041] To facilitate understanding of this reference correspondence, for example, when the reference word is "person", the corresponding similar reference words are "everyone", "human heart", "human body", "human beings", "person's name", "person's image", "person", "life", "man", "woman", "pedestrian", etc.

[0042] In the example above, the content words such as "person," "cold," "sneezing," "runny nose," and "what medicine" are used as indices. By searching for similar words corresponding to each content word in the reference correspondence, a word segmentation set is formed by combining the content words and their similar words in the short text. This word segmentation set includes: "person, cold, sneezing, runny nose, what medicine, human heart, everyone, human being, human body, talent, person's name, human image, human, human brain, Human, humanity, that person, person, that person, People." e, being a person, woman, life, man, people, human world, family, pedestrian, sick, catch a cold, get sick, disease, cold medicine, chills, cold from wind and cold, sweating, stuffy nose, influenza, infection, adventure, cough, rhinitis, smallpox, typhoid fever, runny nose, treat a cold, upper respiratory tract infection, avian influenza, bronchitis, sneeze, flu, fever, hiccups, snoring, snore, nosebleed, spray, sneeze, nosebleed, sinusitis, cry, nosebleed, cough up phlegm, runny nose, DT, medicine, medicine.

[0043] In this embodiment, performing word segmentation on the short text to obtain corresponding content words can refer to: using a preset natural language processing toolkit to perform word segmentation on the short text to obtain the segmented words and their parts of speech; and selecting the segmented words that meet the preset parts of speech from the segmented words as content words based on the parts of speech.

[0044] The default natural language processing toolkit can actually be a word segmentation tool, such as HanLP (Han Language Processing). HanLP is a Java toolkit composed of a series of models and algorithms, aiming to promote the application of natural language processing in production environments. HanLP can be used to segment short texts. It can segment words that conform to preset parts of speech, such as noun segments and verb segments.

[0045] Preset word segmentation parts of speech such as nouns, verbs, and noun-verbs. Nouns include personal names, time words, place names, directional place names, spatial words, other proper nouns, organization names, job titles, industry terms, foreign language words, brand names, food names, medicine names, product names, abbreviations, and numeral nouns.

[0046] The preset part-of-speech of segmented words does not include function words, and the function words include, for example, auxiliary words (such as "de", "le", "zhe", "guo"), prepositions (such as "zai", "cong", "xiang", "dui", "ba"), conjunctions (such as "he", "danshi", "ruguo"), modal particles (such as "a", "ba", "ne"), some adverbs (such as "jiu", "cai", "yijing"), articles (such as English "the", "a") and so on.

[0047] In this embodiment, actually the preset part-of-speech of segmented words may be the part-of-speech corresponding to notional words, and then the segmented words that do not conform to the preset part-of-speech of segmented words are function words.

[0048] This embodiment further includes: determining whether the short text contains a segmented word that conforms to the preset part-of-speech of segmented words based on the segmented words obtained after segmentation processing and the part-of-speech of the segmented words; if the short text does not contain a segmented word that conforms to the preset part-of-speech of segmented words, deleting the short text from the plurality of short texts. For example, the short text is: "Hello", in fact, this short text is only used as a greeting term, and thus there is no need to perform text similarity calculation, and it is deleted from the plurality of short texts to further reduce the amount of text similarity calculation.

[0049] This embodiment further includes: pre-constructing a reference correspondence between a reference notional word and a reference approximate word that is approximate to the reference notional word. As one manner of constructing the reference correspondence between the reference notional word and the reference approximate word that is approximate to the reference notional word: first, a short text sample set within a preset historical time period is obtained; then, segmentation processing and deduplication processing are performed on the short text samples in the short text sample set to obtain a notional word sample set; after that, vectorization is performed on each notional word sample in the notional word sample set to obtain a notional word sample vector of each notional word sample; after obtaining the notional word sample vector of each notional word sample, notional word samples similar to each notional word sample are determined in the notional word sample set based on the notional word sample vector of each notional word sample and a preset similarity threshold; each notional word sample is taken as a reference notional word, the notional word samples similar to each notional word sample are taken as reference approximate words, and the reference notional word and the reference approximate words are associated to construct the reference correspondence.

[0050] Obtaining the short text sample set within the preset historical time period is, for example, acquiring historical query short texts of the user in the recent 180 days as the short text sample set; performing segmentation processing and deduplication processing on the short text samples in the short text sample set to obtain the notional word sample set is, for example, performing segmentation processing and deduplication processing on each historical query short text in the historical query short texts of the recent 180 days to obtain all historical notional words, and the segmentation processing herein specifically refers to the segmentation processing described above.

[0051] Vectorize each real word sample in the real word sample set to obtain the real word sample vector of each real word sample. Specifically, this can mean calling the BGE vector large model (BGE, i.e., BAAI General Embedding) or other types of vectorized large models to vectorize each real word sample in the real word sample set to obtain the real word sample vector of each real word sample.

[0052] Specifically, based on the real word sample vector of each real word sample and a preset similarity threshold, real word samples similar to each real word sample are determined in the real word sample set, including: calculating the real word sample similarity between each real word sample and other real word samples in the real word sample set based on the real word sample vector of each real word sample and the real word sample vector of other real word samples in the real word sample set; and determining real word samples similar to each real word sample in the real word sample set based on the real word sample similarity and the preset similarity threshold.

[0053] In practice, calculating the similarity between each real-word sample and the real-word sample vectors of other real-word samples in the set can be achieved using a cosine similarity algorithm. This involves calculating the pairwise similarity between all real-word samples in the set and identifying real-word samples with a similarity greater than or equal to a preset similarity threshold as those similar to each real-word sample. Using a cosine similarity algorithm to calculate the pairwise similarity between all real-word samples in the set essentially involves calculating the cosine of the angle between the real-word sample vector of each real-word sample and the real-word sample vectors of other real-word samples in the set, thus evaluating the pairwise similarity between all real-word samples in the set.

[0054] It should be noted that, in this embodiment, the method of using the cosine similarity algorithm to calculate the cosine value of the angle between two vectors can also be to directly calculate the Euclidean distance between the two vectors.

[0055] For example, when a certain content word sample is "hairstyle", similar content word samples include "short hair", "hairstyle design", "long hair", "perm", "hairstyle stylist", etc. It should be noted that there may be many more content word samples similar to this content word sample that are not listed here.

[0056] Each content word sample is used as a reference content word, and similar content word samples are used as reference approximations. A mapping is established between the reference content words and their approximations. Essentially, this involves caching the reference content words and their approximations, such as in a key-value database or in memory, for later, rapid searching of approximations. A key-value database is a non-relational database that stores data in key-value pairs; key-value refers to these pairs.

[0057] Step S103: Select candidate short texts from multiple short texts; determine related short texts that are related to the candidate short texts based on the candidate content words and their candidate similar words; combine the candidate short texts and related short texts as text combinations to be compared for text similarity.

[0058] In this paper, the content words of the candidate short text after word segmentation and similar words can also be referred to as the candidate content words and their candidate similar words of the candidate short text.

[0059] Specifically, based on the candidate content words and their candidate similar words corresponding to the candidate short text, related short texts that are associated with the candidate short text are determined among multiple short texts, including: using the candidate content words and candidate similar words as an inverted index, searching among multiple short texts for short texts that contain at least one of the candidate content words and candidate similar words as related short texts.

[0060] In fact, since the corresponding content words and similar words have been obtained for multiple short texts, when searching for related short texts that are related to a candidate short text, the candidate content words and candidate similar words corresponding to the candidate short text are used as an inverted index to search for all related short texts that may be related to the candidate short text. It should be noted that the related short texts contain at least one of the candidate content words and candidate similar words.

[0061] For example, there are five short texts: "What medicine should I take for a cold with sneezing and a runny nose?", "What medicine should I take for a cold with headache, runny nose, stuffy nose, sore throat, and cough?", "What medicine should I take for a cold with sneezing, stuffy nose, runny nose, headache, and fever?", "Please design a hairstyle that is easy to manage and suits my face shape?", and "If I stop paying into my pension insurance, will my pension be cleared in the past six months?"

[0062] The current candidate short text is: "What medicine should I take if I have a cold, sneeze, and have a runny nose?" One related short text found is: "What medicine should I take for a headache, cold, runny nose, stuffy nose, sore throat, and cough?" Both contain the content words "cold", "runny nose", and "what medicine".

[0063] Of course, it is understandable that when searching for related short texts that are associated with a candidate short text among multiple short texts, it is not necessary to consider a candidate short text as its own related short text.

[0064] In this embodiment, the reason for using candidate content words and candidate similar words as inverted indexes to search for short texts containing at least one of the candidate content words and candidate similar words among multiple short texts as associated short texts, instead of using only candidate content words as inverted indexes to search for short texts containing at least one of the candidate content words and candidate similar words among multiple short texts as associated short texts, is to avoid insufficient recall of associated short texts when searching for associated short texts among multiple short texts.

[0065] Step S104: Calculate the text similarity between the candidate short text and the associated short text based on text combination.

[0066] Specifically, calculating the text similarity between candidate short texts and related short texts based on text combination can refer to: vectorizing the candidate short texts and related short texts respectively to obtain candidate short text vectors and related short text vectors; and then using a preset text similarity algorithm and based on the candidate short text vectors and related short text vectors to calculate the text similarity between the candidate short texts and related short texts. The preset text similarity algorithm could be a cosine similarity algorithm, which calculates the cosine of the angle between the candidate short text vector and the related short text vector to evaluate the text similarity between them.

[0067] In this embodiment, there are multiple text combinations. To further reduce the computational load of text similarity, the method also includes: determining whether there is overlap in the combined texts of each text combination; if there is overlap in the combined texts of different text combinations, then the text combinations are deduplicated to obtain deduplicated text combinations, and then text similarity calculation is performed on the deduplicated text combinations.

[0068] In this embodiment, after calculating the text similarity between candidate short texts and associated short texts, based on a preset similarity threshold, candidate short texts that meet the preset similarity threshold can be aggregated with associated short texts to determine the query text hotspots in the user query scenario. To facilitate the distinction between the preset similarity threshold here and the preset similarity threshold mentioned earlier, the preset similarity threshold here can be a second preset similarity threshold, while the preset similarity threshold mentioned earlier can be a first preset similarity threshold.

[0069] To facilitate understanding of the method used in this embodiment, the five short texts in Table 1 are used as examples of multiple short texts in this embodiment. The correspondence between short texts and word segments is shown in Table 1; the correspondence between short texts and content words is shown in Table 2; the correspondence between short texts and content words and similar words is shown in Table 3; the correspondence between text combinations and related words is shown in Table 4; and the correspondence between text combinations for which text similarity needs to be calculated and the calculated text similarity is shown in Table 5.

[0070] Table 1. Correspondence between short texts and word segments It should be noted that Table 1 shows the short text segmented using a pre-defined natural language processing toolkit, resulting in segmented words and their parts of speech. Based on these parts of speech, words that match the pre-defined parts of speech are selected as content words, which are the content words in Table 2.

[0071] In Table 1, word segmentation actually includes punctuation marks in the short text, but punctuation marks are not treated as content words; in Table 1, the parts of speech involved are represented by part-of-speech symbols.

[0072] The part-of-speech symbols include: "n", "nn", "nnt", "nr", "nrf", "ns", "nsf", "nz", "nh", "ni", "nis", "nl", "nt", "ntc", "ntcb", "nto", "ntu", "nz", "nba", "nf", "nhd", "nhm", "nit", "nmc", "nnd", "nnt", "v", "vn", "vi", "nx", "gb", "gm", "gc", "gg", "gi", etc.; corresponding to the following Chinese parts of speech: noun, personal name, time word, place name, directional place name, spatial word, directional spatial word, other proper noun, organization name, job title, industry terminology, foreign language word, and time word (such as dynasty, festival, etc.). Time terms (century), time terms (dynasty), time terms (other), time terms (units), other proper nouns, brand names, food names, medicine names, product names, terms, abbreviations, numeral nouns, time terms, verbs, nominal verbs, intransitive verbs, news terms, country names, currencies, disciplines, geography, institutions, etc.

[0073] Table 2. Correspondence between short texts and content words After obtaining the content words as shown in Table 2, we can obtain similar words based on the reference correspondence. Table 3 records the content words and similar words corresponding to each of the above 5 short texts.

[0074] Table 3. Correspondence between short texts and content words and similar words After obtaining the content words and similar words corresponding to the short texts in Table 3, we can then find the associated short texts associated with each short text. For example, short text 1 and short text 2 in the second row of Table 4 can be considered as a pair of texts. The associated word in this row represents the text combination found by short text 1 and short text 2 through the associated word. In order to facilitate the recording of associated word representatives, they are labeled, i.e., associated word representative sequence numbers.

[0075] Table 4. Correspondence between text combinations and related words As shown in Table 4, some text combinations are repeated. This is because the text combinations are represented by different conjunctions, such as the text combinations in the second, fifth, and eighth rows of Table 4. To avoid repeatedly calculating the text similarity of text combinations, deduplication is performed on potentially repeated text combinations to obtain the text combinations for which text similarity needs to be calculated. See Table 5 for details, which records the correspondence between the text combinations for which text similarity needs to be calculated and the calculated text similarity scores.

[0076] Table 5. Correspondence between text combinations for which text similarity needs to be calculated and the calculated text similarity. Using semantically meaningful content words in the text and their similar content words as the association conditions for calculating text similarity greatly reduces the amount of computation. Taking the five short texts in this example as an example, if no text indexing and text association are done, a Cartesian product is required, and the number of combinations of text similarity calculations would be 5x5=25. However, using this method, only 3 calculations are needed.

[0077] In this embodiment, after obtaining multiple short texts for comparison, instead of directly obtaining pairwise text combinations and calculating the similarity between them, each short text is segmented to obtain corresponding content words and their approximate words. Then, candidate short texts are selected from the multiple texts to identify related short texts based on their content words and approximate words. These candidate short texts and their related short texts are then used as text combinations for comparison. Based on these text combinations, the text similarity between the candidate short texts and their related short texts is calculated. This approach only calculates the text similarity between potentially related text combinations, eliminating the need to calculate the text similarity between unrelated text combinations. This significantly reduces the computational load of text similarity calculations and improves the efficiency of text similarity calculations.

[0078] Second Embodiment Corresponding to the text similarity determination method provided in the first embodiment of this application, the second embodiment of this application also provides a text similarity determination device. Since the device embodiment is basically similar to the first embodiment, the description is relatively simple; relevant details can be found in the description of the first embodiment. The device embodiments described below are merely illustrative.

[0079] Please refer to Figure 3 This is a schematic diagram of the text similarity determination device provided in the second embodiment of this application.

[0080] The text similarity determination device 300 includes: a short text acquisition unit 301, used to acquire multiple short texts for comparison; a content word and similar word acquisition unit 302, used to perform word segmentation on the short texts to acquire content words corresponding to the short texts and to acquire similar words of the content words; a text combination determination unit 303, used to select candidate short texts from the multiple short texts, and determine associated short texts that are related to the candidate short texts based on the candidate content words and their candidate similar words; and to use the candidate short texts and the associated short texts as a text combination for comparison; and a text similarity calculation unit 304, used to calculate the text similarity between the candidate short texts and the associated short texts based on the text combination.

[0081] Third Embodiment Corresponding to the method of the first embodiment of this application, the third embodiment of this application also provides an electronic device.

[0082] The electronic device includes: a processor; and a memory for storing a computer program. After the electronic device is powered on and runs the computer program through the processor, it executes the method of the first embodiment. Figure 4As shown, Figure 4 This is a schematic diagram of an electronic device provided according to a third embodiment of this application. The electronic device specifically includes: at least one processor 401, at least one communication interface 402, at least one memory 403, and at least one communication bus 404. Optionally, the communication interface 402 can be an interface for a communication module, such as an interface for a GSM module. The processor 401 may be a CPU, an Application Specific Integrated Circuit (ASIC), or one or more integrated circuits configured to implement embodiments of this application. The memory 403 may include high-speed RAM and may also include non-volatile memory, such as at least one disk storage device. The memory 403 stores a program, and the processor 401 calls the program stored in the memory 403 to execute the method of the first embodiment.

[0083] Fourth embodiment Corresponding to the method of the first embodiment of this application, the fourth embodiment of this application also provides a computer storage medium storing computer execution instructions, which are executed by a processor to perform the method of the first embodiment of this application.

[0084] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0085] Memory may include non-persistent storage in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM. Memory is an example of computer-readable media.

[0086] 1. Computer-readable media includes both permanent and non-permanent, removable and non-removable media, which can store information by any method or technology. Information can be computer-readable instructions, data structures, program modules, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other non-transferable medium that can be used to store information accessible by a computing device. As defined in this application, computer-readable media does not include non-transitory computer-readable media, such as modulated data signals and carrier waves.

[0087] 2. Those skilled in the art will understand that embodiments of this application can provide methods, systems, or computer program products. Therefore, embodiments of this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, embodiments of this application can take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0088] 3. This application embodiment may involve the use of user data. In practical applications, user-specific personal data may be used within the scope permitted by applicable laws and regulations of the country in which the application is located (e.g., with the user's explicit consent and effective notification to the user, etc.). Furthermore, the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties. The collection, use and processing of related data must comply with the relevant laws, regulations and standards of the relevant countries and regions, and corresponding operation entry points are provided for users to choose to authorize or refuse.

[0089] Although this application discloses preferred embodiments as described above, it is not intended to limit this application. Any person skilled in the art can make possible changes and modifications without departing from the spirit and scope of this application. Therefore, the scope of protection of this application should be determined by the scope defined in the claims of this application.

Claims

1. A method for determining text similarity, characterized in that, include: Obtain multiple short texts for comparison to assess text similarity; The short text is segmented to obtain the content words corresponding to the short text, and the approximate words of the content words are obtained; Candidate short texts are selected from multiple short texts. Based on the candidate content words and their candidate similar words corresponding to the candidate short texts, associated short texts that are related to the candidate short texts are determined from the multiple short texts. The candidate short texts and the associated short texts are used as a text combination to compare the text similarity. Based on the text combination, the text similarity between the candidate short text and the associated short text is calculated.

2. The method according to claim 1, characterized in that, Also includes: A reference correspondence relationship is pre-constructed between reference content words and reference similar words that are similar to the reference content words; The step of obtaining approximate words of the content words includes: using the content words as indexes, searching for reference approximate words corresponding to the content words in the reference correspondence relationship as the approximate words.

3. The method according to claim 2, characterized in that, The construction of the reference correspondence between reference content words and reference similar words that are similar to the reference content words includes: Obtain a short text sample set within a preset historical time period; The short text samples in the short text sample set are segmented and deduplicated to obtain a content word sample set; Each real word sample in the real word sample set is vectorized to obtain the real word sample vector for each real word sample; Based on the real word sample vector of each real word sample and a preset similarity threshold, real word samples similar to each real word sample are determined in the real word sample set; Each content word sample is used as a reference content word, and the content word samples similar to each content word sample are used as reference approximate words. The reference content words and the reference approximate words are mapped to each other to construct the reference mapping relationship.

4. The method according to claim 3, characterized in that, The step of determining real word samples similar to each real word sample in the real word sample set based on the real word sample vector of each real word sample and a preset similarity threshold includes: Based on the real word sample vector of each real word sample and the real word sample vector of other real word samples in the real word sample set, calculate the real word sample similarity between each real word sample and other real word samples in the real word sample set; Based on the similarity of the real word samples and the preset similarity threshold, real word samples similar to each real word sample are determined in the real word sample set.

5. The method according to claim 1, characterized in that, The step of calculating the text similarity between the candidate short text and the associated short text based on the text combination includes: The candidate short text and the associated short text are vectorized respectively to obtain the candidate short text vector and the associated short text vector; A preset text similarity algorithm is used to calculate the text similarity between the candidate short text and the associated short text based on the candidate short text vector and the associated short text vector.

6. The method according to claim 1, characterized in that, The step of segmenting the short text to obtain the corresponding content words includes: The short text is segmented using a pre-defined natural language processing toolkit to obtain the segmented words and their parts of speech. Based on the word segmentation part-of-speech, word segments that conform to the preset word segmentation part-of-speech are selected from the word segments after the word segmentation process as the content words.

7. The method according to claim 6, characterized in that, Also includes: Based on the word segmentation process and the word segmentation part of speech, determine whether the short text contains word segments that conform to the preset word segmentation part of speech; If the short text does not contain any words that match the preset word segmentation part of speech, then the short text will be deleted from the plurality of short texts.

8. The method according to claim 1, characterized in that, The step of determining the associated short texts that are related to the candidate short texts from among the multiple short texts based on the candidate content words and their candidate similar words corresponding to the candidate short texts includes: Using the candidate content words and the candidate similar words as an inverted index, the short text containing at least one of the candidate content words and the candidate similar words is searched among the multiple short texts and is used as the associated short text.

9. The method according to claim 1, characterized in that, The number of text combinations is multiple; the method further includes: Determine whether there is any overlap in the combined text within each text combination; If the combined texts in different text combinations overlap, the text combinations are deduplicated to obtain deduplicated text combinations, and text similarity calculations are performed on the deduplicated text combinations.

10. A text similarity determination device, characterized in that, include: The short text acquisition unit is used to obtain multiple short texts for comparing the similarity of the texts to be compared. The unit for obtaining content words and similar words is used to perform word segmentation on the short text, obtain the content words corresponding to the short text, and obtain the similar words of the content words; A text combination determination unit is used to select candidate short texts from multiple short texts, and determine related short texts that are associated with the candidate short texts based on the candidate content words and their candidate similar words corresponding to the candidate short texts; and use the candidate short texts and the related short texts as a text combination to be compared for text similarity. A text similarity calculation unit is used to calculate the text similarity between the candidate short text and the associated short text based on the text combination.

11. An electronic device, characterized in that, include: processor; And a memory for storing a computer program, wherein after the electronic device is powered on and the computer program is run by the processor, it performs the method according to any one of claims 1-9.

12. A computer storage medium, characterized in that, The computer storage medium stores computer execution instructions, which are executed by a processor to perform the method described in any one of claims 1-9.

Images