Text similarity calculation method and device, computer device and storage medium

Abstract

The application relates to the technical field of semantic analysis, and discloses a text similarity calculation method and device, computer equipment and a storage medium. The method comprises the following steps: obtaining a first text and a second text, performing word segmentation processing, and obtaining a plurality of first segmented words and a plurality of second segmented words; obtaining a first word position parameter and a first word normalization weight of each first segmented word; obtaining a second word position parameter and a second word normalization weight of each second segmented word; determining a feature weight according to the first word normalization weight and the second word normalization weight; determining a relative position weight according to the first word position parameter and the second word position parameter; determining a context weight according to the first word normalization weight, a first position difference of the same word in the first text and a second position difference of the same word in the second text; and determining the similarity of the first text relative to the second text according to the feature weight, the relative position weight and the context weight. The application improves the accuracy of text similarity calculation.

Description

Technical Field

[0001] This invention relates to the field of semantic parsing technology, and in particular to a text similarity calculation method, apparatus, computer device, and storage medium. Background Technology

[0002] In natural language processing tasks, it is often necessary to determine whether two texts or statements are similar and to calculate their similarity score. For example, in a question-answering system, the system prepares some classic questions and their corresponding answers. When a user asks a question, the system retrieves the classic question with the highest similarity to the user's question and then returns the corresponding answer to the user.

[0003] Existing text similarity calculation methods include Jaccard similarity calculation, edit distance-based similarity calculation, and semantic model-based similarity calculation. Jaccard similarity calculation calculates the number of common characters (or words) between two sets of sentences; the more common elements, the higher the similarity. Edit distance-based similarity calculation involves editing two strings through substitution, insertion, and deletion operations; the fewer the number of edits required to transform one string into the other, the higher the similarity. Semantic model-based similarity calculation calculates the vector representations of the two sentences (e.g., [1,0,0,1,1,1,0,1]) and then calculates the cosine similarity between the two vectors to evaluate their similarity.

[0004] Both the Jaccard similarity calculation method and the edit distance-based similarity calculation method ignore the weight of words in a sentence and the influence of word order, and are highly sensitive to sentence length, resulting in inaccurate calculations. Semantic model-based similarity calculation methods rely excessively on the accuracy of the model's vector calculations, easily misinterpreting completely unrelated sentences as similar. Therefore, it is necessary to propose a more accurate text similarity calculation method. Summary of the Invention

[0005] Therefore, it is necessary to provide a text similarity calculation method, apparatus, computer device, and storage medium to address the above-mentioned technical problems, so as to solve the problems that text similarity calculation does not consider the weight and order of words in the text, and that the reliance on model vector calculation leads to poor accuracy.

[0006] A text similarity calculation method includes:

[0007] Obtain a first text and a second text; perform word segmentation on the first text to obtain multiple first words; perform word segmentation on the second text to obtain multiple second words;

[0008] Obtain the first word position parameters and first word normalization weights for each of the first word segments; obtain the second word position parameters and second word normalization weights for each of the second word segments;

[0009] The feature weights of each of the first words are determined based on the normalized weights of the first words of each of the first words and the normalized weights of the second words of each of the second words.

[0010] The relative position weight of each first word is determined based on the first word position parameters of each first word and the second word position parameters of each second word.

[0011] The context weight of each first word is determined based on the first word normalization weight of each first word, the first position difference of the same word in the first text, and the second position difference in the second text.

[0012] The similarity between the first text and the second text is determined based on the feature weights, the relative position weights, and the context weights.

[0013] A text similarity calculation device, comprising:

[0014] The word segmentation module is used to obtain a first text and a second text, perform word segmentation on the first text to obtain multiple first words, and perform word segmentation on the second text to obtain multiple second words.

[0015] The weight normalization module is used to obtain the first word position parameters and the first word normalization weights of each of the first word segments; and to obtain the second word position parameters and the second word normalization weights of each of the second word segments.

[0016] The feature weight calculation module is used to determine the feature weight of each of the first words based on the normalized weight of the first word of each of the first words and the normalized weight of the second word of each of the second words;

[0017] The relative position weight calculation module is used to determine the relative position weight of each first word based on the first word position parameters of each first word and the second word position parameters of each second word.

[0018] The context weight calculation module is used to determine the context weight of each of the first words based on the first word normalization weight of each of the first words, the first position difference of the same word in the first text and the second position difference in the second text.

[0019] The similarity calculation module is used to determine the similarity between the first text and the second text based on the feature weights, the relative position weights, and the context weights.

[0020] A computer device includes a memory, a processor, and computer-readable instructions stored in the memory and executable on the processor, wherein the processor implements the above-described text similarity calculation method when executing the computer-readable instructions.

[0021] A computer-readable storage medium storing computer-readable instructions, characterized in that, when executed by one or more processors, the computer-readable instructions cause the one or more processors to perform the text similarity calculation method described above.

[0022] The aforementioned text similarity calculation method, apparatus, computer equipment, and storage medium obtain multiple first and second word segments by acquiring a first and a second text and performing word segmentation; obtain the first word position parameters and normalized weights of each first word segment; obtain the second word position parameters and normalized weights of each second word segment; determine the feature weights of each first word segment based on the first and second word normalized weights; determine the relative position weights of each first word segment based on the first and second word position parameters; determine the context weights of each first word segment based on the first word normalized weights, the first position difference of the same word in the first text, and the second position difference in the second text; and determine the similarity between the first and second texts based on the feature weights, relative position weights, and context weights. The text similarity calculation method provided by this invention considers the weights of words in the text by calculating feature weights after word segmentation and penalizes words that are not matched; considers the order of words in the text by calculating relative position weights after word segmentation; considers the density of words in the text by calculating context weights after word segmentation; and finally considers the weights of words in the text through weighted calculation. The text similarity calculation method of the present invention comprehensively considers multiple factors of words in the text, thereby improving the accuracy of the calculation. Attached Figure Description

[0023] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the description of the embodiments of the present invention will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0024] Figure 1 This is a flowchart illustrating a text similarity calculation method in one embodiment of the present invention;

[0025] Figure 2 This is a schematic diagram of a text similarity calculation device in one embodiment of the present invention;

[0026] Figure 3This is a schematic diagram of a computer device according to an embodiment of the present invention. Detailed Implementation

[0027] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0028] In one embodiment, such as Figure 1 As shown, a text similarity calculation method is provided, including the following steps S10-S60.

[0029] S10. Obtain a first text and a second text; perform word segmentation on the first text to obtain multiple first words; perform word segmentation on the second text to obtain multiple second words.

[0030] In a text similarity calculation, the input is two texts, and the output is a similarity value. That is, the input is the target text and the comparison text, and the output is the similarity value between the target text and the comparison text. The input text can be long, such as a paragraph; or short, such as a sentence. When calculating the similarity of long texts, preprocessing is required to divide them into sentence texts. Therefore, the text similarity calculation is ultimately based on the sentence text. Word segmentation is a text segmentation method. The purpose of word segmentation is to divide a text sequence into individual character units. A character unit is a series of consecutive characters with syntactic meaning and high relevance, making it easier for humans and machines to understand the content of the text. Word segmentation is a fundamental task in text similarity calculation. Existing word segmentation methods include string matching-based methods, understanding-based methods, and statistical methods. Specific word segmentation tools include jieba, SnowNLP, THULAC, and NLPIR. The appropriate tool can be selected as needed. Word segmentation occurs at positions where character relevance is low, resulting in relatively little information loss.

[0031] Text similarity can be applied in search and input recommendation scenarios. For example, a customer service robot's question-and-answer system is a typical search scenario. The database of the question-and-answer system consists of question-and-answer data pairs created according to business content. The database data includes standard questions, similar questions, and answers. After receiving a user's question, the system searches the database to obtain several questions that are most similar to the user's question (which can be either standard questions or similar questions), and then returns the corresponding answers to the user.

[0032] In an example, the first text is the target text, that is, "Can one still purchase insurance with hypertension?", and the second text is the comparison text, that is, "Purchase insurance with hypertension"; the first word segmentation is the target word segmentation, and the second word segmentation is the comparison word segmentation. The target text and the text to be compared are obtained, and the target text is segmented through a pre-set word segmentation tool to obtain multiple target word segmentations corresponding to the target text, that is, "get, have, hypertension, still can, purchase, insurance, right?"; the comparison text is segmented through a pre-set word segmentation tool to obtain multiple comparison word segmentations corresponding to the comparison text, that is, "hypertension, purchase, insurance".

[0033] S20. Obtain the first word position parameter and the first word normalized weight of each of the first word segmentations; obtain the second word position parameter and the second word normalized weight of each of the second word segmentations.

[0034] Understandably, the first word position parameter is the position encoding corresponding to each first word segmentation in the first text. The first word segmentation is the target word segmentation. Through a pre-trained position recognition algorithm, the position recognition and marking of multiple target word segmentations corresponding to the target text are performed to obtain the position parameter of each target word segmentation, that is, "get = 1, have = 2, hypertension = 3, still can = 4, purchase = 5, insurance = 6, right? = 7"; similarly, the position recognition and marking of multiple comparison word segmentations corresponding to the comparison text are performed to obtain the position parameter of each comparison word segmentation, that is, "hypertension = 1, purchase = 2, insurance = 3". The first word normalized weight is the result of normalizing the word weight of each first word segmentation in the first text. The word weight can be calculated through a pre-trained TermWeight word weight algorithm, indicating the importance degree of the first word segmentation in the first text. The normalization process is to scale the data proportionally and map the data to the range of 0 to 1. Through formal changes in expression, it is beneficial for essential comparison and calculation. The first word segmentation is the target word segmentation. Through a pre-trained TermWeight word weight algorithm, the weight calculation and normalization of multiple target word segmentations corresponding to the target text are performed to obtain the normalized weight of each target word segmentation, and the weight calculation and normalization of multiple comparison word segmentations corresponding to the comparison text are performed to obtain the normalized weight of each comparison word segmentation.

[0035] S30. Determine the feature weight of each of the first word segmentations according to the first word normalized weight of each of the first word segmentations and the second word normalized weight of each of the second word segmentations.

[0036] Understandably, the first word segment is the target word, and the second word segment is the comparison word. The target word and the comparison word may contain the same words. Feature weight refers to the degree of feature calculated by matching the target word and the comparison word, and adjusting the normalized weights based on the degree of similarity in the matches. In the process of matching each word, the feature weight is higher when the matches are the same, and lower when the matches are different. The feature weight considers the contribution of the same words when the target word and the comparison word have the same match, as well as the contribution of the importance of the same words in the target word and the comparison word. Based on the normalized weights of the target word and the comparison word, the feature weights corresponding to each target word can be determined. For example, based on the normalized weights of "hypertension" in the target word and the comparison word, the feature weights corresponding to "hypertension" in the target word can be determined.

[0037] S40. Determine the relative position weight of each first segment based on the first word position parameter of each first segment and the second word position parameter of each second segment.

[0038] Understandably, the first word segmentation is the target word segmentation, and the second word segmentation is the comparison word segmentation. The target word segmentation and the comparison word segmentation may contain identical words. The appearance of identical words indicates a certain similarity between the target text and the comparison text. However, if the identical words appear in different positions in the target text and the comparison text, the similarity between the target text and the comparison text may be significantly reduced. Relative position refers to the difference in position between the identical words in the target text and the comparison text. The relative position weight is the result of calculating the relative difference between the position parameters of the identical words in the target text and the position parameters of the comparison text. Based on the position parameters of the target word segmentation and the comparison word segmentation, the relative position weight corresponding to each target word segmentation can be determined. For example, based on the position parameters of "hypertension" in the target word segmentation and the position parameters of "hypertension" in the comparison word segmentation, the relative position weight corresponding to "hypertension" in the target word segmentation can be determined.

[0039] S50. Determine the context weight of each of the first word segments based on the first word normalization weight of each first word segment, the first position difference of the same word in the first text, and the second position difference in the second text.

[0040] Understandably, the first participle is the target participle, and the second participle is the comparison participle. There may be identical words between the target participle and the comparison participle. The appearance of identical words indicates a certain similarity relationship between the target text and the comparison text. However, if there is more than one identical word in the target text and the comparison text, the closeness of two identical words that are contextually related to each other may affect the similarity relationship between the target text and the comparison text. Context refers to the positional relationship before and after two identical words in the same text, that is, the two identical words appear in the target text or the comparison text at the same time; the context weight is the result of calculating the normalized weight of the specified target participle, the positional difference of the identical words that are contextually related to each other in the target text, and the positional difference in the comparison text. Based on the normalized weight of each target participle, the first positional difference of the identical words that are contextually related to each other in the target text, and the second positional difference in the comparison text, the context weight of the target participle can be determined. For example, based on the normalized weight of "hypertension" in the target participle, the first positional difference of the identical words "purchase - hypertension" that are contextually related to each other in the target text, and the second positional difference in the comparison text, the context weight of "purchase - hypertension" can be determined. Similarly, the context weight of "insurance - hypertension" can be determined, and finally the context weight of "hypertension" in the target participle can be obtained.

[0041] S60. Determine the similarity of the first text relative to the second text according to the feature weight, the relative position weight, and the context weight.

[0042] Understandably, the first text is the target text, and the second text is the comparison text. The weight coefficient is the proportion of the feature weight, the relative position weight, and the context weight. After obtaining the feature weight, the relative position weight, and the context weight, the similarity of the first text relative to the second text can be obtained through weighted summation calculation using the weight coefficient. For example, obtain the feature weight, the relative position weight, and the context weight of each target participle "get, already, hypertension, can also, purchase, insurance, right" in the target text, and perform weighted summation calculation according to the weighting coefficient to obtain the similarity of the target text relative to the comparison text.

[0043] This embodiment obtains multiple first and second word segments by acquiring a first text and performing word segmentation on a second text. It then acquires the first word position parameters and normalized weights for each first word segment; acquires the second word position parameters and normalized weights for each second word segment; determines the feature weights of each first word segment based on the first and second word normalized weights; determines the relative position weights of each first word segment based on the first and second word position parameters; determines the context weights of each first word segment based on the first word normalized weights, the first position difference of the same word in the first text, and the second position difference in the second text; and determines the similarity between the first text and the second text based on the feature weights, relative position weights, and context weights. The text similarity calculation method in this embodiment considers the weights of words in the text by calculating feature weights after word segmentation and penalizes words that do not match; considers the order of words in the text by calculating relative position weights after word segmentation; considers the density of words in the text by calculating context weights after word segmentation; and finally, considers the weights of words in the text through weighted calculation. This embodiment takes into account multiple factors of words in the text, thus improving the accuracy of the calculation.

[0044] Optionally, step S20, namely obtaining the normalized weights of the first words in each of the first word segments, includes:

[0045] S201. Calculate the weight of each first word by performing a pre-trained weight algorithm on each first word segment to obtain the first word weight of each first word segment.

[0046] S202. Normalize the weight of the first word to obtain the normalized weight of the first word for each of the first word segments.

[0047] Understandably, the first word normalization weight is the result of normalizing the word weights of each first word segmentation in the first text. The word weight can be calculated by the pre-trained TermWeight word weight algorithm. The first word segmentation is the target word segmentation, and the TermWeight word weight algorithm is used to calculate the relative importance degree between each target word segmentation in the target text, and assign corresponding scores according to the importance degree. The index for distinguishing the importance degree is TF-IDF (term frequency-inverse document frequency), which is the result of calculating two aspects of term frequency (TF) and inverse document frequency (IDF). The result of TF-IDF is normalized to obtain the normalized weight of the target word segmentation, that is, the first word normalization weight. In the text set, the more times a word segmentation appears, the less information it indicates, and the lower its importance degree; on the contrary, the fewer times a word segmentation appears, the more information it indicates, and the higher its importance degree. As shown in Table 1 for the position parameters and normalized weights corresponding to the target word segmentation and the comparison word segmentation, the words "了" and "吗" appear very frequently in the training text set, but the amount of information is very small. Therefore, the trained TermWeight word weight algorithm gives very low scores to the importance degrees of "了" and "吗" in the target word segmentation.

[0048] Table 1 Position parameters and normalized weights corresponding to the target word segmentation and the comparison word segmentation

[0049] Target word segmentation have to Already hypertension Still Buy Insurance Ma Target word segmentation position parameters 1 2 3 4 5 6 7 Target segmentation normalization weights 0.0055 0.0071 0.2954 0.0022 0.3414 0.3484 0.0000 Comparison of word segments hypertension Buy Insurance - - - - Compare word segmentation position parameters 1 2 3 - - - - Compare the normalized weights of word segmentation 0.4050 0.1321 0.4629 - - - -

[0050] This embodiment considers the word weight in text similarity calculation, obtains the importance degree of each word segmentation in the text by calculating the weight of the word segmentation in the text, and normalizes the data at the same time, which facilitates the weight calculation in the subsequent steps.

[0051] Optionally, the first word segmentations include identical words and different words; before step S30, that is, before determining the feature weights of each first word segmentation according to the first word normalization weights of each first word segmentation and the second word normalization weights of each second word segmentation, it includes:

[0052] S301. Compare each of the first word segmentations with each of the second word segmentations one by one;

[0053] S302. Mark the first word segmentation that is the same as any one of the second word segmentations as an identical word, and mark the first word segmentation that is different from all the second word segmentations as a different word.

[0054] Understandably, the first participle is the target participle, and the second participle is the comparison participle. There may be identical words between the target participle and the comparison participle. In one example, each target participle ("obtained", "already", "hypertension", "can still", "purchase", "insurance", "right") is compared one by one with each comparison participle ("hypertension", "purchase", "insurance"); when the comparison is the same, the corresponding target participle is marked as an identical word ("hypertension", "purchase", "insurance"), and when the comparison is different, the corresponding target participle is marked as a different word ("obtained", "already", "can still", "right"). The appearance of identical words indicates a certain similarity relationship between the target text and the comparison text. When calculating the contribution degrees of various aspects of similarity, different treatments need to be carried out for the contribution degrees of identical words and different words.

[0055] In this embodiment, by comparing one by one to distinguish identical words and different words, it is beneficial for the subsequent steps to calculate the differences between identical words and different words.

[0056] Optionally, the first participle includes identical words and different words; in step S30, that is, determining the feature weights of each of the first participles according to the first word normalization weights of each of the first participles and the second word normalization weights of each of the second participles includes:

[0057] S303. When the first participle is an identical word, perform a multiplication operation on the first word normalization weight and the second word normalization weight corresponding to the identical word to obtain the feature weight of the identical word;

[0058] S304. When the first participle is a different word, take the negative value of the first word normalization weight corresponding to the different word to obtain the feature weight of the different word.

[0059] Understandably, in one example, according to the normalization weights corresponding to the target participles and the comparison participles in Table 1, for the identical words ("hypertension", "purchase", "insurance"), perform a multiplication operation on the target participle normalization weight and the comparison participle normalization weight corresponding to each identical word as the feature weight; for the different words ("obtained", "already", "can still", "right"), directly take the negative value of the corresponding target participle normalization weight as the feature weight. The calculation results of the feature weights of each target participle are shown in Table 2.

[0060] Table 2 Feature Weights of Target Participles

[0061] Target word segmentation have to Already hypertension Still Buy Insurance Ma Target word segmentation feature weights -0.0055 -0.0071 0.1196 -0.0022 0.0451 0.1613 0.0000

[0062] This embodiment takes into account that the importance contribution degrees of identical words and different words are different in the similarity calculation of target participles and comparison participles, and differentiates between identical words and different words, making the text similarity calculation more accurate.

[0063] Optionally, the first word segmentation includes identical words and different words; in step S40, that is, determining the relative position weights of each of the first word segmentations according to the first word position parameters of each of the first word segmentations and the second word position parameters of each of the second word segmentations includes:

[0064] S401. When the first word segmentation is an identical word, perform a first exponential operation on the first word position parameter and the second word position parameter corresponding to the identical word to obtain the relative position weight of the identical word;

[0065] S402. When the first word segmentation is a different word, set the relative position weight of the different word to zero.

[0066] Understandably, in one example, according to the position parameters corresponding to the target word segmentation and the comparison word segmentation in Table 1, for identical words (hypertension, purchase, insurance), perform a first exponential operation on the target word position parameter and the comparison word position parameter corresponding to each identical word as the relative position weight. The base number in the first exponential operation can be preset, and the default setting is "e", and the exponent is the difference between the target word position parameter and the comparison word position parameter corresponding to each identical word. For example, for the identical word "hypertension", the corresponding target word position parameter is "3", and the corresponding comparison word position parameter is "1", then perform the first exponential operation "e -|(3-10| ", and the calculation result "e -2 " is used as the relative position weight of the identical word "hypertension". For different words (get, have, can still, right), directly assign a zero value as the relative position weight of the corresponding target word segmentation. The calculation results of the relative position weights of each target word segmentation are shown in Table 3.

[0067] Table 3 Relative Position Weights of Target Word Segmentations

[0068]

[0069] This embodiment takes into account that the relative positions of identical words in the similarity calculation of the target word segmentation and the comparison word segmentation are different, resulting in different orders, and differentiates between identical words and different words to obtain relative position weights, making the text similarity calculation more accurate.

[0070] Optionally, the first word segmentation includes identical words and different words; in step S50, that is, determining the context weights of each of the first word segmentations according to the first word normalization weights of each of the first word segmentations, the first position difference of the identical word in the first text, and the second position difference in the second text, includes:

[0071] S501. Obtain two of the multiple identical words and perform context marking to obtain an upper-context identical word and a lower-context identical word;

[0072] S502. Determine the first position difference based on the first word position parameter of the preceding identical word in the first text and the first word position parameter of the following identical word in the first text, and determine the first context position weight based on the first position difference; determine the second position difference based on the second word position parameter of the preceding identical word in the second text and the second word position parameter of the following identical word in the second text, and determine the second context position weight based on the first position difference and the second position difference;

[0073] S503. Obtain the first word normalization weight of the word in the following text, and multiply the first word normalization weight, the first context position weight, and the second context position weight of the word in the following text to obtain the context weight of the word in the preceding text.

[0074] S504. Set the context weight of the different words to zero.

[0075] Understandably, "preceding identical words" refers to a specific identical word, such as "hypertension"; "following identical words" refers to other identical words that have a contextual relationship with the specified identical word, such as "purchase". In one example, based on the positional parameters corresponding to the target word in Table 1, a second exponential operation is performed on the target word positional parameters corresponding to the preceding identical word "hypertension" and the following identical word "purchase" as the target context position weight, i.e., the first context position weight. The base in the second exponential operation can be preset, defaulting to "e", and the exponent is the difference between the target word positional parameters corresponding to the preceding and following identical words. For example, if the target word positional parameter corresponding to the preceding identical word "hypertension" is "3" and the target word positional parameter corresponding to the following identical word "purchase" is "5", then the second exponential operation "e" is performed. -|(5-3)| The calculation result is "e". -2 The target context position weight is used as the weight. Then, based on the position parameters corresponding to the compared words in Table 1, a third exponential operation is performed on the position parameters of the compared words corresponding to the same word "hypertension" in the preceding text and the compared words corresponding to the same word "purchase" in the following text, which is used as the weight of the compared context position, i.e., the second context position weight. The base in the third exponential operation can be preset, and the default value is set to "e". The exponent is the difference between the target word position parameters corresponding to the same word in the preceding text and the same word in the following text and the difference between the corresponding compared word position parameters, i.e., the difference between the first position difference and the second position difference. For example, if the position parameter of the compared words corresponding to the same word "hypertension" in the preceding text is "1" and the position parameter of the compared words corresponding to the same word "purchase" in the following text is "2", then the third exponential operation "e" is performed. -|(5-3)-(2-1))| The calculation result is "e". -1”Use it as the comparison context position weight. Then, according to Table 1, obtain the target word segmentation normalized weight of the same word "purchase" in the following text as "0.3414". Multiply the target word segmentation normalized weight, the target context position weight, and the comparison context position weight of the same word "purchase" in the following text to obtain the context weight of "purchase - hypertension" corresponding to the same word "hypertension" in the above text as "0.3414*e -2 *e -1 ”. Similarly, obtain the context weight of "insurance - hypertension" corresponding to the same word "hypertension" in the above text as "0.3484*e -3 *e -1 ”. Finally, perform an addition operation on the context weight of "purchase - hypertension" and the context weight of "insurance - hypertension" to obtain the context weight of the same word "hypertension" in the above text. For different words (got, le, can also, right), directly assign a zero value as the corresponding target word segmentation context weight. The calculation results of the context weights of each target word segmentation are shown in Table 4.

[0076] Table 4 Context Weights of Target Word Segmentations

[0077]

[0078]

[0079] This embodiment takes into account that the context relationships of the same word in the similarity calculation of the target word segmentation and the comparison word segmentation are different, resulting in different degrees of closeness. It differentiates the same word and different words to obtain the context weight, making the text similarity calculation have higher accuracy.

[0080] Optionally, in step S60, that is, determining the similarity of the first text to the second text according to the feature weight, the relative position weight, and the context weight includes:

[0081] S601. Obtain the feature weighting coefficient, the relative position weighting coefficient, and the context weighting coefficient;

[0082] S602. Perform a weighted calculation on the feature weight, the relative position weight, and the context weight according to the feature weighting coefficient, the relative position weighting coefficient, and the context weighting coefficient to obtain the similarity of the first text to the second text; the weighted calculation includes:

[0083] Sim = λ1 * ΣTW i + λ2 * ΣDW i + λ3 * ΣCW i

[0084] where Sim represents the similarity of the first text to the second text;

[0085] λ1 represents the feature weighting coefficient;

[0086] TW i This represents the feature weight of the i-th word in the first word segmentation;

[0087] i represents the position parameter of the first word in each of the first word segments;

[0088] λ2 represents the relative position weighting coefficient;

[0089] DW i This represents the relative position weight of the i-th word in the first word segmentation;

[0090] λ3 represents the context weighting coefficient;

[0091] CW i This represents the context weight of the i-th word in the first word segmentation.

[0092] Understandably, the feature weighting coefficient, relative position weighting coefficient, and context weighting coefficient can be preset. The default settings are all equal and equal to 1. Alternatively, the weighting coefficients can be adjusted through training a preset algorithm to obtain unequal weighting coefficient values. In one example, the feature weighting coefficient λ1, relative position weighting coefficient λ2, and context weighting coefficient λ2 for text similarity are all set to the default value of 1. The weight parameters (feature weight, relative position weight, and context weight) for each target word segmentation are shown in Table 5. Based on the weight parameters and corresponding weighting coefficients in Table 5, a weighted summation is performed, resulting in a similarity of 0.8434 between the target text and the comparison text.

[0093] Table 5 Weight parameters for text similarity calculation

[0094] Target word segmentation have to Already hypertension Still Buy Insurance Ma Position parameter i 1 2 3 4 5 6 7 <![CDATA[Feature weight TW i > -0.0055 -0.0071 0.1196 -0.0022 0.0451 0.1613 0.0000 <![CDATA[Relative position weight DW i > 0.0000 0.0000 0.1353 0.0000 0.0498 0.0498 0.0000 <![CDATA[Context Weight CW i > 0.0000 0.0000 0.0234 0.0000 0.1429 0.1310 0.0000

[0095] This embodiment considers multiple factors for weighted calculation, and by setting and adjusting the weighting coefficients, the text similarity calculation results have higher accuracy.

[0096] It should be understood that the sequence number of each step in the above embodiments does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.

[0097] In one embodiment, a text similarity calculation device is provided, which corresponds one-to-one with the text similarity calculation method in the above embodiments. For example... Figure 2As shown, the text similarity calculation device includes a word segmentation module 10, a weight normalization module 20, a feature weight calculation module 30, a relative position weight calculation module 40, a context weight calculation module 50, and a similarity calculation module 60. Detailed descriptions of each functional module are as follows:

[0098] The word segmentation module 10 is used to acquire a first text and a second text, perform word segmentation on the first text to obtain multiple first words, and perform word segmentation on the second text to obtain multiple second words.

[0099] The weight normalization module 20 is used to obtain the first word position parameters and first word normalization weights of each of the first word segments; and to obtain the second word position parameters and second word normalization weights of each of the second word segments.

[0100] The feature weight calculation module 30 is used to determine the feature weight of each first word based on the first word normalization weight of each first word and the second word normalization weight of each second word;

[0101] The relative position weight calculation module 40 is used to determine the relative position weight of each first word based on the first word position parameters of each first word and the second word position parameters of each second word.

[0102] The context weight calculation module 50 is used to determine the context weight of each of the first words based on the first word normalization weight of each of the first words, the first position difference of the same word in the first text and the second position difference in the second text.

[0103] The similarity calculation module 60 is used to determine the similarity between the first text and the second text based on the feature weight, the relative position weight and the context weight.

[0104] Optionally, the weight normalization module 20 includes:

[0105] The weight calculation unit is used to calculate the weight of each of the first words using a pre-trained weight algorithm to obtain the first word weight of each of the first words.

[0106] The normalization processing unit is used to normalize the weight of the first word to obtain the normalized weight of the first word for each of the first words.

[0107] Optionally, the feature weight calculation module 30 includes:

[0108] The word segmentation comparison unit is used to compare each of the first word segments with each of the second word segments one by one;

[0109] The word segmentation tagging unit is used to tag a first word that is the same as any of the second word as the same word, and to tag a first word that is different from all the second words as a different word.

[0110] Optionally, the feature weight calculation module 30 also includes:

[0111] The same word feature weight calculation unit is used to multiply the normalized weight of the first word and the normalized weight of the second word corresponding to the same word when the first word segmentation is the same word, to obtain the feature weight of the same word;

[0112] The different word feature weight calculation unit is used to take the negative value of the normalized weight of the first word corresponding to the different words when the first word segmentation is different words, so as to obtain the feature weight of the different words.

[0113] Optionally, the relative position weight calculation module 40 includes:

[0114] The relative position weight calculation unit for the same word is used to perform a first exponential operation on the first word position parameter and the second word position parameter corresponding to the same word when the first word segmentation is the same word, so as to obtain the relative position weight of the same word.

[0115] The relative position weight calculation unit for different words is used to set the relative position weight of the different words to zero when the first word segmentation consists of different words.

[0116] Optionally, the context weight calculation module 50 includes:

[0117] A context marker unit is used to obtain two identical words from a plurality of identical words and perform context markers to obtain the preceding identical word and the following identical word;

[0118] The context position weight calculation unit is used to determine the first position difference based on the first word position parameter of the preceding identical word in the first text and the first word position parameter of the following identical word in the first text, and to determine the first context position weight based on the first position difference; and to determine the second position difference based on the second word position parameter of the preceding identical word in the second text and the second word position parameter of the following identical word in the second text, and to determine the second context position weight based on the first position difference and the second position difference.

[0119] The same word context weight calculation unit is used to obtain the first word normalized weight of the same word in the following text, and to perform a multiplication operation on the first word normalized weight, the first context position weight and the second context position weight of the same word in the following text to obtain the context weight of the same word in the preceding text.

[0120] The different word context weight calculation unit is used to set the context weight of the different words to zero.

[0121] Optionally, the similarity calculation module 60 includes:

[0122] The weighting coefficient acquisition unit is used to acquire feature weighting coefficients, relative position weighting coefficients, and context weighting coefficients;

[0123] A weighted calculation unit is configured to perform a weighted calculation on the feature weights, relative position weights, and context weights based on the feature weighting coefficients, the relative position weighting coefficients, and the context weighting coefficients to obtain the similarity between the first text and the second text; the weighted calculation includes:

[0124]

[0125] Wherein, Sim represents the similarity between the first text and the second text;

[0126] λ1 represents the feature weighting coefficient;

[0127] TW i This represents the feature weight of the i-th word in the first word segmentation;

[0128] i represents the position parameter of the first word in each of the first word segments;

[0129] λ2 represents the relative position weighting coefficient;

[0130] DW i This represents the relative position weight of the i-th word in the first word segmentation;

[0131] λ3 represents the context weighting coefficient;

[0132] CW i This represents the context weight of the i-th word in the first word segmentation.

[0133] Specific limitations regarding the text similarity calculation device can be found in the limitations of the text similarity calculation method described above, and will not be repeated here. Each module in the aforementioned text similarity calculation device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device in hardware form, or stored in the memory of a computer device in software form, so that the processor can call and execute the operations corresponding to each module.

[0134] In one embodiment, a computer device is provided, which may be a terminal, and its internal structure diagram may be as follows: Figure 3As shown, the computer device includes a processor, memory, network interface, display screen, and input device connected via a system bus. The processor provides computational and control capabilities. The memory includes a readable storage medium and internal memory. The non-volatile storage medium stores the operating system and computer-readable instructions. The internal memory provides an environment for the operation of the operating system and computer-readable instructions in the readable storage medium. The network interface is used to communicate with an external server via a network connection. When the computer-readable instructions are executed by the processor, they implement a text similarity calculation method. The readable storage medium provided in this embodiment includes both non-volatile and volatile readable storage media.

[0135] In one embodiment, a computer device is provided, including a memory, a processor, and computer-readable instructions stored in the memory and executable on the processor, wherein the processor performs the following steps when executing the computer-readable instructions:

[0136] Obtain a first text and a second text; perform word segmentation on the first text to obtain multiple first words; perform word segmentation on the second text to obtain multiple second words;

[0137] Obtain the first word position parameters and first word normalization weights for each of the first word segments; obtain the second word position parameters and second word normalization weights for each of the second word segments;

[0138] The feature weights of each of the first words are determined based on the normalized weights of the first words of each of the first words and the normalized weights of the second words of each of the second words.

[0139] The relative position weight of each first word is determined based on the first word position parameters of each first word and the second word position parameters of each second word.

[0140] The context weight of each first word is determined based on the first word normalization weight of each first word, the first position difference of the same word in the first text, and the second position difference in the second text.

[0141] The similarity between the first text and the second text is determined based on the feature weights, the relative position weights, and the context weights.

[0142] In one embodiment, one or more computer-readable storage media storing computer-readable instructions are provided. The readable storage media provided in this embodiment include non-volatile readable storage media and volatile readable storage media. The readable storage media stores computer-readable instructions, which, when executed by one or more processors, perform the following steps:

[0143] Obtain a first text and a second text; perform word segmentation on the first text to obtain multiple first words; perform word segmentation on the second text to obtain multiple second words;

[0144] Obtain the first word position parameters and first word normalization weights for each of the first word segments; obtain the second word position parameters and second word normalization weights for each of the second word segments;

[0145] The feature weights of each of the first words are determined based on the normalized weights of the first words of each of the first words and the normalized weights of the second words of each of the second words.

[0146] The relative position weight of each first word is determined based on the first word position parameters of each first word and the second word position parameters of each second word.

[0147] The context weight of each first word is determined based on the first word normalization weight of each first word, the first position difference of the same word in the first text, and the second position difference in the second text.

[0148] The similarity between the first text and the second text is determined based on the feature weights, the relative position weights, and the context weights.

[0149] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by instructing related hardware with computer-readable instructions. These computer-readable instructions can be stored in a non-volatile readable storage medium or a volatile readable storage medium. When executed, these computer-readable instructions can include the processes of the embodiments of the above methods. Any references to memory, storage, databases, or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in a variety of forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), RAMbus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0150] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the above-described division of functional units and modules is used as an example. In practical applications, the above functions can be assigned to different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above.

[0151] The above-described embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention, and should all be included within the protection scope of the present invention.

Claims

1. A method for calculating text similarity, characterized in that, include: Obtain a first text and a second text; perform word segmentation on the first text to obtain multiple first words; The second text is segmented to obtain multiple second words; Obtain the first word position parameters and first word normalization weights for each of the first word segments; obtain the second word position parameters and second word normalization weights for each of the second word segments; The feature weights of each of the first words are determined based on the normalized weights of the first words of each of the first words and the normalized weights of the second words of each of the second words. The relative position weight of each first word is determined based on the first word position parameters of each first word and the second word position parameters of each second word. The context weight of each first word is determined based on the first word normalization weight of each first word, the first position difference of the same word in the first text, and the second position difference in the second text. The similarity between the first text and the second text is determined based on the feature weights, the relative position weights, and the context weights. The first word segmentation includes identical words and different words; The step of determining the relative position weight of each first segment based on the first word position parameters of each first segment and the second word position parameters of each second segment includes: When the first word segment is the same word, the first word position parameter and the second word position parameter corresponding to the same word are subjected to a first exponential operation to obtain the relative position weight of the same word; When the first word segmentation consists of different words, the relative position weights of the different words are set to zero; Determining the similarity between the first text and the second text based on the feature weights, the relative position weights, and the context weights includes: Obtain feature weighting coefficients, relative position weighting coefficients, and context weighting coefficients; The feature weights, relative position weights, and context weights are weighted and calculated based on the feature weighting coefficients, relative position weighting coefficients, and context weighting coefficients to obtain the similarity between the first text and the second text; the weighting calculation includes: in, This indicates the similarity between the first text and the second text; Indicates the weighting coefficients of the features; Indicates the first The feature weights of the first word segmentation; This represents the position parameter of the first word in each of the first word segments; This represents the relative position weighting coefficient; Indicates the first The relative position weight of the first word segmentation; Indicates the context-weighted coefficient; Indicates the first The context weight of the first word segment.

2. The text similarity calculation method as described in claim 1, characterized in that, The step of obtaining the first word normalization weight of each of the first word segments includes: The weights of each first word are obtained by calculating the weights of each first word through a pre-trained weighting algorithm. The weights of the first word are normalized to obtain the normalized weights of the first word for each of the first word segments.

3. The text similarity calculation method as described in claim 1, characterized in that, The first word segmentation includes identical words and different words; Before determining the feature weights of each of the first word segments based on the first word normalization weights and the second word normalization weights of each of the second word segments, the process includes: Each of the first word segments is compared with each of the second word segments one by one; First segment words that are the same as any of the second segment words are marked as identical words, and first segment words that are different from all the second segment words are marked as different words.

4. The text similarity calculation method as described in claim 1, characterized in that, The first word segmentation includes identical words and different words; The step of determining the feature weights of each of the first word segments based on the first word normalization weights of each of the first word segments and the second word normalization weights of each of the second word segments includes: When the first word segment is the same word, the normalized weight of the first word and the normalized weight of the second word corresponding to the same word are multiplied to obtain the feature weight of the same word. When the first word segmentation consists of different words, the normalized weights of the first words corresponding to the different words are negatively evaluated to obtain the feature weights of the different words.

5. The text similarity calculation method as described in claim 1, characterized in that, The first word segmentation includes identical words and different words; The step of determining the context weight of each first word based on its first word normalization weight, the first position difference of the same word in the first text, and the second position difference in the second text includes: Obtain two identical words from among multiple identical words and perform context tagging to obtain the preceding identical word and the following identical word; The first position difference is determined based on the first word position parameter of the preceding identical word in the first text and the first word position parameter of the following identical word in the first text, and the first context position weight is determined based on the first position difference; the second position difference is determined based on the second word position parameter of the preceding identical word in the second text and the second word position parameter of the following identical word in the second text, and the second context position weight is determined based on the first position difference and the second position difference. Obtain the first word normalized weight of the word in the following text, and multiply the first word normalized weight, the first context position weight, and the second context position weight of the word in the following text to obtain the context weight of the word in the preceding text. Set the context weights of the different words to zero.

6. A text similarity calculation device, characterized in that, include: The word segmentation module is used to obtain a first text and a second text, perform word segmentation on the first text, and obtain multiple first words. The second text is segmented to obtain multiple second words; The weight normalization module is used to obtain the first word position parameters and the first word normalization weights of each of the first word segments; and to obtain the second word position parameters and the second word normalization weights of each of the second word segments. The feature weight calculation module is used to determine the feature weight of each of the first words based on the normalized weight of the first word of each of the first words and the normalized weight of the second word of each of the second words; The relative position weight calculation module is used to determine the relative position weight of each first word based on the first word position parameters of each first word and the second word position parameters of each second word. The context weight calculation module is used to determine the context weight of each of the first words based on the first word normalization weight of each of the first words, the first position difference of the same word in the first text and the second position difference in the second text. A similarity calculation module is used to determine the similarity between the first text and the second text based on the feature weights, the relative position weights, and the context weights. The relative position weight calculation module includes: The relative position weight calculation unit for the same word is used to perform a first exponential operation on the first word position parameter and the second word position parameter corresponding to the same word when the first word segmentation is the same word, so as to obtain the relative position weight of the same word. The relative position weight calculation unit for different words is used to set the relative position weight of the different words to zero when the first word segmentation consists of different words; The similarity calculation module includes: The weighting coefficient acquisition unit is used to acquire feature weighting coefficients, relative position weighting coefficients, and context weighting coefficients; A weighted calculation unit is configured to perform a weighted calculation on the feature weights, relative position weights, and context weights based on the feature weighting coefficients, the relative position weighting coefficients, and the context weighting coefficients to obtain the similarity between the first text and the second text; the weighted calculation includes: in, This indicates the similarity between the first text and the second text; Indicates the weighting coefficients of the features; Indicates the first The feature weights of the first word segmentation; This represents the position parameter of the first word in each of the first word segments; This represents the relative position weighting coefficient; Indicates the first The relative position weight of the first word segmentation; Indicates the context-weighted coefficient; Indicates the first The context weight of the first word segment.

7. A computer device comprising a memory, a processor, and computer-readable instructions stored in the memory and executable on the processor, characterized in that, When the processor executes the computer-readable instructions, it implements the text similarity calculation method as described in any one of claims 1 to 5.

8. A computer-readable storage medium storing computer-readable instructions, characterized in that, When the computer-readable instructions are executed by one or more processors, the one or more processors cause the text similarity calculation method as described in any one of claims 1 to 5 to be performed.

Images