A similarity calculation method based on semantic editing fusion

Abstract

The application discloses a similarity calculation method based on semantic editing fusion, comprising the following steps: calculating the global semantic similarity between a source text and a target text by using a semantic coding model; obtaining a first keyword list of the source text and a second keyword list of the target text through word segmentation processing, taking each first keyword in the first keyword list as a target word, and respectively combining the target word with a corresponding keyword in the second keyword list and a plurality of second keywords adjacent to the corresponding keyword to form a plurality of to-be-compared keyword pairs; calculating the semantic similarity between each group of target words and corresponding keywords based on the to-be-compared keyword pairs by using the semantic coding model, and determining the editing distance between the source text and the target text; determining the local semantic similarity between the source text and the target text according to the editing distance, and obtaining the final text matching similarity by combining the global semantic similarity. The application solves the problem that text matching only depends on surface matching at the character level and ignores the semantic association between keywords, and improves the text matching precision.

Description

Technical Field

[0001] This invention belongs to the field of text matching technology, specifically relating to a similarity calculation method based on semantic editing fusion. Background Technology

[0002] In text matching tasks, accurately identifying texts with different characters but similar meanings is crucial for improving matching performance. Traditional methods primarily focus on two dimensions: character differences and semantic similarity. Character differences can be measured using edit distance, such as Levenshtein distance, to effectively capture spelling changes and character substitutions in short texts, thus performing well in handling input errors and non-standard spellings. However, edit distance only focuses on surface-level string changes and cannot identify expressions that are semantically similar but have significant surface differences, limiting its application in semantically driven text matching scenarios.

[0003] To address this deficiency, those skilled in the art have proposed semantic similarity methods. These methods rely on word vectors, semantic graphs, or sentence vector models, and can model word meaning and contextual relationships to a certain extent, making them particularly suitable for matching long texts or sentences with complex structures. However, these methods are not sensitive to surface perturbations such as spelling errors and word order changes, and are prone to mismatches when user input is not standardized, thus reducing the accuracy of text matching.

[0004] In recent years, Transformer-based pre-trained language models (such as BERT) have significantly improved text matching accuracy by deeply modeling contextual semantics, capturing semantic information at both the lexical and sentence levels. However, these models are computationally complex and often suffer from unstable accuracy and insufficient generalization ability when faced with real-world text matching tasks. Summary of the Invention

[0005] To address the aforementioned problems in existing technologies, this invention provides a similarity calculation method based on semantic editing fusion. The technical problem to be solved by this invention is achieved through the following technical solution: This invention provides a similarity calculation method based on semantic editing fusion, comprising: Calculate the global semantic similarity between the source and target texts using a semantic encoding model. ; The source text's first keyword list and the target text's second keyword list are obtained through word segmentation. Each first keyword in the first keyword list is taken as a target word. For each target word, multiple word pairs are formed with its corresponding word in the second keyword list and its adjacent second keywords. The corresponding word is the first keyword in the second keyword list... The second keyword, , Indicates the number of secondary keywords; Based on the word pairs to be compared, the semantic similarity between each target word and its corresponding word is calculated using a semantic coding model; Based on the semantic similarity, the edit distance between the source text and the target text is determined; The local semantic similarity between the source text and the target text is determined based on the edit distance, and then combined with the global semantic similarity to obtain the text matching similarity between the source text and the target text.

[0006] In one embodiment of the present invention, the step of calculating the semantic similarity between each pair of target words and their corresponding words using a semantic coding model based on the word pairs to be compared includes: For each target word, the semantic similarity of each word pair to be compared containing that target word is calculated using a semantic encoding model; Calculate the mean semantic similarity of all word pairs to be compared that contain the target word, and obtain the semantic similarity between the target word and the corresponding word.

[0007] In one embodiment of the present invention, the step of determining the edit distance between the source text and the target text based on the semantic similarity includes: When both the source text and the target text are non-empty texts, a dynamic cost function for the target word and the corresponding word is constructed using the semantic similarity. The edit distance between the source text and the target text is determined based on the dynamic cost function.

[0008] In one embodiment of the present invention, the dynamic cost function is expressed as: ; In the formula, Indicates the first in the source text The first keyword, This indicates the first in the target text. The second keyword, Indicated by For target words, with When they are corresponding words, and semantic similarity, This represents the first similarity threshold. This represents the second similarity threshold.

[0009] In one embodiment of the present invention, the step of determining the edit distance between the source text and the target text based on the dynamic cost function includes: Initial size is The edit distance matrix, This indicates the number of the first keyword; make ; make ; Using dynamic path planning algorithms, calculate the source text Center front First keyword and target text Center front Edit distance between secondary keywords ; The edit distance As the first in the edit distance matrix line, number The values ​​of the elements in the column; make And determine whether it satisfies If satisfied Then return to the method described above, which uses dynamic path planning algorithm to calculate the source text. Center front First keyword and target text Center front Edit distance between secondary keywords The steps; if not satisfied Then let And further determine whether it meets the requirements. ; If satisfied Then return the command. The steps; if not satisfied Then the edit distance matrix is ​​constructed; The first in the edit distance matrix line, number The values ​​of the column elements represent the edit distance between the source text and the target text. .

[0010] In one embodiment of the present invention, a dynamic path planning algorithm is used to calculate the source text. Center front First keyword and target text Center front Edit distance between secondary keywords The steps include: Using dynamic path planning algorithms, calculate respectively , and ; Will , and The minimum value in the source text is determined. Center front First keyword and target text Center front Edit distance between secondary keywords .

[0011] In one embodiment of the present invention, when the source text or the target text is empty text, the source text... Center front First keyword and target text Center front Edit distance between secondary keywords The length of the non-empty text in the source text and the target text.

[0012] In one embodiment of the present invention, the step of determining the local semantic similarity between the source text and the target text based on the edit distance, and further combining it with the global semantic similarity to obtain the text matching similarity between the source text and the target text, includes: Based on the edit distance, calculate the local semantic similarity between the source text and the target text. ; Using preset weighting coefficients For the local semantic similarity With the global semantic similarity The weighted summation yields the text matching similarity between the source text and the target text.

[0013] In one embodiment of the present invention, the local semantic similarity Represented as: ; In the formula, This represents the preset smoothing coefficient. This indicates the edit distance between the source text and the target text.

[0014] In one embodiment of the present invention, before the step of obtaining the first keyword list of the source text and the second keyword list of the target text through word segmentation, the method further includes: Determine the global semantic similarity Is it greater than or equal to 95%? If yes, return the similarity result; otherwise, perform the steps described above to obtain the first keyword list of the source text and the second keyword list of the target text through word segmentation.

[0015] Compared with the prior art, the beneficial effects of the present invention are as follows: 1. It should be understood that the existing Levenshtein Distence algorithm measures similarity by calculating the minimum number of edits (such as insertion, deletion, and replacement of characters) between two strings. This method only focuses on surface character differences, ignores contextual semantics, and cannot handle synonymous expressions. This application first uses a language model to calculate the global semantic similarity between the target text and the source text. It can directly capture deep semantic relationships. Then, after segmenting the source text and target text, it constructs word pairs containing the target words by taking each first keyword as the target word. Since the word pairs to be compared not only include word pairs formed by the target word and the corresponding word, but also word pairs formed by the target word and several second keywords adjacent to the corresponding word, the deviation of isolated word matching can be effectively avoided when calculating the semantic similarity between the target word and the corresponding word.

[0016] Furthermore, by introducing a dynamic cost function, this application also effectively improves the matching tolerance of semantically related text.

[0017] 2. By combining global semantic similarity and local semantic similarity, this application can achieve more accurate matching results in different scenarios, especially when dealing with texts that are semantically similar but have large surface differences.

[0018] The present invention will be further described in detail below with reference to the accompanying drawings and embodiments. Attached Figure Description

[0019] Figure 1 This is a flowchart of a similarity calculation method based on semantic editing fusion provided in an embodiment of the present invention; Figure 2 This is a schematic diagram illustrating the semantic similarity between word pairs to be compared, provided in an embodiment of the present invention. Detailed Implementation

[0020] The present invention will be further described in detail below with reference to specific embodiments, but the implementation of the present invention is not limited thereto.

[0021] Figure 1 This is a flowchart of a similarity calculation method based on semantic editing fusion provided in an embodiment of the present invention. Figure 1 As shown, this embodiment of the invention provides a similarity calculation method based on semantic editing fusion, including: S1. Calculate the global semantic similarity between the source text and the target text using a semantic coding model. .

[0022] Specifically, the source text refers to the text to be matched or queried, while the target text is the text to be compared with the source text, typically from a database, document collection, etc. In this embodiment, the semantic encoding model SentenceTransformer can be selectively used to calculate the global semantic similarity between the source and target texts. The Sentence Transformer model captures the semantic information of sentences by converting sentences or short texts into high-dimensional vector representations.

[0023] S2. Obtain a list of first keywords from the source text and a list of second keywords from the target text through word segmentation. Using each first keyword in the first keyword list as a target word, form multiple word pairs to be compared with the corresponding word in the second keyword list and its adjacent pre-defined number of second keywords. The corresponding word is the first keyword in the second keyword list... The second keyword, , This indicates the number of second keywords.

[0024] Optionally, before performing step S2, the global semantic similarity is determined. Is it greater than or equal to 95%? If yes, return the similarity result directly; otherwise, proceed to step S2, using the jieba word segmentation tool to analyze the source text. and target text Perform word segmentation on each part to obtain a list of the first keywords of the source text. and a list of secondary keywords for the target text , Represents source text The The first keyword, , Represents target text The The second keyword.

[0025] Next, using the first keyword from the list of keywords... The first keyword The target word is selected, and the second keywords from the second keyword list are selected sequentially. As the corresponding word, the target word With each corresponding word, and that corresponding word in the target text Several adjacent second keywords in the middle form word pairs to be compared: By iterating through the first keyword list, all the word pairs to be compared can be obtained. It should be noted that this only considers the corresponding words... Select from both the left and right sides Taking the case of a second keyword as an example, in reality, the number of second keywords selected on the left and right sides of the corresponding word can be equal or unequal, and this application does not limit this.

[0026] S3. Based on the word pairs to be compared, use a semantic coding model to calculate the semantic similarity between each target word and its corresponding word.

[0027] Specifically, step S3 includes: S301. For each target word, use a semantic encoding model such as Sentence Transformer to calculate the semantic similarity of each word pair to be compared that contains the target word.

[0028] Figure 2 This is a schematic diagram illustrating the semantic similarity between word pairs to be compared, provided in an embodiment of the present invention. Please refer to... Figure 2 Taking the source text "How to charge a parking space" and the target text "Charging a parking space now" as an example, after word segmentation, the first keyword list includes: parking space, charging, how to do it, and the second keyword list includes: now, parking space, charging. When the target word is the first keyword "parking space", the word pairs to be compared are: {(parking space, now), (parking space, parking space), (parking space, charging)}. When the target word is the first keyword "charging", the word pairs to be compared are: {(charging, now), (charging, parking space), (charging, charging)}. When the target word is "how to do it", the word pairs to be compared are: {(how to do it, now), (how to do it, parking space), (how to do it, charging)}.

[0029] Furthermore, the semantic similarity of each word pair to be compared can be determined through a semantic coding model. For example, the semantic similarity of the word pair to be compared (how to do, now) is 0.77, and the semantic similarity of the word pair to be compared (parking space, parking space) is 1.

[0030] S302. Calculate the mean semantic similarity of all word pairs to be compared that contain the target word, and obtain the semantic similarity between the target word and the corresponding word.

[0031] S4. Determine the edit distance between the source text and the target text based on semantic similarity.

[0032] Optionally, in step S4, when both the source text and the target text are non-empty texts, a dynamic cost function between the target word and its corresponding word is constructed using semantic similarity, and the edit distance between the source text and the target text is further determined based on the dynamic cost function.

[0033] Specifically, the cost function is used to quantify the cost of each editing operation, such as insertion, deletion, and replacement, during text matching. Existing cost functions are as follows: ; In the formula, Indicated by For target words, with When they are corresponding words, and semantic similarity, This represents a power factor that adjusts the magnitude of editing operation costs, used to control the non-linearity of cost growth.

[0034] In this embodiment, different calculation measures are adopted for different local semantic relevance: for highly similar matching strings, the editing operation cost is reduced, while for highly similar matching strings, the calculation cost is increased. This allows for better differentiation of textual relevance and capture of semantic relevance, thereby improving semantic separation. Combining semantic similarity... By updating the cost function, we can obtain the dynamic cost function: ; In the formula, Indicates the first in the source text The first keyword, Indicates the first in the target text The second keyword, Indicated by For target words, with When they are corresponding words, and semantic similarity, This represents the first similarity threshold. This represents the second similarity threshold.

[0035] This embodiment employs a dynamic path planning (DP) algorithm to calculate the edit distance between the source and target texts. By introducing a dynamic cost function based on semantic similarity, the calculation of editing operations is adaptively adjusted, thereby more accurately reflecting the semantic similarity between texts. By dynamically adjusting the edit distance weights, the matching tolerance for semantically similar texts, such as synonyms and related expressions, is improved, while maintaining strict discrimination against irrelevant texts and finely distinguishing text relationships at different similarity levels.

[0036] Further, in step S4, the step of determining the edit distance between the source text and the target text based on the dynamic cost function includes: S401, Initial size is The edit distance matrix, This indicates the number of primary keywords.

[0037] S402, Order .

[0038] S403, Order .

[0039] S404. Using dynamic path planning algorithm, calculate the source text. Center front First keyword and target text Center front Edit distance between secondary keywords .

[0040] Specifically, step S404 includes: Using dynamic path planning algorithms, calculate respectively , and ,Will , and The minimum value in the source text is determined. Center front First keyword and target text Center front Edit distance between secondary keywords The corresponding formula is expressed as follows: .

[0041] On the other hand, when the source text or target text is empty, the source text Center front First keyword and target text Center front Edit distance between secondary keywords This represents the length of the non-empty text in both the source and target texts.

[0042] S405, Edit Distance As the first in the edit distance matrix line, number The values ​​of the elements in the column.

[0043] S406, Order .

[0044] S407, Determine if the condition is met. If satisfied Then, the result is returned by using a dynamic path planning algorithm to calculate the source text. Center front First keyword and target text Center front Edit distance between secondary keywords The steps.

[0045] S408, if not satisfied Then let And further determine whether it meets the requirements. If satisfied Then return the command. The steps; if not satisfied Then the edit distance matrix is ​​constructed.

[0046] S409, Edit the distance matrix in the first... line, number The values ​​of the column elements represent the edit distance between the source and target text. .

[0047] It should be understood that the dynamic path planning algorithm accumulates the cost of each editing step by constructing an edit distance matrix. The final result is reflected in the value in the lower right corner of the matrix. The value of this element represents the total cost of the overall optimal matching path from the first character to the last character. The smaller the value, the higher the degree of matching between the two source texts and the target text.

[0048] S5. Determine the local semantic similarity between the source text and the target text based on the edit distance, and further combine it with the global semantic similarity to obtain the text matching similarity between the source text and the target text.

[0049] Specifically, step S5 includes: S501. Calculate the local semantic similarity between the source text and the target text based on the edit distance. Local semantic similarity Represented as: ; In the formula, This represents the preset smoothing coefficient. This indicates the edit distance between the source text and the target text.

[0050] S502, using preset weighting coefficients For local semantic similarity With global semantic similarity The weighted sum is used to obtain the text matching similarity between the source text and the target text.

[0051] For example, the text matching similarity between the source text and the target text is represented as: .

[0052] It is used to balance the contribution ratio of global semantic similarity and local semantic similarity in text matching similarity calculation, and its value range is [0,1].

[0053] Taking the source text "How to charge a parking space" and the target text "Charging a parking space now" as an example, this invention and the existing Levenshtein Distence algorithm are used to calculate the first... The first keyword and the target text The edit distances between the secondary keywords are shown in Tables 1 and 2 below: Table 1

[0054] Table 2

[0055] Furthermore, the text similarity between the source text "How to charge a parking space" and the target text "Charging a parking space now" was calculated using the semantic editing fusion-based similarity calculation method provided in this invention, the existing Levenshtein Distence algorithm, and the SentenceTransformer language model, respectively. The results are shown in Table 3. Table 3

[0056] As can be seen from the above embodiments, the beneficial effects of the present invention are as follows: 1. It should be understood that the existing Levenshtein Distence algorithm measures similarity by calculating the minimum number of edits (such as insertion, deletion, and replacement of characters) between two strings. This method only focuses on surface character differences, ignores contextual semantics, and cannot handle synonymous expressions. This application first uses a language model to calculate the global semantic similarity between the target text and the source text. It can directly capture deep semantic relationships. Then, after segmenting the source text and target text, it constructs word pairs containing the target words by taking each first keyword as the target word. Since the word pairs to be compared not only include word pairs formed by the target word and the corresponding word, but also word pairs formed by the target word and several second keywords adjacent to the corresponding word, the deviation of isolated word matching can be effectively avoided when calculating the semantic similarity between the target word and the corresponding word.

[0057] Furthermore, by introducing a dynamic cost function, this application also effectively improves the matching tolerance of semantically related text.

[0058] 2. By combining global semantic similarity and local semantic similarity, this application can achieve more accurate matching results in different scenarios, especially when dealing with texts that are semantically similar but have large surface differences.

[0059] In the description of this invention, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of indicated technical features. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of that feature. In the description of this invention, "a plurality of" means two or more, unless otherwise explicitly specified.

[0060] The use of terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples" indicates that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of the present invention. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples. In addition, those skilled in the art can combine and integrate the different embodiments or examples described in this specification.

[0061] The above description, in conjunction with specific preferred embodiments, provides a further detailed explanation of the present invention. It should not be construed that the specific implementation of the present invention is limited to these descriptions. For those skilled in the art, various simple deductions or substitutions can be made without departing from the concept of the present invention, and all such modifications and substitutions should be considered within the scope of protection of the present invention.

Claims

1. A similarity calculation method based on semantic editing fusion, characterized in that, include: Calculate the global semantic similarity between the source and target texts using a semantic encoding model. ; The source text's first keyword list and the target text's second keyword list are obtained through word segmentation. Each first keyword in the first keyword list is taken as a target word. For each target word, multiple word pairs are formed with its corresponding word in the second keyword list and its adjacent second keywords. The corresponding word is the first keyword in the second keyword list... The second keyword, , Indicates the number of secondary keywords; For each target word, the semantic similarity of each word pair to be compared containing that target word is calculated using a semantic encoding model; Calculate the mean semantic similarity of all word pairs to be compared that contain the target word, and obtain the semantic similarity between the target word and the corresponding word; When both the source text and the target text are non-empty texts, a dynamic cost function for the target word and its corresponding word is constructed using the semantic similarity; the dynamic cost function is expressed as: ； In the formula, Indicates the first in the source text The first keyword, This indicates the first in the target text. The second keyword, Indicated by For target words, with When they are corresponding words, and semantic similarity, This represents the first similarity threshold. This represents the second similarity threshold; The edit distance between the source text and the target text is determined based on the dynamic cost function. Based on the edit distance, calculate the local semantic similarity between the source text and the target text. The local semantic similarity Represented as: ； In the formula, This represents the preset smoothing coefficient. This indicates the edit distance between the source text and the target text. This indicates the number of the first keyword; Using preset weighting coefficients For the local semantic similarity With the global semantic similarity The weighted summation yields the text matching similarity between the source text and the target text.

2. The similarity calculation method based on semantic editing fusion according to claim 1, characterized in that, The step of determining the edit distance between the source text and the target text based on the dynamic cost function includes: Initial size is The edit distance matrix, Indicates the number of second keywords. This indicates the number of the first keyword; make ; make ; Using dynamic path planning algorithms, calculate the source text Center front First keyword and target text Center front Edit distance between secondary keywords ; The edit distance As the first in the edit distance matrix line, number The values ​​of the elements in the column; make And determine whether it satisfies If satisfied Then return to the method described above, which uses dynamic path planning algorithm to calculate the source text. Center front First keyword and target text Center front Edit distance between secondary keywords The steps; if not satisfied Then let And further determine whether it meets the requirements. ; If satisfied Then return the command. The steps; if not satisfied Then the edit distance matrix is ​​constructed; The first in the edit distance matrix line, number The values ​​of the column elements represent the edit distance between the source text and the target text. .

3. The similarity calculation method based on semantic editing fusion according to claim 2, characterized in that, Using dynamic path planning algorithms, calculate the source text Center front First keyword and target text Center front Edit distance between secondary keywords The steps include: Using dynamic path planning algorithms, calculate respectively , and ; Will , and The minimum value in the source text is determined. Center front First keyword and target text Center front Edit distance between secondary keywords .

4. The similarity calculation method based on semantic editing fusion according to claim 1, characterized in that, When the source text or the target text is empty text, the source text Center front First keyword and target text Center front Edit distance between secondary keywords The length of the non-empty text in the source text and the target text.

5. The similarity calculation method based on semantic editing fusion according to claim 1, characterized in that, Before the step of obtaining the first keyword list of the source text and the second keyword list of the target text through word segmentation, the method further includes: Determine the global semantic similarity Is it greater than or equal to 95%? If yes, return the similarity result; otherwise, perform the steps described above to obtain the first keyword list of the source text and the second keyword list of the target text through word segmentation.

Images