A text similarity calculation method based on scene information enhancement

Abstract

The application discloses a text similarity calculation method based on scene information enhancement, comprising the following steps: obtaining two texts, generating scene posterior and target label and calculating consistency coefficient; using pre-training Transform coding and applying scene gating to obtain two scene weighted embeddings; performing scene modulation and counterfactual construction, combining intervention invariance and adversarial debiasing training to obtain debiased embedding; calculating weighted Jaccard lexical difference according to the target scene; fusing lexical difference and semantic distance to construct cost, solving transmission plan by entropy regularization to obtain scene-coupled optimal transmission distance; calculating semantic similarity and fusing conditional value at risk to obtain robust semantic similarity; mapping the transmission distance to alignment similarity, and weighting the robust semantic similarity to output comprehensive similarity. The application improves the accuracy and robustness of cross-scene similarity determination.

Description

Technical Field

[0001] This invention relates to the field of natural language processing technology, and in particular to a text similarity calculation method based on scene information enhancement. Background Technology

[0002] In existing text similarity research, common approaches are divided into two categories: lexical metrics and semantic metrics. Lexical metrics are based on word segmentation and word frequency, combined with stop word processing and normalization, to calculate indicators such as Jaccard, overlap rate, and edit distance. Semantic metrics are based on pre-trained Transformers, extracting sentence vectors or word vectors, and scoring them using cosine similarity. Some methods introduce interactive encoding or attention matching. Both types of methods are usable on general corpora, but they lack stability when facing polysemy, domain transfer, and long-tail expressions. They tend to compress scene differences into a unified vector space, leading to cross-scene matching bias.

[0003] Research on scene factors often remains at the level of domain adaptation or prior label weighting. Common strategies involve constructing prior weights based on domain vocabularies, topic distributions, or historical clicks, and then simply fusing them in the vector space. Existing solutions typically lack clear scene posterior distribution modeling and target scene label selection processes, lack consistency quantification indicators that simultaneously characterize two inputs, and lack a scene-related lexical weighting system. Lexical similarity and semantic distance are often used in parallel but lack a unified cost construction and transmission mechanism. The resulting mismatch phenomenon is particularly prominent in e-commerce retrieval, question-answering matching, and public opinion merging. Texts that are semantically similar but have inconsistent scenes are easily judged as highly similar, while texts with significant semantic differences but consistent scenes are given lower scores.

[0004] In terms of robustness, common methods lack targeted handling of bias sources. The training process usually does not distinguish between target scenes and counterfactual scenes, lacks a systematic generation process for scene modulation functions and counterfactual embeddings, and is difficult to suppress scene-irrelevant factors by intervening invariance constraints and adversarial debiasing constraints. The similarity fusion stage often adopts simple weighted averaging without introducing risk-sensitive criteria and lacks control over tail mismatches. At the alignment level, common optimal transmission practices are mostly based on fixed costs or only semantic distance, and do not unify scene-weighted Jaccard lexical differences and scene-weighted semantic distance into the cost matrix. The transmission kernel lacks temperature adjustment based on scene consistency coefficients, making it difficult to stably present alignment strength in cross-scene matching.

[0005] Therefore, how to provide a text similarity calculation method based on scene information enhancement is a problem that urgently needs to be solved by those skilled in the art. Summary of the Invention

[0006] One objective of this invention is to propose a text similarity calculation method based on scene information enhancement. This invention utilizes techniques such as scene posterior distribution modeling, scene-gated embedding modulation, joint training of intervention invariance and adversarial debiasing, scene-weighted Jaccard lexical differences, and entropy regularization optimal transmission to achieve collaborative modeling of text semantics and scene factors. It has the advantages of more accurate semantic matching, higher cross-scene robustness, and stronger consistency in similarity calculation.

[0007] A text similarity calculation method based on scene information enhancement according to an embodiment of the present invention includes the following steps:

[0008] Obtain the first text and the second text, generate their respective posterior distributions of the scene based on the predefined scene library and historical behavior, determine their respective target scene labels, and calculate the scene consistency coefficient;

[0009] An initial embedding is generated by pre-training a Transformer. Gating weights are applied to the embedding dimension according to the target scene to obtain the first scene weighted embedding and the second scene weighted embedding.

[0010] Based on the target scene label, the initial embedded application scene modulation function is used to generate counterfactual embeddings. The first debiased embedding and the second debiased embedding are obtained by joint training with intervention invariance constraints and adversarial debiasing constraints.

[0011] Assign weights to terms based on the target scenario and calculate scenario-weighted Jaccard lexical differences.

[0012] The scene weighted semantic distance is calculated based on the first scene weighted embedding and the second scene weighted embedding. The optimal transmission cost matrix is ​​constructed with the scene weighted Jaccard lexical difference and the scene weighted semantic distance. The scene consistency coefficient is used to modulate the entropy regularization optimal transmission. The transmission plan is solved and the scene coupling optimal transmission distance is obtained.

[0013] Multiple sets of semantic similarities are calculated between the first debiased embedding, the second debiased embedding and the corresponding counterfactual pair, and risk-sensitive fusion is performed using the conditional value at risk method to obtain robust semantic similarity.

[0014] The optimal transmission distance of scene coupling is converted into alignment similarity, and then weighted and fused with robust semantic similarity to obtain a comprehensive similarity.

[0015] Optionally, the generation of the scene posterior distribution, the determination of the target scene label, and the calculation of the scene consistency coefficient specifically include:

[0016] Two input channels are set up. The first input channel is connected to the first data source, and the second input channel is connected to the second data source. Two natural language texts are received, which are denoted as the first text and the second text, respectively.

[0017] Read the predefined scene library, create a scene tag set, and set a unique number and name for each scene tag;

[0018] Four types of scene matching features are generated for the first text: keyword matching score, semantic matching score, context information score, and historical behavior score. These features are then aggregated into each scene label according to preset weights to form a comprehensive scoring sequence for the first text.

[0019] For the second text, keyword matching score, semantic matching score, context information score, and historical behavior score are generated and aggregated into each scene tag according to the same weight system to form a comprehensive scoring sequence for the second text;

[0020] The exponential mapping and sum-normalization processes are performed on the two comprehensive scoring sequences respectively to obtain the scene posterior distribution of the first text and the scene posterior distribution of the second text.

[0021] The scene with the highest probability in the posterior distribution of the first text is selected as the target scene label of the first text, and the scene with the highest probability in the posterior distribution of the second text is selected as the target scene label of the second text.

[0022] Calculate the scene consistency coefficient: For each scene label, take the square root of the posterior probability of the first text and the second text on the corresponding scene label and multiply them. Add up the products of all scene labels and use the result as the scene consistency coefficient, which ranges from zero to one.

[0023] Optionally, the generation of the first scene-weighted embedding and the second scene-weighted embedding specifically includes:

[0024] Load the pre-trained Transformer and set the embedding dimensions;

[0025] The first text and the second text are encoded respectively to obtain the first initial embedding and the second initial embedding;

[0026] Establish a gating weight table indexed by scene labels, and configure a gating weight vector of the same length as the embedding dimension for each scene label;

[0027] Based on the target scene label of the first text, the gate weight vector is read from the gate weight table, and weights are applied dimension by dimension in the embedding dimension to obtain the weighted embedding of the first scene.

[0028] Based on the target scene label of the second text, the gate weight vector is read from the gate weight table, and weights are applied dimension by dimension in the embedding dimension to obtain the weighted embedding of the second scene.

[0029] Optionally, the scene modulation function and counterfactual embedding generation and bias removal training specifically include:

[0030] A scene modulation function is established, and a parameter table indexed by scene labels is set. Each scene label in the parameter table is associated with four types of parameters, which are consistent with the embedding dimension length. These four types of parameters are the dimensional scaling vector, the dimensional translation vector, the gating vector, and the temperature coefficient. The scene modulation function takes the initial embedding as input and performs normalization, dimensional scaling, dimensional translation, dimensional gating, temperature scaling, and interval clipping in sequence. The output scene modulation embedding is then used. The dimensional scaling vector is used for scaling, the dimensional translation vector is used for bias adjustment, the gating vector is used for dimensional selection, and the temperature coefficient is used for numerical stretching and shrinking.

[0031] Based on the target scene labels of the first text and the second text, two target scene modulation embeddings are obtained by applying the two initial embedding application scene modulation functions respectively.

[0032] A set of scene tags that is not equal to the target scene tag of the first text is selected from the set of scene tags according to a preset number, and a first counterfactual embedding set is generated by applying a scene modulation function to the initial embedding of the first text. A set of scene tags that is not equal to the target scene tag of the second text is selected from the set of scene tags according to a preset number, and a second counterfactual embedding set is generated by applying a scene modulation function to the initial embedding of the second text.

[0033] Set up a projection mapping to map the two target scene modulation embeddings and the two sets of counterfactual embeddings to a fixed-dimensional projection space;

[0034] Define the intervention invariance constraint as follows:

[0035] ;

[0036] in, To intervene in invariance loss, For the initial embedding of the first text, For the initial embedding of the second text, The target scene label for the first text. The target scene label for the second text. and For the scene modulation function instance associated with the target scene label, To match scene tags Associated scene modulation function instance, For projection mapping, This is the first set of counterfactual scenario labels. This is the second set of counterfactual scenario labels. For each set of counterfactual scenario labels, the number of labels. The formula for the L2 norm, which is calculated in the projection space, is based on the principle of measuring the difference between the modulated embedding of the target scene and the corresponding counterfactual embedding in the unified projection space by means of samples.

[0037] A discriminator is set up to perform scene label discrimination on the projection map output to form an adversarial debiasing constraint. This constraint, together with the intervention invariance constraint, forms a joint training objective. The scene modulation function, projection map, and discriminator parameters are updated through gradient descent to obtain the first debiasing embedding and the second debiasing embedding.

[0038] Optionally, the generation of the scenario-weighted Jaccard lexical differences specifically includes:

[0039] The first and second texts are segmented, stop words are removed, and normalization is performed to form the first term table and the second term table, and the word frequency of each term is recorded;

[0040] A first scene term weight table is generated based on the target scene labels of the first text, and a second scene term weight table is generated based on the target scene labels of the second text.

[0041] In the first term table and the second term table, the term frequency of each term is multiplied by the corresponding scenario term weight to obtain the first scenario weighted term value and the second scenario weighted term value;

[0042] A unified term set is established, which contains all different terms appearing in the first and second texts. The term set is then calculated according to the "scene-weighted Jaccard lexical method": for each term, the first scene-weighted term value is compared with the second scene-weighted term value. The smaller value is accumulated as the intersection sum value, and the larger value is accumulated as the union sum value. The ratio of the intersection sum value to the union sum value is used as the scene-weighted Jaccard lexical similarity.

[0043] Subtract the scene-weighted Jaccard lexical similarity from the first value to obtain the scene-weighted Jaccard lexical difference.

[0044] Optionally, the generation of the optimal transmission distance for scene coupling specifically includes:

[0045] An index is built using a unified term set to locate the corresponding term positions in the first and second texts. The term embeddings output by the pre-trained Transformer are extracted. Gating weights are applied to the embedding dimension according to the target scene to obtain the scene-weighted embeddings of the first term and the second term. The distance between any term pair is calculated according to the rule of summing the squared differences of each dimension and then taking the square root, forming a scene-weighted semantic distance matrix.

[0046] Based on a unified term set, the weighted term values ​​of the first and second scenarios are read. For the same term, the ratio of the sum of the intersection and the sum of the union is calculated to obtain the scenario-weighted Jaccard lexical similarity. In order to uniformly map the similarity as a cost, the similarity is subtracted by one to make the value fall between zero and one and maintain the same order: zero for completely identical terms, zero for partial overlap, and one for completely disjoint terms. For different terms, the intersection of the weighted Jaccard is zero and the union is positive, the similarity is zero, and the difference is one. The lexical difference is obtained by subtracting the current similarity from one. The lexical difference is set to one for different terms to form a scenario-weighted Jaccard lexical difference matrix.

[0047] By setting the lexical difference weight and semantic distance weight to non-negative constants, the scene-weighted Jaccard lexical difference matrix and the scene-weighted semantic distance matrix are linearly weighted and synthesized at the corresponding positions to obtain the optimal transmission cost matrix.

[0048] The temperature coefficient is calculated from the input scene consistency coefficient. The temperature coefficient is then combined with the entropy regularization coefficient to construct the transmission kernel and complete the transmission kernel initialization.

[0049] Perform Sinkhorn iterations on the transmission kernel, alternating between scaling the row and column directions, so that the row edge distribution is equal to the probability vector obtained by summing and normalizing the weighted term values ​​of the first scenario, and the column edge distribution is equal to the probability vector obtained by summing and normalizing the weighted term values ​​of the second scenario. After convergence, the transmission plan is obtained.

[0050] Multiply the optimal transmission cost matrix and the transmission plan item by item at the corresponding positions, and sum them up over the entire matrix to obtain a single value, which is then denoted as the optimal transmission distance for scene coupling.

[0051] Optionally, the generation of robust semantic similarity specifically includes:

[0052] The input objects are explicitly defined as the first debiased embedding, the second debiased embedding, the first counterfactual embedding set, and the second counterfactual embedding set;

[0053] Pairings are established in a fixed order: the first debiased embedding is paired with the second debiased embedding, the first debiased embedding is paired with each element in the second counterfactual embedding set, each element in the first counterfactual embedding set is paired with the second debiased embedding, and each element in the first counterfactual embedding set is paired with each element in the second counterfactual embedding set.

[0054] For each pair, semantic similarity is calculated. The calculation rule is to multiply the two vectors dimension by dimension and sum them to obtain the inner product. Then, the lengths of the two vectors are calculated separately. The cosine similarity is obtained by the ratio of the inner product to the product of the two lengths. The results are recorded as a semantic similarity sequence according to the pairing order.

[0055] The semantic similarity sequence is converted into a risk metric sequence. The conversion rule is to subtract the corresponding value from each semantic similarity to obtain the risk metric, while maintaining consistency with the pairing order.

[0056] The conditional risk value is calculated by setting quantile parameters and taking values ​​sequentially within the allowed range. For each value, the tail excess average is calculated. The tail excess average is calculated by comparing each risk measure with the threshold. If the value is greater than the threshold, the difference is taken; if it is not greater than the threshold, the difference is taken. The sum is divided by the sample size and then divided by the quantile parameter. Since the conditional risk value is averaged based on the tail probability quality, it needs to be normalized by the quantile parameter. The excess portion above the threshold is converted into the average level of the tail interval. The current threshold is added to the tail excess average to form the target value. The smallest target value obtained from all the values ​​is selected as the conditional risk value.

[0057] Robust semantic similarity is obtained by subtracting the conditional risk value from one. Since the conditional risk value describes the degree of risk, subtracting the conditional risk value from one can map the risk scale to the similarity scale, so that a smaller risk corresponds to a larger robust semantic similarity.

[0058] Optionally, the generation of the comprehensive similarity specifically includes:

[0059] The input parameters are explicitly defined as the optimal transmission distance for scene coupling and robust semantic similarity.

[0060] The optimal transmission distance for scene coupling is converted into alignment similarity through a monotonically decreasing mapping, and the mapping output is limited to the range of zero to one.

[0061] Set alignment similarity weight and robust semantic similarity weight, with both weights being non-negative and adding up to one;

[0062] The alignment similarity and robust semantic similarity are weighted and fused to obtain the comprehensive similarity.

[0063] The beneficial effects of this invention are:

[0064] This invention introduces scene posterior distribution and target scene labels, enabling similarity calculation to maintain stable judgment under conditions of polysemy and domain migration. Scene consistency coefficient is used to quantify the scene proximity of two text segments. Scene gating is finely adjusted in the embedding dimension. Scene-weighted term weights are used to construct scene-weighted Jaccard lexical differences. The two clues jointly participate in the construction of the optimal transmission cost matrix, aligning lexical and semantic functions within the same framework. The entropy regularization transmission kernel uses the scene consistency coefficient as the temperature regulation input. The transmission plan is solved under probability conservation constraints. The obtained scene coupling optimal transmission distance can reflect the alignment strength across texts and can be monotonically mapped to alignment similarity for fusion.

[0065] At the representation end, target scene modulation embeddings are generated through scene modulation functions, and intervention pairs are constructed using counterfactual embeddings. By jointly using intervention invariance constraints and adversarial debiasing constraints, a purer debiased embedding is obtained. At the similarity end, a similarity sample set is formed using debiased embeddings and counterfactual pairs. Risk-sensitive fusion is performed using conditional risk values. Robust semantic similarity is more sensitive to tail mismatches. At the fusion end, alignment similarity and robust semantic similarity are combined into a comprehensive similarity according to weights. The entire process, from scene recognition, representation modulation, alignment transmission to risk fusion, is interconnected, enabling relevant texts to be matched more accurately and irrelevant texts to be effectively separated. The system maintains stable performance in multi-scenario applications. Attached Figure Description

[0066] The accompanying drawings are provided to further illustrate the invention and form part of the specification. They are used in conjunction with embodiments of the invention to explain the invention and do not constitute a limitation thereof. In the drawings:

[0067] Figure 1 This is a flowchart of a text similarity calculation method based on scene information enhancement proposed in this invention;

[0068] Figure 2 This is a schematic diagram illustrating the generation of the scene posterior distribution for a text similarity calculation method based on scene information enhancement proposed in this invention.

[0069] Figure 3 This is a schematic diagram illustrating the optimal transmission solution for a text similarity calculation method based on scene information enhancement proposed in this invention. Detailed Implementation

[0070] The present invention will now be described in further detail with reference to the accompanying drawings. These drawings are simplified schematic diagrams, illustrating only the basic structure of the invention, and therefore only show the components relevant to the invention.

[0071] refer to Figures 1-3 A text similarity calculation method based on scene information enhancement includes the following steps:

[0072] Obtain the first text and the second text, generate their respective posterior distributions of the scene based on the predefined scene library and historical behavior, determine their respective target scene labels, and calculate the scene consistency coefficient;

[0073] An initial embedding is generated by pre-training a Transformer. Gating weights are applied to the embedding dimension according to the target scene to obtain the first scene weighted embedding and the second scene weighted embedding.

[0074] Based on the target scene label, the initial embedded application scene modulation function is used to generate counterfactual embeddings. The first debiased embedding and the second debiased embedding are obtained by joint training with intervention invariance constraints and adversarial debiasing constraints.

[0075] Assign weights to terms based on the target scenario and calculate scenario-weighted Jaccard lexical differences;

[0076] The scene weighted semantic distance is calculated based on the first scene weighted embedding and the second scene weighted embedding. The optimal transmission cost matrix is ​​constructed with the scene weighted Jaccard lexical difference and the scene weighted semantic distance. The scene consistency coefficient is used to modulate the entropy regularization optimal transmission. The transmission plan is solved and the scene coupling optimal transmission distance is obtained.

[0077] Multiple sets of semantic similarities are calculated between the first debiased embedding, the second debiased embedding and the corresponding counterfactual pair, and risk-sensitive fusion is performed using the conditional value at risk method to obtain robust semantic similarity.

[0078] The optimal transmission distance of scene coupling is converted into alignment similarity, and then weighted and fused with robust semantic similarity to obtain a comprehensive similarity.

[0079] In this embodiment, the generation of the scene posterior distribution, the determination of the target scene label, and the calculation of the scene consistency coefficient specifically include:

[0080] Two input channels are set up. The first input channel is connected to the first data source, and the second input channel is connected to the second data source. Two natural language texts are received, which are denoted as the first text and the second text, respectively.

[0081] Read the predefined scene library, create a scene tag set, and set a unique number and name for each scene tag;

[0082] Four types of scene matching features are generated for the first text: keyword matching score, semantic matching score, context information score, and historical behavior score. These features are then aggregated into each scene label according to preset weights to form a comprehensive scoring sequence for the first text.

[0083] For the second text, keyword matching score, semantic matching score, context information score, and historical behavior score are generated and aggregated into each scene tag according to the same weight system to form a comprehensive scoring sequence for the second text;

[0084] The exponential mapping and sum-normalization processes are performed on the two comprehensive scoring sequences respectively to obtain the scene posterior distribution of the first text and the scene posterior distribution of the second text.

[0085] The scene with the highest probability in the posterior distribution of the first text is selected as the target scene label of the first text, and the scene with the highest probability in the posterior distribution of the second text is selected as the target scene label of the second text.

[0086] Calculate the scene consistency coefficient: For each scene label, take the square root of the posterior probability of the first text and the second text on the corresponding scene label and multiply them. Add up the products of all scene labels and use the result as the scene consistency coefficient, which ranges from zero to one.

[0087] In this embodiment, the generation of the first scene-weighted embedding and the second scene-weighted embedding specifically includes:

[0088] Load the pre-trained Transformer and set the embedding dimensions;

[0089] The first text and the second text are encoded respectively to obtain the first initial embedding and the second initial embedding;

[0090] Establish a gating weight table indexed by scene labels, and configure a gating weight vector of the same length as the embedding dimension for each scene label;

[0091] Based on the target scene label of the first text, the gate weight vector is read from the gate weight table, and weights are applied dimension by dimension in the embedding dimension to obtain the weighted embedding of the first scene.

[0092] Based on the target scene label of the second text, the gate weight vector is read from the gate weight table, and weights are applied dimension by dimension in the embedding dimension to obtain the weighted embedding of the second scene.

[0093] In this embodiment, the scene modulation function and counterfactual embedding generation and bias removal training specifically include:

[0094] A scene modulation function is established, and a parameter table indexed by scene labels is set. Each scene label in the parameter table is associated with four types of parameters, which are consistent with the embedding dimension length. These four types of parameters are the dimensional scaling vector, the dimensional translation vector, the gating vector, and the temperature coefficient. The scene modulation function takes the initial embedding as input and performs normalization, dimensional scaling, dimensional translation, dimensional gating, temperature scaling, and interval clipping in sequence. The output scene modulation embedding is then used. The dimensional scaling vector is used for scaling, the dimensional translation vector is used for bias adjustment, the gating vector is used for dimensional selection, and the temperature coefficient is used for numerical stretching and shrinking.

[0095] Based on the target scene labels of the first text and the second text, two target scene modulation embeddings are obtained by applying the two initial embedding application scene modulation functions respectively.

[0096] A set of scene tags that is not equal to the target scene tag of the first text is selected from the set of scene tags according to a preset number, and a first counterfactual embedding set is generated by applying a scene modulation function to the initial embedding of the first text. A set of scene tags that is not equal to the target scene tag of the second text is selected from the set of scene tags according to a preset number, and a second counterfactual embedding set is generated by applying a scene modulation function to the initial embedding of the second text.

[0097] Set up a projection mapping to map the two target scene modulation embeddings and the two sets of counterfactual embeddings to a fixed-dimensional projection space;

[0098] Define the intervention invariance constraint as follows:

[0099] ;

[0100] in, To intervene in invariance loss, For the initial embedding of the first text, For the initial embedding of the second text, The target scene label for the first text. The target scene label for the second text. and For the scene modulation function instance associated with the target scene label, To match scene tags Associated scene modulation function instance, For projection mapping, This is the first set of counterfactual scenario labels. This is the second set of counterfactual scenario labels. For each set of counterfactual scenario labels, the number of labels. The formula for the L2 norm, which is calculated in the projection space, is based on the principle of measuring the difference between the modulated embedding of the target scene and the corresponding counterfactual embedding in the unified projection space by means of samples.

[0101] A discriminator is set up to perform scene label discrimination on the projection map output to form an adversarial debiasing constraint. This constraint, together with the intervention invariance constraint, forms a joint training objective. The scene modulation function, projection map, and discriminator parameters are updated through gradient descent to obtain the first debiasing embedding and the second debiasing embedding.

[0102] In this embodiment, the generation of the scene-weighted Jaccard lexical difference specifically includes:

[0103] The first and second texts are segmented, stop words are removed, and normalization is performed to form the first term table and the second term table, and the word frequency of each term is recorded;

[0104] A first scene term weight table is generated based on the target scene labels of the first text, and a second scene term weight table is generated based on the target scene labels of the second text.

[0105] In the first term table and the second term table, the term frequency of each term is multiplied by the corresponding scenario term weight to obtain the first scenario weighted term value and the second scenario weighted term value;

[0106] A unified term set is established, which contains all different terms appearing in the first and second texts. The term set is then calculated according to the "scene-weighted Jaccard lexical method": for each term, the first scene-weighted term value is compared with the second scene-weighted term value. The smaller value is accumulated as the intersection sum value, and the larger value is accumulated as the union sum value. The ratio of the intersection sum value to the union sum value is used as the scene-weighted Jaccard lexical similarity.

[0107] Subtract the scene-weighted Jaccard lexical similarity from the first value to obtain the scene-weighted Jaccard lexical difference.

[0108] In this embodiment, the generation of the optimal transmission distance for scene coupling specifically includes:

[0109] An index is built using a unified term set to locate the corresponding term positions in the first and second texts. The term embeddings output by the pre-trained Transformer are extracted. Gating weights are applied to the embedding dimension according to the target scene to obtain the scene-weighted embeddings of the first term and the second term. The distance between any term pair is calculated according to the rule of summing the squared differences of each dimension and then taking the square root, forming a scene-weighted semantic distance matrix.

[0110] Based on a unified term set, the weighted term values ​​of the first and second scenarios are read. For the same term, the ratio of the sum of the intersection and the sum of the union is calculated to obtain the scenario-weighted Jaccard lexical similarity. In order to uniformly map the similarity as a cost, the similarity is subtracted by one to make the value fall between zero and one and maintain the same order: zero for completely identical terms, zero for partial overlap, and one for completely disjoint terms. For different terms, the intersection of the weighted Jaccard is zero and the union is positive, the similarity is zero, and the difference is one. The lexical difference is obtained by subtracting the current similarity from one. The lexical difference is set to one for different terms to form a scenario-weighted Jaccard lexical difference matrix.

[0111] By setting the lexical difference weight and semantic distance weight to non-negative constants, the scene-weighted Jaccard lexical difference matrix and the scene-weighted semantic distance matrix are linearly weighted and synthesized at the corresponding positions to obtain the optimal transmission cost matrix.

[0112] The temperature coefficient is calculated from the input scene consistency coefficient. The temperature coefficient is then combined with the entropy regularization coefficient to construct the transmission kernel and complete the transmission kernel initialization.

[0113] Perform Sinkhorn iterations on the transmission kernel, alternating between scaling the row and column directions, so that the row edge distribution is equal to the probability vector obtained by summing and normalizing the weighted term values ​​of the first scenario, and the column edge distribution is equal to the probability vector obtained by summing and normalizing the weighted term values ​​of the second scenario. After convergence, the transmission plan is obtained.

[0114] Multiply the optimal transmission cost matrix and the transmission plan item by item at the corresponding positions, and sum them up over the entire matrix to obtain a single value, which is then denoted as the optimal transmission distance for scene coupling.

[0115] In this embodiment, the generation of robust semantic similarity specifically includes:

[0116] The input objects are explicitly defined as the first debiased embedding, the second debiased embedding, the first counterfactual embedding set, and the second counterfactual embedding set;

[0117] Pairings are established in a fixed order: the first debiased embedding is paired with the second debiased embedding, the first debiased embedding is paired with each element in the second counterfactual embedding set, each element in the first counterfactual embedding set is paired with the second debiased embedding, and each element in the first counterfactual embedding set is paired with each element in the second counterfactual embedding set.

[0118] For each pair, semantic similarity is calculated. The calculation rule is to multiply the two vectors dimension by dimension and sum them to obtain the inner product. Then, the lengths of the two vectors are calculated separately. The cosine similarity is obtained by the ratio of the inner product to the product of the two lengths. The results are recorded as a semantic similarity sequence according to the pairing order.

[0119] The semantic similarity sequence is converted into a risk metric sequence. The conversion rule is to subtract the corresponding value from each semantic similarity to obtain the risk metric, while maintaining consistency with the pairing order.

[0120] The conditional risk value is calculated by setting quantile parameters and taking values ​​sequentially within the allowed range. For each value, the tail excess average is calculated. The tail excess average is calculated by comparing each risk measure with the threshold. If the value is greater than the threshold, the difference is taken; if it is not greater than the threshold, the difference is taken. The sum is divided by the sample size and then divided by the quantile parameter. Since the conditional risk value is averaged based on the tail probability quality, it needs to be normalized by the quantile parameter. The excess portion above the threshold is converted into the average level of the tail interval. The current threshold is added to the tail excess average to form the target value. The smallest target value obtained from all the values ​​is selected as the conditional risk value.

[0121] Robust semantic similarity is obtained by subtracting the conditional risk value from one. Since the conditional risk value describes the degree of risk, subtracting the conditional risk value from one can map the risk scale to the similarity scale, so that a smaller risk corresponds to a larger robust semantic similarity.

[0122] In this embodiment, the generation of the comprehensive similarity specifically includes:

[0123] The input parameters are explicitly defined as the optimal transmission distance for scene coupling and robust semantic similarity.

[0124] The optimal transmission distance for scene coupling is converted into alignment similarity through a monotonically decreasing mapping, and the mapping output is limited to the range of zero to one.

[0125] Set alignment similarity weight and robust semantic similarity weight, with both weights being non-negative and adding up to one;

[0126] The alignment similarity and robust semantic similarity are weighted and fused to obtain the comprehensive similarity.

[0127] Example 1:

[0128] To verify the feasibility of this invention in practice, it was applied to the text matching task of an intelligent customer service question-and-answer system to test its semantic judgment accuracy and robustness in multiple scenarios. This system involves three typical business scenarios: e-commerce shopping, logistics consultation, and after-sales service. The natural language questions input by users are semantically similar but have different scenarios, such as "When will the apples be shipped?" and "When will the apples ripen?". The former refers to the product shipping scenario, while the latter refers to the crop growth scenario. Traditional similarity algorithms are mainly based on lexical or semantic embedding and do not explicitly model scenario factors, often resulting in cross-scenario misjudgments, leading to a decrease in question-and-answer matching accuracy. In this environment, this invention is used to identify semantic differences in multiple scenarios and improve matching accuracy.

[0129] The system first annotates the question-and-answer dataset with 50,000 text pairs, covering five sub-scenarios: e-commerce shopping, logistics tracking, after-sales feedback, payment issues, and product inquiries. During the testing phase, user questions and candidate question texts are input. The system calculates the posterior distribution of scenarios using a predefined scenario library and historical interaction records, determines the target scenario label, and calculates the scenario consistency coefficient. A pre-trained Transformer generates text embeddings and applies scenario-gating weights, allowing the model to weight different scenarios along the embedding dimension. For each scenario label, the model executes a scenario modulation function to generate counterfactual embeddings and applies intervention invariance and adversarial debiasing training to obtain debiased embedding representations. Then, the optimal transmission cost matrix is ​​constructed by combining scenario-weighted Jaccard lexical differences with semantic distance. Entropy regularization is used to optimize the transmission plan, and finally, the optimal transmission distance for scenario coupling is calculated. Robust semantic similarity is obtained through conditional risk fusion and weighted with alignment similarity to output the final similarity score.

[0130] In the comparative experiment, three models were set up: a traditional semantic similarity model (based on BERT cosine similarity), a domain-adaptive model (based on label weighting), and the method of this invention. The test set contained 10,000 samples, of which 3,000 were cross-scene text pairs and 7,000 were text pairs within the same scene. Evaluation metrics included Top1 hit rate (the proportion of correct matches), Top3 recall rate (the proportion of candidates containing the correct answer), cross-scene false positive rate (the proportion of high similarity judgments across different scenes), computation time, and stability. The experimental results are shown in Table 1.

[0131] Table 1. Comparison of Text Similarity Determination Results of Different Methods

[0132]

[0133] As shown in Table 1, the method of this invention significantly outperforms the control model in both Top1 hit rate and Top3 recall rate, reducing the cross-scene misclassification rate by more than 50% and the standard deviation by nearly 60%, indicating more stable results. Although the computation time increases slightly, it is within an acceptable range. Further analysis shows that when two texts are semantically similar but have different scenarios (e.g., "refund arrival time" and "course learning time"), the traditional model has a misclassification probability of about 12%, while the model of this invention has a misclassification probability of only 4%. When two texts have significant semantic differences but consistent scenarios (e.g., "view order details" and "cancel order operation"), the traditional model underestimates the similarity probability by 18%, while the model of this invention underestimates it by only 6%. Therefore, by introducing scenario consistency and optimal transmission alignment mechanisms, the model can more effectively distinguish cross-scenario texts while preserving semantic consistency within a scenario.

[0134] Furthermore, the method of this invention demonstrates outstanding robustness in multi-scenario robustness tests. The model was deployed to a customer service system and ran continuously for 72 hours. User requests from different business scenarios were randomly selected, and the system automatically determined the question's attribution and the matching of the answer. Statistical results show that in the question-answer alignment task, the correct matching rate remained stable at over 94%, and the misjudgment rate did not exceed 5% in highly similar but cross-scenario corpora. This indicates that the invention has strong generalization ability in complex semantic environments and can adapt to different corpus scenarios such as e-commerce, finance, and education, achieving high-precision text semantic matching.

[0135] The above description is only a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any equivalent substitutions or modifications made by those skilled in the art within the scope of the technology disclosed in the present invention, based on the technical solution and inventive concept of the present invention, should be covered within the scope of protection of the present invention.

Claims

1. A text similarity calculation method based on scene information enhancement, characterized in that, Includes the following steps: Obtain the first text and the second text, generate their respective posterior distributions of the scene based on the predefined scene library and historical behavior, determine their respective target scene labels, and calculate the scene consistency coefficient; An initial embedding is generated by pre-training a Transformer. Gating weights are applied to the embedding dimension according to the target scene to obtain the first scene weighted embedding and the second scene weighted embedding. Based on the target scene label, the initial embedded application scene modulation function is used to generate counterfactual embeddings. The first debiased embedding and the second debiased embedding are obtained by joint training with intervention invariance constraints and adversarial debiasing constraints. Assign weights to terms based on the target scenario and calculate scenario-weighted Jaccard lexical differences. The scene weighted semantic distance is calculated based on the first scene weighted embedding and the second scene weighted embedding. The optimal transmission cost matrix is ​​constructed with the scene weighted Jaccard lexical difference and the scene weighted semantic distance. The scene consistency coefficient is used to modulate the entropy regularization optimal transmission. The transmission plan is solved and the scene coupling optimal transmission distance is obtained. Multiple sets of semantic similarities are calculated between the first debiased embedding, the second debiased embedding and the corresponding counterfactual pair, and risk-sensitive fusion is performed using the conditional value at risk method to obtain robust semantic similarity. The optimal transmission distance of scene coupling is converted into alignment similarity, and then weighted and fused with robust semantic similarity to obtain a comprehensive similarity.

2. The text similarity calculation method based on scene information enhancement according to claim 1, characterized in that, The generation of the scene posterior distribution, the determination of the target scene label, and the calculation of the scene consistency coefficient specifically include: Two input channels are set up. The first input channel is connected to the first data source, and the second input channel is connected to the second data source. Two natural language texts are received, which are denoted as the first text and the second text, respectively. Read the predefined scene library, create a scene tag set, and set a unique number and name for each scene tag; Four types of scene matching features are generated for the first text: keyword matching score, semantic matching score, context information score, and historical behavior score. These features are then aggregated into each scene label according to preset weights to form a comprehensive scoring sequence for the first text. For the second text, keyword matching score, semantic matching score, context information score, and historical behavior score are generated and aggregated into each scene tag according to the same weight system to form a comprehensive scoring sequence for the second text; The exponential mapping and sum-normalization processes are performed on the two comprehensive scoring sequences respectively to obtain the scene posterior distribution of the first text and the scene posterior distribution of the second text. The scene with the highest probability in the posterior distribution of the first text is selected as the target scene label of the first text, and the scene with the highest probability in the posterior distribution of the second text is selected as the target scene label of the second text. Calculate the scene consistency coefficient: For each scene label, take the square root of the posterior probability of the first text and the second text on the corresponding scene label and multiply them. Add up the products of all scene labels and use the result as the scene consistency coefficient, which ranges from zero to one.

3. The text similarity calculation method based on scene information enhancement according to claim 1, characterized in that, The generation of the first scene weighted embedding and the second scene weighted embedding specifically includes: Load the pre-trained Transformer and set the embedding dimensions; The first text and the second text are encoded respectively to obtain the first initial embedding and the second initial embedding; Establish a gating weight table indexed by scene labels, and configure a gating weight vector of the same length as the embedding dimension for each scene label; Based on the target scene label of the first text, the gate weight vector is read from the gate weight table, and weights are applied dimension by dimension in the embedding dimension to obtain the weighted embedding of the first scene. Based on the target scene label of the second text, the gate weight vector is read from the gate weight table, and weights are applied dimension by dimension in the embedding dimension to obtain the weighted embedding of the second scene.

4. The text similarity calculation method based on scene information enhancement according to claim 1, characterized in that, The scene modulation function and counterfactual embedding generation and bias-reduction training specifically include: Establish a scene modulation function, set a parameter table indexed by scene labels, and take the initial embedding as input. The scene modulation function performs normalization, dimensional scaling, dimensional translation, dimensional gating, temperature scaling and interval clipping in sequence, and outputs the scene modulation embedding. Based on the target scene labels of the first text and the second text, two target scene modulation embeddings are obtained by applying the two initial embedding application scene modulation functions respectively. A set of scene tags that is not equal to the target scene tag of the first text is selected from the set of scene tags according to a preset number, and a first counterfactual embedding set is generated by applying a scene modulation function to the initial embedding of the first text. A set of scene tags that is not equal to the target scene tag of the second text is selected from the set of scene tags according to a preset number, and a second counterfactual embedding set is generated by applying a scene modulation function to the initial embedding of the second text. Set up a projection mapping to map the two target scene modulation embeddings and the two sets of counterfactual embeddings to a fixed-dimensional projection space; The intervention invariance constraint is defined based on the principle of measuring the difference between the target scene modulation embedding and the corresponding counterfactual embedding in the unified projection space by means of samples. A discriminator is set up to perform scene label discrimination on the projection map output to form an adversarial debiasing constraint. This constraint, together with the intervention invariance constraint, forms a joint training objective. The scene modulation function, projection map, and discriminator parameters are updated through gradient descent to obtain the first debiasing embedding and the second debiasing embedding.

5. The text similarity calculation method based on scene information enhancement according to claim 1, characterized in that, The generation of the scenario-weighted Jaccard lexical differences specifically includes: The first and second texts are segmented, stop words are removed, and normalization is performed to form the first term table and the second term table, and the word frequency of each term is recorded; A first scene term weight table is generated based on the target scene labels of the first text, and a second scene term weight table is generated based on the target scene labels of the second text. In the first term table and the second term table, the term frequency of each term is multiplied by the corresponding scenario term weight to obtain the first scenario weighted term value and the second scenario weighted term value; Establish a unified term set. For each term, compare the weighted term value of the first scenario with the weighted term value of the second scenario. The smaller value is accumulated as the intersection sum, and the larger value is accumulated as the union sum. The ratio of the intersection sum to the union sum is used as the scenario-weighted Jaccard lexical similarity. Subtract the scene-weighted Jaccard lexical similarity from the first value to obtain the scene-weighted Jaccard lexical difference.

6. The text similarity calculation method based on scene information enhancement according to claim 1, characterized in that, The generation of the optimal transmission distance for scene coupling specifically includes: An index is built using a unified term set to locate the corresponding term positions in the first and second texts. The term embeddings output by the pre-trained Transformer are extracted. Gating weights are applied to the embedding dimension according to the target scene to obtain the scene-weighted embeddings of the first term and the second term. The distance between any term pair is calculated according to the rule of summing the squared differences of each dimension and then taking the square root, forming a scene-weighted semantic distance matrix. Based on a unified term set, the weighted term values ​​of the first and second scenarios are read. For the same term, the ratio of the sum of the intersection and the sum of the union is calculated to obtain the scenario-weighted Jaccard lexical similarity. The lexical difference is obtained by subtracting the current similarity from one. The lexical difference is set to one for different terms to form a scenario-weighted Jaccard lexical difference matrix. By setting the lexical difference weight and semantic distance weight to non-negative constants, the scene-weighted Jaccard lexical difference matrix and the scene-weighted semantic distance matrix are linearly weighted and synthesized at the corresponding positions to obtain the optimal transmission cost matrix. The temperature coefficient is calculated from the input scene consistency coefficient. The temperature coefficient is then combined with the entropy regularization coefficient to construct the transmission kernel and complete the transmission kernel initialization. Perform Sinkhorn iterations on the transmission kernel, alternating between scaling the row and column directions, so that the row edge distribution is equal to the probability vector obtained by summing and normalizing the weighted term values ​​of the first scenario, and the column edge distribution is equal to the probability vector obtained by summing and normalizing the weighted term values ​​of the second scenario. After convergence, the transmission plan is obtained. Multiply the optimal transmission cost matrix and the transmission plan item by item at the corresponding positions, and sum them up over the entire matrix to obtain a single value, which is then denoted as the optimal transmission distance for scene coupling.

7. The text similarity calculation method based on scene information enhancement according to claim 1, characterized in that, The generation of robust semantic similarity specifically includes: The input objects are explicitly defined as the first debiased embedding, the second debiased embedding, the first counterfactual embedding set, and the second counterfactual embedding set; Pairings are established in a fixed order: the first debiased embedding is paired with the second debiased embedding, the first debiased embedding is paired with each element in the second counterfactual embedding set, each element in the first counterfactual embedding set is paired with the second debiased embedding, and each element in the first counterfactual embedding set is paired with each element in the second counterfactual embedding set. For each pair, semantic similarity is calculated. The calculation rule is to multiply the two vectors dimension by dimension and sum them to obtain the inner product. Then, the lengths of the two vectors are calculated separately. The cosine similarity is obtained by the ratio of the inner product to the product of the two lengths. The results are recorded as a semantic similarity sequence according to the pairing order. The semantic similarity sequence is converted into a risk metric sequence. The conversion rule is to subtract the corresponding value from each semantic similarity to obtain the risk metric, while maintaining consistency with the pairing order. The process involves setting quantile parameters and calculating the conditional risk value. The calculation process is as follows: a threshold variable is introduced and values ​​are taken sequentially within the allowed range. For each value, the tail excess average is calculated first. The tail excess average is calculated by comparing the risk measure with the threshold one by one. If the value is greater than the threshold, the difference is taken; if it is not greater than the threshold, the value is zero. The sum is divided by the sample size and then divided by one minus the quantile parameter. The current threshold is added to the tail excess average to form the target value. The smallest target value obtained from all the values ​​is selected as the conditional risk value. The robust semantic similarity is obtained by subtracting the value of the conditional risk from one.

8. The text similarity calculation method based on scene information enhancement according to claim 1, characterized in that, The generation of the comprehensive similarity specifically includes: The input parameters are explicitly defined as the optimal transmission distance for scene coupling and robust semantic similarity. The optimal transmission distance for scene coupling is converted into alignment similarity through a monotonically decreasing mapping, and the mapping output is limited to the range of zero to one. Set alignment similarity weight and robust semantic similarity weight, with both weights being non-negative and adding up to one; The alignment similarity and robust semantic similarity are weighted and fused to obtain the comprehensive similarity.

Images