Text similarity calculation method based on scene information enhancement

Abstract

The invention discloses a text similarity calculation method based on scene information enhancement, and the method comprises the following steps: obtaining two texts, generating a scene posteriori and a target label, and calculating a consistency coefficient; coding is carried out by using a pre-training Transform, and scene gating is applied, so that weighted embedding of two scenes is obtained; executing scene modulation and anti-fact construction, and combining intervention invariance and adversarial depolarization training to obtain depolarization embedding; calculating a weighted Jaccard lexical difference according to a target scene; fusing the lexical difference and the semantic distance construction cost, and solving a transmission plan through entropy regularization to obtain a scene coupling optimal transmission distance; calculating semantic similarity and fusing the semantic similarity with a condition in a risk value to obtain robust semantic similarity; and mapping the transmission distance into an alignment similarity, and weighting the alignment similarity with the robust semantic similarity to output a comprehensive similarity. According to the method, the accuracy and robustness of cross-scene similarity judgment are improved.

Description

A Text Similarity Calculation Method Based on Scene Information Enhancement Technical Field

[0001] This invention relates to the field of natural language processing technology, and in particular to a text similarity calculation method based on scene information enhancement. Background Technology

[0002] In existing text similarity research, common approaches are divided into two categories: lexical metrics and semantic metrics. Lexical metrics are based on word segmentation and word frequency, combined with stop word processing and normalization, to calculate indicators such as Jaccard, overlap rate, and edit distance. Semantic metrics are based on pre-trained Transformers, extracting sentence vectors or word vectors, and scoring them using cosine similarity. Some methods introduce interactive encoding or attention matching. Both types of methods are usable on general corpora, but they lack stability when facing polysemy, domain transfer, and long-tail expressions. They tend to compress scene differences into a unified vector space, leading to cross-scene matching bias.

[0003] Research on scene factors often remains at the level of domain adaptation or prior label weighting. Common strategies involve constructing prior weights based on domain vocabularies, topic distributions, or historical clicks, and then simply fusing them in the vector space. Existing solutions typically lack clear scene posterior distribution modeling and target scene label selection processes, lack consistency quantification indicators that simultaneously characterize two inputs, and lack a scene-related lexical weighting system. Lexical similarity and semantic distance are often used in parallel but lack a unified cost construction and transmission mechanism. The resulting mismatch phenomenon is particularly prominent in e-commerce retrieval, question-answering matching, and public opinion merging. Texts that are semantically similar but have inconsistent scenes are easily judged as highly similar, while texts with significant semantic differences but consistent scenes are given lower scores.

[0004] In terms of robustness, common methods lack targeted handling of bias sources. The training process usually does not distinguish between target scenes and counterfactual scenes, lacks a systematic generation process for scene modulation functions and counterfactual embeddings, and is difficult to suppress scene-irrelevant factors by intervening invariance constraints and adversarial debiasing constraints. The similarity fusion stage often adopts simple weighted averaging without introducing risk-sensitive criteria and lacks control over tail mismatches. At the alignment level, common optimal transmission practices are mostly based on fixed costs or only semantic distance, and do not unify scene-weighted Jaccard lexical differences and scene-weighted semantic distance into the cost matrix. The transmission kernel lacks temperature adjustment based on scene consistency coefficients, making it difficult to stably present alignment strength in cross-scene matching.

[0005] Therefore, how to provide a text similarity calculation method based on scene information enhancement is a problem that urgently needs to be solved by those skilled in the art. Summary of the Invention

[0006] One objective of this invention is to propose a text similarity calculation method based on scene information enhancement. This invention utilizes techniques such as scene posterior distribution modeling, scene-gated embedding modulation, joint training of intervention invariance and adversarial debiasing, scene-weighted Jaccard lexical differences, and entropy regularization optimal transmission to achieve collaborative modeling of text semantics and scene factors. It has the advantages of more accurate semantic matching, higher cross-scene robustness, and stronger consistency in similarity calculation.

[0007] According to an embodiment of the present invention, a text similarity calculation method based on scene information enhancement includes the following steps: acquiring a first text and a second text; generating their respective scene posterior distributions based on a predefined scene library and historical behavior; determining their respective target scene labels; and calculating scene consistency coefficients; generating initial embeddings through pre-trained Transformers; applying gating weights to the embedding dimension according to the target scene to obtain a first scene-weighted embedding and a second scene-weighted embedding; applying a scene modulation function to the initial embeddings based on the target scene labels to generate counterfactual embeddings; and jointly training using intervention invariance constraints and adversarial debiasing constraints to obtain a first debiased embedding and a second debiased embedding; and using the target scene as a term. Weights are assigned, and scene-weighted Jaccard lexical differences are calculated. Based on the first scene-weighted embedding and the second scene-weighted embedding, the scene-weighted semantic distance is calculated. The optimal transmission cost matrix is ​​constructed using the scene-weighted Jaccard lexical differences and the scene-weighted semantic distance. The scene consistency coefficient is used to modulate the entropy regularization optimal transmission, and the transmission plan is solved to obtain the scene-coupled optimal transmission distance. Multiple sets of semantic similarities are calculated between the first debiased embedding, the second debiased embedding, and the corresponding counterfactual pairs. The conditional at-risk method is used for risk-sensitive fusion to obtain robust semantic similarity. The scene-coupled optimal transmission distance is converted into alignment similarity and weighted fusion with robust semantic similarity to obtain the comprehensive similarity.

[0008] Optionally, the generation of the scene posterior distribution, the determination of the target scene label, and the calculation of the scene consistency coefficient specifically include: setting two input channels, the first input channel connected to a first data source and the second input channel connected to a second data source, receiving two segments of natural language text, denoted as the first text and the second text respectively; reading a predefined scene library, establishing a scene label set, and assigning a unique number and name to each scene label; generating four types of scene matching features for the first text: keyword matching score, semantic matching score, context information score, and historical behavior score, summarizing them according to preset weights to each scene label to form a comprehensive scoring sequence for the first text; generating keyword matching score, semantic matching score, and context information score for the second text. Historical behavior scores are aggregated into each scene label according to the same weighting system to form a comprehensive scoring sequence for the second text. Exponential mapping and sum-normalization are performed on the two comprehensive scoring sequences to obtain the scene posterior distributions of the first and second texts. The scene posterior distribution with the highest probability in the first text is selected as the target scene label for the first text, and the scene posterior distribution with the highest probability in the second text is selected as the target scene label for the second text. The scene consistency coefficient is calculated as follows: for each scene label, the square roots of the posterior probabilities of the first and second texts on the corresponding scene label are taken and multiplied. The products obtained for all scene labels are summed, and the result is used as the scene consistency coefficient, with a value ranging from zero to one.

[0009] Optionally, the generation of the first scene-weighted embedding and the second scene-weighted embedding specifically includes: loading a pre-trained Transformer and setting the embedding dimension; encoding the first text and the second text respectively to obtain the first initial embedding and the second initial embedding; establishing a gating weight table indexed by scene labels, configuring a gating weight vector of the same length as the embedding dimension for each scene label; reading the gating weight vector from the gating weight table according to the target scene label of the first text, and applying weights dimension by dimension in the embedding dimension to obtain the first scene-weighted embedding; reading the gating weight vector from the gating weight table according to the target scene label of the second text, and applying weights dimension by dimension in the embedding dimension to obtain the second scene-weighted embedding.

[0010] Optionally, the scene modulation function and counterfactual embedding generation and bias removal training specifically include: establishing a scene modulation function, setting a parameter table indexed by scene labels, where each scene label is associated with four types of parameters, the four types of parameters having the same length as the embedding dimension, namely, a dimension-wise scaling vector, a dimension-wise translation vector, a gating vector, and a temperature coefficient; the scene modulation function takes the initial embedding as input and sequentially performs normalization, dimension-wise scaling, dimension-wise translation, dimension-wise gating, temperature scaling, and interval clipping, outputting a scene modulation embedding; the dimension-wise scaling vector is used for scaling, the dimension-wise translation vector is used for bias adjustment, the gating vector is used for dimension-wise selection, and the temperature coefficient is used for numerical stretching and shrinking; based on the target of the first text... The scene labels and the target scene labels of the second text are respectively applied to the two initial embeddings using scene modulation functions to obtain two target scene modulation embeddings; a set of scene labels not equal to the target scene labels of the first text are selected from the scene label set according to a preset number and applied to the initial embeddings of the first text using scene modulation functions to generate a first counterfactual embedding set; a set of scene labels not equal to the target scene labels of the second text are selected from the scene label set according to a preset number and applied to the initial embeddings of the second text using scene modulation functions to generate a second counterfactual embedding set; a projection mapping is set to map the two target scene modulation embeddings and the two sets of counterfactual embeddings to a fixed-dimensional projection space; an intervention invariance constraint is defined, given by the following formula: ;in, To intervene in invariance loss, For the initial embedding of the first text, For the initial embedding of the second text, The target scene label for the first text. The target scene label for the second text. and For the scene modulation function instance associated with the target scene label, To match scene tags Associated scene modulation function instance, For projection mapping, This is the first set of counterfactual scenario labels. This is the second set of counterfactual scenario labels. For each set of counterfactual scenario labels, the number of labels. The L2 norm is calculated in the projection space. This formula is based on the principle of measuring the difference between the target scene modulation embedding and the corresponding counterfactual embedding in the unified projection space by means of samples. A discriminator is set up to perform scene label discrimination on the projection map output to form an adversarial debiasing constraint. This constraint, together with the intervention invariance constraint, forms a joint training objective. The scene modulation function, projection map, and discriminator parameters are updated by gradient descent to obtain the first debiasing embedding and the second debiasing embedding.

[0011] Optionally, the generation of the scene-weighted Jaccard lexical differences specifically includes: performing word segmentation, stop word removal, and normalization on the first and second texts to form a first term table and a second term table, and recording the word frequency of each term; generating a first scene term weight table based on the target scene tags of the first text, and generating a second scene term weight table based on the target scene tags of the second text; multiplying the word frequency of each term by the corresponding scene term weight in the first and second term tables to obtain the first scene-weighted term value and the second scene-weighted term value; establishing a unified... The term set contains all the different terms appearing in the first and second texts. Calculations are performed on this unified term set according to the "scene-weighted Jaccard lexical similarity" rule: For each term, the first scene-weighted term value is compared with the second scene-weighted term value. The smaller value is accumulated as the intersection sum, and the larger value is accumulated as the union sum. The ratio of the intersection sum to the union sum is used as the scene-weighted Jaccard lexical similarity. The scene-weighted Jaccard lexical difference is obtained by subtracting one from the scene-weighted Jaccard lexical similarity.

[0012] Optionally, the generation of the optimal transmission distance for scene coupling specifically includes: establishing an index using a unified term set, locating the corresponding term positions in the first and second texts, extracting term embeddings from the pre-trained Transformer output, applying gate weights to the embedding dimension according to the target scene to obtain the scene-weighted embeddings of the first and second terms, calculating the distance between any term pairs according to the rule of summing the squared differences in each dimension and then taking the square root, forming a scene-weighted semantic distance matrix; reading the first and second scene-weighted term values ​​based on the unified term set, calculating the ratio of the intersection sum to the union sum for the same terms to obtain the scene-weighted Jaccard lexical similarity, and to uniformly map the similarity as a cost, subtracting one from the similarity to make the value fall between zero and one while maintaining a consistent order: zero for completely identical terms, zero for partially overlapping terms, and one for completely disjoint terms. For different terms, the intersection of the weighted Jaccard is zero, the union is positive, the similarity is zero, and the difference is zero. First, subtract the current similarity from 1 to obtain the lexical difference. Set the lexical difference to 1 for different terms to form a scene-weighted Jaccard lexical difference matrix. Set the lexical difference weight and semantic distance weight to non-negative constants. Perform linear weighted synthesis of the scene-weighted Jaccard lexical difference matrix and the scene-weighted semantic distance matrix at the corresponding positions to obtain the optimal transmission cost matrix. Input the scene consistency coefficient to calculate the temperature coefficient. Combine the temperature coefficient with the entropy regularization coefficient to construct the transmission kernel and complete the initialization of the transmission kernel. Perform Sinkhorn iteration on the transmission kernel, alternately scaling the row and column directions so that the row edge distribution is equal to the probability vector obtained by summing and normalizing the first scene-weighted term values, and the column edge distribution is equal to the probability vector obtained by summing and normalizing the second scene-weighted term values. After convergence, the transmission plan is obtained. Multiply the optimal transmission cost matrix and the transmission plan item by item at the corresponding positions and sum them over the entire matrix to obtain a single value, which is recorded as the scene-coupled optimal transmission distance.

[0013] Optionally, the generation of robust semantic similarity specifically includes: defining the input objects as a first debiased embedding, a second debiased embedding, a first counterfactual embedding set, and a second counterfactual embedding set; establishing pairings in a fixed order, namely, pairing the first debiased embedding with the second debiased embedding, pairing the first debiased embedding with each element in the second counterfactual embedding set, pairing each element in the first counterfactual embedding set with the second debiased embedding, and pairing each element in the first counterfactual embedding set with each element in the second counterfactual embedding set; calculating semantic similarity for each pairing, the calculation rule being that the two vectors are multiplied dimension by dimension and summed to obtain the inner product, then the lengths of the two vectors are calculated separately, and the cosine similarity is obtained by the ratio of the inner product to the product of the two lengths, and recorded as a semantic similarity sequence according to the pairing order; converting the semantic similarity sequence into a risk metric sequence, the conversion rule being that the risk metric is obtained by subtracting the corresponding value from each semantic similarity, while maintaining consistency with the pairing order. The process involves setting quantile parameters and calculating the conditional risk value. The calculation process involves introducing a threshold variable and taking values ​​sequentially within the allowed range. For each value, the tail excess average is calculated. This tail excess average is calculated by comparing each risk measure with the threshold; values ​​greater than the threshold are taken as the difference, and values ​​less than or equal to the threshold are taken as zero. The sum of these values ​​is divided by the sample size, and then by the quantile parameter. Because the conditional risk value is averaged based on the tail probability quality, it needs to be normalized by subtracting the quantile parameter, converting the excess portion above the threshold into the average level of the tail interval. The current threshold is added to the tail excess average to form the target value. The smallest target value obtained from all the selected values ​​is taken as the conditional risk value. The robust semantic similarity is obtained by subtracting the conditional risk value. Since the conditional risk value characterizes the degree of risk, subtracting the conditional risk value maps the risk scale to a similarity scale, allowing smaller risks to correspond to larger robust semantic similarities.

[0014] Optionally, the generation of the comprehensive similarity specifically includes: specifying the input quantities as the optimal transmission distance of scene coupling and robust semantic similarity; converting the optimal transmission distance of scene coupling into alignment similarity through a monotonically decreasing mapping, with the mapping output limited to the range of zero to one; setting alignment similarity weights and robust semantic similarity weights, with the two weights being non-negative and adding up to one; and weighting and fusing the alignment similarity and robust semantic similarity according to the weights to obtain the comprehensive similarity.

[0015] The beneficial effects of this invention are as follows: By introducing scene posterior distribution and target scene labels, similarity calculation can maintain stable judgment under conditions of polysemy and domain migration. Scene consistency coefficient is used to quantify the scene proximity of two text segments. Scene gating is finely adjusted in the embedding dimension. Scene-weighted term weights are used to construct scene-weighted Jaccard lexical differences. The two clues jointly participate in the construction of the optimal transmission cost matrix, aligning lexical and semantic functions within the same framework. The entropy regularization transmission kernel uses the scene consistency coefficient as the temperature regulation input. The transmission plan is solved under probability conservation constraints. The obtained scene coupling optimal transmission distance can reflect the alignment strength across texts and can be monotonically mapped to alignment similarity to participate in fusion.

[0016] At the representation end, target scene modulation embeddings are generated through scene modulation functions, and intervention pairs are constructed using counterfactual embeddings. By jointly using intervention invariance constraints and adversarial debiasing constraints, a purer debiased embedding is obtained. At the similarity end, a similarity sample set is formed using debiased embeddings and counterfactual pairs. Risk-sensitive fusion is performed using conditional risk values. Robust semantic similarity is more sensitive to tail mismatches. At the fusion end, alignment similarity and robust semantic similarity are combined into a comprehensive similarity according to weights. The entire process, from scene recognition, representation modulation, alignment transmission to risk fusion, is interconnected, enabling relevant texts to be matched more accurately and irrelevant texts to be effectively separated. The system maintains stable performance in multi-scenario applications. Attached Figure Description

[0017] The accompanying drawings are provided to further illustrate the invention and form part of the specification. They are used together with the embodiments of the invention to explain the invention and do not constitute a limitation thereof. In the drawings: Figure 1 is a flowchart of a text similarity calculation method based on scene information enhancement proposed in this invention; Figure 2 is a schematic diagram of scene posterior distribution generation in the text similarity calculation method based on scene information enhancement proposed in this invention; Figure 3 is a schematic diagram of optimal transmission solution in the text similarity calculation method based on scene information enhancement proposed in this invention. Detailed Implementation

[0018] The present invention will now be described in further detail with reference to the accompanying drawings. These drawings are simplified schematic diagrams, illustrating only the basic structure of the invention, and therefore only show the components relevant to the invention.

[0019] Referring to Figures 1-3, a text similarity calculation method based on scene information enhancement includes the following steps: acquiring a first text and a second text; generating their respective scene posterior distributions based on a predefined scene library and historical behavior; determining their respective target scene labels; and calculating scene consistency coefficients; generating initial embeddings through pre-trained Transformers; applying gating weights to the embedding dimension according to the target scene to obtain first scene weighted embeddings and second scene weighted embeddings; applying a scene modulation function to the initial embeddings based on the target scene labels to generate counterfactual embeddings; and jointly training with intervention invariance constraints and adversarial debiasing constraints to obtain first debiased embeddings and second debiased embeddings; and segmenting terms according to the target scene. We assign weights and calculate scene-weighted Jaccard lexical differences; based on the first scene-weighted embedding and the second scene-weighted embedding, we calculate the scene-weighted semantic distance; we construct the optimal transmission cost matrix using the scene-weighted Jaccard lexical differences and the scene-weighted semantic distance; we modulate the optimal transmission using the scene consistency coefficient and entropy regularization, solve the transmission plan, and obtain the scene-coupled optimal transmission distance; we calculate multiple sets of semantic similarities between the first and second debiased embeddings and their corresponding counterfactual pairs, and use the conditional at-risk method for risk-sensitive fusion to obtain robust semantic similarity; we convert the scene-coupled optimal transmission distance into alignment similarity, and perform weighted fusion with the robust semantic similarity to obtain the comprehensive similarity.

[0020] In this embodiment, the generation of the scene posterior distribution, the determination of the target scene label, and the calculation of the scene consistency coefficient specifically include: setting up two input channels, the first input channel connected to the first data source and the second input channel connected to the second data source, receiving two segments of natural language text, denoted as the first text and the second text respectively; reading a predefined scene library, establishing a scene label set, and assigning a unique number and name to each scene label; generating four types of scene matching features for the first text: keyword matching score, semantic matching score, context information score, and historical behavior score, summarizing them according to preset weights to each scene label to form a comprehensive scoring sequence for the first text; generating keyword matching score, semantic matching score, and context information score for the second text. The scores for the first text and historical behavior are aggregated into each scene label according to the same weighting system to form the comprehensive scoring sequence of the second text. Exponential mapping and sum-normalization are performed on the two comprehensive scoring sequences respectively to obtain the scene posterior distributions of the first text and the second text. The scene posterior distribution with the highest probability in the first text is selected as the target scene label for the first text, and the scene posterior distribution with the highest probability in the second text is selected as the target scene label for the second text. The scene consistency coefficient is calculated as follows: for each scene label, the square roots of the posterior probabilities of the first and second texts on the corresponding scene label are taken and multiplied. The products obtained for all scene labels are added together, and the result is used as the scene consistency coefficient, with a value ranging from zero to one.

[0021] In this embodiment, the generation of the first scene-weighted embedding and the second scene-weighted embedding specifically includes: loading a pre-trained Transformer and setting the embedding dimension; encoding the first text and the second text respectively to obtain the first initial embedding and the second initial embedding; establishing a gating weight table indexed by scene labels, and configuring a gating weight vector of the same length as the embedding dimension for each scene label; reading the gating weight vector from the gating weight table according to the target scene label of the first text, and applying weights dimension by dimension in the embedding dimension to obtain the first scene-weighted embedding; reading the gating weight vector from the gating weight table according to the target scene label of the second text, and applying weights dimension by dimension in the embedding dimension to obtain the second scene-weighted embedding.

[0022] In this embodiment, the scene modulation function and counterfactual embedding generation and bias removal training specifically include: establishing a scene modulation function, setting a parameter table indexed by scene labels, where each scene label is associated with four types of parameters, the four types of parameters having the same length as the embedding dimension, namely, a dimension-wise scaling vector, a dimension-wise translation vector, a gating vector, and a temperature coefficient; the scene modulation function takes the initial embedding as input and sequentially performs normalization, dimension-wise scaling, dimension-wise translation, dimension-wise gating, temperature scaling, and interval clipping, outputting a scene modulation embedding; the dimension-wise scaling vector is used for scaling, the dimension-wise translation vector is used for bias adjustment, the gating vector is used for dimension-wise selection, and the temperature coefficient is used for numerical stretching and shrinking; based on the purpose of the first text... Using the target scene labels of the first text and the target scene labels of the second text, respectively, apply the scene modulation function to the two initial embeddings to obtain two target scene modulation embeddings; select a set of scene labels from the scene label set that are not equal to the target scene labels of the first text according to a preset number, and apply the scene modulation function to the initial embedding of the first text to generate a first counterfactual embedding set; select a set of scene labels from the scene label set that are not equal to the target scene labels of the second text according to a preset number, and apply the scene modulation function to the initial embedding of the second text to generate a second counterfactual embedding set; set a projection mapping to map the two target scene modulation embeddings and the two sets of counterfactual embeddings to a fixed-dimensional projection space; define the intervention invariance constraint, given by the following formula: ;in, To intervene in invariance loss, For the initial embedding of the first text, For the initial embedding of the second text, The target scene label for the first text. The target scene label for the second text. and For the scene modulation function instance associated with the target scene label, To match scene tags Associated scene modulation function instance, For projection mapping, This is the first set of counterfactual scenario labels. This is the second set of counterfactual scenario labels. For each set of counterfactual scenario labels, the number of labels. The L2 norm is calculated in the projection space. This formula is based on the principle of measuring the difference between the target scene modulation embedding and the corresponding counterfactual embedding in the unified projection space by means of samples. A discriminator is set up to perform scene label discrimination on the projection map output to form an adversarial debiasing constraint. This constraint, together with the intervention invariance constraint, forms a joint training objective. The scene modulation function, projection map, and discriminator parameters are updated by gradient descent to obtain the first debiasing embedding and the second debiasing embedding.

[0023] In this embodiment, the generation of the scene-weighted Jaccard lexical difference specifically includes: performing word segmentation, stop word removal, and normalization on the first and second texts to form a first term table and a second term table, and recording the word frequency of each term; generating a first scene term weight table based on the target scene tags of the first text, and generating a second scene term weight table based on the target scene tags of the second text; multiplying the word frequency of each term by the corresponding scene term weight in the first and second term tables to obtain the first scene-weighted term value and the second scene-weighted term value; establishing a unified... A term set, containing all different terms appearing in the first and second texts, is used for calculation according to the "scene-weighted Jaccard lexical similarity" rule on a unified term set: for each term, the first scene-weighted term value is compared with the second scene-weighted term value, the smaller one is accumulated as the intersection sum, and the larger one is accumulated as the union sum. The ratio of the intersection sum to the union sum is used as the scene-weighted Jaccard lexical similarity; the scene-weighted Jaccard lexical difference is obtained by subtracting the scene-weighted Jaccard lexical similarity from one.

[0024] In this embodiment, the generation of the optimal transmission distance for scene coupling specifically includes: establishing an index using a unified term set, locating the corresponding term positions in the first and second texts, extracting term embeddings from the pre-trained Transformer output, applying gate weights to the embedding dimension according to the target scene to obtain the first term scene-weighted embedding and the second term scene-weighted embedding, calculating the distance between any term pairs according to the rule of summing the squared differences in each dimension and then taking the square root, forming a scene-weighted semantic distance matrix; reading the first scene-weighted term values ​​and the second scene-weighted term values ​​based on the unified term set, calculating the ratio of the intersection sum to the union sum for the same term to obtain the scene-weighted Jaccard lexical similarity, and to uniformly map the similarity as a cost, subtracting one from the similarity to make the value fall between zero and one while maintaining a consistent order: zero for completely identical terms, zero for partially overlapping terms, and one for completely disjoint terms. For different terms, the intersection of the weighted Jaccard is zero, the union is positive, the similarity is zero, and the difference is zero. The lexical difference is obtained by subtracting the current similarity from the difference and setting the lexical difference to 1 for different terms, forming a scene-weighted Jaccard lexical difference matrix. The lexical difference weight and semantic distance weight are set to non-negative constants. The scene-weighted Jaccard lexical difference matrix and the scene-weighted semantic distance matrix are linearly weighted and synthesized at the corresponding positions to obtain the optimal transmission cost matrix. The temperature coefficient is calculated by inputting the scene consistency coefficient. The temperature coefficient is combined with the entropy regularization coefficient to construct the transmission kernel and complete the initialization of the transmission kernel. The Sinkhorn iteration is performed on the transmission kernel, alternately scaling the row and column directions so that the row edge distribution is equal to the probability vector obtained by summing and normalizing the first scene-weighted term values, and the column edge distribution is equal to the probability vector obtained by summing and normalizing the second scene-weighted term values. After convergence, the transmission plan is obtained. The optimal transmission cost matrix and the transmission plan are multiplied item by item at the corresponding positions and summed over the entire matrix to obtain a single value, which is recorded as the scene-coupled optimal transmission distance.

[0025] In this embodiment, the generation of robust semantic similarity specifically includes: defining the input objects as a first debiased embedding, a second debiased embedding, a first counterfactual embedding set, and a second counterfactual embedding set; establishing pairings in a fixed order, namely, pairing the first debiased embedding with the second debiased embedding, pairing the first debiased embedding with each element in the second counterfactual embedding set, pairing each element in the first counterfactual embedding set with the second debiased embedding, and pairing each element in the first counterfactual embedding set with each element in the second counterfactual embedding set; calculating semantic similarity for each pairing, the calculation rule being that the two vectors are multiplied dimension by dimension and summed to obtain the inner product, then the lengths of the two vectors are calculated separately, and the cosine similarity is obtained by the ratio of the inner product to the product of the two lengths, and recorded as a semantic similarity sequence according to the pairing order; converting the semantic similarity sequence into a risk metric sequence, the conversion rule being that the risk metric is obtained by subtracting the corresponding value from each semantic similarity, while maintaining the pairing order. Consistency; quantile parameters are set and conditional risk values ​​are calculated. The calculation process involves introducing a threshold variable and taking values ​​sequentially within the allowed range. For each value, the tail excess average is calculated first. The tail excess average is calculated by comparing each risk measure with the threshold. If the value is greater than the threshold, the difference is taken; if it is not greater than the threshold, zero is taken. The sum is divided by the sample size, and then divided by the quantile parameter. Because the conditional risk value is averaged based on the tail probability quality, it needs to be normalized by the quantile parameter. The excess portion above the threshold is converted into the average level of the tail interval. The current threshold is added to the tail excess average to form the target value. The smallest target value obtained from all the values ​​is selected as the conditional risk value. The robust semantic similarity is obtained by subtracting the conditional risk value. Because the conditional risk value characterizes the degree of risk, subtracting the conditional risk value can map the risk scale to the similarity scale, so that a smaller risk corresponds to a larger robust semantic similarity.

[0026] In this embodiment, the generation of the comprehensive similarity specifically includes: defining the input quantities as the optimal transmission distance of scene coupling and robust semantic similarity; converting the optimal transmission distance of scene coupling into alignment similarity through a monotonically decreasing mapping, with the mapping output limited to the range of zero to one; setting alignment similarity weights and robust semantic similarity weights, with the two weights being non-negative and adding up to one; and weighting and fusing the alignment similarity and robust semantic similarity according to the weights to obtain the comprehensive similarity.

[0027] Example 1: To verify the feasibility of this invention in practice, it was applied to the text matching task of an intelligent customer service question-and-answer system to test its semantic judgment accuracy and robustness in multiple scenarios. The system involves three typical business scenarios: e-commerce shopping, logistics consultation, and after-sales service. The natural language questions input by users are semantically similar but have different scenarios, such as "When will the apples be shipped?" and "When will the apples ripen?". The former refers to the product shipping scenario, while the latter refers to the crop growth scenario. Traditional similarity algorithms are mainly based on lexical or semantic embedding and do not explicitly model scenario factors, often resulting in cross-scenario misjudgments, leading to a decrease in question-and-answer matching accuracy. This invention is used in this environment to identify semantic differences in multiple scenarios and improve matching accuracy.

[0028] The system first annotates the question-and-answer dataset with 50,000 text pairs, covering five sub-scenarios: e-commerce shopping, logistics tracking, after-sales feedback, payment issues, and product inquiries. During the testing phase, user questions and candidate question texts are input. The system calculates the posterior distribution of scenarios using a predefined scenario library and historical interaction records, determines the target scenario label, and calculates the scenario consistency coefficient. A pre-trained Transformer generates text embeddings and applies scenario-gating weights, allowing the model to weight different scenarios along the embedding dimension. For each scenario label, the model executes a scenario modulation function to generate counterfactual embeddings and applies intervention invariance and adversarial debiasing training to obtain debiased embedding representations. Then, the optimal transmission cost matrix is ​​constructed by combining scenario-weighted Jaccard lexical differences with semantic distance. Entropy regularization is used to optimize the transmission plan, and finally, the optimal transmission distance for scenario coupling is calculated. Robust semantic similarity is obtained through conditional risk fusion and weighted with alignment similarity to output the final similarity score.

[0029] In the comparative experiment, three models were set up: a traditional semantic similarity model (based on BERT cosine similarity), a domain-adaptive model (based on label weighting), and the method of this invention. The test set contained 10,000 samples, of which 3,000 were cross-scene text pairs and 7,000 were text pairs within the same scene. Evaluation metrics included Top1 hit rate (the proportion of correct matches), Top3 recall rate (the proportion of candidates containing the correct answer), cross-scene false positive rate (the proportion of high similarity judgments in different scenes), computation time, and stability. The experimental results are shown in Table 1: Table 1 Comparison of Text Similarity Judgment Performance of Different Methods

[0030] As shown in Table 1, the method of this invention significantly outperforms the control model in both Top1 hit rate and Top3 recall rate, reducing the cross-scene misclassification rate by more than 50% and the standard deviation by nearly 60%, indicating more stable results. Although the computation time increases slightly, it is within an acceptable range. Further analysis shows that when two texts are semantically similar but have different scenarios (e.g., "refund arrival time" and "course learning time"), the traditional model has a misclassification probability of about 12%, while the model of this invention has a misclassification probability of only 4%. When two texts have significant semantic differences but consistent scenarios (e.g., "view order details" and "cancel order operation"), the traditional model underestimates the similarity probability by 18%, while the model of this invention underestimates it by only 6%. Therefore, by introducing scenario consistency and optimal transmission alignment mechanisms, the model can more effectively distinguish cross-scenario texts while preserving semantic consistency within a scenario.

[0031] Furthermore, the method of this invention demonstrates outstanding robustness in multi-scenario robustness tests. The model was deployed to a customer service system and ran continuously for 72 hours. User requests from different business scenarios were randomly selected, and the system automatically determined the question's attribution and the matching of the answer. Statistical results show that in the question-answer alignment task, the correct matching rate remained stable at over 94%, and the misjudgment rate did not exceed 5% in highly similar but cross-scenario corpora. This indicates that the invention has strong generalization ability in complex semantic environments and can adapt to different corpus scenarios such as e-commerce, finance, and education, achieving high-precision text semantic matching.

[0032] The above description is only a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any equivalent substitutions or modifications made by those skilled in the art within the scope of the technology disclosed in the present invention, based on the technical solution and inventive concept of the present invention, should be covered within the scope of protection of the present invention.

Claims

1. A text similarity calculation method based on scene information enhancement, characterized in that, The steps include: obtaining the first text and the second text, generating their respective posterior distributions based on a predefined scene library and historical behavior, determining their respective target scene labels, and calculating the scene consistency coefficient; generating initial embeddings through a pre-trained Transformer, applying gating weights to the embedding dimension according to the target scene, and obtaining the first scene weighted embedding and the second scene weighted embedding. Based on the target scene label, the initial embedded application scene modulation function is used to generate counterfactual embeddings. Joint training with intervention invariance constraints and adversarial debiasing constraints is employed to obtain the first and second debiased embeddings. Weights are assigned to terms according to the target scene, and scene-weighted Jaccard lexical differences are calculated. Scene-weighted semantic distance is calculated based on the first and second scene-weighted embeddings. An optimal transmission cost matrix is ​​constructed using the scene-weighted Jaccard lexical differences and scene-weighted semantic distance. The optimal transmission is modulated using the scene consistency coefficient to obtain the scene-coupled optimal transmission distance. Multiple sets of semantic similarities are calculated between the first and second debiased embeddings and their corresponding counterfactual pairs. A conditional risk-sensitive fusion method is used to obtain robust semantic similarity. The scene-coupled optimal transmission distance is converted into alignment similarity and weighted fusion with the robust semantic similarity to obtain a comprehensive similarity.

2. The text similarity calculation method based on scene information enhancement according to claim 1, characterized in that, The generation of the scene posterior distribution, the determination of the target scene label, and the calculation of the scene consistency coefficient specifically include: setting up two input channels, the first input channel connected to the first data source and the second input channel connected to the second data source, receiving two segments of natural language text, denoted as the first text and the second text respectively; reading a predefined scene library, establishing a scene label set, and assigning a unique number and name to each scene label; generating four types of scene matching features for the first text: keyword matching score, semantic matching score, context information score, and historical behavior score, which are then aggregated into each scene label according to preset weights to form a comprehensive scoring sequence for the first text; generating keyword matching score, semantic matching score, context information score, and historical behavior score for the second text. Historical scores are aggregated into each scene label using the same weighting system to form a comprehensive scoring sequence for the second text. Exponential mapping and sum-normalization are then performed on the two comprehensive scoring sequences to obtain the scene posterior distributions for the first and second texts. The scene posterior distribution with the highest probability in the first text is selected as the target scene label for the first text, and the scene posterior distribution with the highest probability in the second text is selected as the target scene label for the second text. The scene consistency coefficient is calculated as follows: for each scene label, the square roots of the posterior probabilities of the first and second texts on the corresponding scene label are taken and multiplied. The products obtained for all scene labels are summed, and the result is used as the scene consistency coefficient, with a value ranging from zero to one.

3. The text similarity calculation method based on scene information enhancement according to claim 1, characterized in that, The generation of the first scene-weighted embedding and the second scene-weighted embedding specifically includes: loading a pre-trained Transformer and setting the embedding dimension; encoding the first text and the second text respectively to obtain the first initial embedding and the second initial embedding; establishing a gating weight table indexed by scene labels, and configuring a gating weight vector of the same length as the embedding dimension for each scene label; reading the gating weight vector from the gating weight table according to the target scene label of the first text, and applying weights dimension by dimension in the embedding dimension to obtain the first scene-weighted embedding; reading the gating weight vector from the gating weight table according to the target scene label of the second text, and applying weights dimension by dimension in the embedding dimension to obtain the second scene-weighted embedding.

4. The text similarity calculation method based on scene information enhancement according to claim 1, characterized in that, The scene modulation function and counterfactual embedding generation and debiasing training specifically include: establishing a scene modulation function, setting a parameter table indexed by scene labels, the scene modulation function taking the initial embedding as input, and sequentially performing normalization, dimensional scaling, dimensional translation, dimensional gating, temperature scaling, and interval pruning to output the scene modulation embedding; applying the scene modulation function to the two initial embeddings based on the target scene labels of the first text and the target scene labels of the second text to obtain two target scene modulation embeddings; selecting a set of scene labels from the scene label set that are not equal to the target scene labels of the first text according to a preset number, and applying the scene modulation function to the initial embedding of the first text to generate a first counterfactual embedding set; and then selecting a set of scene labels from the scene label set according to a preset number to generate a first counterfactual embedding set. A set of scene labels with a preset quantity that is not equal to the target scene label of the second text is selected, and a second counterfactual embedding set is generated by applying a scene modulation function to the initial embedding of the second text. A projection mapping is set to map the two target scene modulation embeddings and the two sets of counterfactual embeddings to a fixed-dimensional projection space. An intervention invariance constraint is defined based on the principle of measuring the difference between the target scene modulation embedding and the corresponding counterfactual embedding in a unified projection space by means of samples. A discriminator is set to perform scene label discrimination on the projection mapping output to form an adversarial debiasing constraint, which, together with the intervention invariance constraint, forms a joint training objective. The scene modulation function, projection mapping, and discriminator parameters are updated by gradient descent to obtain the first debiased embedding and the second debiased embedding.

5. The text similarity calculation method based on scene information enhancement according to claim 1, characterized in that, The generation of the scene-weighted Jaccard lexical difference specifically includes: performing word segmentation, stop word removal, and normalization on the first and second texts to form a first term table and a second term table, and recording the word frequency of each term; generating a first scene term weight table based on the target scene label of the first text, and generating a second scene term weight table based on the target scene label of the second text; multiplying the word frequency of each term by the corresponding scene term weight in the first and second term tables to obtain the first scene-weighted term value and the second scene-weighted term value; establishing a unified term set, comparing the first scene-weighted term value and the second scene-weighted term value for each term, accumulating the smaller one as the intersection summation value, and accumulating the larger one as the union summation value, using the ratio of the intersection summation value to the union summation value as the scene-weighted Jaccard lexical similarity; subtracting the scene-weighted Jaccard lexical similarity value from one to obtain the scene-weighted Jaccard lexical difference.

6. The text similarity calculation method based on scene information enhancement according to claim 1, characterized in that, The generation of the optimal transmission distance for scene coupling specifically includes: establishing an index using a unified term set, locating the corresponding term positions in the first and second texts, extracting term embeddings from the pre-trained Transformer output, applying gate weights to the embedding dimension according to the target scene to obtain the scene-weighted embeddings of the first and second terms, calculating the distance between any term pairs according to the rule of summing and square-taking the square root of the differences in each dimension, forming a scene-weighted semantic distance matrix; reading the first and second scene-weighted term values ​​based on the unified term set, calculating the ratio of the intersection sum to the union sum for the same terms to obtain the scene-weighted Jaccard lexical similarity, subtracting the current similarity from one to obtain the lexical difference, setting the lexical difference to one for different terms, forming a scene-weighted Jaccard lexical difference matrix. The optimal transmission cost matrix is ​​obtained by setting the lexical difference weight and semantic distance weight to non-negative constants and performing linear weighted synthesis of the scene-weighted Jaccard lexical difference matrix and the scene-weighted semantic distance matrix at corresponding positions. The temperature coefficient is calculated by inputting the scene consistency coefficient and combining the temperature coefficient with the entropy regularization coefficient to construct the transmission kernel and complete the initialization of the transmission kernel. The Sinkhorn iteration is performed on the transmission kernel, alternatingly scaling the row and column directions so that the row edge distribution is equal to the probability vector obtained by summing and normalizing the first scene-weighted term values, and the column edge distribution is equal to the probability vector obtained by summing and normalizing the second scene-weighted term values. After convergence, the transmission plan is obtained. The optimal transmission cost matrix and the transmission plan are multiplied item by item at corresponding positions and summed over the entire matrix range to obtain a single value, which is recorded as the scene-coupled optimal transmission distance.

7. The text similarity calculation method based on scene information enhancement according to claim 1, characterized in that, The robust semantic similarity generation specifically includes: defining the input objects as a first debiased embedding, a second debiased embedding, a first counterfactual embedding set, and a second counterfactual embedding set; establishing pairings in a fixed order: pairing the first debiased embedding with the second debiased embedding, pairing the first debiased embedding with each element in the second counterfactual embedding set, pairing each element in the first counterfactual embedding set with the second debiased embedding, and pairing each element in the first counterfactual embedding set with each element in the second counterfactual embedding set; calculating the semantic similarity for each pairing, using the following rule: multiplying the two vectors dimension by dimension and summing the results to obtain the inner product; then calculating the lengths of the two vectors separately; using the ratio of the inner product to the product of the two lengths to obtain the cosine similarity; and recording this as a semantic similarity sequence according to the pairing order. The semantic similarity sequence is converted into a risk metric sequence. The conversion rule is to subtract the corresponding value from each semantic similarity to obtain the risk metric, while maintaining consistency with the pairing order. A quantile parameter is set and the conditional risk value is calculated. The calculation process is as follows: a threshold variable is introduced and values ​​are taken sequentially within the allowed range. For each value, the tail excess average term is calculated. The tail excess average term is calculated by comparing each risk metric with the threshold. If it is greater than the threshold, the difference is taken; if it is not greater than the threshold, zero is taken. The sum is divided by the number of samples, and then divided by the quantile parameter. The current threshold is added to the tail excess average term to form the target value. The smallest target value obtained from all the values ​​is selected as the conditional risk value. The robust semantic similarity is obtained by subtracting the conditional risk value from the threshold.

8. The text similarity calculation method based on scene information enhancement according to claim 1, characterized in that, The generation of the comprehensive similarity specifically includes: defining the input quantities as the optimal transmission distance of scene coupling and robust semantic similarity; converting the optimal transmission distance of scene coupling into alignment similarity through a monotonically decreasing mapping, with the mapping output limited to the range of zero to one; setting alignment similarity weights and robust semantic similarity weights, with both weights being non-negative and adding up to one; and weighting and fusing the alignment similarity and robust semantic similarity according to the weights to obtain the comprehensive similarity.

Images