Method and device for generating difficult negative samples

Abstract

The application provides a method for generating difficult negative samples. A set of negative sample candidates corresponding to a query text sample is coarsely screened from a plurality of text segment samples. Multi-dimensional heterogeneous features of each candidate text segment sample are determined. The multi-dimensional heterogeneous features are input into a three-dimensional Gaussian mixture model. Parameters of the three-dimensional Gaussian mixture model are estimated by a maximum likelihood algorithm. The set of negative sample candidates is divided into at least three sample distribution clusters. Decision boundaries between different sample distribution clusters are adjusted. A dynamic weight distribution with a learnable bias is performed on the multi-dimensional heterogeneous features by an attention mechanism. A synergistic inhibition relationship score between the candidate text segment sample and the query text sample is obtained. The candidate text segment sample within the decision boundary corresponding to the difficult negative sample cluster is taken as a difficult negative sample. Each difficult negative sample is constructed into a set of difficult negative samples corresponding to the query text sample. The application can generate high-quality difficult negative samples and improve the training performance of a model.

Description

Technical Field

[0001] This application relates to the fields of machine learning and natural language processing, and in particular to a method and apparatus for generating difficult negative samples. Background Technology

[0002] In machine learning and natural language processing, especially for cross-domain task transfer scenarios, such as finance, law, and product recommendation, LLM (Large Language Model) needs to learn within that domain. During this transfer, the model needs to learn specific domain knowledge to become practical. This learning process requires both positive and negative samples. Therefore, generating negative samples is crucial for model training. Without sufficient negative samples, the model may overfit to positive samples, leading to performance degradation in practical applications. Simply obtaining negative samples through random sampling offers limited performance improvement. Therefore, obtaining high-quality, challenging negative samples to better enhance model performance is a pressing issue. Summary of the Invention

[0003] This application provides a method for generating difficult negative samples, which solves the problem of insufficient high-quality difficult negative samples, leading to model overfitting and insufficient recognition ability.

[0004] In a first aspect, embodiments of this application provide a method for generating difficult negative samples, the method comprising:

[0005] For each query text sample:

[0006] Based on the vector distance between the query text sample and multiple text segment samples, a candidate set of negative text segment samples is coarsely selected from multiple text segment samples;

[0007] Based on the multiple correlation features between the query text sample and each candidate text sample in the negative sample candidate set, the multidimensional heterogeneous features of each candidate text sample are determined.

[0008] Multidimensional heterogeneous features are input into a three-dimensional Gaussian mixture model. The parameters of the three-dimensional Gaussian mixture model are estimated using the expectation-maximization (EM) algorithm. The physical constraint range is used as the threshold for out-of-bounds limitation. The negative sample candidate set of the text is divided into at least three sample distribution clusters. The decision boundary between different sample distribution clusters is adjusted. The at least three sample distribution clusters include easily distinguishable negative sample clusters, difficult negative sample clusters, and potential positive sample clusters.

[0009] For each candidate text segment sample, an attention mechanism is used to dynamically assign weights with learnable biases to the multidimensional heterogeneous features corresponding to the candidate text segment sample, thereby obtaining a score of the collaborative inhibition relationship between the candidate text segment sample and the corresponding query text sample.

[0010] Candidate text samples whose collaborative inhibition relationship scores fall within the decision boundary corresponding to the difficult negative sample cluster are taken as difficult negative samples, and each obtained difficult negative sample is used to construct a difficult negative sample set corresponding to the query text sample.

[0011] Optionally, the multidimensional heterogeneous features are input into a three-dimensional Gaussian mixture model, and the parameters of the three-dimensional Gaussian mixture model are estimated using the EM algorithm. Using the physical constraint range as a threshold for out-of-bounds limitation, the negative sample candidate set of the text segment is divided into at least three sample distribution clusters. The decision boundaries between different sample distribution clusters are adjusted, including:

[0012] Using a three-dimensional Gaussian mixture model, the input multidimensional heterogeneous features are transformed into at least three Gaussian distributions;

[0013] In the process of iteratively calculating the parameters of a 3D Gaussian mixture model based on the EM algorithm,

[0014] For a low-similarity Gaussian distribution with the largest variance among at least three Gaussian distributions, the corresponding sample distribution cluster is taken as an easily distinguishable negative sample cluster;

[0015] For a highly similar Gaussian distribution with the smallest variance among at least three Gaussian distributions, the corresponding sample distribution cluster is taken as the potential positive sample cluster.

[0016] The sample distribution clusters corresponding to at least three Gaussian distributions with moderate similarity are taken as the difficult negative sample clusters;

[0017] Obtain the leftmost standard deviation of the mean of the potential positive sample cluster and the leftmost standard deviation of the mean of the difficult negative sample cluster. Use the leftmost standard deviation of the mean as the upper limit of the negative sample decision boundary and the leftmost standard deviation of the mean as the lower limit of the negative sample decision boundary. Use physical constraints to limit the threshold range of the upper and lower limits.

[0018] Optionally, threshold out-of-bounds limits that restrict the upper and lower limits based on physical constraints include:

[0019] The threshold range for the negative sample decision boundary is calculated based on the lower bound of the 95% confidence interval of the moderately similar Gaussian distribution.

[0020] The upper limit of the negative sample decision boundary is calculated based on the lower limit of the 68% confidence interval of the high-similarity Gaussian distribution, which is the threshold limit range for exceeding the boundary.

[0021] Optionally, through an attention mechanism, dynamic weights with learnable biases are assigned to the multidimensional heterogeneous features corresponding to the candidate text sample to obtain a score of the collaborative inhibition relationship between the candidate text sample and the corresponding query text sample, including:

[0022] For each candidate text sample, each dimension of the heterogeneous features in the multidimensional heterogeneous features is encoded to obtain the corresponding heterogeneous feature encoding vector.

[0023] Each heterogeneous feature encoding vector is processed by a self-attention mechanism to obtain the attention distribution corresponding to each heterogeneous feature dimension.

[0024] Based on the attention distribution corresponding to each heterogeneous feature and the attention distribution of all heterogeneous features, dynamic weights with learnable biases corresponding to each heterogeneous feature are obtained through activation function processing.

[0025] For each heterogeneous feature of the candidate text sample, calculate the product of the dynamic weight of the heterogeneous feature with a learnable bias and the normalized heterogeneous feature.

[0026] The products are summed to obtain the collaborative inhibition relationship score between the candidate text sample and the corresponding query text sample.

[0027] Optionally, based on the attention distribution corresponding to each heterogeneous feature and the attention distribution of all heterogeneous features, dynamic weights with learnable biases corresponding to each heterogeneous feature are obtained through activation function processing, including:

[0028] The feature weight α corresponding to the i-th heterogeneous feature has a learnable bias. i Represented as:

[0029]

[0030] Where σ(·) represents the sigmoid function, Attention i Let L represent the attention distribution corresponding to the i-th heterogeneous feature, L represent the dimension of the multidimensional heterogeneous feature, b represent the learnable bias parameter, and ε represent a constant.

[0031] Optionally, based on the vector distance between the query text sample and multiple text segment samples, a candidate set of negative text segment samples is coarsely filtered from the multiple text segment samples, including:

[0032] An embedding model is used to vectorize each query text sample and each text segment sample in the sample set, resulting in the query vector corresponding to each query text sample and the text segment vector corresponding to each text segment sample.

[0033] For each query text sample, determine the similarity between the query vector and each text segment vector, and construct the K text segment samples with the highest similarity as the candidate set of negative text segment samples corresponding to the query text sample.

[0034] Optionally, based on multiple correlation features between the query text sample and each candidate text sample in the negative sample candidate set, the multidimensional heterogeneous features of each candidate text sample are determined, including:

[0035] For each candidate text sample,

[0036] The rearrangement model is used to obtain the rearrangement score between the query text sample and the candidate text sample, so as to determine the global statistical correlation between the query text sample and the candidate text sample.

[0037] Calculate the semantic similarity between the query text sample and the candidate text sample to determine the deep semantic association between the query text sample and the candidate text sample;

[0038] Calculate the Jaccard coefficient between the query text sample and the candidate text sample to determine the character-level similarity between them;

[0039] Determine the longest common subsequence between the query text sample and the candidate text sample to determine the consistency of word order structure between the query text sample and the candidate text sample;

[0040] Calculate the edit distance between the query text sample and the candidate text sample to determine the morphological change tolerance between the query text sample and the candidate text sample;

[0041] Multidimensional heterogeneous features of the candidate text sample are constructed using rearrangement score, semantic similarity, Jaccard coefficient, longest common subsequence, and edit distance.

[0042] Optionally, the method further includes:

[0043] For each candidate text sample corresponding to a query text sample, the candidate text sample whose collaborative inhibition relationship score is greater than the upper limit of the negative sample decision boundary is regarded as a positive sample and added to the positive sample set corresponding to the query text sample.

[0044] Candidate text samples whose collaborative inhibition relationship scores are less than the lower limit of the negative sample decision boundary are designated as negative samples, and candidate text samples whose collaborative inhibition relationship scores are within the negative sample decision boundary are designated as difficult negative samples. Negative samples and difficult negative samples are added to the negative sample set corresponding to the query text sample.

[0045] A training set is constructed based on each query text sample, the corresponding positive sample set, and the negative sample set. The large language model is then trained using the training set to obtain the trained large language model.

[0046] Secondly, embodiments of this application provide an apparatus for generating difficult negative samples, the apparatus comprising:

[0047] The coarse screening module is used to coarsely screen out a candidate set of negative text samples from multiple text samples for each query text sample based on the vector distance between the query text sample and multiple text sample samples.

[0048] The heterogeneous feature construction module is used to determine the multidimensional heterogeneous features of each candidate text segment based on multiple correlation features between the query text sample and each candidate text segment sample in the negative sample candidate set.

[0049] The negative sample boundary confirmation module is used to input multidimensional heterogeneous features into a three-dimensional Gaussian mixture model, estimate the parameters of the three-dimensional Gaussian mixture model using the expectation-maximization (EM) algorithm, limit the out-of-bounds range using the physical constraint range, divide the candidate set of negative samples of the text into at least three sample distribution clusters, and adjust the decision boundary between different sample distribution clusters. The at least three sample distribution clusters include easily distinguishable negative sample clusters, difficult negative sample clusters, and potential positive sample clusters.

[0050] The scoring and confirmation module is used to dynamically assign weights with learnable biases to the multidimensional heterogeneous features corresponding to each candidate text sample through an attention mechanism, thereby obtaining a score of the collaborative inhibition relationship between the candidate text sample and the corresponding query text sample.

[0051] The difficult negative sample generation module is used to take candidate text samples whose collaborative inhibition relationship scores are within the decision boundary corresponding to the difficult negative sample cluster as difficult negative samples, and construct the difficult negative sample set corresponding to the query text sample from the obtained difficult negative samples.

[0052] Secondly, embodiments of this application provide an electronic device, including a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the steps of the method for generating difficult negative samples as described above.

[0053] This application employs a two-stage progressive sample selection process. First, candidate text samples are coarsely screened based on vector distance. Then, the coarse screening results are transformed into multidimensional heterogeneous features. Based on these multidimensional heterogeneous features, a set of difficult negative samples is dynamically refined according to the different candidate text samples. Because the filtering granularity is dynamically adjusted according to different sample data, the accuracy of sample data recall is maximized. Furthermore, by using multidimensional heterogeneous features to adjust the decision boundaries between different sample distribution clusters in real time within a three-dimensional Gaussian mixture model, a dynamic decision boundary can be obtained to replace the traditional fixed threshold, significantly improving the model's adaptability in complex environments. Moreover, based on an attention mechanism, dynamic weight allocation with learnable biases is applied to the multidimensional heterogeneous features to obtain a score of the collaborative inhibition relationship between different samples. This allows for more flexible identification of difficult negative samples that better meet the needs of different query texts, thereby using these higher-quality difficult negative samples to improve the model's understanding of cross-domain query texts and ensuring the performance of the cross-domain knowledge question answering model after domain changes. Attached Figure Description

[0054] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0055] Figure 1 This is a first flowchart illustrating the method for generating difficult negative samples provided in this application embodiment;

[0056] Figure 2 This is a second flowchart illustrating the method for generating difficult negative samples provided in the embodiments of this application;

[0057] Figure 3 This is a schematic diagram of the third process of the method for generating difficult negative samples provided in the embodiments of this application;

[0058] Figure 4 This is a schematic diagram of the fourth process of the method for generating difficult negative samples provided in the embodiments of this application;

[0059] Figure 5 This is a fifth flowchart illustrating the method for generating difficult negative samples provided in this application embodiment;

[0060] Figure 6 This is a sixth flowchart illustrating the method for generating difficult negative samples provided in this application embodiment;

[0061] Figure 7 This is a system block diagram of the apparatus for generating difficult negative samples provided in the embodiments of this application;

[0062] Figure 8 This is a system block diagram of the electronic device provided in the embodiments of this application. Detailed Implementation

[0063] The present application will be described in detail below with reference to the specific embodiments shown in the accompanying drawings. However, these embodiments do not limit the present application. Any structural, methodological, or functional modifications made by those skilled in the art based on these embodiments are included within the protection scope of the present application.

[0064] Please refer to Figure 1 This application provides a method for generating difficult negative samples, which includes steps S101-S105.

[0065] S101, for each query text sample, based on the vector distance between the query text sample and multiple text segment samples, a preliminary set of negative text sample candidates is selected from multiple text segment samples.

[0066] A sample set is constructed for model training, such as for training a large language model. This sample set includes multiple query text samples and multiple text segment samples. Preprocessing of the sample set filters out the candidate set of negative text segments corresponding to each query text sample. Query text samples can also be called question text samples. Text segment samples can also be called answer text samples. For example, by calculating the similarity between each query text sample and each text segment sample, and filtering the text segment samples based on this similarity, a preliminary selection of the candidate set of negative text segments corresponding to each query text sample can be achieved from all text segments in the sample set, providing candidate samples for the subsequent determination of difficult samples.

[0067] S102, based on the multiple correlation features between the query text sample and each candidate text sample in the negative sample candidate set, determine the multidimensional heterogeneous features of each candidate text sample.

[0068] For each candidate set of negative text samples corresponding to a query text sample, determine the correlation features between the query text sample and each candidate text sample in the candidate set of negative text samples, and determine the multidimensional heterogeneous features of each candidate text sample.

[0069] S103, input the multidimensional heterogeneous features into the three-dimensional Gaussian mixture model, estimate the parameters of the three-dimensional Gaussian mixture model using the expectation-maximization (EM) algorithm, use the physical constraint range as the threshold to limit the out-of-bounds range, divide the candidate set of negative samples of the text into at least three sample distribution clusters, and adjust the decision boundary between different sample distribution clusters. The at least three sample distribution clusters include easily distinguishable negative sample clusters, difficult negative sample clusters, and potential positive sample clusters.

[0070] In machine learning and data mining, a negative sample is a sample that does not meet specific conditions or belongs to the target class. In contrast, a positive sample is a sample that meets the conditions or belongs to the target class. A hard negative sample is a sample that is most similar to a positive sample but does not belong to the target class. Hard negative samples can improve the robustness of the model, allowing it to learn more complex features to distinguish between positive and negative samples.

[0071] In this embodiment, the 3D Gaussian Mixture Model is an unsupervised machine learning algorithm based on a probability model. It is a generative model that assumes all data are generated by mixing one or more different Gaussian distributions (normal distributions), with each Gaussian distribution forming a cluster. Each data point is assigned a probability of belonging to each cluster. The 3D Gaussian Mixture Model learns the parameters of these Gaussian distributions through the Expectation-Maximization (EM) algorithm.

[0072] In this embodiment, the number of clusters is set to at least three, including easily distinguishable negative sample clusters, difficult negative sample clusters, and potentially positive sample clusters. Based on the multidimensional heterogeneous features corresponding to each candidate text sample, a three-dimensional Gaussian mixture model is used to cluster each candidate text sample. The probability of each candidate text sample belonging to one of the three sample clusters is determined, i.e., the probability of belonging to the easily distinguishable negative sample cluster, the probability of belonging to the difficult negative sample cluster, and the probability of belonging to the potentially positive sample cluster. The cluster with the highest probability value is selected as the sample cluster corresponding to that candidate text sample, i.e., whether the candidate text sample belongs to the easily distinguishable negative sample cluster, the difficult negative sample cluster, or the potentially positive sample cluster. This assigns multiple candidate text samples to different sample clusters. Since each sample cluster can be considered a Gaussian distribution, the Gaussian model parameters corresponding to each sample cluster can be determined using the EM algorithm. Using the physical constraint range as a threshold for out-of-bounds limitation, the candidate negative sample set is divided into at least three sample distribution clusters, and the decision boundaries between different sample distribution clusters are adjusted.

[0073] S104. For each candidate text sample, an attention mechanism is used to dynamically assign weights with learnable biases to the multidimensional heterogeneous features corresponding to the candidate text sample, thereby obtaining a score of the collaborative inhibition relationship between the candidate text sample and the corresponding query text sample.

[0074] For each candidate text segment corresponding to the query text sample, an attention mechanism is used to process the multidimensional heterogeneous features corresponding to the candidate text segment sample, and dynamic weights with learnable biases are assigned to these multidimensional heterogeneous features. The learnable bias parameter can be a parameter variable, learned through training the features of the candidate text segments during model training, to improve the model's expressive power. Through the dynamic feature weight allocation algorithm of the attention mechanism, the importance of features is adjusted based on the features between the query text sample and each candidate text segment sample, capturing the synergistic / inhibitionary relationships between features, and obtaining the fused feature weights of heterogeneous features. This determines the synergistic inhibition relationship score between the candidate text segment sample and the corresponding query text sample. In this embodiment, by combining the attention mechanism with the learnable bias parameter, the combined enhancing effect of the captured multidimensional heterogeneous feature combination on the effectiveness of hard negative sample discrimination is considered a synergistic relationship, and the mutual cancellation effect of the multidimensional heterogeneous feature combination on the effectiveness of hard negative sample discrimination is considered an inhibition relationship. The synergistic inhibition relationship is scored, thereby obtaining a question-answering model more suitable for the domain of the current query text.

[0075] S105, the candidate text samples whose collaborative inhibition relationship scores are within the decision boundary corresponding to the difficult negative sample cluster are taken as difficult negative samples, and the obtained difficult negative samples are constructed into a difficult negative sample set corresponding to the query text sample.

[0076] Based on the scores of each candidate text sample, those scores within the negative sample decision boundary are designated as difficult negative samples. These difficult negative samples are then used to construct a difficult negative sample set corresponding to each query text sample. This process is repeated to obtain the difficult negative sample set for each query text sample.

[0077] like Figure 2The flowchart of this embodiment of the application is shown. The sample set includes multiple query text samples and multiple text segment samples. Data preprocessing is performed on the sample set to filter out the text segment negative sample candidate set corresponding to each query text sample. The data preprocessing includes: text vectorization of each query text sample and each text segment sample to obtain each query vector corresponding to each query text sample and each text segment vector corresponding to each text segment sample. Text recall processing is performed on each text segment sample. For each query text sample, the similarity between the corresponding query vector and each text segment vector is calculated. The Top K text segment samples with the highest similarity are used as the text segment negative sample candidate set corresponding to that query text sample, thus obtaining the text segment negative sample candidate set corresponding to each query text sample. The heterogeneous feature matrix of each candidate text segment sample in each text segment negative sample candidate set is then calculated. The process involves screening difficult negative samples from each candidate negative sample set, determining the negative sample decision boundary for each query text sample using an adaptive algorithm based on a three-dimensional Gaussian mixture model, and determining the collaborative inhibition relationship score of each candidate text sample in the candidate negative sample set corresponding to each query text sample using a dynamic weight algorithm based on an attention mechanism. Finally, the process involves screening each candidate text sample based on the collaborative inhibition relationship score and the negative sample decision boundary to obtain the difficult negative sample set corresponding to each query text sample.

[0078] This embodiment employs a two-stage progressive sample selection process. First, candidate text samples are coarsely screened based on vector distance. Then, the coarse screening results are transformed into multidimensional heterogeneous features. Based on these multidimensional heterogeneous features, a set of difficult negative samples is dynamically refined according to the different candidate text samples. Because the filtering granularity is dynamically adjusted according to different sample data, the maximum accuracy of sample data recall can be guaranteed. At the same time, by using multidimensional heterogeneous features to adjust the decision boundaries between different sample distribution clusters in real time in a three-dimensional Gaussian mixture model, a dynamic decision boundary can be obtained to replace the traditional fixed threshold, significantly improving the model's adaptability in complex environments. Furthermore, based on the attention mechanism, dynamic weight allocation with learnable biases is applied to the multidimensional heterogeneous features to obtain a score of the cooperative inhibition relationship between different samples. This can correspond to the needs of different query texts and more flexibly find difficult negative samples that better fit the needs of the query text. Thus, these higher-quality difficult negative samples can be used to improve the model's ability to understand cross-domain query texts, thereby improving model performance.

[0079] One embodiment of this application, such as Figure 3 As shown, based on the vector distance between the query text sample and multiple text segment samples, a candidate set of negative text segment samples is coarsely screened from multiple text segment samples, including steps S301-S302.

[0080] S301, an embedding model is used to vectorize each query text sample and each text segment sample in the sample set to obtain the query vector corresponding to each query text sample and the text segment vector corresponding to each text segment sample.

[0081] S302, for each query text sample corresponding to the query vector, determine the similarity between the query vector and each text segment vector, and take the K text segment samples with the highest similarity as the candidate set of negative text segment samples corresponding to the query text sample.

[0082] For example, an embedding model can be used to vectorize each query text sample and each text segment sample. For instance, the embedding model can be a BGE Embedding model, which uses BGE Embedding model to vectorize each query text sample and each text segment sample to obtain each query vector corresponding to each query text sample and each text segment vector corresponding to each text segment sample, thereby obtaining the semantic information of the query text samples and text segment samples.

[0083] For example, the sample set is represented as {Q,D}, where Q represents the set of query text samples and D represents the set of text segment samples. The BGE Embedding model is used to represent each query text sample and each text segment sample in the sample set as a vector, resulting in the query vector representation Q = [q1,q2,…,q...]. n ], and obtain the text vector representation D = [d1, d2, ..., d] for each text sample. m Let ] represent the number of query text samples and m represent the number of text segment samples. For each query vector, calculate the cosine similarity between the query vector and each text segment vector. The K text segment samples with the highest similarity are used as the candidate set of negative text segment samples corresponding to the query text sample. For example, K is 100. Based on this implementation method, the candidate set of negative text segment samples corresponding to each query text sample can be obtained.

[0084] This embodiment filters each text segment sample based on the similarity between each query text sample and each text segment sample, coarsely selecting candidate sets of text segment samples and negative sample candidate sets corresponding to each query text sample. This provides candidate samples for the subsequent determination of difficult samples. A two-level filtering mechanism of vector distance coarse screening and heterogeneous feature fine screening is used to screen text segment samples. The statistical model (Gaussian mixture model) is combined with deep learning (attention mechanism) to improve the accuracy of difficult sample identification and achieve orthogonal optimization of processing efficiency and quality.

[0085] One embodiment of this application, such as Figure 4As shown, based on the multiple correlation features between the query text sample and each candidate text sample in the text sample candidate set, the multidimensional heterogeneous features of each candidate text sample are determined, including steps S401-S406.

[0086] S401, For each candidate text sample, use the rearrangement model to obtain the rearrangement score between the query text sample and the candidate text sample, so as to determine the global statistical correlation between the query text sample and the candidate text sample.

[0087] S402, Based on the semantic similarity between the query text sample and the candidate text sample, determine the deep semantic association between the query text sample and the candidate text sample;

[0088] S403, Based on the Jaccard coefficient between the query text sample and the candidate text sample, determine the character correlation degree between the query text sample and the candidate text sample;

[0089] S404, determine the longest common subsequence between the query text sample and the candidate text sample to determine the character-level similarity between the query text sample and the candidate text sample;

[0090] S405, calculate the edit distance between the query text sample and the candidate text sample to determine the morphological change tolerance between the query text sample and the candidate text sample;

[0091] S406 uses rearrangement score, semantic similarity, Jaccard coefficient, longest common subsequence, and edit distance to construct multidimensional heterogeneous features of the candidate text sample.

[0092] For example, for each candidate text sample corresponding to the query text sample, a re-ranking model is used to calculate the re-ranking score between the query text sample and the candidate text sample, in order to determine the global statistical relevance between the query text sample and the candidate text sample. For example, the re-ranking model can be Cohere Rerank, Bge-reranker, Bge-reranker-large, etc. Taking the BGE ReRanker model as an example, it calculates the similarity between the input query text sample and each candidate text sample, and ranks each candidate text sample based on the similarity, so that candidate text samples more relevant to the query text sample are ranked higher. The re-ranking results output by the re-ranking model typically arrange the candidate text samples in descending order of similarity.

[0093] For example, the semantic similarity between the query text sample and the candidate text sample is determined to ascertain the deep semantic association between them. An embedding model is used to obtain the query vector corresponding to the query text sample and the text vector of the candidate text sample. The semantic similarity between the query vector and the text vector is then calculated. This semantic similarity can be calculated using a correlation function, such as cosine similarity.

[0094] For example, the Jaccard coefficient between the query text sample and the candidate text sample is calculated to determine the character correlation between the query text sample and the candidate text sample.

[0095] For example, the longest common subsequence (LCS) between the query text sample and the candidate text sample is determined to assess the consistency of word order structure between them. The longest common subsequence (LCS) is the longest subsequence in which one string appears in another string without changing the order of characters. A longer LCS indicates higher similarity. For instance, if the query is "Is it going to rain today?" and the candidate text sample is "According to the latest weather forecast, it may rain today, please bring an umbrella", the longest common subsequence would be "It's going to rain today".

[0096] For example, the edit distance between the query text sample and the candidate text sample is calculated to determine the tolerance for morphological changes such as spelling or morphological variations between the query text sample and the candidate text sample. Edit distance refers to the minimum number of editing operations required to transform one string into another, quantifying the degree of difference between the two strings. A larger edit distance indicates a greater difference in character values. Transformation operations include adding a character, deleting a character, and replacing a character. For example, if the query is "Does the weather forecast say it will rain today?" and the candidate text sample is "The weather forecast shows light rain today", the edit distance is 6.

[0097] Five dimensions—rearrangement score, semantic similarity, Jaccard coefficient, longest common subsequence, and edit distance—were used as the multidimensional heterogeneous features of the candidate text sample.

[0098] One embodiment of this application, such as Figure 5 As shown, multidimensional heterogeneous features are input into a three-dimensional Gaussian mixture model, the parameters of the three-dimensional Gaussian mixture model are estimated using the EM algorithm, and the negative sample candidate set of the text is divided into at least three sample distribution clusters using the physical constraint range as the threshold for out-of-bounds limitation. This includes steps S501-S505.

[0099] S501 uses a three-dimensional Gaussian mixture model to transform the input multidimensional heterogeneous features into at least three Gaussian distributions;

[0100] S502, In the process of iteratively calculating the parameters of the three-dimensional Gaussian mixture model based on the EM algorithm, for the low similarity Gaussian distribution with the largest variance among at least three Gaussian distributions, the corresponding sample distribution cluster is taken as the easily distinguishable negative sample cluster.

[0101] S503: For at least three Gaussian distributions with the smallest variance and high similarity, the corresponding sample distribution cluster is taken as the potential positive sample cluster.

[0102] S504, take the sample distribution clusters corresponding to at least three Gaussian distributions with moderate similarity as the difficult negative sample clusters;

[0103] S505, obtain the leftmost standard deviation of the mean of the potential positive sample cluster and the leftmost standard deviation of the mean of the difficult negative sample cluster respectively, use the leftmost standard deviation of the mean as the upper limit of the negative sample decision boundary, and use the leftmost standard deviation of the mean as the lower limit of the negative sample decision boundary, and use the physical constraint range to limit the threshold of the upper and lower limits.

[0104] Obtain the 5-dimensional heterogeneous features of each candidate text segment sample, namely, rearrangement score, semantic similarity, Jaccard coefficient, longest common subsequence, and edit distance, and construct a heterogeneous feature matrix. For example, assuming the query text sample corresponds to 100 candidate text segments, a 5*100 heterogeneous feature matrix is ​​constructed based on the 5-dimensional heterogeneous features of these 100 candidate text segments, denoted as X=[X rank ,X cos ,X jac ,X lcs ,X edit Each column corresponds to the feature values ​​of 100 candidate text samples. This indicates the score for rearrangement. Indicates semantic similarity. Represents the Jaccard coefficient. Represents the longest common subsequence. This represents the edit distance. A 3D Gaussian mixture model is used to transform the input heterogeneous feature matrix into at least three Gaussian distributions.

[0105] For example, a three-dimensional Gaussian mixture model is constructed using heterogeneous feature matrices. The number of clusters is set to 3. To easily distinguish negative sample clusters, difficult negative sample clusters, and potential positive sample clusters, the Gaussian mixture model is represented as follows:

[0106]

[0107] Where k = 1, 2, 3 represent easily distinguishable negative sample clusters, difficult negative sample clusters, and potential positive sample clusters, respectively; x represents the multidimensional heterogeneous features of the candidate text sample; π k This represents the weight of the k-th Gaussian distribution (sample cluster). Let μ represent the probability density function of the k-th Gaussian distribution. k Let σ represent the mean vector of the k-th Gaussian distribution. k Let represent the covariance matrix of the k-th Gaussian distribution.

[0108] For example, the Gaussian mixture model is iteratively processed using the expectation-maximization algorithm to estimate the model parameters, which are then updated to obtain a three-dimensional Gaussian mixture model. This process may include the following steps:

[0109] Step a: Initialize π k μ k and σ k Initialize weights

[0110] Step b: Based on π k μ k and σ k The value of and the formula for calculating the posterior probability are used to calculate the posterior probability that the i-th candidate text sample belongs to the k-th cluster.

[0111] Step c: Based on the posterior probability and π k μ k and σ k The calculation formulas for π are respectively used for... k μ k and σ k Update the value;

[0112] Repeat steps b and c above until π is reached. k μ k and σ k When the value reaches the convergence threshold or the maximum number of iterations, π will be obtained. k μ k and σ k The final result is used as the model parameters of the Gaussian mixture model. The probability of each candidate text sample belonging to one of the three sample clusters is output, and the cluster with the highest probability value is selected as the sample cluster corresponding to that candidate text sample. That is, it determines whether the candidate text sample belongs to the easily distinguishable negative sample cluster, the difficult negative sample cluster, or the potentially positive sample cluster. The π value of each sample cluster is also output. k μ k and σ k value.

[0113] In the process of iteratively calculating the parameters of the 3D Gaussian mixture model based on the EM algorithm, for the low-similarity Gaussian distribution among at least three Gaussian distributions, the corresponding sample distribution cluster is regarded as the easily distinguishable negative sample cluster. The sample distribution cluster corresponding to the high-similarity Gaussian distribution among at least three Gaussian distributions is regarded as the potential positive sample cluster. The sample distribution cluster corresponding to the medium-similarity Gaussian distribution among at least three Gaussian distributions is regarded as the difficult negative sample cluster. The left first standard deviation of the mean of the potential positive sample cluster and the left second standard deviation of the mean of the difficult negative sample cluster are obtained respectively. The left first standard deviation of the mean is used as the upper limit of the negative sample decision boundary, and the left second standard deviation of the mean is used as the lower limit of the negative sample decision boundary. The threshold limit range of the upper and lower limits is restricted by the physical constraint range.

[0114] For example, based on the Gaussian distribution of easily distinguishable negative sample clusters, a first mean vector and a first covariance matrix corresponding to the easily distinguishable negative sample clusters are determined; based on the Gaussian distribution of difficult negative sample clusters, a second mean vector and a second covariance matrix corresponding to the difficult negative sample clusters are determined, and the second standard deviation from the left of the mean of the difficult negative sample clusters is determined based on the second mean vector and the second covariance matrix; based on the Gaussian distribution of potential positive sample clusters, a third mean vector and a third covariance matrix corresponding to the potential positive sample clusters are determined, and the first standard deviation from the left of the mean of the potential positive sample clusters is determined based on the third mean vector and the third covariance matrix; the first standard deviation from the left of the mean is used as the upper limit of the negative sample decision boundary, and the second standard deviation from the left of the mean is used as the lower limit of the negative sample decision boundary.

[0115] For example, the second standard deviation from the left of the mean of the difficult negative sample cluster is used as the lower bound d1 of the negative sample decision boundary, and d1 is expressed as:

[0116] d1 = μ2 - 2σ2;

[0117] Where μ2 represents the second mean vector and σ2 represents the second covariance matrix;

[0118] The leftmost standard deviation of the mean of the potential positive sample cluster is used as the upper bound d2 of the decision boundary for negative samples, and d2 is expressed as:

[0119] d2=μ3-σ3;

[0120] Where μ3 represents the third mean vector and σ3 represents the third covariance matrix.

[0121] The lower bound d1 of the negative sample decision boundary represents the lower limit of the 95% confidence interval covering a moderately similar Gaussian distribution (difficult negative sample cluster), indicating the minimum threshold for difficult negative samples. The upper bound d2 of the negative sample decision boundary represents the lower bound of the 68% confidence interval covering a highly similar Gaussian distribution (potentially positive sample cluster), indicating the threshold for positive sample confusion risk control. Using the above formula, the negative sample decision boundary corresponding to this query text sample can be obtained as [d1, d2]. Similarly, the upper and lower bounds of the negative sample decision boundary corresponding to each query text sample can be determined.

[0122] One embodiment of this application defines a threshold out-of-bounds limitation range for the upper and lower limits of the physical constraint range, including: a threshold out-of-bounds limitation range for the lower limit of the negative sample decision boundary calculated according to the lower limit of the 95% confidence interval of the moderately similar Gaussian distribution; and a threshold out-of-bounds limitation range for the upper limit of the negative sample decision boundary calculated according to the lower limit of the 68% confidence interval of the high-similarity Gaussian distribution.

[0123] To avoid the risk of the lower and upper bounds of the negative sample decision boundary exceeding their limits, physical constraints are set for the lower and upper bounds to ensure sample quality. For example, the lower bound d1 satisfies: d1 = clip(d1, 0.2, 0.6), meaning the threshold exceedance range of the lower bound d1 is between 0.2 and 0.6; the upper bound d2 satisfies: d2 = clip(d2, 0.65, 0.85), meaning the threshold exceedance range of the upper bound d2 is between 0.65 and 0.85. In this way, the lower bound of the negative sample decision boundary is always lower than the upper bound, thereby avoiding physically incorrect settings and improving the real-time availability of the dynamic negative sample decision boundary automatically acquired by the model.

[0124] In this embodiment, the adaptive boundary optimization algorithm of Gaussian mixture model is used to automatically learn the best negative sample decision boundary for each query text sample based on different candidate text samples corresponding to different query text samples through a data-driven approach. The corresponding negative sample decision boundary is dynamically adjusted to accurately screen difficult negative samples in an adaptive manner, thereby improving the recognition accuracy of difficult negative samples and improving the training effect and performance of the model.

[0125] One embodiment of this application, such as Figure 6 As shown, through the attention mechanism, dynamic weights with learnable biases are assigned to the multidimensional heterogeneous features corresponding to the candidate text sample to obtain the collaborative inhibition relationship score between the candidate text sample and the corresponding query text sample, including steps S601-S605.

[0126] S601, For each candidate text sample, encode each dimension of the heterogeneous features in the multidimensional heterogeneous features to obtain the corresponding heterogeneous feature encoding vector.

[0127] S602, perform self-attention mechanism processing on each heterogeneous feature encoding vector to obtain the attention distribution corresponding to each dimension of heterogeneous feature;

[0128] S603, based on the attention distribution corresponding to each heterogeneous feature and the attention distribution of all heterogeneous features, the dynamic weights with learnable biases corresponding to each heterogeneous feature are obtained through activation function processing.

[0129] S604, For each heterogeneous feature of the candidate text sample, calculate the product of the dynamic weight of the heterogeneous feature with learnable bias and the normalized heterogeneous feature.

[0130] S605, sum up the products to obtain the collaborative inhibition relationship score between the candidate text sample and the corresponding query text sample.

[0131] For example, for each candidate text sample, the heterogeneous features of each dimension are encoded through an embedding layer network, namely, the rearrangement score, semantic similarity, Jaccard coefficient, longest common subsequence, and edit distance are encoded. Through layer normalization, the corresponding rearrangement score encoding vector, semantic similarity encoding vector, Jaccard coefficient encoding vector, longest common subsequence encoding vector, and edit distance encoding vector are obtained.

[0132] For example, the attention network includes 5 attention heads. The above-mentioned rearrangement score encoding vector, semantic similarity encoding vector, Jaccard coefficient encoding vector, longest common subsequence encoding vector, and edit distance encoding vector are input into the corresponding attention heads. In each attention head, a self-attention mechanism is performed to obtain the corresponding attention distribution.

[0133] For example, the attention distribution of the i-th dimension heterogeneous feature. i The calculation formula is expressed as:

[0134]

[0135] Q i =X i W Q ;

[0136] K i =X i W K ;

[0137] V i =V i W V ;

[0138] Among them, X iLet Q represent the i-th heterogeneous feature encoding vector of the input, where i = 1, 2, 3, 4, 5, and Q represent the rearrangement score encoding vector, semantic similarity encoding vector, Jaccard coefficient encoding vector, longest common subsequence encoding vector, and edit distance encoding vector, respectively. i K represents the query vector corresponding to the i-th heterogeneous feature. i V represents the key vector corresponding to the i-th heterogeneous feature. i W represents the value vector corresponding to the i-th heterogeneous feature. Q W represents the query weight matrix. K W represents the key weight matrix. V d represents the value weight matrix. l =5, This represents the scaling factor.

[0139] Based on the attention distribution corresponding to each heterogeneous feature and the attention distribution of all heterogeneous features, dynamic weights with learnable biases corresponding to each heterogeneous feature are obtained through activation function processing.

[0140] For example, the feature weights α with learnable biases corresponding to the i-th heterogeneous feature. i Represented as:

[0141]

[0142] Where σ(·) represents the sigmoid function, Attention i Let L represent the attention distribution corresponding to the i-th heterogeneous feature, L represent the dimension of the multidimensional heterogeneous feature, b represent the learnable bias parameter, and ε represent a constant, ∈ = 1e-8, a small constant to prevent division by zero errors. For example, L can be set to 5. This formula maps the attention distribution of heterogeneous features to dynamic weights on the feature dimensions of the heterogeneous features. The learnable bias parameter is a variable value obtained by learning the features of candidate text samples during model training, thereby improving the model's expressive power.

[0143] For each heterogeneous feature of the candidate text sample, the product of the dynamic weight of the heterogeneous feature with learnable bias and the normalized heterogeneous feature is calculated. The products are then summed to obtain the collaborative inhibition relationship score between the candidate text sample and the corresponding query text sample.

[0144] For example, the collaborative inhibition score X of candidate text samples fused Represented as:

[0145]

[0146] Where, α iLet L represent the feature weights with learnable biases corresponding to the i-th heterogeneous feature, and let L represent the dimension of the multidimensional heterogeneous feature. Let i represent the heterogeneous feature encoding vector of the i-th heterogeneous feature, where i = 1, 2, 3, 4, 5, and i represent the rearrangement score encoding vector, semantic similarity encoding vector, Jaccard coefficient encoding vector, longest common subsequence encoding vector, and edit distance encoding vector, respectively.

[0147] Based on the collaborative inhibition score X of the candidate text sample fused If d1≤X fused If ≤d2, then the candidate text sample is taken as a difficult negative sample, and the obtained difficult negative samples are used to construct the difficult negative sample set corresponding to the query text sample.

[0148] In this embodiment, an attention mechanism is used to fuse the multidimensional heterogeneous features of candidate text samples. Based on the dynamic feature weight allocation algorithm of the attention mechanism, the importance of features is adjusted according to the features between the current query text and candidate text samples. It can dynamically adjust the importance of different features according to the data and the scene, capture the cooperative inhibition relationship between features, and combine the three-dimensional Gaussian mixture model and the attention mechanism to improve the accuracy of identifying difficult negative samples and significantly improve the adaptability of the model in complex environments.

[0149] In one embodiment of this application, the method further includes: for each candidate text sample corresponding to a query text sample, selecting candidate text samples whose collaborative inhibition relationship score is greater than the upper limit of the negative sample decision boundary as positive samples, and adding these positive samples to the positive sample set corresponding to the query text sample; selecting candidate text samples whose collaborative inhibition relationship score is less than the lower limit of the negative sample decision boundary as negative samples, and selecting candidate text samples whose collaborative inhibition relationship score is within the negative sample decision boundary as difficult negative samples, and adding these negative samples and difficult negative samples to the negative sample set corresponding to the query text sample; constructing a training set based on each query text sample, the corresponding positive sample set, and the negative sample set, and training the large language model with the training set to obtain the trained large language model. It should be noted that, based on the implementation method of this embodiment, positive samples, negative samples, and difficult negative samples are selected from the candidate text sample set for model training. Other positive samples in the positive sample set and other negative samples in the negative sample set can be obtained, but are not limited to, using methods such as similarity or random sampling.

[0150] like Figure 7 As shown in the figure, this application provides an apparatus for generating difficult negative samples, the apparatus comprising:

[0151] The coarse screening module 701 is used to coarsely screen out a candidate set of negative text samples from multiple text samples based on the vector distance between the query text sample and multiple text samples for each query text sample.

[0152] The heterogeneous feature construction module 702 is used to determine the multidimensional heterogeneous features of each candidate text segment based on multiple correlation features between the query text sample and each candidate text segment sample in the negative sample candidate set.

[0153] The negative sample boundary confirmation module 703 is used to input multidimensional heterogeneous features into a three-dimensional Gaussian mixture model, estimate the parameters of the three-dimensional Gaussian mixture model using the expectation-maximization (EM) algorithm, limit the out-of-bounds range using the physical constraint range, divide the candidate set of negative samples of the text into at least three sample distribution clusters, and adjust the decision boundary between different sample distribution clusters. The at least three sample distribution clusters include easily distinguishable negative sample clusters, difficult negative sample clusters, and potential positive sample clusters.

[0154] The scoring confirmation module 704 is used to dynamically assign weights with learnable biases to the multidimensional heterogeneous features corresponding to each candidate text sample through an attention mechanism, so as to obtain the collaborative inhibition relationship score between the candidate text sample and the corresponding query text sample.

[0155] The difficult negative sample generation module 705 is used to take candidate text samples whose collaborative inhibition relationship scores are within the decision boundary corresponding to the difficult negative sample cluster as difficult negative samples, and construct the difficult negative sample set corresponding to the query text sample from each of the obtained difficult negative samples.

[0156] Preferably, the coarse screening module 701 is specifically used for:

[0157] An embedding model is used to vectorize each query text sample and each text segment sample in the sample set, resulting in the query vector corresponding to each query text sample and the text segment vector corresponding to each text segment sample.

[0158] For each query text sample, determine the similarity between the query vector and each text segment vector, and construct the K text segment samples with the highest similarity as the candidate set of negative text segment samples corresponding to the query text sample.

[0159] Preferably, the heterogeneous feature construction module 702 is specifically used for:

[0160] For each candidate text sample,

[0161] The rearrangement model is used to obtain the rearrangement score between the query text sample and the candidate text sample, so as to determine the global statistical correlation between the query text sample and the candidate text sample.

[0162] Calculate the semantic similarity between the query text sample and the candidate text sample to determine the deep semantic association between the query text sample and the candidate text sample;

[0163] Calculate the Jaccard coefficient between the query text sample and the candidate text sample to determine the character-level similarity between them;

[0164] Determine the longest common subsequence between the query text sample and the candidate text sample to determine the consistency of word order structure between the query text sample and the candidate text sample;

[0165] Calculate the edit distance between the query text sample and the candidate text sample to determine the morphological change tolerance between the query text sample and the candidate text sample;

[0166] Multidimensional heterogeneous features of the candidate text sample are constructed using rearrangement score, semantic similarity, Jaccard coefficient, longest common subsequence, and edit distance.

[0167] Preferably, the negative sample boundary confirmation module 703 is specifically used for:

[0168] Using a three-dimensional Gaussian mixture model, the input multidimensional heterogeneous features are transformed into at least three Gaussian distributions;

[0169] In the process of iteratively calculating the parameters of a 3D Gaussian mixture model based on the EM algorithm,

[0170] For a low-similarity Gaussian distribution with the largest variance among at least three Gaussian distributions, the corresponding sample distribution cluster is taken as an easily distinguishable negative sample cluster;

[0171] For a highly similar Gaussian distribution with the smallest variance among at least three Gaussian distributions, the corresponding sample distribution cluster is taken as the potential positive sample cluster.

[0172] The sample distribution clusters corresponding to at least three Gaussian distributions with moderate similarity are taken as the difficult negative sample clusters;

[0173] Obtain the leftmost standard deviation of the mean of the potential positive sample cluster and the leftmost standard deviation of the mean of the difficult negative sample cluster. Use the leftmost standard deviation of the mean as the upper limit of the negative sample decision boundary and the leftmost standard deviation of the mean as the lower limit of the negative sample decision boundary. Use physical constraints to limit the threshold range of the upper and lower limits.

[0174] Preferably, the negative sample boundary confirmation module 703 is specifically used for:

[0175] The threshold range for the lower bound of the negative sample decision boundary is calculated based on the lower bound of the 95% confidence interval of the Gaussian distribution with moderate similarity; the threshold range for the upper bound of the negative sample decision boundary is calculated based on the lower bound of the 68% confidence interval of the Gaussian distribution with high similarity.

[0176] Preferably, the scoring confirmation module 704 is specifically used for:

[0177] For each candidate text sample, each dimension of the heterogeneous features in the multidimensional heterogeneous features is encoded to obtain the corresponding heterogeneous feature encoding vector.

[0178] Each heterogeneous feature encoding vector is processed by a self-attention mechanism to obtain the attention distribution corresponding to each heterogeneous feature dimension.

[0179] Based on the attention distribution corresponding to each heterogeneous feature and the attention distribution of all heterogeneous features, dynamic weights with learnable biases corresponding to each heterogeneous feature are obtained through activation function processing.

[0180] For each heterogeneous feature of the candidate text sample, calculate the product of the dynamic weight of the heterogeneous feature with a learnable bias and the normalized heterogeneous feature.

[0181] The products are summed to obtain the collaborative inhibition relationship score between the candidate text sample and the corresponding query text sample.

[0182] For example: the feature weight α corresponding to the i-th heterogeneous feature with a learnable bias i Represented as:

[0183]

[0184] Where σ(·) represents the sigmoid function, Attention i Let L represent the attention distribution corresponding to the i-th heterogeneous feature, L represent the dimension of the multidimensional heterogeneous feature, b represent the learnable bias parameter, and ε represent a constant.

[0185] This application also provides a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the steps of any of the above-described methods for generating difficult negative samples.

[0186] Figure 8 This is a schematic diagram of the hardware structure of an electronic device provided in an embodiment of this application. Figure 8 The illustrated electronic device includes: a processor 801, a communication interface 802, a memory 803, and a communication bus 804. The processor 801, communication interface 802, and memory 803 communicate with each other via the communication bus 804. Figure 8 The connection method between the processor 801, communication interface 802, and memory 803 shown is merely exemplary. In the implementation process, the processor 801, communication interface 802, and memory 803 may also communicate with each other using other connection methods besides the communication bus 804.

[0187] The memory 803 can be used to store computer programs, which may include instructions and data to implement the steps of any of the above-described methods for generating difficult negative samples. In this embodiment, the memory 803 can be various types of storage media, such as random access memory (RAM), read-only memory (ROM), non-volatile RAM (NVRAM), programmable ROM (PROM), erasable PROM (EPROM), electrically erasable PROM (EEPROM), flash memory, optical memory, and registers. The memory 803 may include a hard disk and / or RAM.

[0188] Processor 801 can be a general-purpose processor, which can be a processor that performs specific steps and / or operations by reading and executing a computer program (e.g., a computer program) stored in memory (e.g., memory 803). The general-purpose processor may use data stored in memory (e.g., memory 803) during the execution of said steps and / or operations. The general-purpose processor can be, for example, but not limited to, a central processing unit (CPU). Furthermore, processor 801 can also be a special-purpose processor, which can be a processor specifically designed to perform specific steps and / or operations. Special-purpose processors can be, for example, but not limited to, ASICs and FPGAs. Additionally, processor 801 can also be a combination of multiple processors, such as a multi-core processor.

[0189] Communication interface 802 may include input / output (I / O) interfaces, physical interfaces, and logical interfaces for interconnecting devices within the network device, as well as interfaces for interconnecting the network device with other devices (e.g., network devices). The communication network may be Ethernet, a radio access network (RAN), a wireless local area network (WLAN), etc. Communication interface 802 may be a module, circuit, transceiver, or any device capable of enabling communication.

[0190] In implementation, each step of the above method can be completed by the integrated logic circuits in the hardware of the processor 801 or by instructions in software. The method disclosed in the embodiments of this application can be directly implemented by the hardware processor, or by a combination of hardware and software modules in the processor. The software modules can reside in readily available storage media in the art, such as random access memory (RAM), read-only memory (ROM), programmable read-only memory (ROM), electrically erasable programmable memory (EPR), or registers. This storage medium is located in memory 803, and the processor 801 reads the information in memory 803 and, in conjunction with its hardware, completes the steps of the above method. To avoid repetition, detailed descriptions are omitted here.

[0191] It should be understood that the term "and / or" in this article is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, or B existing alone. A and B can be singular or plural. Additionally, the character " / " in this article generally indicates an "or" relationship between the preceding and following related objects, but it can also represent an "and / or" relationship. Please refer to the context for a more accurate understanding.

[0192] In this invention, "at least one" means one or more, and "more than one" means two or more. "At least one of the following" or similar expressions refer to any combination of these items, including any combination of a single item or a plurality of items. For example, at least one of a, b, or c can represent: a, b, c, ab, ac, bc, or abc, where a, b, and c can be single or multiple.

[0193] It should be noted that, in this document, relational terms such as "first" and "second" are used merely to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes the element.

[0194] The various embodiments in this specification are described in a related manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, the apparatus embodiments are basically similar to the method embodiments, so the description is relatively simple; relevant parts can be referred to the descriptions of the method embodiments.

[0195] The above are merely preferred embodiments of this application and are not intended to limit the scope of protection of this application. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application are included within the scope of protection of this application.

Claims

1. A method for generating difficult negative samples, characterized in that, The method includes: For each query text sample: Based on the vector distance between the query text sample and multiple text segment samples, a candidate set of negative text segment samples is coarsely filtered out from the multiple text segment samples. Based on the multiple correlation features between the query text sample and each candidate text sample in the negative sample candidate set, the multidimensional heterogeneous features of each candidate text sample are determined. The multidimensional heterogeneous features are input into a three-dimensional Gaussian mixture model, and the parameters of the three-dimensional Gaussian mixture model are estimated using the expectation-maximization (EM) algorithm. The physical constraint range is used as the threshold for limiting the out-of-bounds range. The negative sample candidate set of the text segment is divided into at least three sample distribution clusters. The decision boundary between different sample distribution clusters is adjusted. The at least three sample distribution clusters include easily distinguishable negative sample clusters, difficult negative sample clusters, and potential positive sample clusters. For each candidate text segment sample, an attention mechanism is used to dynamically assign weights with learnable biases to the multidimensional heterogeneous features corresponding to the candidate text segment sample, thereby obtaining a score of the collaborative inhibition relationship between the candidate text segment sample and the corresponding query text sample. The candidate text samples whose collaborative inhibition relationship scores are within the decision boundary corresponding to the difficult negative sample cluster are taken as difficult negative samples, and the obtained difficult negative samples are used to construct the difficult negative sample set corresponding to the query text sample.

2. The method for generating difficult negative samples as described in claim 1, characterized in that, The multidimensional heterogeneous features are input into a three-dimensional Gaussian mixture model, and the parameters of the three-dimensional Gaussian mixture model are estimated using the EM algorithm. Using the physical constraint range as a threshold for out-of-bounds limitation, the negative sample candidate set of the text segment is divided into at least three sample distribution clusters. The decision boundaries between different sample distribution clusters are adjusted, including: Using a three-dimensional Gaussian mixture model, the input multidimensional heterogeneous features are transformed into at least three Gaussian distributions; During the iterative calculation of the parameters of the three-dimensional Gaussian mixture model based on the EM algorithm, For the low-similarity Gaussian distribution with the largest variance among the at least three Gaussian distributions, the corresponding sample distribution cluster is taken as the easily distinguishable negative sample cluster; For the at least three Gaussian distributions with the smallest variance, the corresponding sample distribution cluster is taken as the potential positive sample cluster; The sample distribution cluster corresponding to the moderately similar Gaussian distribution among the at least three Gaussian distributions is taken as the difficult negative sample cluster; The leftmost standard deviation of the mean of the potential positive sample cluster and the leftmost standard deviation of the mean of the difficult negative sample cluster are obtained respectively. The leftmost standard deviation of the mean is used as the upper limit of the negative sample decision boundary, and the leftmost standard deviation of the mean is used as the lower limit of the negative sample decision boundary. The threshold over-limit range of the upper and lower limits is restricted by the physical constraint range.

3. The method for generating difficult negative samples as described in claim 2, characterized in that, The threshold out-of-bounds limit range of the upper and lower limits is restricted by physical constraints: The threshold limit for the negative sample decision boundary is calculated based on the lower limit of the 95% confidence interval of a moderately similar Gaussian distribution. The threshold limit for the upper limit of the negative sample decision boundary is calculated based on the lower limit of the 68% confidence interval of the high-similarity Gaussian distribution.

4. The method for generating difficult negative samples as described in claim 1, characterized in that, Through an attention mechanism, dynamic weights with learnable biases are assigned to the multidimensional heterogeneous features corresponding to the candidate text sample, resulting in a score for the collaborative inhibition relationship between the candidate text sample and the corresponding query text sample, including: For each candidate text segment sample, each dimension of the heterogeneous features in the multidimensional heterogeneous features is encoded to obtain the corresponding heterogeneous feature encoding vector. Each heterogeneous feature encoding vector is processed by a self-attention mechanism to obtain the attention distribution corresponding to each heterogeneous feature dimension. Based on the attention distribution corresponding to each heterogeneous feature and the attention distribution of all heterogeneous features, dynamic weights with learnable biases corresponding to each heterogeneous feature are obtained through activation function processing. For each heterogeneous feature of the candidate text sample, calculate the product of the dynamic weight of the heterogeneous feature with a learnable bias and the normalized heterogeneous feature. The products are summed to obtain the collaborative inhibition relationship score between the candidate text sample and the corresponding query text sample.

5. The method for generating difficult negative samples as described in claim 4, characterized in that, Based on the attention distribution corresponding to each heterogeneous feature and the attention distribution of all heterogeneous features, dynamic weights with learnable biases corresponding to each heterogeneous feature are obtained through activation function processing, including: The dynamic weights α corresponding to the i-th heterogeneous feature have learnable biases. i Represented as: Where σ(·) represents the sigmoid function, Attention i Let L represent the attention distribution corresponding to the i-th heterogeneous feature, L represent the dimension of the multidimensional heterogeneous feature, b represent the learnable bias parameter, and ε represent a constant.

6. The method for generating difficult negative samples as described in claim 1, characterized in that, Based on the vector distance between the query text sample and multiple text segment samples, a candidate set of negative text segment samples is coarsely filtered from the multiple text segment samples, including: An embedding model is used to vectorize each query text sample and each text segment sample in the sample set, resulting in the query vector corresponding to each query text sample and the text segment vector corresponding to each text segment sample. For each query text sample, determine the similarity between the query vector and each text segment vector, and construct the K text segment samples with the highest similarity as the candidate set of negative text segment samples corresponding to the query text sample.

7. The method for generating difficult negative samples as described in claim 1, characterized in that, Based on the multiple correlation features between the query text sample and each candidate text sample in the negative sample candidate set, the multidimensional heterogeneous features of each candidate text sample are determined, including: For each candidate text sample, The rearrangement model is used to obtain the rearrangement score between the query text sample and the candidate text sample, so as to determine the global statistical correlation between the query text sample and the candidate text sample. Calculate the semantic similarity between the query text sample and the candidate text sample to determine the deep semantic association between the query text sample and the candidate text sample; Calculate the Jaccard coefficient between the query text sample and the candidate text sample to determine the character-level similarity between them; Determine the longest common subsequence between the query text sample and the candidate text sample to determine the consistency of word order structure between the query text sample and the candidate text sample; Calculate the edit distance between the query text sample and the candidate text sample to determine the morphological change tolerance between the query text sample and the candidate text sample; The multidimensional heterogeneous features of the candidate text sample are constructed using the rearrangement score, semantic similarity, Jaccard coefficient, longest common subsequence, and edit distance.

8. The method for generating difficult negative samples as described in claim 1, characterized in that, The method further includes: For each candidate text sample corresponding to a query text sample, the candidate text sample whose collaborative inhibition relationship score is greater than the upper limit of the negative sample decision boundary is regarded as a positive sample and added to the positive sample set corresponding to the query text sample. Candidate text samples whose collaborative inhibition relationship score is less than the lower limit of the negative sample decision boundary are designated as negative samples, and candidate text samples whose collaborative inhibition relationship score is within the negative sample decision boundary are designated as difficult negative samples. Negative samples and difficult negative samples are added to the negative sample set corresponding to the query text sample. A training set is constructed based on each query text sample, the corresponding positive sample set, and the negative sample set. The large language model is trained using the training set to obtain the trained large language model.

9. A device for generating difficult negative samples, characterized in that, The device includes: The coarse screening module is used to coarsely screen out a candidate set of negative text samples from the multiple text samples for each query text sample based on the vector distance between the query text sample and the multiple text sample samples. The heterogeneous feature construction module is used to determine the multidimensional heterogeneous features of each candidate segment sample based on multiple correlation features between the query text sample and each candidate segment sample in the negative sample candidate set of the segment. The negative sample boundary confirmation module is used to input the multidimensional heterogeneous features into the three-dimensional Gaussian mixture model, estimate the parameters of the three-dimensional Gaussian mixture model using the expectation-maximization (EM) algorithm, and divide the negative sample candidate set of the text into at least three sample distribution clusters using the physical constraint range as the threshold for out-of-bounds limitation. The module also adjusts the decision boundary between different sample distribution clusters, wherein the at least three sample distribution clusters include easily distinguishable negative sample clusters, difficult negative sample clusters, and potential positive sample clusters. The scoring and confirmation module is used to dynamically assign weights with learnable biases to the multidimensional heterogeneous features corresponding to each candidate text sample through an attention mechanism, thereby obtaining a score of the collaborative inhibition relationship between the candidate text sample and the corresponding query text sample. The difficult negative sample generation module is used to take candidate text samples with collaborative inhibition relationship scores within the decision boundary corresponding to the difficult negative sample cluster as difficult negative samples, and construct the difficult negative sample set corresponding to the query text sample from each of the obtained difficult negative samples.

10. An electronic device comprising a memory and a processor, characterized in that, The memory stores a computer program, characterized in that, when the processor executes the computer program, it implements the steps of the method for generating difficult negative samples according to any one of claims 1 to 8.

Images