A training method and device of a query expansion model

Abstract

The embodiment of the specification provides a training method of a query expansion model, comprising the following steps: obtaining a first query text and a label document set corresponding to the first query text in a knowledge base, wherein each document in the label document set contains an answer corresponding to the first query text; determining a training loss, and training the query expansion model with the objective of minimizing the training loss; the training loss is negatively correlated with retrieval quality corresponding to the first query text; the retrieval quality is used to measure the recall quality of the query text in the knowledge base for the label document set; and the trained query expansion model is used to expand the query text to perform retrieval in the knowledge base.

Description

Technical Field

[0001] This specification relates to the field of machine learning technology, and in particular to a method and apparatus for training a query expansion model. Background Technology

[0002] Retrieval-Augmented Generation (RAG) systems, by introducing external knowledge sources, can alleviate the shortcomings of large language models in terms of knowledge coverage and factual accuracy to some extent. In RAG systems, the recall rate in the retrieval phase often directly affects the quality of subsequent answer generation: if the initial query fails to find relevant documents in the knowledge base, the large language model may struggle to provide an accurate answer due to a lack of necessary contextual information.

[0003] Query expansion, a key technology for improving retrieval recall, generates query statements that better express the user's true intent by semantically supplementing and structurally optimizing the user's original query. This narrows the gap between the user's original input and the semantics of documents in the knowledge base, thereby increasing the probability of relevant documents being retrieved.

[0004] One related technique employs a rule-based and thesaurus-based approach to query expansion. This method relies on a pre-built thesaurus, domain terminology dictionary, or grammatical rules to perform synonym replacements or sentence restructuring on keywords in the original query, generating several expanded queries. However, this method's expansion capability depends heavily on the completeness and domain relevance of the dictionary. It often struggles to generate effective expansion results for new words not included in the dictionary, colloquial expressions, or queries with complex semantic implications. Furthermore, this technique uses a uniform expansion strategy for all queries, paying little attention to the differences in retrieval tasks between different queries.

[0005] In another related technique, query expansion employs a generative method based on Supervised Fine-Tuning (SFT). Using pre-constructed pairs of original and expanded queries, a large language model is trained under supervision, enabling it to generate corresponding expanded versions based on the input query. This deep learning-based approach offers stronger semantic understanding and generalization capabilities compared to rule-based methods. However, this method prioritizes maximizing the probability of the large language model during training, focusing on generating text similar to the surface form of the training labels, rather than optimizing for retrieval performance. Therefore, the model may generate expanded results that are linguistically fluent but deviate from actual retrieval needs.

[0006] This indicates that existing query expansion model training methods generally suffer from a discrepancy between training objectives and retrieval performance metrics. Therefore, this paper aims to provide a solution that, through technical means, enables query expansion models to improve their ability to expand various types of queries, especially difficult queries, thereby enhancing the retrieval quality of the expanded query text in the knowledge base. Summary of the Invention

[0007] One or more embodiments of this specification describe a method and apparatus for training a query expansion model, which can solve the above-mentioned technical problems.

[0008] Based on the first aspect, a method for training a query expansion model is provided, including:

[0009] Obtain the first query text and the set of tagged documents corresponding to the first query text in the knowledge base, wherein each document in the set of tagged documents contains an answer corresponding to the first query text.

[0010] A training loss is determined, and the query expansion model is trained with the goal of minimizing the training loss; the training loss is negatively correlated with the retrieval quality corresponding to the first query text; the retrieval quality is used to measure the recall quality of the query text in the knowledge base for the tagged document set; the trained query expansion model is used to expand the query text to perform retrieval in the knowledge base.

[0011] According to the second aspect, this specification provides a training apparatus for a query expansion model, comprising:

[0012] The acquisition module is configured to acquire a first query text and a set of tag documents corresponding to the first query text in the knowledge base, wherein each document in the set of tag documents contains an answer corresponding to the first query text.

[0013] The training module is configured to determine a training loss and train the query expansion model with the goal of minimizing the training loss; the training loss is negatively correlated with the retrieval quality corresponding to the first query text; the retrieval quality is used to measure the recall quality of the query text in the knowledge base for the tagged document set; the trained query expansion model is used to expand the query text to perform a retrieval in the knowledge base.

[0014] According to a third aspect, a computer program product is provided, including a computer program / instructions that, when executed by a processor, implement the steps of the method described in the first aspect.

[0015] According to a fourth aspect, a computing device is provided, including a memory and a processor, characterized in that the memory stores executable code, and when the processor executes the executable code, it implements the method described in the first aspect.

[0016] In summary, the methods and apparatus provided in the embodiments of this specification include a method for training a query expansion model. During the training process, the contribution of the training sample to the model training can be adaptively adjusted based on the retrieval quality of the first query text used as the training sample. Lower retrieval quality, i.e., greater retrieval difficulty, results in a greater contribution of the training sample to the loss function and a more significant update of the model parameters; conversely, higher retrieval quality, i.e., lower retrieval difficulty, results in a smaller contribution of the training sample to the loss function. Therefore, the training of the query expansion model can focus on difficult samples that contribute significantly to improving model performance, thereby enhancing the model's ability to expand various queries, especially complex, fuzzy, or long-tail queries. Attached Figure Description

[0017] To more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are merely some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without any creative effort.

[0018] Figure 1 This is a schematic diagram illustrating the implementation framework of a query expansion model training method disclosed in this specification;

[0019] Figure 2 This is a flowchart of a training method for a query expansion model disclosed in this specification;

[0020] Figure 3 This is a schematic diagram illustrating a training method for a query expansion model according to an embodiment of this specification;

[0021] Figure 4 This is a schematic diagram of a training device for a query expansion model according to an embodiment of this specification. Detailed Implementation

[0022] The solutions provided in the embodiments of this specification will now be described with reference to the accompanying drawings.

[0023] Retrieval-Augmented Generation (RAG) systems, by introducing external knowledge sources, can alleviate the shortcomings of large language models in terms of knowledge coverage and factual accuracy to some extent. In RAG systems, the recall rate during the retrieval phase directly impacts the quality of subsequent answer generation: if the initial query fails to find relevant documents in the knowledge base, the large language model may struggle to provide an accurate answer due to a lack of necessary contextual information. Query expansion, a key technology for improving retrieval recall, generates query statements that better express the user's true intent by semantically supplementing and structurally optimizing the user's original query. This narrows the gap between the user's original input and the semantics of documents in the knowledge base, increasing the probability of relevant documents being retrieved.

[0024] A query expansion model is a model that performs the aforementioned query expansion task. It is typically trained on a large language model to achieve this. The query expansion model receives the original query text input by the user and outputs expanded query text, which has undergone semantic supplementation or structural reorganization, for use in subsequent retrieval processes. For a query expansion model, the recall capability of its output expanded query text for target documents in the knowledge base is one of the core performance metrics. Therefore, the training objective of a query expansion model should not be limited to generating outputs that are textually similar to the training labels, but should ensure that the expanded queries output by the model can effectively improve the recall quality of knowledge base retrieval.

[0025] In one related technique, query expansion employs a rule-based and thesaurus-based approach. This method relies on a pre-built thesaurus, domain terminology dictionary, or grammatical rules to perform synonym replacements or sentence restructuring on keywords in the original query, generating several expanded queries. However, this method's expansion capability depends on the completeness and domain relevance of the dictionary. For new words not included in the dictionary, colloquial expressions, or queries with complex semantic implications, it often struggles to generate effective expansion results. Furthermore, this related technique applies a uniform expansion strategy to all queries, paying little attention to the differences in retrieval tasks between different queries.

[0026] In another related technique, query expansion employs a generative expansion method based on Supervised Fine-Tuning (SFT). This method uses pre-constructed pairs of original and expanded queries to train a large language model, enabling it to generate corresponding expanded versions based on input queries. This deep learning-based approach offers stronger semantic understanding and generalization capabilities compared to rule-based methods. However, this method prioritizes maximizing the probability of the large language model during training, focusing on generating text similar to the surface form of the training labels, rather than directly optimizing for retrieval performance metrics. Therefore, the model may generate fluent expanded results that deviate from actual retrieval needs. Furthermore, the supervised fine-tuning training process treats all training samples equally, failing to consider the differences in retrieval tasks between different queries. For difficult samples where the original query alone is insufficient to retrieve relevant documents, the model cannot receive differentiated learning attention during training.

[0027] Besides the problems described above in the training process of query expansion models, another issue lies in the construction of training data. High-quality expansion labels typically require manual annotation, with annotators writing one or more high-quality expanded queries for each original query. This manual annotation method is costly, and the annotation quality is affected by the professional level of the annotators, making it difficult to implement on a large scale.

[0028] In view of this, the inventors propose a training method for a query expansion model in this specification. This method can guide the training and optimization direction of the query expansion model based on the retrieval quality indicators of the training samples. During the training process, the model can learn differently from samples with different retrieval difficulties (based on retrieval quality), thereby improving the query expansion model's ability to expand various queries, especially difficult queries.

[0029] Figure 1 A schematic diagram illustrating the implementation framework of the above method is shown. (See also...) Figure 1In one embodiment, the training loss during the query expansion model training process can be negatively correlated with the retrieval quality of the original query. First, for each original query, a set of tagged documents corresponding to it in the knowledge base is obtained. Each document in the tagged document set contains the answer to the original query, representing the documents expected to be retrieved. Based on this, a retrieval quality metric is defined to measure the recall quality of the query text in the knowledge base for that tagged document set. Furthermore, in the design of the training loss, it is negatively correlated with the retrieval quality of the original query: if the retrieval quality of the original query itself is high, the sample contributes less to the training loss; if the retrieval quality of the original query itself is low, the sample contributes more to the training loss. Thus, the training process can differentially adjust the training signal according to the retrieval quality of the query; in other words, difficult queries receive a stronger training gradient, and simple queries receive a weaker training gradient, thereby favoring the training of the query expansion model in improving its ability to expand difficult queries.

[0030] In this embodiment, the training loss of the query expansion model is negatively correlated with the retrieval quality of the original query. It can automatically identify difficult queries during training and apply stronger optimization signals to them, thus addressing the problem of existing training methods lacking differentiated attention to difficult samples. The trained query expansion model can generate more effective expansions for various queries, especially complex, fuzzy, or long-tail queries, improving the recall quality of knowledge base retrieval.

[0031] Based on the above technical framework Figure 2 A flowchart illustrating a training method for a query expansion model according to an embodiment of this specification is shown. It is understood that the method disclosed in the embodiments of this specification can be executed by any device, apparatus, platform, or cluster of devices with computing and processing capabilities.

[0032] See Figure 2 In one embodiment, the method includes at least the following steps: Step S201: Obtain a first query text and a set of tagged documents corresponding to the first query text in a knowledge base, wherein each document in the tagged document set contains an answer corresponding to the first query text. Step S203: Determine a training loss and train the query expansion model with the goal of minimizing the training loss; the training loss is negatively correlated with the retrieval quality corresponding to the first query text; the retrieval quality is used to measure the recall quality of the query text in the knowledge base for the tagged document set; the trained query expansion model is used to expand the query text to perform a retrieval in the knowledge base.

[0033] The above steps will be explained in detail below.

[0034] Step S201: Obtain the first query text and the tag document set corresponding to the first query text in the knowledge base, wherein each document in the tag document set contains the answer corresponding to the first query text.

[0035] In this step, a training sample may include a first query text and a corresponding set of tagged documents. The first query text is the user's original query statement, which can be a complete natural language question, a combination of keywords, or query text containing colloquial expressions, vague statements, or long-tail terms. The knowledge base stores a large number of documents, which can be structured documents, unstructured text, question-answer pairs, etc. The tagged document set is the collection of documents in the knowledge base related to the first query text, and each document in the tagged document set contains an answer corresponding to the first query text. It can be understood that the tagged document set essentially defines the expected retrieval target for the first query text; that is, the documents that the first query text, after expansion, is expected to effectively retrieve from the knowledge base.

[0036] In a specific implementation, the tagged document set can be obtained as follows: For the first query text, a fusion recall based on a first strategy and a second strategy is performed in the knowledge base to obtain a candidate document set. The first strategy can be a keyword-based recall strategy, and the second strategy can be a semantic-based recall strategy.

[0037] Specifically, keyword-based recall strategies can employ the BM25 (Best Match 25) algorithm. BM25 is a text retrieval algorithm based on the bag-of-words model. Its core idea is: for a given query text, it calculates the relevance score of each candidate document to the query based on the term frequency (TF) of each term in the candidate documents and the inverse document frequency (IDF) of that term in the entire document set. BM25 performs saturation processing on term frequency—that is, the more frequently a term appears in a document, the more its contribution to the score gradually flattens out rather than increasing linearly, thus avoiding a few high-frequency terms from excessively dominating the score. At the same time, BM25 introduces a document length normalization mechanism, so that longer documents do not naturally have an advantage in ranking simply because they contain more terms. The advantages of BM25 are its high computational efficiency, sensitivity to precise term matching, and direct use without training. However, its limitation is that it only focuses on the surface matching of terms and cannot capture the semantic relationships between words—for example, BM25 has difficulty recognizing the semantic equivalence of synonyms, near-synonyms, or sentence paraphrasing.

[0038] Semantic-based recall strategies can employ dense retrieval methods based on BERT embeddings. BERT (Bidirectional Encoder Representations from Transformers) is a pre-trained language model based on the Transformer architecture. Through self-supervised pre-training tasks such as masked language modeling and next-sentence prediction on massive text corpora, it can learn deep semantic representations of text. In dense retrieval scenarios, the query text and documents in the knowledge base can be mapped to fixed-dimensional dense vectors (embeddings) using a BERT encoder. Then, the semantic relevance between the query vector and the document vector is measured by calculating the cosine similarity or inner product between them. Unlike keyword-based BM25, BERT embeddings can capture the semantic associations between terms—even if the query and document are not completely identical in terms of surface wording, as long as they are close in semantic space, they can be effectively retrieved. For example, the query "subsidy application" and the document "grant application" may not match under BM25, but their semantic similarity can be captured through BERT embedding.

[0039] A candidate document set is obtained by fusing the search results based on the first strategy and the search results based on the second strategy. The fusion method can be either taking the union of the two recall results or normalizing the scores of each recall result and then weighting and summing them, selecting the top few documents based on the overall score from highest to lowest to form the candidate document set. The two retrieval strategies based on different principles are complementary in recall. The keyword-based strategy is more effective for precise term matching but lacks robustness to synonyms or sentence variations; the semantic-based strategy can capture semantic similarity but may be insufficiently sensitive to rare words or proper nouns. Fusing the two results yields a more comprehensive candidate document set.

[0040] After obtaining the candidate document set, documents that meet the first condition are selected and added to the labeled document set. The first condition includes that the document contains an answer corresponding to the first query text. This selection operation can be performed by a large language model. Specifically, each document in the candidate document set can be provided to the large language model, which then determines whether the document contains information that can answer the first query text question. This includes, but is not limited to, the document containing content that can completely answer the first query text, or the document containing key information used to answer the first query text. After selection, the documents that meet the condition constitute the labeled document set. As an example, appropriate prompts can be designed to instruct the large language model to read the candidate documents and determine their relevance to the first query text, and output a binary classification (relevant / irrelevant) or hierarchical (highly relevant / moderately relevant / irrelevant) judgment result. Furthermore, the large language model can also be required to extract key paragraphs or information fragments that directly answer the first query text from the document when it is determined to be relevant, so as to provide more accurate contextual reference when generating the first labeled text later.

[0041] After obtaining the tagged document set, the retrieval quality of the first query text can be calculated. Retrieval quality measures the recall quality of any query text within the knowledge base for the tagged document set. In a specific implementation, retrieval quality can be determined based on a weighted sum of a hit rate metric and a ranking quality metric.

[0042] The hit rate metric indicates whether the search results obtained by retrieving the query text in a knowledge base contain at least one document from the tagged document set. For example, the hit rate metric could be Hit@K, which examines whether at least one document from the tagged document set is found among the top K returned documents. In one embodiment, Hit@K can be calculated as follows: assuming the knowledge base contains documents related to the query text... Return to the front The set of search results is The tagged document set is Then Hit@K is defined as:

[0043]

[0044] in, This is an indicator function that takes the value 1 if the condition within the parentheses is true, and 0 otherwise. This represents the intersection operation of sets; This represents the empty set. The meaning of this formula is: [The formula is incomplete and requires further context to be fully translated.] In each search result, as long as at least one document belongs to the tag document set. Hit@K is 1 (hit) or 0 (miss). A hit means the query text successfully guided the retrieval system to find relevant documents; a miss means the retrieval system failed to associate the query text with the target document.

[0045] In a specific application, Hit@K can also take the form of continuous values, for example, previously... The percentage of documents belonging to the tagged document set in each search result is used as the indicator value, so that Hit@K takes continuous values ​​between 0 and 1 to provide more granular information on the hit rate.

[0046] Ranking quality metrics are used to represent how closely a search result ranks relative to a set of tagged documents. For example, a ranking quality metric can be the Normalized Discounted Cumulative Gain (NDCG). In one embodiment, NDCG can be calculated as follows.

[0047] First, before calculation Discounted Cumulative Gain (DCG) of each search result:

[0048]

[0049] in, Indicates the sorting position in the search results; Indicates the order in the order of The relevance score of a document if it belongs to the tagged document set. ,but You can choose 1 (or a higher score depending on the relevance), otherwise... =0; As a decay factor, the contribution of documents ranked lower in the search results to DCG is logarithmically attenuated, reflecting the intuitive understanding that higher-ranked documents are more valuable to users in search scenarios.

[0050] Then, calculate the Ideal DCG (IDCG). IDCG is the cumulative gain under ideal ordering (i.e., the set of tagged documents). DCG value when all documents are ranked at the top of the search results:

[0051]

[0052] in, This indicates the number of documents in the tagged document set. IDCG represents the current query's position within the given document set. The maximum DCG value that can be achieved.

[0053] Based on this, NDCG is defined as the ratio of DCG to IDCG:

[0054]

[0055] The NDCG value ranges from 0 to 1. The closer the value is to 1, the closer the ranking of the search results is to the ideal ranking, meaning that documents in the tagged document set rank higher in the search results. Unlike the hit rate metric, the ranking quality metric not only focuses on whether a document was hit, but also on the quality of the position of the hit document in the ranking, thus providing a more granular evaluation of search quality.

[0056] Search quality can be expressed as a weighted combination of hit rate and ranking quality metrics:

[0057]

[0058] in, This represents the query text to be evaluated. Represents a set of tagged documents; Indicates the use of query text When searching in the knowledge base, the first Hit in the search results The index values ​​of the Chinese document; Indicates the use of query text When searching in the knowledge base, Normalized loss cumulative gain of Chinese documents in search results; and The weighting coefficients for the hit rate metric and the ranking quality metric are respectively, satisfying... The specific value can be flexibly configured according to the different emphasis requirements of hit rate and sorting quality in the application scenario. As an example, 0.6 can be taken. The value can be 0.4; it can also be adjusted to other values ​​based on experience. This specification does not limit the implementation of this embodiment. To determine the number of truncations for the search results being examined, for example, we can take... , wait.

[0059] Based on the above method, after calculating the retrieval quality of the first query text, the first tag text corresponding to the first query text can be further generated. The first tag text can be used as a tag in the query expansion model training process, that is, the high-quality version that the expanded query output by the query expansion model should refer to.

[0060] In a specific implementation, the first tag text can be generated as follows: based on the first query text and the tag document set, multiple first candidate expanded texts are generated through a large language model; the first tag text is determined according to the ranking of the retrieval quality of the multiple first candidate expanded texts.

[0061] Specifically, the first query text and each document in the tag document set can be used as input, and prompt words can be set to enable the large language model to generate... This allows for better matching and recall of candidate expanded queries from documents in the tagged document set. The prompts might include instructions such as requiring the large language model to semantically supplement, clarify, or structurally optimize the first query text based on the content of each document in the tagged document set, generating multiple candidate expanded queries with different emphases in expression. Because the large language model possesses semantic understanding and text generation capabilities, it can generate multiple first candidate expanded texts based on its understanding of the semantics of the first query text and the content of each document in the tagged document set. For example, assuming the first query text is a vague, colloquial expression of a subsidy policy, and the tagged document set contains formal explanatory documents of that subsidy policy, the large language model could generate the following candidate expanded texts: one focusing on the name of the subsidy policy, one focusing on the application conditions, one focusing on the subsidy recipients, and so on.

[0062] For each generated first candidate expanded text Calculate their retrieval quality respectively ,in Indicates the first First, select the candidate expanded texts. Then, rank the candidate expanded texts according to their retrieval quality, and select the candidate with the highest retrieval quality as the first tag text. :

[0063]

[0064] in, For generated The first candidate expanded text. Through the above methods, the first tag text... The quality of the search results can be determined based on objective search quality metrics, rather than human judgment. High search quality means that the candidate expanded text is more likely to effectively match the tagged document set. The documents in the document can be used as labels for model training.

[0065] Thus, through the above operations, the data required to train the query expansion model can be obtained, including: the first query text. , tag document set Search quality indicators First tag text And the retrieval quality of the first query text itself. This data provides the foundation for training the query expansion model in subsequent steps.

[0066] Step S203: Determine the training loss and train the query expansion model with the goal of minimizing the training loss.

[0067] After obtaining the data related to the training of the query expansion model, in this step, the training loss can be determined based on this data, and the parameters of the query expansion model can be updated accordingly.

[0068] In one embodiment, the training loss can be negatively correlated with the retrieval quality corresponding to the first query text. Specifically, in step S201, the first query text is obtained. Its own search quality This retrieval quality quantifies the quality of the first query text in the knowledge base for the set of tagged documents, without expansion. The ability to recall [products]. The lower the value, the more difficult it is to find relevant documents for the first query text, indicating that it is a difficult sample with a high retrieval difficulty. The higher the value, the more effectively the first query text can hit relevant documents, indicating lower retrieval difficulty. In this embodiment, the training loss is designed so that training samples with lower retrieval quality (higher retrieval difficulty) receive a relatively large training loss, while training samples with higher retrieval quality (lower retrieval difficulty) receive a relatively small training loss. Therefore, the training process can differentiate the optimization signal according to the retrieval difficulty of the training samples, allowing the parameter updates of the query expansion model to focus more on improving the expansion ability for difficult queries.

[0069] In a specific application, the training loss can include at least one of the following two loss terms: a first loss and a second loss. The first loss can be based on supervised learning, measuring the difference between the output of the query expansion model and the first-labeled text; the second loss can be based on reinforcement learning, using retrieval quality as a reward signal to guide the model to directly optimize the retrieval quality of its output prediction results. These two loss terms can be dynamically weighted and correlated with the retrieval quality of the first query text to constitute the final training loss. Figure 3 The diagram illustrates the query expansion model training method in this embodiment. The query expansion model can be trained based on the first loss, the second loss, and the retrieval quality of the first query text. The two loss terms are described in detail below.

[0070] Before describing the detailed calculation methods of the two loss terms, the generation process of the query expansion model will first be explained. The parameters of the query expansion model are set as follows: It defines the conditional probability distribution from the input query to the output expanded query, denoted as . The first query text After inputting the query expansion model, the model autoregressively generates expanded query text word by word based on the conditional probability distribution. In a specific implementation, the query expansion model can be implemented based on the architecture of a Large Language Model (LLM), for example, using a Transformer-based decoder structure, whose parameters... After large-scale pre-training, further optimization can be performed using the training methods described in the embodiments of this specification.

[0071] First loss: The first query text can be expanded using the query expansion model. The second query text is obtained. The first loss is based on the first tag text. The first loss is determined by the difference between the first predicted output and the second query text. In a specific implementation, the first loss can be based on the difference between the first predicted output and the first labeled text. It is determined by the cross-entropy between them. For example, for the first... For each training sample, the first loss... It can be represented as:

[0072]

[0073] Among them, superscript Indicates the first One training sample; Indicates the first The first query text of each training sample; Indicates the first The first label text of each training sample; Indicates the first tag text The sequence length (in terms of lexical units); Indicates the position index of a word in the sequence; Indicates the first tag text The Middle The word element at each position; Indicates the first tag text Center front A sequence of lexical units, i.e., prefix context; Indicates that given the first query text and the generated prefix lexicon sequence Under the condition that the query expansion model predicts the next term exactly as The probability. This loss function calculates the cross-entropy between the model's predicted distribution and the actual words in the first-labeled text on a word-by-word basis in an autoregressive manner. When the model assigns... When the generation probability of each word element is relatively high, To obtain a smaller value; conversely, if the model is... If the prediction probability of a word element is low, then Larger.

[0074] Using the first loss, the query expansion model can learn the first label text by minimizing the cross-entropy. The language patterns and semantic expressions in the model establish a stable language generation capability. The gradient signal of the cross-entropy loss is relatively smooth and usually does not oscillate violently during training, providing reasonable policy initialization for subsequent reinforcement learning branches. It can be understood that the first loss focuses on the similarity between the model's output text and the labeled text, rather than measuring the retrieval quality of the output in the knowledge base.

[0075] The second loss, unlike the first loss, does not rely on fixed label text. Instead, it uses the actual retrieval quality of the second query text output by the query expansion model in the knowledge base as the evaluation criterion. In a specific implementation, the process of determining the second loss may include: expanding the first query text multiple times through the query expansion model to obtain multiple second query texts; for any second query text, using its corresponding retrieval quality as a reward score; and determining the second loss based on the multiple reward scores corresponding to the multiple second query texts.

[0076] Specifically, for the first query text Generate by querying the expanded model using random sampling. A different second query text, For example, an integer greater than 1. , Wait. This The second query text can be denoted as ,in Indicates the first The second query text, Due to the randomness of the sampling process, different second query texts may differ in terms of expression, information focus, and semantic completeness, which in turn leads to different retrieval quality in the knowledge base.

[0077] For any one of the second query texts Using this as the query, a retrieval is performed in the knowledge base, and its target document set is calculated. The search quality is used as the reward score for the second query text. :

[0078]

[0079] The higher the search quality, the higher the reward score. The larger the value, the better the second query text performs in the actual retrieval task. By using retrieval quality as a reward, the second loss can introduce the feedback signal of the retrieval system into the training process of the query expansion model, so that the update direction of the model parameters is aligned with the improvement of retrieval quality.

[0080] In obtaining Each reward point Subsequently, a second loss can be determined based on these reward scores to guide the model towards updating in the direction of generating high-quality expanded queries. Since the absolute value range and distribution of retrieval quality may differ among different training samples, directly using the original reward scores as the optimization signal may cause inconsistencies in the scale of the training signal across different training samples. Therefore, an advantage function can be introduced to normalize the within-group reward scores.

[0081] In a specific implementation, the process of determining the second loss based on multiple reward scores may include: determining the mean and standard deviation of the multiple reward scores; for any second query text, normalizing its reward score based on the mean and standard deviation to obtain an advantage value; and determining the second loss based on the multiple advantage values. For example, first calculate... Mean and standard deviation of each reward score:

[0082]

[0083]

[0084] in, The average of the reward scores within the group; The standard deviation of the group's bonus scores; For a very small positive number (e.g.) This is used to prevent division by zero errors when the standard deviation is zero.

[0085] Then, for each second query text, its dominance value is calculated. :

[0086]

[0087] Advantage value Reflects the first The retrieval quality of the second query text relative to the group average. If A positive value indicates that the retrieval quality of the second query text is better than the group average and should be encouraged during training; if... A negative value indicates that the retrieval quality of the second query text is below the group average and should be suppressed during training. Normalization by dividing by the standard deviation maps reward scores from different batches and training samples to a similar scale, improving the stability of the training process. Using this method, it is unnecessary to train a separate Critic model to estimate state values, thereby reducing training overhead and simplifying the training process.

[0088] Based on the advantage value, a second loss can be constructed. In a specific implementation, for the... For each training sample, the second loss can be a clipped surrogate objective function. The clipping function controls the magnitude of the second loss's variation, preventing large single policy updates from causing training instability. Second loss It can be represented as:

[0089]

[0090] in, Indicates the first The first query text of each training sample; Indicates the current policy (i.e., the current parameters) The query expansion model below is for the first Training samples generated The conditional probability; This indicates that the old strategy (i.e., the query expansion model under the parameters before the update, or a snapshot strategy) is for the first... Training samples generated The conditional probability; The importance sampling ratio represents the ratio of the probability of the current strategy to the probability of the old strategy generating the same second query text. This is the clipping factor, for example, it can be taken as... Used to limit the importance sampling ratio to Within the range; This indicates taking the smaller of the two values; Indicates will Crop to range inside, if Then take ,like Then take Otherwise take ; The KL divergence term is used to constrain the current policy. Do not deviate from the reference strategy Too far; The reference strategy can be a query expansion model under the initial parameters; The weight coefficient of the KL divergence penalty term is used to control the strength of the KL constraint, and its value can be adjusted according to the training stability requirements.

[0091] The principle behind this pruning mechanism can be described as follows: when the importance sampling ratio exceeds the pruning interval, the pruned ratio value (rather than the original ratio value) is multiplied by the dominance value. Because The existence of the operation, if the advantage value If the value is positive, it represents the upper limit of the pruning ratio, preventing the strategy from excessively amplifying the probability of high-reward outputs in a single update; if the advantage value is positive... A negative value represents the lower bound of the pruning ratio, preventing the policy from excessively suppressing the probability of low-reward outputs. This allows control over the single-step update magnitude of the policy while maintaining an effective learning signal, avoiding divergence or instability during training. The introduction of KL divergence can limit the language generation distribution of the query expansion model from deviating too far from the initial language capability while maximizing retrieval quality—if the distribution of the current policy differs significantly from the reference policy, the value of the KL divergence term is larger, suppressing excessive policy shift through feedback in the loss function; when… When the value is large, the model tends to maintain a close relationship with the reference policy; when... When the size is smaller, the model has greater freedom to explore generation strategies that yield better retrieval quality.

[0092] By leveraging the synergistic effects of multiple mechanisms such as group sampling, advantage normalization, pruning, and KL constraints, the second loss enables the query expansion model to update its parameters in a direction that improves retrieval quality under relatively stable training conditions. Unlike the first loss (cross-entropy), the training signal of the second loss comes directly from the feedback of the knowledge base retrieval system. Therefore, it can effectively overcome the possible deviation between the training objective and retrieval quality in supervised training. Even if the second query text generated by the query expansion model is not completely consistent with the first tag text in terms of textual form, as long as the second query text can achieve high retrieval quality in the knowledge base, the query expansion model will obtain a positive optimization signal.

[0093] The first loss was calculated separately. Second loss Then, the two losses can be weighted to obtain the final training loss. Furthermore, to achieve a negative correlation between the training loss and the quality of the first query text retrieval, retrieval difficulty can be introduced as a weighting coefficient into the training loss.

[0094] In one embodiment, search difficulty can be quantified based on search quality. Search difficulty For example, it can be based on the first The first query text of each training sample The retrieval quality is calculated and defined as:

[0095]

[0096] in, For the first The first query text of each training sample The search quality itself refers to the search quality index value obtained by searching the knowledge base directly with the original query without expansion. The value of is between 0 and 1 (or normalized to that range). The lower the value, the more difficult it is to retrieve the tagged document set from the knowledge base using the original query. The greater the difficulty of retrieving documents within the training sample, the better. The closer to 1, the less likely they are to be; conversely, The higher the value, the more likely the original query can effectively find relevant documents, indicating a lower retrieval difficulty. The closer it is to 0, the better. Therefore, This quantifies the difficulty of performing a retrieval task using training samples: for difficult queries that are fuzzy, colloquial, or long-tailed, Larger; for simple queries that are clear, standardized, and consistent with the knowledge base documentation, Smaller.

[0097] Based on this, the search difficulty Introducing the calculation of training loss can make difficult samples ( Larger samples contribute more to the training loss, while simpler samples ( The smaller loss contributes less to the training loss. Meanwhile, the weights of the first and second losses dynamically change with the number of training steps to achieve a smooth transition of the query expansion model from language capability to retrieval quality optimization.

[0098] In a specific implementation, the training loss can be determined as follows: The weights of both the first and second losses change dynamically with the number of training steps, with the weight of the first loss negatively correlated with the number of training steps, and the weight of the second loss positively correlated with the number of training steps. Let the current training step number be... Total training steps: The weighting coefficient corresponding to the first loss is The weighting coefficient corresponding to the second loss is .

[0099] In a specific implementation The weights can be updated based on the number of training steps and the initial weights using the cosine annealing algorithm. Specifically:

[0100]

[0101] in, The initial weights, i.e., the number of training steps. Preset weights for time; Total training steps; It is a cosine function; Let π be the constant of pi. This formula makes... In the early stages of training ( (Approaching 0) ,along with Increases and then decreases smoothly until the end of training ( near The decay rate is close to 0. Cosine annealing is characterized by a slower decay rate at the beginning and end of training and a faster decay rate in the middle. This helps to maintain sufficient supervised learning signals in the early stage of training, leave enough room for optimization of reinforcement learning in the late stage of training, and achieve a natural switch in the middle of training.

[0102] Accordingly, According to Confirmed. In one implementation, the sum of the weights of the first loss and the second loss can be 1, i.e.:

[0103]

[0104] thus, Follow The increase synchronously grew from near 0 to near 1, and They form a complementary relationship.

[0105] In a specific implementation, the initial weights It can take the value 1, which is the value during the initial stage of training. , Model training is primarily driven by the first loss. In this phase, the query expansion model uses the first-labeled text... The goal is to quickly learn the language structure, syntactic features, and basic semantic expressions of expanded queries, avoiding a significant drop in generation quality caused by random sampling in the early stages of reinforcement learning exploration. As training progresses, when the number of training steps reaches half of the total steps, i.e. hour, and The values ​​can be equal (e.g., both 0.5), and while maintaining language generation capabilities, the model begins to explore expanded expressions that can obtain higher retrieval rewards. Towards the end of training, near , , Training is primarily driven by retrieval quality, even if the second query text generated by the query expansion model matches the first tag text. While not entirely consistent at the lexical level, the model receives a positive optimization signal as long as its retrieval quality is higher. This dynamic transition mechanism allows for a balance between training stability and retrieval quality optimization.

[0106] Search difficulty With dynamic weight coefficients , By combining these factors, a training loss for a single training sample can be obtained. In one embodiment, an overall training loss function is derived that integrates the aforementioned retrieval difficulty, dynamic weights, the first loss, and the second loss. It can be represented as:

[0107]

[0108] in, Using the index of the training sample, the summation operation iterates through all training samples in the current training batch; For the first The difficulty of retrieving each training sample; For the first The weighted sum of the first and second losses for each training sample.

[0109] It can be seen that the training loss is related to the retrieval quality of the training samples, specifically in two aspects. One aspect is the retrieval difficulty of the training samples, determined based on the retrieval quality. Scaling of the training loss, i.e. The lower, The larger the value, the amplified the overall training loss of that training sample, resulting in a stronger gradient for model parameter updates. On another level: reward signal That is, retrieval quality In group sampling and advantage comparison, the model is directly guided to update towards generating higher-quality retrieval.

[0110] Next, with the goal of minimizing the training loss, in each training step, the training loss for the current batch is calculated according to the formula above, and the effect of the loss on the model parameters is calculated through backpropagation. The gradient is calculated, and the parameters of the query expansion model are updated using an optimizer (e.g., the AdamW optimizer). Steps S201 and S203 are repeated, iterating through each training sample in the training dataset, until the preset number of training steps is reached. Alternatively, it may satisfy other convergence conditions. Once trained, the query expansion model possesses strong expansion capabilities. During the training process described above, for difficult query samples with low retrieval quality, the query expansion model can generate more effective expanded queries to improve retrieval recall; for simple query samples with high retrieval quality, the query expansion model can receive relatively less attention during training to save computational resources.

[0111] The above describes in detail a training method for a query expansion model based on one or more embodiments. It can be understood that the training method consisting of steps S201 and S203 can adaptively adjust the contribution of the training sample to the model training based on the retrieval quality of the first query text used as the training sample during the training process. The lower the retrieval quality, i.e., the greater the retrieval difficulty, the greater the contribution of the training sample to the loss function, and the more significant the model parameter update; conversely, the higher the retrieval quality, i.e., the lower the retrieval difficulty, the smaller the contribution of the training sample to the loss function. Therefore, the training of the query expansion model can focus on difficult samples that contribute significantly to improving model performance, thereby enhancing the model's ability to expand various queries, especially complex, fuzzy, or long-tail queries. Simultaneously, the supervised learning loss (i.e., the first loss) provides a stable benchmark for the model's language generation ability, while the reinforcement learning loss (i.e., the second loss) uses retrieval quality as the optimization objective. The two losses achieve a smooth transition from language ability learning to retrieval quality optimization through dynamic weights. Furthermore, the first tag text in the training data... It is automatically generated based on search feedback, and its quality is based on objective search quality, which reduces the reliance on manual annotation to a certain extent.

[0112] In this specification, the terms "first" in the first label text, first query text, etc., as well as the corresponding "second" and "third" (if they exist) in the text, are merely for the convenience of distinction and description, and do not have any limiting meaning.

[0113] The foregoing description describes specific embodiments of this specification; other embodiments are within the scope of the appended claims. In some cases, the actions or steps recited in the claims may be performed in a different order than those shown in the embodiments, and the desired result may still be achieved. Furthermore, the processes depicted in the drawings do not necessarily need to follow the specific or sequential order shown to achieve the desired result. In some embodiments, multitasking and parallel processing are possible or may be advantageous.

[0114] Figure 4This is a schematic diagram of a training device for a query expansion model according to an embodiment of this specification. The device 400 is deployed in a computing device, which can be implemented using any device, equipment, platform, device cluster, etc., with computing and processing capabilities. The device 400 includes:

[0115] The acquisition module 401 is configured to acquire a first query text and a set of tag documents corresponding to the first query text in a knowledge base, wherein each document in the set of tag documents contains an answer corresponding to the first query text.

[0116] Training module 402 is configured to determine a training loss and train the query expansion model with the goal of minimizing the training loss; the training loss is negatively correlated with the retrieval quality corresponding to the first query text; the retrieval quality is used to measure the recall quality of the query text in the knowledge base for the tagged document set; the trained query expansion model is used to expand the query text to perform a retrieval in the knowledge base.

[0117] According to another embodiment, this specification also provides a computer program product, including a computer program / instructions that, when executed by a processor, implement the foregoing combinations. Figure 2 The steps of the method are described.

[0118] According to yet another embodiment, this specification also provides a computing device, including a memory and a processor, characterized in that the memory stores executable code, and when the processor executes the executable code, it implements the foregoing combination. Figure 2 The steps of the method are described.

[0119] Those skilled in the art will recognize that, in one or more of the examples above, the functions described in the embodiments of the present invention can be implemented using hardware, software, firmware, or any combination thereof. When implemented in software, these functions can be stored in a computer-readable medium or transmitted as one or more instructions or code on a computer-readable medium.

[0120] The specific embodiments described above further illustrate the purpose, technical solution, and beneficial effects of the present invention. It should be understood that the above descriptions are merely specific embodiments of the present invention and are not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, or improvements made based on the technical solutions of the present invention should be included within the scope of protection of the present invention.

Claims

1. A method for training a query expansion model, comprising: Obtain the first query text and the tag document set corresponding to the first query text in the knowledge base, wherein each document in the tag document set contains the answer corresponding to the first query text; A training loss is determined, and the query expansion model is trained with the goal of minimizing the training loss; the training loss is negatively correlated with the retrieval quality corresponding to the first query text. The retrieval quality is used to measure the recall quality of the query text in the knowledge base for the set of tagged documents; the trained query expansion model is used to expand the query text to perform retrieval in the knowledge base.

2. The method of claim 1, wherein, The determination of training loss includes: The first query text is expanded using the query expansion model to obtain the second query text; A first loss is determined based on the first tag text and the second query text; the first tag text is a query text generated based on the first query text and the tag document set. The retrieval quality corresponding to the second query text is used as the reward score to determine the second loss; The training loss is determined by a weighted sum of the first loss and the second loss; wherein the weight of the first loss is negatively correlated with the number of training steps, and the weight of the second loss is positively correlated with the number of training steps.

3. The method of claim 1, wherein, The tagged document set is obtained in the following way: For the first query text, a fusion recall based on the first strategy and the second strategy is performed in the knowledge base to obtain a candidate document set; the first strategy is a keyword-based recall strategy, and the second strategy is a semantic-based recall strategy. Documents that meet the first condition are selected from the candidate document set and included in the tagged document set; the first condition includes that the document contains an answer corresponding to the first query text.

4. The method of claim 1, wherein, The retrieval quality is determined based on a weighted sum of a hit rate metric and a ranking quality metric; the hit rate metric indicates whether the retrieval results obtained by searching the knowledge base using the query text contain at least one document from the tagged document set; the ranking quality metric indicates the degree of similarity in document ranking between the retrieval results and the tagged document set.

5. The method of claim 2, wherein, The first tag text is generated in the following way: Based on the first query text and the set of tagged documents, multiple first candidate expanded texts are generated using a large language model; The first tag text is determined based on the ranking of the retrieval quality of the plurality of first candidate expanded texts.

6. The method of claim 2, wherein, The determination of the first loss includes: The first query text is expanded using the query expansion model to obtain the first predicted output; The first loss is determined based on the cross-entropy between the first predicted output and the first labeled text.

7. The method of claim 2, wherein, The determination of the second loss includes: The query expansion model is used to expand the first query text multiple times to obtain multiple second query texts. For any second query text, its corresponding retrieval quality is used as a reward score; The second loss is determined based on the multiple reward scores corresponding to the multiple second query texts.

8. The method of claim 7, wherein, The second loss is determined based on the multiple reward scores corresponding to the multiple second query texts, including: Determine the mean and standard deviation of the plurality of reward scores; For any second query text, its reward score is normalized based on the mean and the standard deviation to obtain the advantage value; The second loss is determined based on the multiple advantage values ​​corresponding to the multiple second query texts.

9. The method of claim 2, wherein, The step of using the retrieval quality corresponding to the second query text as a reward score to determine the second loss includes: The retrieval quality corresponding to the second query text is used as the reward score, and a second loss is determined based on a pruning function; the pruning function is used to control the variation range of the second loss.

10. The method of claim 2, wherein, The second loss also includes a KL divergence penalty term, used to constrain the probability distribution difference between the updated policy and the reference policy after the query expansion model is updated.

11. The method of claim 2, wherein, The weight of the first loss is calculated in the following way: According to the cosine annealing algorithm, the weights of the first loss are updated based on the number of training steps and the initial weights; the initial weights are the preset weights when the number of training steps is 0.

12. The method of claim 2, wherein, The sum of the weights of the first loss and the second loss is 1.

13. The method of claim 2, wherein, The total number of training steps for the query expansion model is N; the method further includes: When the number of training steps is N / 2, the weights of the first loss and the second loss are equal.

14. A training apparatus for a query expansion model, comprising: The acquisition module is configured to acquire a first query text and a set of tag documents corresponding to the first query text in the knowledge base, wherein each document in the set of tag documents contains an answer corresponding to the first query text. The training module is configured to determine a training loss and train the query expansion model with the objective of minimizing the training loss; the training loss is negatively correlated with the retrieval quality corresponding to the first query text. The retrieval quality is used to measure the recall quality of the query text in the knowledge base for the set of tagged documents; the trained query expansion model is used to expand the query text to perform retrieval in the knowledge base.

15. A computer program product comprising a computer program / instructions that, when executed by a processor, implement the steps of the method according to any one of claims 1-13.

16. A computing device, comprising a memory and a processor, characterized in that, The memory stores executable code, and when the processor executes the executable code, it implements the method of any one of claims 1-13.

Images