Model training method and device, query method and device, server and storage medium

Abstract

The invention relates to a model training method and device, a query method and device, a server and a storage medium. The large model training method comprises the following steps: based on a sample query problem, retrieving an auxiliary document related to the sample query problem from a document library; inputting the sample query question, the auxiliary document and the sample candidate document into an initial generative model to obtain sample reply information of the sample query question; the sample reply information is output by the initial generative model based on user intention information and sample candidate documents, and the user intention information corresponding to the sample query problem is determined by the initial generative model based on the sample query problem and the auxiliary documents; obtaining a target reward according to the sample reply information; and training the initial generative model based on the target reward and reinforcement learning algorithm to obtain a target generative model. By adopting the model training method, the accuracy of the recommended content output by the generative model can be improved.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, and in particular to a model training method, query method, apparatus, server, and storage medium. Background Technology

[0002] When Retrieval-Augmented Generation (RAG) is applied on User-Generated Content (UGC) platforms, it generates answers and provides intelligent recommendations by searching for content such as travel guides and food notes, greatly improving the user experience. The core of RAG lies in its relevance assessment module, that is, how to accurately determine the semantic matching degree between the query and the document.

[0003] Traditional techniques primarily rely on supervised learning to match queries with documents and train generative models. However, traditional techniques may suffer from inaccurate recommendations output by generative models. Summary of the Invention

[0004] Therefore, it is necessary to provide a model training method, query method, device, server, and storage medium that can improve the accuracy of the recommended content output by the generative model, addressing the aforementioned technical problems.

[0005] Firstly, this application provides a model training method, the method comprising:

[0006] Based on the sample query question, retrieve auxiliary documents related to the sample query question from the document library;

[0007] The sample query question, the auxiliary document, and the sample candidate document are input into the initial generative model to obtain sample response information for the sample query question. The sample response information is the user intent information corresponding to the sample query question determined by the initial generative model based on the sample query question and the auxiliary document, and output based on the user intent information and the sample candidate document.

[0008] The target reward is obtained based on the sample response information;

[0009] The initial generative model is trained based on the target reward and reinforcement learning algorithm to obtain the target generative model.

[0010] In one embodiment, the sample response information includes sample document fragments from the sample candidate documents and / or a sample relevance metric between the sample candidate documents and the user intent information; or...

[0011] The sample response information includes preset document fragments and / or preset relevance metrics.

[0012] In one embodiment, the sample response information is output by the initial generative model based on any of the following:

[0013] If it is determined based on the user intent information that the sample document fragment exists in the sample candidate document, then based on the sample candidate document and the user intent information, the sample document fragment and / or the sample relevance metric value are output;

[0014] If the initial generative model determines, based on the user intent information, that the sample document fragment does not exist in the sample candidate documents, then it outputs the preset document fragment and / or the preset relevance metric.

[0015] In one embodiment, obtaining the target reward based on the sample response information includes:

[0016] A format reward is obtained based on the format information of the document fragment and / or the format information of the relevance metric; the document fragment includes the sample document fragment or the preset document fragment; the relevance metric includes the sample relevance metric or the preset relevance metric.

[0017] A score reward is obtained based on the relevance metric and the standard relevance metric of the sample candidate documents;

[0018] The target reward is obtained based on the format reward and / or the score reward.

[0019] In one embodiment, obtaining the target reward based on the format reward and / or the score reward includes:

[0020] The product of the format reward and the score reward is determined as the target reward.

[0021] In one embodiment, training the initial generative model based on the target reward and reinforcement learning algorithm to obtain the target generative model includes:

[0022] The loss function of the reinforcement learning algorithm is constructed based on the target reward;

[0023] The target generative model is obtained by optimizing the parameters of the initial generative model by minimizing the value of the loss function.

[0024] In one embodiment, the method further includes:

[0025] The document corresponding to the query question is structured and labeled using a large language model to obtain structured training samples; the structured training samples include target document fragments and / or target relevance measures between the query question and the document;

[0026] The query question and the document are input into a preset generative model to obtain response information for the query question; the response information includes document fragments and / or a predicted value of the relevance measure between the query question and the document;

[0027] The preset generative model is trained based on the response information and the structured training samples to obtain the initial generative model.

[0028] Secondly, this application provides a query method, the method comprising:

[0029] Based on the query question, candidate documents are retrieved from the document library;

[0030] The query question and the candidate documents are input into the target generative model to obtain the response information for the query question;

[0031] The target generative model is obtained by training the method described in the first aspect.

[0032] In one embodiment, the response information includes document fragments from the candidate documents and / or a relevance metric between the candidate documents and the query question; or,

[0033] The response information includes preset document fragments and / or preset relevance metrics.

[0034] In one embodiment, the response information is output by the target generative model based on any of the following:

[0035] If it is determined that the document fragment exists in the candidate document based on the query question, then based on the candidate document and the query question, the document fragment and / or the relevance measure value are output;

[0036] If it is determined based on the query question that the document fragment does not exist in the candidate documents, then the preset document fragment and / or the preset relevance metric value are output.

[0037] Thirdly, this application also provides a model training apparatus, the apparatus comprising:

[0038] The retrieval module is used to retrieve auxiliary documents related to the sample query question from the document library;

[0039] The first acquisition module is used to input the sample query question, the auxiliary document, and the sample candidate document into the initial generative model to obtain sample response information for the sample query question; the sample response information is the user intent information corresponding to the sample query question determined by the initial generative model based on the sample query question and the auxiliary document, and output based on the user intent information and the sample candidate document.

[0040] The second acquisition module is used to obtain the target reward based on the sample response information;

[0041] The training module is used to train the initial generative model based on the target reward and reinforcement learning algorithm to obtain the target generative model.

[0042] Fourthly, this application also provides a query device, the device comprising:

[0043] The retrieval module is used to retrieve candidate documents from the document library based on the query question.

[0044] The query module is used to input the query question and the candidate documents into the target generative model to obtain the response information for the query question;

[0045] The target generative model is obtained by training the method described in the first aspect.

[0046] Fifthly, this application also provides a server, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps of the methods described in the first and second aspects.

[0047] Sixthly, this application also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the methods described in the first and second aspects.

[0048] In a seventh aspect, this application also provides a computer program product, including a computer program that, when executed by a processor, implements the steps of the methods described in the first and second aspects.

[0049] The aforementioned model training method, query method, device, server, and storage medium include a model training method that involves retrieving auxiliary documents related to a sample query question from a document library, inputting the sample query question, auxiliary documents, and candidate sample documents into an initial generative model to obtain sample response information for the sample query question. The sample response information is generated by the initial generative model based on the sample query question and auxiliary documents, determining the user intent information corresponding to the sample query question, and outputting it based on the user intent information and candidate sample documents. The initial generative model can complete the implicit intent of the sample query question based on the sample query question and auxiliary documents, ensuring the accuracy of the obtained user intent information. Furthermore, it can accurately obtain the sample response information for the sample query question based on the user intent information and candidate sample documents. Based on the sample response information output by the initial generative model, a target reward is obtained. The initial generative model is then accurately trained using the target reward and a reinforcement learning algorithm to obtain a target generative model. This target generative model enables the accurate acquisition of sample response information for the query question, improving the accuracy of the recommended content output by the generative model. Attached Figure Description

[0050] To more clearly illustrate the technical solutions in the embodiments of this application or related technologies, the drawings used in the description of the embodiments of this application or related technologies will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0051] Figure 1 This is a diagram illustrating the application environment of the model training method and the query method in one embodiment.

[0052] Figure 2 This is a flowchart illustrating a model training method in one embodiment;

[0053] Figure 3 This is a flowchart illustrating the model training method in another embodiment;

[0054] Figure 4 This is a flowchart illustrating the model training method in another embodiment;

[0055] Figure 5 This is a flowchart illustrating the model training method in another embodiment;

[0056] Figure 6 This is a schematic diagram of the training process of the initial generative model in one embodiment;

[0057] Figure 7This is a flowchart illustrating the query method in one embodiment;

[0058] Figure 8 This is a structural block diagram of a model training device in one embodiment;

[0059] Figure 9 This is a structural block diagram of the model training device in another embodiment;

[0060] Figure 10 This is a structural block diagram of the model training device in another embodiment;

[0061] Figure 11 This is a structural block diagram of the model training device in another embodiment;

[0062] Figure 12 This is a structural block diagram of the query device in one embodiment;

[0063] Figure 13 This is an internal structural diagram of a computer device in one embodiment. Detailed Implementation

[0064] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0065] The model training method and query method provided in this application embodiment can be applied to, for example... Figure 1 In the application environment shown, terminal 102 communicates with server 104 via a network. A data storage system can store the data that server 104 needs to process. The data storage system can be integrated onto server 104 or located on the cloud or other network servers. Terminal 102 can send a query to server 104, requesting the server to respond and process the query. Terminal 102 can be, but is not limited to, various personal computers, laptops, smartphones, tablets, IoT devices, and portable wearable devices. IoT devices can include smart TVs, smart in-vehicle devices, projection devices, etc. Portable wearable devices can include smartwatches, smart bracelets, etc. Server 104 can be a standalone physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing cloud computing services.

[0066] First, the technical concept of this application's embodiments is summarized. In the training process of the generative model, this application's embodiments introduce a two-round inference architecture, decomposing the traditional relevance assessment task into two stages: "inferring user intent" and "document relevance judgment." This two-round inference architecture effectively alleviates the deficiency in user-generated content platforms where accurate recommended content cannot be output when there is ambiguity in the query. It can infer potential intent and complete the implicit intent of the query by using auxiliary documents (document content strongly related to the query retrieved using the user's input) even when the user has not explicitly expressed a complete query requirement. Furthermore, during training, the generative model is required to extract original text fragments from sample candidate documents as scoring criteria. When no original text fragment strongly related to the user's intent information exists in the sample candidate documents, a "None output mechanism" is introduced. This allows the generative model to explicitly indicate "no relevant content" rather than forcibly scoring, significantly improving the model's ability to identify unstructured, superficially relevant but substantially unrelated content, thereby reducing the misjudgment rate caused by noise information (such as emoticons and tags).

[0067] In one exemplary embodiment, such as Figure 2 As shown, a model training method is provided, which can be applied to... Figure 1 Taking the server in the example, the following steps are included:

[0068] S201, Based on the sample query question, retrieve auxiliary documents related to the sample query question from the document library.

[0069] In this embodiment, the document library can be any type of document library generated based on content published by an application (APP). For example, the document library can be a user-generated content document library. Exemplarily, published content can be travel guides, food notes, user experiences, product videos, product links, notes, videos, etc. Exemplarily, the aforementioned application can be a social e-commerce application, a social application, an educational application, etc.

[0070] For example, the sample query question can be of various types, such as information query questions and decision-making suggestion questions. Information query questions can be, for example, questions like "What does an infinite series mean?" or "What are some highly rated food recommendations in the attachments?" which are questions that obtain various kinds of knowledge, data, or real-time information. Decision-making suggestion questions can be, for example, questions like "I'm going to XX for tourism, what are some must-see attractions?" or "Can you recommend some indoor attractions I can visit on the weekend?" which are questions that obtain suggestions, plans, or solutions.

[0071] In this embodiment, as an example, the sample query question can be determined based on a large number of query questions entered by the user during historical terminal usage. This query question can be entered by the user through any of the following methods: voice, text, image, or gesture. Optionally, the query question can be a natural language query question, and the language of the query question can be any of the following languages: Chinese, English, German, Japanese, etc. It is understood that, to balance the training samples, different types of historical natural language query questions can be pre-selected from the user's historical natural language query questions to construct a sample query question set. For example, 40% could be information query questions, 40% could be decision suggestion questions, and 20% could be other types of questions to construct the sample query question set. Alternatively, based on users' historical usage habits, a sample query question set can be constructed by selecting different types of historical natural language query questions from the user's input. For example, if it is determined that users frequently use information query questions in their historical usage, then 60% of information query questions, 30% of decision-making advice questions, and 10% of other types of questions can be selected to construct the sample query question set; similarly, if it is determined that users frequently use decision-making advice questions in their historical usage, then 60% of decision-making advice questions, 20% of information query questions, and 20% of other types of questions can be selected to construct the sample query question set.

[0072] In this embodiment, the auxiliary document is a document used to assist the initial generative model in completing the implicit intent of the query based on the sample query question. As one feasible implementation, after obtaining the sample query question, firstly, the sample query question can be segmented to extract keywords. Then, based on the keywords, a filter is used to retrieve initial documents related to the sample query question from the document library (for example, documents in the document library whose keywords overlap with the keywords of the sample query question can be identified as initial documents). Then, the initial documents are sorted, and the top-ranked initial documents are identified as auxiliary documents. In this embodiment, the auxiliary document can include at least one document. As an example, the top 5 ranked initial documents can be identified as auxiliary documents.

[0073] S202, Input the sample query question, auxiliary documents, and sample candidate documents into the initial generative model to obtain sample response information for the sample query question; The sample response information is the user intent information corresponding to the sample query question determined by the initial generative model based on the sample query question and auxiliary documents, and output based on the user intent information and sample candidate documents.

[0074] In this embodiment, the process of constructing the dataset for training the initial generative model is first explained: Different types of natural language query questions can be selected from a large number of natural language query questions input by users in the past as sample query questions in the dataset, and sample candidate documents corresponding to each sample query question are determined. If the sample candidate document is highly relevant to the sample query question, the standard relevance metric between the sample candidate document and the sample query question is marked as 2 points; if the sample candidate document is moderately relevant to the sample query question, the standard relevance metric between the sample candidate document and the sample query question is marked as 1 point; if the sample candidate document is unrelated to the sample query question, the standard relevance metric between the sample candidate document and the sample query question is marked as 0 points, thus establishing a one-to-one correspondence between the sample query questions and the corresponding sample candidate documents. Furthermore, to balance the data types in the dataset, taking a dataset containing 3000 data sets as an example, the first data set could include 1000 pairs of sample query questions and corresponding candidate documents with a standard relevance score of 2; the second data set could include 1000 pairs of sample query questions and corresponding candidate documents with a standard relevance score of 1; and the third data set could include 1000 pairs of sample query questions and corresponding candidate documents with a standard relevance score of 0. It should be noted that the sample query questions in the first, second, and third data sets are all different, and the individual sample query questions in the first, second, and third data sets are also different. In other words, the dataset can include 3000 different sample query questions. When training the initial generative model, after selecting a sample query question from the dataset, the corresponding candidate documents for the sample query question are determined, as are the standard relevance measures of the candidate documents. For example, if a sample query question A is selected from the dataset, the candidate documents corresponding to sample query question A can be determined as the sample candidate documents, and the relevance measures labeled on the candidate documents corresponding to sample query question A can be determined as the standard relevance measures corresponding to sample query question A.

[0075] It is understandable that the sample candidate documents are a part of the dataset used to evaluate the relevance of the initial generative model. The aforementioned auxiliary documents can be understood as additional supplements to the dataset. The auxiliary documents are used to help the initial generative model determine more accurate user intent information based on the sample query question.

[0076] As an example, when constructing the first set of data in the dataset, documents that are helpful to the sample query question can be selected from the document library, and one of the higher-ranked documents can be selected as the sample candidate document corresponding to the sample query question. Similarly, when constructing the second set of data in the dataset, documents that are helpful to the sample query question can be selected from the document library, and one of the lower-ranked documents can be selected as the sample candidate document corresponding to the sample query question. When constructing the third set of data in the dataset, documents that are helpful to the sample query question can be selected from the document library, and then some additional documents can be selected from the unselected documents. One of these additional documents can then be selected as the sample candidate document corresponding to the sample query question.

[0077] In this embodiment, the initial generative model can extract key information from the sample query question to clarify the user's focus and needs. Then, the initial generative model can supplement the query scenario and domain knowledge corresponding to the sample query question with auxiliary documents, and combine the original text statements in the auxiliary documents to reason about the sample query question and predict user intent information. For example, user intent information can be used to characterize the expected query content of the sample query question. As an example, taking the sample query question "What does an infinite series mean?" as an example, the initial generative model analyzes the sample query question to clarify that the user wants to know information about "infinite series". Then, the initial generative model can combine the relevant information about "infinite series" in the auxiliary documents (such as "basic definition of infinite series → properties → convergence determination → applications") to infer the user's intent. Since the sample query question is "What does an infinite series mean?", the user may be encountering infinite series for the first time and wants to know the basic definition of "infinite series" and gain a preliminary understanding of "infinite series", rather than wanting to know the properties and solution methods of "infinite series". Through this reasoning process, the initial generative model can determine the user intent information corresponding to the sample query question as "to obtain the basic definition of infinite series".

[0078] For example, in this embodiment, the object processed by the initial generative model in the intent reasoning stage can be: a sample query question + auxiliary documents, and the generated content can be:

[0079] <think>Content analysis of supporting documents;

[0080] <intent>: Predicted user intent information.

[0081] For example, in this embodiment, the object processed by the initial generative model in the relevance judgment stage can be: user intent information + sample candidate documents, and the generated content can be:

[0082] <extract>Extract the most relevant original sentences from the sample candidate documents.

[0083] <score>: Correlation measure predicted value (0 = no correlation, 1 = partially correlated, 2 = strongly correlated)

[0084] It should be noted that in this embodiment, the inference mechanism of the initial generative model requires it to select only "original sentences" from the sample candidate documents as the basis for judgment, preventing the generative model from "imagine" anything out of thin air; in addition, if the content of the sample candidate document does not match any user intent information, it should return... <extract> :none< / extract> To prevent noise-induced noise.

[0085] In this embodiment, before inputting the sample query question, auxiliary documents, and sample candidate documents into the initial generative model, a cold start approach can be used to initialize the preset generative model, resulting in the aforementioned initial generative model. This initial generative model possesses basic reasoning and format compliance capabilities, providing a foundation for subsequent reinforcement learning training and avoiding problems such as policy instability and convergence difficulties in the early stages of training. Figure 3 As shown, the initialization process of a pre-defined generative model may include:

[0086] S301, The document corresponding to the query question is structurally labeled using a large language model to obtain a structured training sample; the structured training sample includes the target document fragment and / or the target relevance measure between the query question and the document.

[0087] In this embodiment, when initializing the initial generative model, some documents can be selected from the document library, and these selected documents can be annotated in a structured way using a large language model. The documents are formatted into "reasoning content", "relevance score", etc., thereby constructing a structured training sample that includes target document fragments and / or target relevance measures between query questions and documents.

[0088] S302, Input the query question and document into the preset generative model to obtain the response information for the query question; the response information includes document fragments and / or, the predicted value of the relevance measure between the query question and the document.

[0089] S303, Train the preset generative model based on the response information and structured training samples to obtain the initial generative model.

[0090] Furthermore, after constructing the structured training samples, the query question and the corresponding document can be input into the preset generative model to obtain the response information of the query question. The response information may include document fragments and / or the predicted value of the relevance measure between the query question and the document. Then, a supervised fine-tuning strategy is adopted to train the preset generative model based on the response information output by the preset generative model and the constructed structured training samples, so that it learns the basic format of the inference chain, thereby enhancing the stability of the initial generative model in subsequent training.

[0091] S203, Based on the sample response information, the target reward is obtained.

[0092] As an example, a format reward metric can be used to judge the output format of the sample response information output by the initial generative model, that is, to judge whether the sample response information output by the initial generative model conforms to the predetermined format, thereby obtaining the format reward, and using the format reward as the target reward.

[0093] As another example, a score reward metric can be used to judge the accuracy of the sample response information output by the initial generative model, that is, to judge whether the response information output by the initial generative model is relevant to the sample query question, thereby obtaining a score reward, which can be used as the target reward.

[0094] Based on this, as another example, the above-mentioned format rewards and score rewards can be weighted and merged, and the merged reward value can be used as the target reward.

[0095] S204. The initial generative model is trained based on the target reward and reinforcement learning algorithm to obtain the target generative model.

[0096] In this embodiment, the initial generative model is trained primarily based on target rewards and reinforcement learning algorithms to improve its decision quality and the clarity of its discrimination boundaries. For example, the initial generative model can be trained using a combination of generative modeling and reinforcement learning policy optimization (GRPO) algorithms. The two rounds of reasoning—the intent reasoning stage and the relevance judgment stage—are considered as a complete "policy trajectory." The average reward of multiple sampled outputs is used as a baseline, and the sample response information is fed back to update the current policy (i.e., the initial generative model), reducing output errors caused by user intent ambiguity and content noise.

[0097] In the above model training method, based on the sample query question, auxiliary documents related to the sample query question are retrieved from the document library. The sample query question, auxiliary documents, and sample candidate documents are input into the initial generative model to obtain sample response information for the sample query question. The sample response information is the user intent information corresponding to the sample query question determined by the initial generative model based on the sample query question and auxiliary documents, and output based on the user intent information and sample candidate documents. The initial generative model can complete the implicit intent of the sample query question based on the sample query question and auxiliary documents, thus obtaining the user intent information corresponding to the sample query question, ensuring the accuracy of the obtained user intent information. Therefore, based on the user intent information and sample candidate documents, the sample response information for the sample query question can be accurately obtained. The target reward can then be obtained based on the sample response information output by the initial generative model. Based on the target reward and reinforcement learning algorithm, the initial generative model is accurately trained to obtain the target generative model. This allows the target generative model to accurately obtain the sample response information for the query question, improving the accuracy of the recommended content output by the generative model.

[0098] This embodiment will explain the process of the initial generative model outputting sample response information. In an exemplary embodiment, the sample response information includes sample document fragments in the sample candidate documents and / or sample relevance measures between the sample candidate documents and user intent information; or, the sample response information includes preset document fragments and / or preset relevance measures. As an example, the sample response information can be output by the initial generative model based on any of the following:

[0099] If it is determined that there is a sample document fragment in the sample candidate document based on the user intent information, then the sample document fragment and / or sample relevance measure are output based on the sample candidate document and the user intent information.

[0100] If the initial generative model determines, based on user intent information, that there is no sample document fragment in the candidate documents, it outputs a preset document fragment and / or a preset relevance metric.

[0101] In this embodiment, if the initial generative model determines that a sample document fragment exists in the candidate documents based on user intent information, it can predict the relevance metric between the sample query question and the candidate documents, determine the sample relevance metric, and output the sample document fragment and / or the sample relevance metric in the candidate documents. If the initial generative model determines that no sample document fragment related to the user intent information exists in the candidate documents based on user intent information, it can output a preset document fragment and / or a preset relevance metric. For example, the preset document fragment can be... <extract> :none< / extract> The default correlation metric value can be 0.

[0102] In this embodiment, the sample response information output by the initial generative model includes sample document fragments in the sample candidate documents and / or sample relevance measurement values ​​between the sample candidate documents and user intent information; or, the sample response information includes preset document fragments and / or preset relevance measurement values. Through the two-stage reasoning process of the initial generative model, the implicit intent of the sample query question is first completed using auxiliary documents, and then, based on the user intent information and sample candidate documents with high accuracy, sample document fragments of the sample query question and / or sample relevance measurement values ​​between the sample query question and the sample candidate documents are output, effectively reducing errors caused by ambiguous user intent and content noise. If the content of the sample candidate documents does not match any user intent information, preset document fragments and / or preset relevance measurement values ​​are output, preventing noise-induced errors.

[0103] Because user behavior feedback for user-generated content documents is sparse, discontinuous, and mostly at the document level, it cannot accurately correspond to content fragments. Therefore, existing weakly supervised training methods struggle to obtain high-quality labels to guide the initial generative model's learning and training, limiting its generalization ability in complex and noisy environments. Thus, in this embodiment, reinforcement learning can be used to train the initial generative model, aiming to improve the accuracy of query-document matching in the retrieval enhancement generation system for user-generated content scenarios. A "step-by-step inference + reinforcement learning" design approach is adopted for training the initial generative model. The detailed process of obtaining the target reward for reinforcement learning will be explained below. In one embodiment, such as... Figure 4 As shown, the above S203 includes:

[0104] S401, Receive a format reward based on the format information of the document fragment and / or the format information of the relevance metric; the document fragment includes a sample document fragment or a preset document fragment; the relevance metric includes a sample relevance metric or a preset relevance metric.

[0105] S402, a score reward is obtained based on the relevance metric and the standard relevance metric of the sample candidate documents.

[0106] In this embodiment, the sample response information output by the initial generative model is scored and used as a reward source for reinforcement learning training. As an example, the output of the initial generative model can be scored from two dimensions: "format compliance" and "semantic judgment accuracy," to determine the value of the reward function of the initial generative model.

[0107] For format rewards: A format reward can be obtained based on the format information of the document fragments output by the initial generative model and / or the format information of the sample relevance metrics; where document fragments include sample document fragments or preset document fragments; and relevance metrics include sample relevance metrics or preset relevance metrics. If the output of the initial generative model meets the preset structure requirements, the format reward is... It can be 1; otherwise, if there are any deviations in the format of the initial generative model output (e.g., missing labels, incorrect label nesting, incorrect label order, etc.), then... It is 0.

[0108] For example, the preset structured requirements can be:

[0109] <think>

[0110] XXXXXXXXXXXXXXXXXXX

[0111] < / think>

[0112] <extract>

[0113] XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX

[0114] < / extract>

[0115] <score>

[0116] XXX

[0117] < / score>

[0118] In other words, the pre-defined structured requirements stipulate that the output of the initial generative model includes a clear thought process, document fragments corresponding to the sample query question, and a measure of sample relevance between the sample query question and the sample candidate documents. For example, the thought process can be expressed as follows: <think> ...< / think> Labels are used to define the scope of the application. <think>Tags can be used to enclose internal thinking / reasoning processes; document fragments can be used as... <extract> ...< / extract> Labels are used to define the scope of the application. <extract>The document fragments output by the label-enclosing model, and the sample relevance measure between the sample query question and the sample candidate documents can be used as follows: <score> ...< / score> .tags are used to define the scope of the application. <score>The labels enclose the output sample query question and the sample candidate documents, measuring the sample relevance. As an example, continuing with the sample query question "What does an infinite series mean?", the initial generative model output that meets the preset structure requirements can be:

[0119] <think>

[0120] First, I need to assess the relevance between the user's query and the sample candidate documents, and assign a score based on the scoring criteria. The query is "What does an infinite series mean?", and the user's need is to understand the definition or basic concept of an infinite series.

[0121] Next, let's examine the content of the sample candidate documents. Most of the sample candidate documents contain user guides and source courses for the calculus glossary. In the "Infinite Series" section, it mentions "convergence tests for series" and "power series and Taylor series." This information may cover some criteria for infinite series and the content of power series, but it does not provide a definition or basic concepts of infinite series.

[0122] According to the scoring criteria, a score of 2 requires the document to directly and completely address the core needs of the query. The query requires a definition or basic concept of infinite series, but the sample candidate documents do not provide this information. A score of 1 indicates that the document partially answers the question or requires specific conditions. In this case, the document does not provide a definition or basic concept of infinite series, but rather touches on some discrimination methods and power series. Therefore, the document neither directly answers the search term's question nor provides a complete answer.

[0123] Therefore, the document is irrelevant to the main needs of the search terms and should be scored as 0 points.

[0124] < / think>

[0125] <extract> none< / extract>

[0126] <answer> 0< / answer>

[0127] If the initial generative model output meets the preset structuring requirements, then the output should contain... <think>, <extract> 、 <score>Tags such as [list of tags]. It's understandable that the format reward for the initial generative model can constrain the standardization of its output, preventing it from "playing freely" and outputting responses that don't meet the standards.

[0128] For the score reward: The relevance metric output by the initial generative model can be compared with the standard relevance metric of the sample candidate documents to determine the score reward. If the relevance metric matches the standard relevance metric, the score reward can be 1; if the relevance metric does not match the standard relevance metric (e.g., the relevance metric is greater than or less than the standard relevance metric), the score reward can be 0. As an example, the expression for the score reward can be:

[0129]

[0130] S403, obtain the target reward based on format rewards and / or score rewards.

[0131] As an example, the target reward can be defined as: In the formula, R represents the target reward. Represents the format reward, r 分数 This refers to a score-based reward; that is, the target reward can be determined by multiplying the format reward and the score reward. Alternatively, the target reward can be determined by a weighted average of the format reward and the score reward.

[0132] It is understandable that the target reward for the initial generative model can cause the initial generative model to optimize at least one of the following simultaneously during training:

[0133] The normalization of the structured output of generative models (contributes to system stability);

[0134] Improving the semantic accuracy of discrimination (enhancing the quality of business relevance assessment);

[0135] Abnormal sample handling capability (avoiding interference through a mechanism that returns "none").

[0136] In this embodiment, a format reward can be obtained based on the format information of the document fragments and / or the format information of the relevance measure in the sample response information. In addition, a score reward can be obtained based on the relevance measure in the sample response information and the standard relevance measure of the sample candidate documents. The document fragments in the sample response information may include sample document fragments or preset document fragments, and the relevance measure may include sample relevance measure or preset relevance measure. This ensures that the format reward and score reward can be accurately obtained for different outputs of the initial generative model, thereby ensuring the accuracy of obtaining the target reward based on the format reward and / or score reward.

[0137] This embodiment will explain in detail the process of training the initial generative model based on target reward and reinforcement learning algorithms. In one embodiment, such as Figure 5 As shown, the above S204 includes:

[0138] S501, the loss function of the reinforcement learning algorithm is constructed based on the target reward.

[0139] S502 optimizes the parameters of the initial generative model by minimizing the loss function to obtain the target generative model.

[0140] Understandably, in reinforcement learning, the target reward is the core factor guiding the model optimization process and shaping the model's behavior. In this embodiment, the format reward and score reward of the initial generative model's output are combined to provide an effective reward signal for reinforcement learning training, enabling policy optimization of the initial generative model through reinforcement learning. As a specific example, in this embodiment, the two-round inference process can be regarded as a complete "policy trajectory." Multiple sets of sample query questions and sample candidate documents are used to obtain the output of the initial generative model for each set of inputs. Then, the target reward for each set of outputs is calculated, and the GRPO algorithm is used to obtain the value of the loss function of the initial generative model. The initial generative model is trained with the goal of minimizing the value of the loss function.

[0141] For example, firstly, it can be based on the formula Calculate the standardized odds value, A, of the current output of the initial generative model. i The standardized advantage value represents the current output i. The standardized advantage value measures the performance of the current output relative to the average. Standardizing the advantage improves training stability and ensures fair reward feedback to all samples. In the formula, i represents the current output, and r represents the average advantage value. i μ represents the target reward for the current output. r and σ r These are the mean and standard deviation of the reward for the group containing the current output, respectively. Then, based on A... i The expression value of the loss function corresponding to the GRPO algorithm is obtained, the value of the loss function of the initial generative model is calculated, and the value of the loss function is used to train the initial generative model.

[0142] For example, the loss function of the GRPO algorithm can be expressed as:

[0143]

[0144] In the formula, J GRPO (θ) represents the value of the loss function currently output by the initial generative model. The idea behind the GRPO algorithm is that for each sample query question q, the loss function is derived from the old policy model π. θold In the previous update, a set of outputs is extracted from the model. Then, using the formatted reward and score reward methods described above, the target reward for this set of outputs is obtained, generating G corresponding rewards (a0, a1, a2…). These rewards are then standardized by subtracting the group mean and dividing by the group standard deviation. Finally, the output strategy of the initial generative model is optimized by minimizing the value of the aforementioned loss function, where E represents the expectation and π… θ The current policy function of the initial generative model, i.e., the optimization of the initial generative model parameters, determines the probability of generating the next word at each step; this is the initial generative model to be trained, π. θold The old policy function represents the model from the previous update step. ε represents the pruning ratio, used in the policy gradient method to control the update magnitude and prevent excessively large policy updates from causing training instability. π ref It is a frozen baseline model, i.e., an untrained initial generative model, used as a reference for comparison, D KL β represents the KL divergence between the initial generative model and the untrained initial generative model, measuring the difference between them. It is used as a penalty term to prevent the new policy from deviating too far from the old policy. This is one of the improvements of the GRPO algorithm. Regularization is achieved by directly adding the KL divergence between the policy model and the reference model to the loss function, instead of adding a KL penalty term to the reward, thus simplifying the training process. β is a hyperparameter in the loss function, q represents the sample query problem, d represents the sample candidate document, and clip represents the clipping function.

[0145] In this embodiment, the training objective of the initial generative model is to minimize the value of the aforementioned loss function. That is, when the value of the loss function reaches its minimum, it indicates that the parameters of the initial generative model are relatively optimal. Using the generative model at this point, the input query question and candidate documents can be accurately analyzed, the user intent information can be accurately determined, and the document fragments corresponding to the query question and / or the relevance measure between the query question and the candidate documents can be output.

[0146] In this embodiment, the target reward is used as the reward source for reinforcement learning training of the initial generative model. The loss function of the reinforcement learning algorithm is constructed based on the target reward of the initial generative model. By minimizing the value of the loss function, the accuracy of the initial generative model evaluation is improved, and errors caused by ambiguity of user intent and noise in candidate document content are effectively reduced.

[0147] The model training method provided in this application will be explained in detail below with a complete embodiment. Figure 6 This diagram illustrates the training process of an initial generative model. Please refer to [link / reference]. Figure 6 The training process of the initial generative model can be divided into a document retrieval module, a two-round inference module, a rule-based reward module, and a reinforcement learning training module. The correspondence between the inputs and outputs of each module can be specifically described as shown in Table 1 below.

[0148] Table 1

[0149]

[0150] This application's embodiment addresses the characteristics of document content in the document library, such as high noise levels (often containing emoticons, informal expressions, tags, etc.) and sparse user feedback data on queries, making it difficult to accurately correspond to content fragments. The core technology of the provided model training method lies in utilizing auxiliary documents to complete the implicit intent of the sample query, accurately determining user intent information, and then using reinforcement learning to progressively reason about the relevance evaluation process between the input sample query and the sample candidate documents. Specifically, this application's embodiment divides the model's reasoning process into two stages: First, using auxiliary documents in the document library that are highly relevant to the sample query, even when the user has not explicitly expressed a complete need, the auxiliary documents, combined with the input sample query, infer the user's potential intent and accurately identify user intent information. Then, based on the user intent information, document fragments strongly related to the user intent information are extracted from the sample candidate documents for judgment, outputting the document fragments in the sample candidate documents, as well as the sample relevance metric between the sample query and the sample candidate documents. A "None output mechanism" is introduced: when no document fragment strongly related to the user intent information exists in the sample candidate documents, "None" is output.< / score> < / extract> The system provides pre-defined document fragments and relevance metrics, allowing the model to explicitly indicate "no relevant content" rather than forcibly assigning a score. This two-stage inference process effectively alleviates the common query ambiguity problem in document repositories. It can infer potential intent from auxiliary documents retrieved based on sample query questions, even when users haven't explicitly expressed their complete needs, outputting high-quality user intent information. This significantly improves the model's ability to identify unstructured, superficially relevant but essentially unrelated content, thereby reducing the misjudgment rate caused by noise (such as emoticons and tags). During training, the GRPO algorithm is used to improve the model's evaluation accuracy, effectively reducing errors caused by user intent ambiguity and content noise. This training method not only solves the problem of existing technologies failing to accurately identify semantic relevance in user-generated content but also significantly improves the model's inference performance.

[0151] In one exemplary embodiment, such as Figure 7 As shown, a query method is provided, which can be applied to... Figure 1 Taking the server in the example, the following steps are included:

[0152] S601, Based on the query question, retrieve candidate documents from the document library.

[0153] S602, Input the query question and candidate documents into the target generative model to obtain the response information for the query question;

[0154] The target generative model is trained using the method described in the above embodiments.

[0155] In this embodiment, candidate documents are retrieved from a document library based on the query question. For example, documents related to the query question can be retrieved from the document library, and one document can be randomly selected as a candidate document. Alternatively, multiple initial documents can be retrieved from the document library based on the query question, and the top-ranked documents (such as the first four documents) can be selected as candidate documents. Since the candidate documents are retrieved from the document library, they may be completely relevant to the query question, only moderately relevant, or completely irrelevant. Furthermore, the target generative model is capable of reasoning and analyzing the different relationships between the query question and the candidate documents. In other words, the relevance metric output by the target generative model in this embodiment can be any value among 0, 1, and 2. Specifically, if the candidate document is completely relevant to the query question, the relevance metric output by the target generative model is 2; if the candidate document is moderately relevant to the query question, the relevance metric output by the target generative model is 1; and if the candidate document is completely irrelevant to the query question, the relevance metric output by the target generative model is 0.

[0156] For example, the query question in this embodiment can be any type of query question, such as an information query question or a decision suggestion question, or it can be other types of query questions. This embodiment does not limit the type of query question. In actual use of the target generative model, the type of query question can correspond to the type of sample query question used when training the initial generative model. If a new type of query question is added during actual use, the initial generative model can be trained accordingly using the new type of query question, and the target generative model can also process this new type of query question.

[0157] For example, the query question in this application embodiment can be input by the user using any of the input methods of voice, text, image, or gesture. The language of the query question can be any of the languages ​​such as Chinese, English, German, or Japanese.

[0158] Understandably, the generative model trained using the above methods is already capable of accurately analyzing the input query question and has the ability to complete the implicit intent information of the input query question. It can accurately determine the user intent information corresponding to the input query question. Therefore, in actual use, it is no longer necessary to input auxiliary documents of the query question into the target generative model so that the target generative model can complete the implicit intent information of the query question and determine the intent information of the input query question.

[0159] For example, the response information for the query may include document fragments in candidate documents and / or relevance measures of candidate documents to the query, or the response information may include preset document fragments and / or preset relevance measures.

[0160] In this embodiment, after inputting the query question and candidate documents into the target generative model, the target generative model can first analyze the query question. If it is determined based on the query question that there are document fragments in the candidate documents that are related to the query question, the target generative model can determine the relevance metric between the query question and the candidate documents. Following the preset structured requirements described in the above embodiment, it outputs the document fragments and the relevance metric between the query question and the candidate documents. As another example, if it is determined based on the query question that there are no document fragments in the candidate documents that are related to the query question, then following the preset structured requirements described in the above embodiment, it outputs preset document fragments and / or preset relevance metrics, for example, using... <think> ... <think>Tags limit the thought process for addressing a query, through <think>Tags can be used to enclose internal thinking / reasoning processes; document fragments can be used as... <extract> ... <extract>Labels are used to define the scope of the application.

[0161] <extract>The document fragments output by the label-enclosing model, and the relevance measure between the query question and the candidate documents can be used... <score> ... <score>Labels are used to define the scope of the application. <score>The tags enclose the relevance measure between the query question and the candidate documents in the output.

[0162] The query method proposed in this embodiment uses the target generative model trained by the model training method in any of the above embodiments. It can perform query tasks in actual query scenarios, improve the accuracy of the response information of the query questions output by the generative model, and at the same time, improve the accuracy of the recommended content output by the generative model.

[0163] It should be understood that although the steps in the flowcharts of the embodiments described above are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the embodiments described above may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the steps or stages in other steps. It is understood that the steps in different embodiments can be freely combined as needed, and all non-contradictory solutions formed by such combinations are within the scope of protection of this application.

[0164] Based on the same inventive concept, this application also provides a model training apparatus for implementing the model training method described above. The solution provided by this apparatus is similar to the implementation scheme described in the above method; therefore, the specific limitations in one or more model training apparatus embodiments provided below can be found in the limitations of the model training method described above, and will not be repeated here.

[0165] In one exemplary embodiment, such as Figure 8 As shown, a model training device is provided, including: a retrieval module 10, a first acquisition module 11, a second acquisition module 12, and a training module 13, wherein:

[0166] The retrieval module 10 is used to retrieve auxiliary documents related to the sample query question from the document library.

[0167] The first acquisition module 11 is used to input the sample query question, auxiliary documents and sample candidate documents into the initial generative model to obtain the sample response information of the sample query question; the sample response information is the user intent information corresponding to the sample query question determined by the initial generative model based on the sample query question and auxiliary documents, and output based on the user intent information and sample candidate documents.

[0168] The second acquisition module 12 is used to obtain the target reward based on the sample response information.

[0169] Training module 13 is used to train the initial generative model based on the target reward and reinforcement learning algorithm to obtain the target generative model.

[0170] The model training apparatus provided in this application retrieves auxiliary documents related to the sample query question from a document library based on the sample query question. The sample query question, auxiliary documents, and candidate sample documents are input into an initial generative model to obtain sample response information for the sample query question. The sample response information is generated by the initial generative model based on the sample query question and auxiliary documents, determining the user intent information corresponding to the sample query question, and outputting it based on the user intent information and candidate sample documents. The initial generative model can complete the implicit intent of the sample query question based on the sample query question and auxiliary documents, thus obtaining the user intent information corresponding to the sample query question, ensuring the accuracy of the obtained user intent information. Therefore, based on the user intent information and candidate sample documents, the sample response information for the sample query question can be accurately obtained. A target reward can then be obtained based on the sample response information output by the initial generative model. The initial generative model is then accurately trained based on the target reward and a reinforcement learning algorithm to obtain a target generative model. This allows the target generative model to accurately obtain the sample response information for the query question, improving the accuracy of the recommended content output by the generative model.

[0171] In one embodiment, the sample response information includes sample document fragments in the sample candidate documents and / or a sample relevance measure between the sample candidate documents and the user intent information; or, the sample response information includes preset document fragments and / or preset relevance measures.

[0172] The model training device provided in this embodiment can execute the above method embodiment, and its implementation principle and technical effect are similar, so it will not be described again here.

[0173] In one embodiment, the above sample response information is output by the initial generative model based on any of the following:

[0174] If it is determined that a sample document fragment exists in the candidate sample document based on the user intent information, then the sample document fragment and / or sample relevance metric are output based on the candidate sample document and the user intent information.

[0175] If the initial generative model determines, based on user intent information, that there is no sample document fragment in the candidate documents, it outputs a preset document fragment and / or a preset relevance metric.

[0176] The model training device provided in this embodiment can execute the above method embodiment, and its implementation principle and technical effect are similar, so it will not be described again here.

[0177] Based on the above embodiments, such as Figure 9 As shown, the second acquisition module 12 includes: a first acquisition unit 121, a second acquisition unit 122, and a third acquisition unit 123, wherein:

[0178] The first acquisition unit 121 is used to obtain a format reward based on the format information of the document fragment and / or the format information of the relevance measure; the document fragment includes a sample document fragment or a preset document fragment; the relevance measure includes a sample relevance measure or a preset relevance measure.

[0179] The second acquisition unit 122 is used to obtain a score reward based on the relevance metric and the standard relevance metric of the sample candidate documents.

[0180] The third acquisition unit 123 is used to obtain the target reward based on the format reward and / or score reward.

[0181] The model training device provided in this embodiment can execute the above method embodiment, and its implementation principle and technical effect are similar, so it will not be described again here.

[0182] In one embodiment, the third acquisition unit 123 is used to determine the target reward as the product of the format reward and the score reward.

[0183] The model training device provided in this embodiment can execute the above method embodiment, and its implementation principle and technical effect are similar, so it will not be described again here.

[0184] Based on the above embodiments, such as Figure 10 As shown, the training module 13 includes: a fourth acquisition unit 131 and a training unit 132, wherein:

[0185] The fourth acquisition unit 131 is used to construct the loss function of the reinforcement learning algorithm based on the target reward.

[0186] Training unit 132 is used to optimize the parameters of the initial generative model by minimizing the value of the loss function to obtain the target generative model.

[0187] The model training device provided in this embodiment can execute the above method embodiment, and its implementation principle and technical effect are similar, so it will not be described again here.

[0188] Based on the above embodiments, such as Figure 11 As shown, the above-mentioned device further includes: a third acquisition module 14, a fourth acquisition module 15, and a fifth acquisition module 16, wherein:

[0189] The third acquisition module 14 is used to perform structured annotation on the documents corresponding to the query question through a large language model to obtain structured training samples; the structured training samples include target document fragments and / or target relevance measures between the query question and the documents.

[0190] The fourth acquisition module 15 is used to input the query question and the document into a preset generative model to obtain the response information of the query question; the response information includes document fragments and / or, the predicted value of the relevance measure between the query question and the document.

[0191] The fifth acquisition module 16 is used to train the preset generative model based on the response information and structured training samples to obtain the initial generative model.

[0192] The model training device provided in this embodiment can execute the above method embodiment, and its implementation principle and technical effect are similar, so it will not be described again here.

[0193] Based on the same inventive concept, this application also provides a query device for implementing the query method described above. The solution provided by this device is similar to the solution described in the above method; therefore, the specific limitations in one or more query device embodiments provided below can be found in the limitations of the query method described above, and will not be repeated here.

[0194] In one exemplary embodiment, such as Figure 12 As shown, a query device is provided, including: a retrieval module 20 and a query module 21, wherein:

[0195] The retrieval module 20 is used to retrieve candidate documents from the document library based on the query question.

[0196] Query module 21 is used to input the query question and candidate documents into the target generative model to obtain the response information for the query question;

[0197] The target generative model is trained using the method described in any of the foregoing embodiments.

[0198] The query device provided in this embodiment can execute the above method embodiment, and its implementation principle and technical effect are similar, so it will not be described again here.

[0199] In one embodiment, the response information includes document fragments from candidate documents and / or a relevance measure between the candidate documents and the query; or,

[0200] The response information includes preset document fragments and / or preset relevance metrics.

[0201] The query device provided in this embodiment can execute the above method embodiment, and its implementation principle and technical effect are similar, so it will not be described again here.

[0202] In one embodiment, the above response information is output by the target generative model based on any of the following:

[0203] If it is determined that document fragments exist in the candidate documents based on the query question, then based on the candidate documents and the query question, output the document fragments and / or relevance measures;

[0204] If it is determined that no document fragment exists in the candidate documents based on the query question, then the preset document fragment and / or preset relevance metric value will be output.

[0205] The query device provided in this embodiment can execute the above method embodiment, and its implementation principle and technical effect are similar, so it will not be described again here.

[0206] Each module in the above-mentioned device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device in hardware form, or stored in the memory of a computer device in software form, so that the processor can call and execute the operations corresponding to each module.

[0207] In one exemplary embodiment, a computer device is provided, which may be a server, and its internal structure diagram may be as follows: Figure 13 As shown, this computer device includes a processor, memory, input / output (I / O) interfaces, and a communication interface. The processor, memory, and I / O interfaces are connected via a system bus, and the communication interface is also connected to the system bus via the I / O interfaces. The processor provides computational and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system, computer programs, and a database. The internal memory provides the environment for the operating system and computer programs stored in the non-volatile storage media. The database stores a library of user-generated content documents. The I / O interfaces are used for exchanging information between the processor and external devices. The communication interface is used for communicating with external terminals via a network connection. When executed by the processor, the computer program implements a model training and query method.

[0208] Those skilled in the art will understand that Figure 13 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0209] In one embodiment, a computer device is provided, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps in the above-described method embodiments.

[0210] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon that, when executed by a processor, implements the steps in the above method embodiments.

[0211] In one embodiment, a computer program product is provided, including a computer program that, when executed by a processor, implements the steps in the above method embodiments.

[0212] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of the relevant data must comply with relevant regulations.

[0213] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. Any references to memory, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile memory and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM). The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, artificial intelligence (AI) processors, etc., and are not limited to these.

[0214] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this application.

[0215] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of this patent application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this application should be determined by the appended claims.< / score> < / score> < / score> < / extract> < / extract> < / extract> < / think> < / think> < / think> < / think> < / score> < / extract> < / think> < / score> < / extract> < / intent> < / think>

Claims

1. A model training method, characterized in that, The method includes: Based on the sample query question, retrieve auxiliary documents related to the sample query question from the document library; The sample query question, the auxiliary document, and the sample candidate document are input into the initial generative model to obtain sample response information for the sample query question. The sample response information is the user intent information corresponding to the sample query question determined by the initial generative model based on the sample query question and the auxiliary document, and output based on the user intent information and the sample candidate document. The target reward is obtained based on the sample response information; The initial generative model is trained based on the target reward and reinforcement learning algorithm to obtain the target generative model.

2. The method according to claim 1, characterized in that, The sample response information includes sample document fragments from the candidate documents and / or a sample relevance metric between the candidate documents and the user intent information; or... The sample response information includes preset document fragments and / or preset relevance metrics.

3. The method according to claim 2, characterized in that, The sample response information is output by the initial generative model based on any of the following conditions: If it is determined based on the user intent information that the sample document fragment exists in the sample candidate document, then based on the sample candidate document and the user intent information, the sample document fragment and / or the sample relevance metric value are output; If the initial generative model determines, based on the user intent information, that the sample document fragment does not exist in the sample candidate documents, then it outputs the preset document fragment and / or the preset relevance metric.

4. The method according to claim 2 or 3, characterized in that, The step of obtaining the target reward based on the sample response information includes: A format reward is obtained based on the format information of the document fragment and / or the format information of the relevance metric; the document fragment includes the sample document fragment or the preset document fragment; the relevance metric includes the sample relevance metric or the preset relevance metric. A score reward is obtained based on the relevance metric and the standard relevance metric of the sample candidate documents; The target reward is obtained based on the format reward and / or the score reward.

5. The method according to claim 4, characterized in that, The process of obtaining the target reward based on the format reward and / or the score reward includes: The product of the format reward and the score reward is determined as the target reward.

6. The method according to claim 1, characterized in that, The step of training the initial generative model based on the target reward and reinforcement learning algorithm to obtain the target generative model includes: The loss function of the reinforcement learning algorithm is constructed based on the target reward; The target generative model is obtained by optimizing the parameters of the initial generative model by minimizing the value of the loss function.

7. The method according to claim 1, characterized in that, The method further includes: The document corresponding to the query question is structured and labeled using a large language model to obtain structured training samples; the structured training samples include target document fragments and / or target relevance measures between the query question and the document; The query question and the document are input into a preset generative model to obtain response information for the query question; the response information includes document fragments and / or a predicted value of the relevance measure between the query question and the document; The preset generative model is trained based on the response information and the structured training samples to obtain the initial generative model.

8. A query method, characterized in that, The method includes: Based on the query question, candidate documents are retrieved from the document library; The query question and the candidate documents are input into the target generative model to obtain the response information for the query question; The target generative model is obtained by training using the method described in any one of claims 1-7.

9. The method according to claim 8, characterized in that, The response information includes document fragments from the candidate documents and / or a relevance metric between the candidate documents and the query question; or... The response information includes preset document fragments and / or preset relevance metrics.

10. The method according to claim 9, characterized in that, The response information is output by the target generative model based on any of the following conditions: If it is determined that the document fragment exists in the candidate document based on the query question, then based on the candidate document and the query question, the document fragment and / or the relevance measure value are output; If it is determined based on the query question that the document fragment does not exist in the candidate documents, then the preset document fragment and / or the preset relevance metric value are output.

11. A model training device, characterized in that, The device includes: The retrieval module is used to retrieve auxiliary documents related to the sample query question from the document library; The first acquisition module is used to input the sample query question, the auxiliary document, and the sample candidate document into the initial generative model to obtain sample response information for the sample query question; the sample response information is the user intent information corresponding to the sample query question determined by the initial generative model based on the sample query question and the auxiliary document, and output based on the user intent information and the sample candidate document. The second acquisition module is used to obtain the target reward based on the sample response information; The training module is used to train the initial generative model based on the target reward and reinforcement learning algorithm to obtain the target generative model.

12. A query device, characterized in that, The device includes: The retrieval module is used to retrieve candidate documents from the document library based on the query question. The query module is used to input the query question and the candidate documents into the target generative model to obtain the response information for the query question; The target generative model is obtained by training using the method described in any one of claims 1-7.

13. A server comprising a memory and a processor, the memory storing a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 10.

14. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 10.

15. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 10.

Images