Self-search reinforcement learning training method and device, electronic equipment and medium

Abstract

The application provides a self-search reinforcement learning training method and device, electronic equipment and medium. The method comprises: obtaining a training sample, the training sample comprising a question text and a corresponding reference answer; inputting the question text into a language model to be trained, and constraining the language model to be trained to generate a response sequence according to a preset structured interaction format; determining a result reward value according to a matching result of an answer field and the reference answer, and determining a format reward value according to a satisfaction of the response sequence to the preset structured interaction format; determining a total reward of the sample based on the result reward value and the format reward value, inputting the response sequence and the total reward of the sample into a reinforcement learning optimization process, updating a strategy parameter of the language model to be trained, repeatedly executing the above process until a stop condition is met, and obtaining a target language model; and deploying the target language model in an online inference process. The application can reduce training cost, improve training stability, and improve strategy migration efficiency.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, and in particular to a self-search reinforcement learning training method, apparatus, electronic device, and medium. Background Technology

[0002] In recent years, large language models have encoded a wealth of general knowledge into their parameters through large-scale pre-training and have been gradually applied to complex question-answering and intelligent decision-making tasks. To enhance the model's ability to process real-time and domain-specific information, existing solutions typically construct large language models as agents with tool-calling capabilities, enabling them to retrieve information from external search services during reasoning and then generate answers based on the search results. Following this approach, existing technologies further employ reinforcement learning to train search and reasoning collaborative strategies, optimizing the model's behavior in the "thinking, querying, information integration, and answering" chain through numerous interaction trajectories.

[0003] While existing technologies can improve performance on some tasks, they still have significant drawbacks. First, the training process heavily relies on external search interfaces, requiring numerous retrieval calls during the reinforcement learning phase, leading to high training costs. Furthermore, the latency, availability, and fluctuations in returned content from external services introduce instability, affecting training consistency. Second, existing solutions generally rely on external retrieval as the primary information source, underutilizing the knowledge already encoded within the model. This results in unnecessary retrievals for problems that could be solved using internal knowledge. Third, some solutions employ different interaction mechanisms between the training and deployment environments, limiting the transfer efficiency and generalization ability of the trained search inference strategies in real-world online scenarios. Therefore, there is an urgent need for a self-search reinforcement learning training method that can improve the efficiency of internal knowledge utilization under low external dependency conditions and support consistent transfer from training to online applications. Summary of the Invention

[0004] In view of this, embodiments of this application provide a self-search reinforcement learning training method, apparatus, electronic device, and medium to solve the problems of high training cost, strong external dependence, and unstable policy transfer in the prior art.

[0005] A first aspect of this application provides a self-searching reinforcement learning training method, comprising: acquiring training samples, the training samples including question text and corresponding reference answers; inputting the question text into a language model to be trained, constraining the language model to be trained to generate a response sequence according to a preset structured interaction format, wherein the response sequence includes a search query field, an information field, and an answer field, the content of the information field being generated by the language model to be trained based on knowledge represented by internal parameters; determining a result reward value based on the matching result of the answer field and the reference answer, and determining a format reward value based on the satisfaction of the response sequence with the preset structured interaction format; determining a total sample reward based on the result reward value and the format reward value, and inputting the response sequence and the total sample reward into a reinforcement learning optimization process to update the policy parameters of the language model to be trained, repeating the above process until a stopping condition is met to obtain a target language model; deploying the target language model in an online inference process, intercepting the query content and calling an external search service to obtain retrieval results when a search query field is detected, writing the retrieval results into the information field, and then continuing to generate the answer field.

[0006] A second aspect of this application provides a self-searching reinforcement learning training apparatus, comprising: an acquisition module for acquiring training samples, the training samples including question text and corresponding reference answers; a constraint module for inputting the question text into a language model to be trained, constraining the language model to be trained to generate a response sequence according to a preset structured interaction format, wherein the response sequence includes a search query field, an information field, and an answer field, the content of the information field being generated by the language model to be trained based on knowledge represented by internal parameters; a determination module for determining a result reward value based on the matching result of the answer field and the reference answer, and determining a format reward value based on the satisfaction of the preset structured interaction format by the response sequence; an update module for determining a total sample reward based on the result reward value and the format reward value, and inputting the response sequence and the total sample reward into a reinforcement learning optimization process to update the policy parameters of the language model to be trained, repeating the above process until a stopping condition is met to obtain a target language model; and a generation module for deploying the target language model in an online inference process, intercepting the query content when a search query field is detected and calling an external search service to obtain retrieval results, writing the retrieval results into the information field, and then continuing to generate the answer field.

[0007] A third aspect of this application provides an electronic device, including a memory, a processor, and a computer program stored in the memory and running on the processor, wherein the processor executes the computer program to implement the steps of the above-described method.

[0008] A fourth aspect of this application provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps of the above-described method.

[0009] The above-described technical solutions adopted in the embodiments of this application can achieve the following beneficial effects:

[0010] By acquiring training samples, including question text and corresponding reference answers, and inputting the question text into the language model to be trained, the language model is constrained to generate a response sequence according to a preset structured interaction format. The response sequence includes a search query field, an information field, and an answer field. The information field content is generated by the language model based on knowledge represented by its internal parameters. A result reward value is determined based on the matching result between the answer field and the reference answer, and a format reward value is determined based on the response sequence's compliance with the preset structured interaction format. The total sample reward is determined based on the result reward value and the format reward value, and the response sequence and the total sample reward are input into a reinforcement learning optimization process to update the policy parameters of the language model to be trained. This process is repeated until a stopping condition is met, resulting in the target language model. The target language model is deployed in an online inference process. When a search query field is detected, the query content is intercepted, and an external search service is called to obtain the retrieval results. The retrieval results are written into the information field before the answer field is generated. This application can reduce training costs, improve training stability, and increase policy transfer efficiency. Attached Figure Description

[0011] To more clearly illustrate the technical solutions in the embodiments of this application, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0012] Figure 1 This is a flowchart illustrating the self-search reinforcement learning training method provided in the embodiments of this application;

[0013] Figure 2 This is a schematic diagram of the structure of the self-search reinforcement learning training device provided in the embodiments of this application;

[0014] Figure 3 This is a schematic diagram of the structure of the electronic device provided in the embodiments of this application. Detailed Implementation

[0015] In the following description, specific details such as particular system architectures and techniques are set forth for illustrative purposes and not for limitation, in order to provide a thorough understanding of the embodiments of this application. However, those skilled in the art will understand that this application may also be implemented in other embodiments without these specific details. In other instances, detailed descriptions of well-known systems, apparatuses, circuits, and methods have been omitted so as not to obscure the description of this application with unnecessary detail.

[0016] In recent years, large-scale language models have encoded massive amounts of world knowledge into their parameters through extensive pre-training. To enable LLMs to solve complex tasks requiring up-to-date or specific information, researchers have attempted to train large language model-enabled agents (LLM agents). In this model, LLMs are endowed with tool-using capabilities, such as searching and using Python compilers. Search ability, as one of the fundamental human abilities, effectively helps humans acquire real-time information to answer questions. Therefore, enabling LLMs to invoke external search engines (such as Google and Bing) and then reason and answer based on the retrieved information is crucial. Existing works, such as Search-R1 and ZeroSearch, have trained LLMs to master this search-reasoning paradigm through reinforcement learning, achieving excellent performance on multiple question-answering benchmarks.

[0017] However, this paradigm that relies on external search still suffers from the following two core bottlenecks:

[0018] 1. High Cost and Instability: Reinforcement learning training requires generating a massive number of rollouts, and each API call to an external search engine incurs costs. This makes training extremely expensive. Furthermore, the availability, latency, and quality of the returned content from external search engines are unpredictable, introducing instability into the training process.

[0019] 2. Waste of pre-trained knowledge: A fundamental fact is that LLMs themselves already contain a wealth of knowledge capable of answering questions. Many questions do not actually require external search; the model's internal knowledge is sufficient to solve them. Forcing reliance on external search not only increases unnecessary overhead but also ignores the enormous potential of the model itself as a "knowledge base."

[0020] Meanwhile, an important observation has sparked new thinking: LLMs can function not only as "inferators" but also as "world models." This means that for a knowledge-intensive problem, a large language model can leverage its vast knowledge of the world to find the answer from its own parameters. Work such as ZeroSearch has demonstrated that a finely tuned LLM can replace a real search engine, providing stable and reliable knowledge for other models, thus creating a "semi-real" search environment. However, whether LLMs can utilize their own world knowledge and the upper limit of their ability to do so remains unclear.

[0021] Based on the above observations, this paper raises a fundamental question: Can we completely break free from dependence on external knowledge sources and complete search-driven tasks using only the internal knowledge of the LLM itself? If feasible, this would not only minimize training costs but also open up a more efficient and autonomous training path for LLM agents, while also proving the feasibility of LLM as a world model.

[0022] To answer this question, the paper first explores the "self-search" capability of LLM—that is, through structured prompts, the model can simulate a complete "think-search-acquire information-rethink" cycle in a single generation. The study finds that by increasing the number of samples during inference, the self-search performance of LLM exhibits strong scalability, with its performance ceiling (pass@k) even surpassing some models that use real-world search.

[0023] However, consistently selecting the optimal answer from multiple sampling results and systematically improving the model's knowledge extraction capabilities remain significant challenges. To address these challenges, this paper proposes a self-search reinforcement learning framework. This framework aims to optimize the utilization efficiency of the model's internal knowledge through reinforcement learning, leveraging feedback generated by the model itself, thereby constructing a fully self-consistent and low-cost new paradigm for agent training.

[0024] In summary, the main technical problems addressed by this application include:

[0025] 1. Activating Internal Knowledge Potential: The aim is to build a new paradigm that can fully activate and efficiently utilize the inherent world knowledge within large language models, making it a reliable and low-cost internal knowledge source, thereby reducing dependence on external tools.

[0026] 2. Optimize self-search capabilities: The aim is to propose a self-search reinforcement learning framework that iteratively optimizes the model through feedback generated by the model itself, systematically improving its ability to simulate search, integrate internal knowledge, and solve problems, making its performance more stable and powerful.

[0027] 3. Achieve seamless transfer from simulation to reality: The aim is to ensure that the core search and reasoning strategies learned by an agent trained in a pure simulation environment can be seamlessly generalized to a real online search environment, achieving the optimal balance between training cost and deployment performance.

[0028] In view of the problems existing in the prior art, this application proposes a self-search reinforcement learning training method, aiming to train a large language model (LLM) into an efficient and autonomous agent capable of completing complex search-based question-answering tasks using only its internally encoded world knowledge. The technical solution of this application includes the following three core operations:

[0029] 1. Construct a self-searching simulation environment:

[0030] First, through structured prompts, the LLM is guided to simulate a complete "think-search-acquire information" cycle in a single generation. The model then... <search>Generate a query in the tags, and then... <information>Relevant information is automatically generated from the tags. The entire process is completed within the model itself, without needing to call any external search engines or databases, thus creating a zero-cost, highly stable, fully simulated training environment.

[0031] 2. Design a dual-reward mechanism for reinforcement learning:

[0032] In the simulated environment described above, the model is trained using reinforcement learning (such as the GRPO algorithm). The key innovation lies in designing a dual reward mechanism that does not rely on external information:

[0033] Results-based rewards: Rewards or penalties are given based on the correctness of the final answer (compared to the standard answer).

[0034] Formatted Rewards: Rewards are given for models that strictly adhere to a pre-defined structured format for thinking, searching, and answering.

[0035] Through this self-feedback reinforcement learning, the model is driven to continuously optimize its internal knowledge retrieval, integration, and reasoning strategies.

[0036] 3. Achieve seamless migration from simulation to reality:

[0037] Because the training process strictly follows the same interaction format as real search tools, the trained model can be directly deployed to the real environment and the powerful reasoning ability learned in the simulation environment can be seamlessly applied to processing real search results, achieving a smooth transition from low-cost simulation to high-performance reality.

[0038] Compared with the aforementioned prior art solutions, the technical solution of this application differs in the following two main aspects:

[0039] 1. Different search environment: Previous technologies relied on external search engines or knowledge bases; the proposed solution fully utilizes the internal knowledge of the model to perform a full simulation search, achieving zero-cost training.

[0040] 2. Different training mechanisms: Previous techniques relied on external supervision signals; the proposed solution adopts self-consistent reinforcement learning (SSRL), which optimizes itself through feedback generated by the model itself.

[0041] Compared with existing technologies, the SSRL framework of this application brings the following significant advantages:

[0042] 1. Significantly reduce training costs and improve training stability:

[0043] By constructing a fully internalized "full-simulation search" environment, the reliance on expensive external search engine APIs is completely eliminated. This not only reduces training costs by several orders of magnitude but also eliminates training instability caused by fluctuations, latency, or content changes in external services, enabling efficient and reliable model training.

[0044] 2. Significantly improve model performance and reduce knowledge "illusions":

[0045] Through self-consistent reinforcement learning, the model is driven to systematically optimize its internal knowledge retrieval and integration strategies. This enables the model to access its existing world knowledge more accurately and reliably, thereby outperforming baseline models that rely on external search on multiple question-answering benchmarks and effectively reducing the "illusion" phenomenon caused by inaccurate or missing knowledge.

[0046] 3. Achieve seamless migration from simulation to reality, combining efficiency and generalization:

[0047] Models trained in simulated environments can seamlessly transfer their learned search and reasoning abilities to real-world online search scenarios. This means that models can be trained internally at low cost and intelligently combine internal knowledge with external search during deployment, achieving a perfect balance between high training efficiency and high deployment performance, greatly expanding their practical value.

[0048] The main user scenarios for the technical solution in this application include, but are not limited to, the following:

[0049] 1. A customized enterprise-level intelligent customer service system;

[0050] 2. Answer questions in cutting-edge fields such as financial services, healthcare, and scientific discovery, especially those requiring knowledge retrieval to obtain the correct answer;

[0051] 3. In environments with limited training resources, achieve comparable or even better search agents at a lower cost.

[0052] The technical solution of this application will now be described in detail with reference to the accompanying drawings and specific embodiments.

[0053] Figure 1 This is a flowchart illustrating the self-search reinforcement learning training method provided in an embodiment of this application. Figure 1 As shown, this self-search reinforcement learning training method may specifically include:

[0054] S101, Obtain training samples, which include question texts and corresponding reference answers;

[0055] S102, Input the question text into the language model to be trained, and constrain the language model to be trained to generate a response sequence according to a preset structured interaction format. The response sequence includes a search query field, an information field, and an answer field. The content of the information field is generated by the language model to be trained based on the knowledge represented by its internal parameters.

[0056] S103, determine the result reward value based on the matching result between the answer field and the reference answer, and determine the format reward value based on the satisfaction of the response sequence with the preset structured interaction format;

[0057] S104. Determine the total sample reward based on the result reward value and the format reward value, and input the response sequence and the total sample reward into the reinforcement learning optimization process to update the policy parameters of the language model to be trained. Repeat the above process until the stopping condition is met to obtain the target language model.

[0058] S105: Deploy the target language model in the online inference process. When a search query field is detected, intercept the query content and call an external search service to obtain the search results. Write the search results into the information field and then continue to generate the answer field.

[0059] In some embodiments, training samples are obtained, which include question text and corresponding reference answers, including:

[0060] Obtain the target task corpus and perform sample parsing on the target task corpus to form sample pairs of question text and reference answers;

[0061] The sample pairs were formatted to standardize the input format of the questions and the format of the reference answers.

[0062] The normalized sample pairs are filtered for validity according to the preset training data rules to obtain a set of training samples used to drive the language model to be trained to generate response sequences.

[0063] Specifically, firstly, the target task corpus is acquired. The target task corpus preferably uses data sources consistent with those used in search-based question-answering tasks, including general factual question-answering corpora, domain-specific question-answering corpora, and historical evaluation corpora with standard answer annotations. In line with the aforementioned technical solution's requirement that "the model only uses internal parameters to represent knowledge and generate information fields during the training phase," the two core elements of the question text and the reference answer are retained during corpus access, without introducing external search results as the main training labels. Taking an enterprise intelligent customer service scenario as an example, a hybrid set of enterprise knowledge question-answering corpora and publicly available factual question-answering corpora can be accessed; the former covers enterprise terminology and business expressions, while the latter maintains the distribution of general knowledge.

[0064] Furthermore, after acquiring the target task corpus, sample parsing is performed to form sample pairs of question texts and reference answers. The sample parsing process includes field extraction, sample segmentation, and correspondence establishment. For multi-round records in the original corpus, the question texts that can be answered independently in the current round are first extracted according to preset segmentation rules, and then the standard answers that semantically correspond to the question texts are determined as reference answers, forming one-to-one corresponding sample pairs.

[0065] In some examples, where multiple acceptable answers exist for the same question, these answers are grouped into a set of reference answers for the same sample pair. The mapping relationship between the main reference answer and equivalent reference answers is recorded in the sample metadata for matching tolerance control in the subsequent reward calculation stage. This parsing method avoids mismatches between the question text and the reference answers, ensuring a clear basis for the reward calculation.

[0066] Furthermore, the sample pairs undergo format standardization to unify the input format of the question text and the format of the reference answer annotation. Standardization on the question text side focuses on character encoding standardization, removal of irrelevant control characters, standardization of punctuation and numerical representation, standardization of time and unit representation, and standardization of question sentence structure. Standardization on the reference answer side focuses on answer boundary standardization, merging of synonyms, standardization of answer enumeration order rules, and standardization of numerical precision rules. To adapt to the aforementioned structured interaction format, the standardization stage can also generate input packaging templates to ensure that the question text has a consistent input appearance before entering the language model to be trained.

[0067] For example, taking a financial Q&A example, the original question "What was the revenue in Q4 2021?" can be standardized into a unified Chinese question expression. The "1.2 billion" and "1.2 billion" in the reference answer can be mapped to the same standard numerical expression, thereby reducing format noise when matching subsequent answers.

[0068] Furthermore, after normalization, the normalized sample pairs are screened for effectiveness according to preset training data rules to obtain a training sample set. The preset training data rules include at least completeness rules, consistency rules, and learnability rules. Completeness rules are used to remove samples with missing question text or missing reference answers; consistency rules are used to identify and remove samples with semantic conflicts between the question text and the reference answer; learnability rules are used to filter samples that are extremely verbose, have unresolved ambiguities, or have indeterminate answers. Furthermore, stratified sampling can be performed on the retained samples according to task distribution rules to control the proportion of samples from different knowledge types and difficulty ranges, ensuring that the training sample set is consistent with the distribution of the target application. Taking a subset of medical question-and-answer materials as an example, samples with outdated conclusions and no version markers are downweighted or removed to avoid incorporating unstable knowledge into the training trajectory evaluation baseline.

[0069] In this embodiment, through the sequential processing of "parsing-normalization-filtering" described above, a training sample set is finally output to drive the language model to be trained to generate response sequences. This training sample set maintains consistency in field semantics, text appearance, and answer annotation, directly supporting the stable calculation of subsequent result reward values ​​and format reward values, and forming a closed loop with the self-search reinforcement learning process. Therefore, it can provide a high-quality sample foundation for the model's internal knowledge retrieval and structured response generation without relying on external retrieval annotations, improving the consistency and reproducibility of the training phase.

[0070] In some embodiments, question text is input into the language model to be trained, and the language model is constrained to generate a response sequence according to a preset structured interaction format, including:

[0071] The model input prompts are built based on the preset structured interaction template. The preset structured interaction template is used to limit the generation order and field boundaries of the search query field, information field and answer field.

[0072] The question text and the model input prompt are concatenated and encoded before being input into the language model to be trained, so as to trigger the generation of field-level responses according to the preset structured interaction template;

[0073] During the generation process, structural constraint checks are performed on the response sequence to obtain candidate response sequences that satisfy field integrity and order consistency.

[0074] The information field content generated by the language model to be trained based on the knowledge represented by the internal parameters is retained in the candidate response sequence to form the target response sequence for subsequent reward calculation.

[0075] Specifically, firstly, input prompts are constructed based on a pre-defined structured interaction template. In this embodiment, the pre-defined structured interaction template employs a fixed-field protocol, including at least a search query field, an information field, and an answer field, and explicitly defines the order and boundaries of the fields. For example, in some examples, a fixed-field protocol can be used. <think> 、 <search> 、 <information> 、 <answer>The organizational forms, among which <think>Used to carry intermediate reasoning segments, <search>Used to carry query expressions <information>Used to carry information content, <answer>This is used to carry the final answer. To avoid field drift during model training, the input prompts simultaneously provide rules for boundary closure, no missing fields, and no interchangeable field order, ensuring that the model is bound to a unified interaction format from the very beginning of generation.

[0076] Subsequently, the question text and model input prompts are concatenated and encoded before being input into the language model to be trained, triggering field-level response generation. The concatenation is preferably organized in the order of "task instruction segment + field template segment + question text segment," where the task instruction segment declares the current self-search training round, the field template segment declares the set and order of fields that must be output, and the question text segment carries the question for the current sample. For example, taking the "knowledge-intensive question answering" example, when the input question is a factual question, the model first... <search>The fields form query phrases focused on the problem, and then... <information>The field generates information fragments based on the knowledge in the internal parameters, and finally... <answer>The field outputs the answer. This process does not call an external search engine, maintaining consistency with the fully simulated training environment.

[0077] Furthermore, during the generation process, structural constraint checks are performed on the response sequences to filter candidate response sequences that meet the requirements of field completeness and order consistency. The structural constraint checks are performed step-by-step: "field existence—field order—field boundaries." Field existence checks confirm that the search query fields, information fields, and answer fields have all been generated; field order checks confirm that fields appear in the order specified in the template; field boundary checks confirm that the start and end markers of each field are paired and closed without overlapping or nesting. For response sequences that do not meet the constraints, the corresponding structural anomaly type is recorded and marked as a low-quality trajectory, and these sequences are not included in the target response sequence set. For response sequences that meet the constraints, they are retained as candidate response sequences and proceed to the next step of processing. This check mechanism cleans up format noise before reward calculation, preventing invalid structures from being introduced into policy updates.

[0078] Furthermore, within the candidate response sequence, the information field content generated by the language model under training based on its internal parameter representation knowledge is retained to form the target response sequence for subsequent reward calculation. The key processing point of this step is to distinguish between "information fields generated by internal knowledge" and "non-template noise fragments," and to perform boundary-based truncation and text straightening on the information fields, enabling them to establish stable reference relationships with the search query field and answer field in the same response. For example, combining the example of self-search capability expanding with the number of samplings, when multiple candidate response sequences can be generated for the same question, their information field content is retained according to the above rules, and then submitted to the subsequent reward module for unified evaluation based on answer correctness and format satisfaction.

[0079] For example, in an enterprise-level intelligent customer service example, the question text relates to product upgrade strategies. The model outputs the search query field "product upgrade compatibility conditions" according to a template, provides version constraints and dependency conditions generated from internal knowledge in the information field, and then outputs the executable conclusion in the answer field. After structural constraint verification confirms that the fields are complete and in the correct order, the response is identified as the target response sequence and enters the reward calculation. Compared to the free generation method without template constraints, this implementation significantly reduces the proportion of missing fields and disordered order, making the trajectory received in the subsequent reinforcement learning process more stable and more comparable.

[0080] Through the technical means of this embodiment, the training phase can stably generate structured response sequences without relying on external retrieval interfaces, and incorporate the information fields generated by internal knowledge into the subsequent optimization process in a verifiable form, thereby providing a consistent input basis for the joint learning of result rewards and format rewards, and improving the stability and trajectory quality of the training process.

[0081] In some embodiments, a result reward value is determined based on the matching result of the answer field and the reference answer, and a format reward value is determined based on the satisfaction of the response sequence with a preset structured interaction format, including:

[0082] The target response sequence is parsed at the field level to locate the target answer fragment corresponding to the answer field, and the target answer fragment is then normalized.

[0083] The standardized target answer fragment is matched with the reference answer to obtain the answer matching degree index, and the result reward value is determined according to the mapping relationship between the answer matching degree index and the preset result reward function.

[0084] Perform structural compliance checks on the target response sequence. The structural compliance checks include field completeness checks, field order consistency checks, and field boundary closure checks to obtain the format compliance index.

[0085] The format reward value is determined based on the mapping relationship between the format compliance index and the preset format reward function. The preset format reward function configures differentiated penalty coefficients for missing fields, out-of-order fields, or abnormal field boundaries.

[0086] Specifically, in this embodiment, the reward calculation stage corresponds to the self-reinforcement stage, employing a dual-channel evaluation mechanism of "outcome reward + format reward" to uniformly quantify the target response sequence, thereby providing a stable and comparable reward signal for the subsequent reinforcement learning optimization process. This implementation continues the aforementioned structured interaction protocol, and the target response sequence at least includes a search query field, an information field, and an answer field, and may also include fields used for intermediate inference.

[0087] First, field-level parsing is performed on the target response sequence to locate the target answer fragment corresponding to the answer field, and then normalization processing is performed on the target answer fragment. Field-level parsing extracts the content of the answer field according to preset field boundary rules to avoid mistakenly merging adjacent field text into the answer fragment. Normalization processing preferably includes character standardization, punctuation correction, numerical unit unification, synonym mapping, and enumeration item sorting standardization. For example, in knowledge-intensive question-answering examples, when the answer appears in different forms such as "1.2 billion" and "1.2 billion", it is first mapped to a unified numerical expression before entering the matching calculation, thereby reducing misjudgments caused by differences in expression.

[0088] Subsequently, the standardized target answer fragment is matched with the reference answer to obtain an answer matching degree index. The result reward value is then determined based on the mapping relationship between the answer matching degree index and a preset result reward function. The matching calculation can employ a combined approach of exact matching and semantic matching: a high matching degree is assigned when the target answer fragment is completely identical to the reference answer; a medium-high matching degree is assigned according to preset rules when they are semantically equivalent but differ in expression; and a low matching degree is assigned when there is a core factual conflict. Based on this matching degree index, the result reward value is obtained through mapping using the preset result reward function. For samples with multiple acceptable reference answers, the "maximum matching degree with any reference answer" is used as the mapping input to ensure that the reward allocation is consistent with the data annotation rules.

[0089] Furthermore, after the result reward calculation is completed, a structural compliance check is performed on the target response sequence to obtain a format compliance index. The structural compliance check is performed in the following order: first, a field completeness check is performed to confirm whether the search query field, information field, and answer field are complete; then, a field order consistency check is performed to confirm that the order in which the fields appear is consistent with the preset structured interaction format; finally, a field boundary closure check is performed to confirm that the field start and end markers are complete and there is no cross-nesting. Referring to the specific example above, if the model output is missing an information field or places the answer field before the information field, it is recorded as a missing field or an out-of-order field error, respectively, and the corresponding score is deducted from the format score.

[0090] Furthermore, the format reward value is determined based on the mapping relationship between the format compliance index and the preset format reward function. The preset format reward function is configured with differentiated penalty coefficients, applying different penalty intensities to field missing, field out-of-order, and field boundary anomalies. The penalty coefficient for field missing is higher than that for field out-of-order, and field boundary anomalies are subject to tiered penalties based on their impact range. This guides the model to prioritize field completeness and boundary standardization before further optimizing field order stability. Taking enterprise-level intelligent customer service training samples as an example, if a response answer is factually correct but the information field boundary is not closed, the result reward value is high while the format reward value is significantly reduced; if another response is approximately factually correct and structurally fully compliant, both types of rewards remain at a high level, and subsequent reinforcement learning tends to retain this type of trajectory.

[0091] In multi-sample training scenarios, this embodiment calculates the result reward value and format reward value for multiple target response sequences generated for the same problem, while maintaining the same reward function and the same penalty coefficient configuration to ensure the comparability of rewards across samples. This approach is consistent with the aforementioned training mechanism of "self-search loop completed within a single generation," which can form a closed-loop reward signal without introducing external retrieval supervision.

[0092] Through the technical means of this embodiment, the reward calculation is expanded from a single answer correctness evaluation to a joint evaluation of "answer correctness + structural compliance". This constrains both the quality of the model output content and the quality of the output structure, enabling reinforcement learning updates to obtain a more stable and interpretable optimization direction, thereby improving the consistency of the training process and reducing the interference of invalid trajectories on parameter updates.

[0093] In some embodiments, the response sequence and the total sample reward are input into the reinforcement learning optimization process to update the policy parameters of the language model to be trained. The above process is repeated until the stopping condition is met to obtain the target language model, including:

[0094] A sample-level policy trajectory is constructed based on the response sequence. The policy trajectory includes the sequence state information corresponding to the search query field generation action, the information field generation action, and the answer field generation action.

[0095] The total reward of the sample is allocated to each generated action of the policy trajectory to obtain the action reward signal for policy optimization, and the advantage estimate relative to the reference policy is calculated based on the action reward signal.

[0096] Based on the advantage estimate and the preset policy constraints, a policy optimization objective is constructed, and the parameters of the language model to be trained are updated. The preset policy constraints are used to limit the offset between the updated policy and the original policy.

[0097] The response generation, reward calculation, and parameter update are repeated for each training round, and the target language model is output when the preset stopping condition is met.

[0098] Specifically, firstly, a sample-level policy trajectory is constructed based on the response sequence. For each training sample, the generation process is segmented into actions according to a preset structured interaction format. The generation process of the search query field is defined as the first type of action, the generation process of the information field is defined as the second type of action, and the generation process of the answer field is defined as the third type of action. The sequence state information corresponding to each action is recorded. The sequence state information includes at least the current field context, the previously generated field content, and the current decoding position. Referring to the aforementioned specific example, in... <think> <search> <information> <answer>Under the protocol, the model first forms a query expression, then generates internal knowledge information fragments, and finally outputs the answer. The three generation processes are mapped to continuous action segments in the same sample-level policy trajectory.

[0099] Furthermore, the total sample reward is allocated to each generated action in the policy trajectory to obtain action reward signals, and the advantage estimate relative to the reference policy is calculated. The reward allocation uses a field relevance weighting method: answer field actions, which are strongly correlated with answer correctness, receive higher weights; field boundary and sequence actions, which are strongly correlated with structural compliance, receive structural weights; and information field actions receive both content and structural weights. For each action, the advantage estimate is calculated based on its action reward signal and the baseline reward of the reference policy under the same state, to characterize the improvement of that action relative to the reference policy. This process avoids the coarse credit allocation problem caused by updating the reward solely for the entire sequence, allowing for targeted learning of "whether the query is valid, whether the information is usable, and whether the answer is accurate."

[0100] Furthermore, after obtaining the dominance estimate, a policy optimization objective is constructed based on the dominance estimate and preset policy constraints, and parameter updates are performed. The preset policy constraints control the magnitude of the shift between the old and new policies, preventing excessively large single-round updates from causing training oscillations. In implementation, a constraint range can be set for the policy ratio, or a penalty term can be introduced for the difference in the distribution of the old and new policies, allowing the update to increase the probability of dominant actions while suppressing policy mutations. Considering the training stability requirements, when the external environment is not involved and the reward is completely endogenous, this constraint mechanism can significantly reduce the risk of overfitting a single batch trajectory and maintain convergence consistency across multiple training rounds.

[0101] Furthermore, response generation, reward calculation, and parameter updates are repeated for each training round. Each round uses the same structured interaction protocol to generate a response sequence, and the total sample reward is obtained by fusing the result reward and format reward before proceeding to trajectory optimization. For multiple sampling results of the same question, multiple policy trajectories can be constructed in parallel and updated uniformly to leverage the aforementioned characteristic that "increasing the number of samplings can increase the upper limit of pass@k". Taking an enterprise-level intelligent customer service scenario as an example, in product compatibility question answering, the model often exhibits problems such as insufficient generalization of query fields and redundancy of information fields in the early stages. After multiple rounds of trajectory optimization, the query fields become more focused, the consistency between information fields and answer fields improves, and the stability of the answer increases with each round.

[0102] Finally, the target language model is output when preset stopping conditions are met. Preset stopping conditions may include the development set comprehensive index reaching a threshold, the policy gain being below a threshold for several consecutive rounds, or the training rounds reaching the upper limit. Before output, a consistency check is performed on the final policy to confirm that it meets the deployment requirements in terms of structural compliance rate and answer matching degree, and it is then solidified into the target language model for subsequent online inference processes.

[0103] Through the technical means of this embodiment, the total sample reward can be refined and allocated to three key action segments: search query, information generation, and answer generation. By combining reference strategy advantage estimation and offset constraint update, stable optimization of the internal knowledge retrieval strategy is achieved. Under training conditions that do not rely on external retrieval supervision, the model can still form convergent and transferable search-based reasoning capabilities, providing a parameter basis for continuous reasoning in subsequent real retrieval backfilling scenarios.

[0104] In some embodiments, the method further includes performing candidate answer selection based on multiple self-search sampling results to determine a target response sequence for the reinforcement learning optimization process, specifically including:

[0105] For the same question text, control the language model to be trained to generate multiple response sequence samples according to a preset structured interaction format;

[0106] A sample quality score is calculated for each response sequence sample. The sample quality score is determined based on the result reward value, the format reward value, and the semantic consistency index between the search query field and the information field.

[0107] Based on the sample quality score, multiple response sequence samples are sorted and filtered to determine the target response sequence, and the total reward of the sample corresponding to the target response sequence is input into the reinforcement learning optimization process.

[0108] Specifically, firstly, for the same question text, the language model to be trained is controlled to generate multiple response sequence samples according to a preset structured interaction format. During implementation, a field protocol consistent with the main training process is maintained, and the response sequence must contain at least a search query field, an information field, and an answer field. To obtain a comparable candidate set, a fixed number of sampling rounds is set for the same question, and the sampling temperature and maximum generation length are controlled within a preset range. Referring to the aforementioned training phase example, the model executes an internal closed loop of "generating query—generating information—generating answer" in each sampling, without calling external search services. Using a financial question-and-answer example, the same question may yield multiple response sequences, some of which may have correct answers but lack proper structure, while others may have proper structure but weak correlation between the information field and the query field, requiring subsequent scoring for unified judgment.

[0109] Furthermore, a sample quality score is calculated for each response sequence sample. The sample quality score is jointly determined by the result reward value, the format reward value, and the semantic consistency index between the search query field and the information field. The result reward value reflects the degree of matching between the answer field and the reference answer, the format reward value reflects the completeness, order consistency, and boundary closure of the fields, and the semantic consistency index is used to measure the degree of coupling between the search query field and the information field in terms of semantic theme, avoiding the deviation trajectory of "query focusing on A, information describing B".

[0110] In some examples, semantic consistency can be calculated using keyword coverage, semantic vector similarity, or topic constraint matching, and then fused according to preset weights to form a sample quality score. Using an enterprise-level intelligent customer service example, if a sample's query field is "upgrade compatibility conditions," but the information field mainly describes "pricing strategy," then even if the format is compliant, its semantic consistency index will still be low, and the overall score will be lowered.

[0111] Furthermore, after scoring, multiple response sequence samples are sorted and filtered based on sample quality scores to determine the target response sequence. In the sorting stage, a candidate queue is formed from highest to lowest score; in the filtering stage, either a "Top-1 single-sample selection" or "Top-K multiple-sample retention" strategy can be adopted. For scenarios prioritizing training stability, Top-1 samples are selected to reduce gradient noise; for scenarios prioritizing exploration capability, Top-K samples are selected and participate in updates with normalized weights.

[0112] In some examples, to prevent pattern collapse, diversity constraints can be added to the selection rules to remove duplicates from samples with highly repetitive semantics. After determining the target response sequence, the total reward of the corresponding samples is written into the reinforcement learning optimization input for subsequent advantage estimation and policy parameter updates.

[0113] Furthermore, the selection threshold can be dynamically adjusted during training rounds. In the early stages of training, the threshold is appropriately relaxed to retain exploratory trajectories, while in the later stages, the threshold is gradually increased to enhance the proportion of high-quality trajectories. This dynamic rule, combined with the aforementioned "increasing the upper limit of sampling counts" mechanism, can improve convergence speed while maintaining exploration. Taking medical question answering as an example, retaining multiple approximately correct samples in the early stages helps the model learn a stable structure; focusing on high-scoring samples in the later stages improves answer accuracy and field consistency.

[0114] In terms of engineering implementation, this embodiment can also record the candidate sample score distribution for each question text and use the score distribution for training diagnosis. When a certain type of question consistently exhibits a "high format, low result" distribution, it indicates insufficient internal knowledge retrieval; when a "high result, low consistency" distribution occurs, it indicates that the coupling strategy between query fields and information fields needs to be strengthened. This diagnostic information can be used in reverse to adjust sampling parameters and weight configurations, forming a closed-loop optimization within the training process.

[0115] Through the technical means of this embodiment, quality-aware optimization of multiple self-search sampling results is achieved without relying on external retrieval supervision, transforming the "performance ceiling brought by sampling expansion" into a "stable, high-quality trajectory that can be used for parameter updates." This reduces update noise introduced by low-quality samples, improves the convergence stability of reinforcement learning, enhances the consistency between the search query field and the information field, and further improves the transferability of the target language model in subsequent online inference scenarios.

[0116] In some embodiments, the target language model is deployed in an online inference process. When a search query field is detected, the query content is intercepted and an external search service is invoked to obtain the search results. After writing the search results into the information field, the answer field is generated, including:

[0117] During online inference, real-time field monitoring is performed on the response sequence output by the target language model. When a search query field is identified, the query content is extracted and an external retrieval process is triggered.

[0118] The query content is sent to an external search service, the corresponding search result set is received, and the search result set is processed. The result processing includes one or more of the following: result deduplication, text truncation, or source sorting.

[0119] The categorized search results are written into the information field according to the preset structured interaction format, generating an intermediate response context containing external search information.

[0120] The intermediate response context is injected back into the target language model to continue generating the answer field and obtain the final response result.

[0121] Specifically, firstly, during online inference, real-time field monitoring is performed on the response sequence output by the target language model. When a search query field is identified, the query content is extracted and an external retrieval process is triggered. The system establishes a field monitor for the inference session, performs incremental parsing on the streaming output, and continuously detects the start and end boundaries of the search query field. When a complete search query field is detected, the query content is immediately extracted and the automatic generation path of subsequent information fields is frozen, before proceeding to the retrieval call branch. Referring to the aforementioned enterprise-level intelligent customer service example, when a user asks "What are the interface compatibility conditions after a certain version upgrade?", the model first generates the search query field, the field monitor captures the query phrase, and triggers the retrieval, avoiding the model relying solely on internal memory to provide outdated conclusions.

[0122] Furthermore, the query content is sent to an external search service, the corresponding search result set is received, and result normalization is performed. The external search service can be a general search engine, an enterprise knowledge retrieval service, or a vertical domain search service. To ensure that the backfilled information can be stably consumed by the model, the system normalizes the search result set. Normalization includes at least one or more of the following: result deduplication, text truncation, or source sorting. Result deduplication is used to merge duplicate entries from the same source; text truncation is used to compress excessively long text segments to a preset length window; source sorting is used to rearrange the results based on source credibility, timeliness, or domain matching. Taking a financial Q&A scenario as an example, when multiple identical announcement reprint links are returned, deduplication is performed first, and then regulatory agencies and official disclosure sources are prioritized to reduce noise interference in answer generation.

[0123] Furthermore, after the results are normalized, the normalized search results are written into the information field according to a preset structured interaction format, generating an intermediate response context containing external search information. The writing process maintains the same field boundaries and order as during training, ensuring that the target language model receives input with the same distribution structure. To enhance traceability, source identifiers and fragment numbers can be retained in the information field without changing the main semantics of the field. For example, combined with the unified interaction format migration example, the model still proceeds with inference based on a fixed structure of "search query field—information field—answer field" during deployment; only the content of the information field is replaced by internally generated information from external search results.

[0124] Furthermore, the intermediate response context is input into the target language model to continue generating the answer field, yielding the final response result. At this stage, the model integrates the previously filled information fields with the existing reasoning state to complete the answer and outputs the answer field content. If necessary, answer generation constraints can be set, such as requiring the answer to cite key evidence fragments from the information fields, or limiting the degree to which the answer deviates from the previously filled information. Taking medical consultation Q&A as an example, after the system fills in the latest guideline entries, the model continues to generate answer fields based on this information, reducing bias caused by answering solely based on internal old knowledge.

[0125] Furthermore, this embodiment allows for the configuration of anomaly handling paths: when external searches yield no results or the reliability of the results is below a threshold, the system can fall back to "internal knowledge-based answering + low confidence alert" or trigger a secondary query rewrite, calling the external search service again. This mechanism, together with field monitoring and result normalization, forms an online inference closed loop, ensuring availability under different network and data conditions.

[0126] Through the technical means of this embodiment, the information source replacement was completed while keeping the structured protocol unchanged during the training period, realizing a smooth migration from full simulation training to real retrieval reasoning; at the same time, through real-time field monitoring, retrieval result normalization and intermediate context backfilling, the timeliness and consistency of online answering were improved, and the interference of external noise on the final answer generation was reduced.

[0127] The following are embodiments of the apparatus described in this application, which can be used to execute the embodiments of the method described in this application. For details not disclosed in the apparatus embodiments of this application, please refer to the embodiments of the method described in this application.

[0128] Figure 2 This is a schematic diagram of the structure of the self-search reinforcement learning training device provided in an embodiment of this application. For example... Figure 2 As shown, the self-search reinforcement learning training device includes:

[0129] The acquisition module 201 is used to acquire training samples, which include question texts and corresponding reference answers.

[0130] The constraint module 202 is used to input the question text into the language model to be trained and to constrain the language model to be trained to generate a response sequence according to a preset structured interaction format. The response sequence includes a search query field, an information field, and an answer field. The content of the information field is generated by the language model to be trained based on the knowledge represented by its internal parameters.

[0131] The determination module 203 is used to determine the result reward value based on the matching result of the answer field and the reference answer, and to determine the format reward value based on the satisfaction of the response sequence with the preset structured interaction format;

[0132] Update module 204 is used to determine the total sample reward based on the result reward value and the format reward value, and input the response sequence and the total sample reward into the reinforcement learning optimization process to update the policy parameters of the language model to be trained. The above process is repeated until the stopping condition is met to obtain the target language model.

[0133] The generation module 205 is used to deploy the target language model in the online inference process. When a search query field is detected, the query content is intercepted and an external search service is called to obtain the search results. After writing the search results into the information field, the answer field is generated.

[0134] In some embodiments, Figure 2 The acquisition module 201 acquires the target task corpus and performs sample parsing on the target task corpus to form sample pairs of question text and reference answer; it performs format standardization processing on the sample pairs to unify the input format of question text and the annotation format of reference answer; it performs validity screening on the standardized sample pairs according to the preset training data rules to obtain a training sample set used to drive the language model to be trained to generate response sequences.

[0135] In some embodiments, Figure 2 The constraint module 202 constructs model input prompts based on a preset structured interaction template. The preset structured interaction template is used to limit the generation order and field boundaries of the search query field, information field, and answer field. The question text and the model input prompt are concatenated and encoded before being input into the language model to be trained, triggering the generation of field-level responses according to the preset structured interaction template. During the generation process, structural constraint verification is performed on the response sequence to obtain candidate response sequences that satisfy field integrity and order consistency. The information field content generated by the language model to be trained based on the knowledge represented by internal parameters is retained in the candidate response sequence to form the target response sequence for subsequent reward calculation.

[0136] In some embodiments, Figure 2 The determination module 203 performs field-level parsing on the target response sequence, locates the target answer fragment corresponding to the answer field, and performs normalization processing on the target answer fragment; it matches the normalized target answer fragment with the reference answer to obtain the answer matching degree index, and determines the result reward value according to the mapping relationship between the answer matching degree index and the preset result reward function; it performs structural compliance verification on the target response sequence, including field completeness verification, field order consistency verification, and field boundary closure verification, to obtain the format compliance index; it determines the format reward value according to the mapping relationship between the format compliance index and the preset format reward function, wherein the preset format reward function configures differentiated penalty coefficients for field missing, field out-of-order, or field boundary abnormalities.

[0137] In some embodiments, Figure 2 The update module 204 constructs a sample-level policy trajectory based on the response sequence. The policy trajectory includes the sequence state information corresponding to the search query field generation action, the information field generation action, and the answer field generation action. The total sample reward is allocated to each generation action of the policy trajectory to obtain the action reward signal for policy optimization, and the advantage estimate relative to the reference policy is calculated based on the action reward signal. The policy optimization objective is constructed according to the advantage estimate and the preset policy constraint term, and the parameters of the language model to be trained are updated. The preset policy constraint term is used to limit the offset of the updated policy from the original policy. The response generation, reward calculation, and parameter update are repeatedly executed according to the training rounds, and the target language model is output when the preset stopping condition is reached.

[0138] In some embodiments, Figure 2 The update module 204 is also used to perform candidate answer selection based on multiple self-search sampling results to determine the target response sequence for the reinforcement learning optimization process. Specifically, it includes: for the same question text, controlling the language model to be trained to generate multiple response sequence samples according to a preset structured interaction format; calculating the sample quality score for each response sequence sample, wherein the sample quality score is determined based on the result reward value, the format reward value, and the semantic consistency index between the search query field and the information field; sorting and filtering the multiple response sequence samples according to the sample quality score to determine the target response sequence, and inputting the total sample reward corresponding to the target response sequence into the reinforcement learning optimization process.

[0139] In some embodiments, Figure 2 The generation module 205 performs real-time field monitoring on the response sequence output by the target language model during online inference. When a search query field is identified, the query content is extracted and an external retrieval process is triggered. The query content is sent to an external search service, the corresponding retrieval result set is received, and the retrieval result set is processed by result normalization, which includes one or more of the following: result deduplication, text truncation, or source sorting. The normalized retrieval results are written into the information field according to a preset structured interaction format to generate an intermediate response context containing external retrieval information. The intermediate response context is injected back into the target language model to continue generating the answer field and obtain the final response result.

[0140] It should be understood that the sequence number of each step in the above embodiments does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application.

[0141] Figure 3 This is a schematic diagram of the electronic device 3 provided in an embodiment of this application. Figure 3 As shown, the electronic device 3 of this embodiment includes: a processor 301, a memory 302, and a computer program 303 stored in the memory 302 and executable on the processor 301. When the processor 301 executes the computer program 303, it implements the steps in the various method embodiments described above. Alternatively, when the processor 301 executes the computer program 303, it implements the functions of each module / unit in the various device embodiments described above.

[0142] Electronic device 3 can be a desktop computer, laptop, handheld computer, cloud server, or other electronic device. Electronic device 3 may include, but is not limited to, processor 301 and memory 302. Those skilled in the art will understand that... Figure 3 This is merely an example of electronic device 3 and does not constitute a limitation on electronic device 3. It may include more or fewer components than shown, or different components.

[0143] The processor 301 may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc.

[0144] The memory 302 can be an internal storage unit of the electronic device 3, such as a hard disk or memory of the electronic device 3. The memory 302 can also be an external storage device of the electronic device 3, such as a plug-in hard disk, smart media card (SMC), secure digital (SD) card, flash card, etc., equipped on the electronic device 3. The memory 302 can also include both internal and external storage units of the electronic device 3. The memory 302 is used to store computer programs and other programs and data required by the electronic device.

[0145] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the above-described division of functional units and modules is merely an example. In practical applications, the above functions can be assigned to different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiments can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0146] If integrated modules / units are implemented as software functional units and sold or used as independent products, they can be stored in a readable storage medium (e.g., a computer-readable storage medium). Based on this understanding, all or part of the processes in the methods of the above embodiments can also be implemented by a computer program instructing related hardware. The computer program can be stored in a computer-readable storage medium, and when executed by a processor, it can implement the steps of the various method embodiments described above. The computer program may include computer program code, which may be in the form of source code, object code, executable files, or certain intermediate forms. The computer-readable storage medium may include: any entity or device capable of carrying computer program code, recording media, USB flash drives, portable hard drives, magnetic disks, optical disks, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signals, telecommunication signals, and software distribution media, etc.

[0147] The above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application, and should all be included within the protection scope of this application.< / answer> < / information> < / search> < / think> < / answer> < / information> < / search> < / answer> < / information> < / search> < / think> < / answer> < / information> < / search> < / think> < / information> < / search>

Claims

1. A self-search reinforcement learning training method, characterized in that, include: Obtain training samples, which include question texts and corresponding reference answers; The question text is input into the language model to be trained, and the language model to be trained is constrained to generate a response sequence according to a preset structured interaction format. The response sequence includes a search query field, an information field, and an answer field. The content of the information field is generated by the language model to be trained based on the knowledge represented by its internal parameters. The result reward value is determined based on the matching result between the answer field and the reference answer, and the format reward value is determined based on the satisfaction of the response sequence with the preset structured interaction format; The total reward for the sample is determined based on the result reward value and the format reward value. The response sequence and the total reward for the sample are then input into the reinforcement learning optimization process to update the policy parameters of the language model to be trained. The above process is repeated until the stopping condition is met to obtain the target language model. The target language model is deployed in an online inference process. When a search query field is detected, the query content is intercepted and an external search service is called to obtain the search results. The search results are written into the information field and then the answer field is generated. The step of inputting the question text into the language model to be trained and constraining the language model to generate a response sequence according to a preset structured interaction format includes: The model input prompts are constructed based on a preset structured interaction template, which is used to limit the generation order and field boundaries of the search query field, information field and answer field; The question text and the model input prompt are concatenated and encoded, and then input into the language model to be trained to trigger the generation of field-level responses according to the preset structured interaction template; During the generation process, structural constraint verification is performed on the response sequence to obtain candidate response sequences that satisfy field integrity and order consistency. The information field content generated by the language model to be trained based on the knowledge represented by the internal parameters is retained in the candidate response sequence to form the target response sequence for subsequent reward calculation.

2. The method according to claim 1, characterized in that, The process of obtaining training samples, which include question text and corresponding reference answers, includes: Obtain the target task corpus and perform sample parsing on the target task corpus to form sample pairs of question text and reference answers; The sample pairs were subjected to format standardization processing to unify the format of question text input and reference answer annotation; The normalized sample pairs are filtered for validity according to the preset training data rules to obtain a training sample set used to drive the language model to be trained to generate the response sequence.

3. The method according to claim 1, characterized in that, The step of determining the result reward value based on the matching result of the answer field and the reference answer, and determining the format reward value based on the satisfaction of the response sequence with the preset structured interaction format, includes: The target response sequence is parsed at the field level to locate the target answer fragment corresponding to the answer field, and the target answer fragment is then normalized. The standardized target answer fragment is matched with the reference answer to obtain the answer matching degree index, and the result reward value is determined according to the mapping relationship between the answer matching degree index and the preset result reward function. Perform structural compliance checks on the target response sequence. The structural compliance checks include field completeness checks, field order consistency checks, and field boundary closure checks to obtain a format compliance index. The format reward value is determined based on the mapping relationship between the format compliance index and the preset format reward function, wherein the preset format reward function configures differentiated penalty coefficients for field missing, field out-of-order, or field boundary abnormalities.

4. The method according to claim 1, characterized in that, The step of inputting the response sequence and the total sample reward into the reinforcement learning optimization process to update the policy parameters of the language model to be trained, and repeating the above process until the stopping condition is met to obtain the target language model, includes: Based on the response sequence, a sample-level strategy trajectory is constructed, which includes sequence state information corresponding to the search query field generation action, the information field generation action, and the answer field generation action. The total reward of the sample is allocated to each generated action of the policy trajectory to obtain the action reward signal for policy optimization, and the advantage estimate relative to the reference policy is calculated based on the action reward signal. Based on the advantage estimate and the preset policy constraint, a policy optimization objective is constructed, and the parameters of the language model to be trained are updated. The preset policy constraint is used to limit the offset between the updated policy and the original policy. The response generation, reward calculation, and parameter update are repeated in each training round, and the target language model is output when a preset stopping condition is met.

5. The method according to claim 4, characterized in that, The method further includes performing candidate answer selection based on multiple self-search sampling results to determine the target response sequence for the reinforcement learning optimization process, specifically including: For the same question text, the language model to be trained is controlled to generate multiple response sequence samples according to the preset structured interaction format; A sample quality score is calculated for each response sequence sample, wherein the sample quality score is determined based on the result reward value, the format reward value, and the semantic consistency index between the search query field and the information field; Based on the sample quality score, multiple response sequence samples are sorted and filtered to determine the target response sequence, and the total reward of the sample corresponding to the target response sequence is input into the reinforcement learning optimization process.

6. The method according to claim 1, characterized in that, The process of deploying the target language model in an online inference flow, intercepting the query content when a search query field is detected, calling an external search service to obtain the search results, writing the search results into the information field, and then generating the answer field includes: During online inference, real-time field monitoring is performed on the response sequence output by the target language model. When a search query field is identified, the query content is extracted and an external retrieval process is triggered. The query content is sent to an external search service, the corresponding search result set is received, and the search result set is processed by result normalization, which includes one or more of the following: result deduplication, text truncation, or source sorting. The rectified search results are written into the information field according to the preset structured interaction format, generating an intermediate response context containing external search information. The intermediate response context is injected back into the target language model to continue generating the answer field and obtain the final response result.

7. A self-searching reinforcement learning training device, characterized in that, include: The acquisition module is used to acquire training samples, which include question text and corresponding reference answers; The constraint module is used to input the question text into the language model to be trained and constrain the language model to generate a response sequence according to a preset structured interaction format. The response sequence includes a search query field, an information field, and an answer field. The content of the information field is generated by the language model to be trained based on the knowledge represented by its internal parameters. The determination module is used to determine the result reward value based on the matching result between the answer field and the reference answer, and to determine the format reward value based on the satisfaction of the response sequence with the preset structured interaction format; The update module is used to determine the total sample reward based on the result reward value and the format reward value, and input the response sequence and the total sample reward into the reinforcement learning optimization process to update the policy parameters of the language model to be trained. The above process is repeated until the stopping condition is met to obtain the target language model. The generation module is used to deploy the target language model in the online inference process. When a search query field is detected, the query content is intercepted and an external search service is called to obtain the search results. The search results are written into the information field and then the answer field is generated. The constraint module is used to construct model input prompts based on a preset structured interaction template. The preset structured interaction template is used to limit the generation order and field boundaries of search query fields, information fields, and answer fields. The question text and the model input prompts are concatenated and encoded before being input into the language model to be trained to trigger the generation of field-level responses according to the preset structured interaction template. During the generation process, structural constraint verification is performed on the response sequence to obtain candidate response sequences that satisfy field integrity and order consistency. The information field content generated by the language model to be trained based on the knowledge represented by internal parameters is retained in the candidate response sequence to form a target response sequence for subsequent reward calculation.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and running on the processor, characterized in that, When the processor executes the computer program, it implements the steps of the method as described in any one of claims 1 to 6.

9. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the method as described in any one of claims 1 to 6.

Images