Language model training method, language model and research report generation method

Abstract

The invention provides a language model training method, a language model and a research report generation method, and belongs to the technical field of artificial intelligence, and the method comprises the steps: carrying out the cold start fine tuning of a model based on a reference trajectory containing an explicit action label, and constructing a single-model multi-role circulation architecture; performing search strengthening training based on the standard answer label, and optimizing the information extraction capability; and writing intensive training is performed based on the multi-dimensional quality score, so that the long text generation quality is improved. According to the method, multi-agent interaction logic is creatively internalized into context reasoning of a single model, through a staged course learning strategy, the problems that in a traditional method, retrieval and writing capacity are separated, and long text generation lacks an effective supervision signal are solved, model illusion is effectively restrained, and the learning efficiency is improved. And the logic depth, the evidence traceability and the content credibility of the deep research report are remarkably improved.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence technology, and in particular to a language model training method, a language model, and a method for generating research reports. Background Technology

[0002] With the improvement of the capabilities of Large Language Models (LLMs), the demand for in-depth research applications oriented towards retrieval, analysis, and writing is growing. These applications require models to be able to autonomously plan tasks, conduct multiple rounds of information retrieval and evidence reading comprehension for open and complex problems, just like human researchers, and ultimately write long research reports that are logically rigorous, detailed, and well-supported by evidence.

[0003] To meet these needs, existing technologies typically employ Retrieval Augmented Generation (RAG) or Multi-Agent System (MAS) approaches. RAG typically retrieves relevant documents and directly appends them to prompts to generate a response in one step. MAS, on the other hand, deploys multiple models with independent functions (such as planning, searching, and writing models), orchestrating the process through message queues or middleware, with each model working collaboratively to complete complex tasks. For model training, Supervised Fine Tuning (SFT) or Reinforcement Learning from Human Feedback (RLHF) are commonly used to enhance the model's general dialogue capabilities.

[0004] However, the aforementioned existing technical solutions still have limitations in practical applications. On the one hand, traditional multi-agent systems rely on the engineered combination of multiple independent models, resulting in high system deployment resource consumption, long inference chains, and high latency. Furthermore, due to the independent parameters of each model, it is difficult to perform end-to-end joint optimization of the entire process from retrieval to writing through a unified gradient descent. On the other hand, under the existing training paradigm, it is often difficult to simultaneously achieve both the model's retrieval capability and its ability to write long articles: simple retrieval enhancement training is insufficient for the model to learn when to stop retrieval or how to distinguish truth from falsehood in complex scenarios; while training for long article generation lacks refined quality supervision signals, leading to problems such as loose logical structure, lack of evidence to support core arguments, or rote application of retrieved content in the generated long reports, making it difficult to produce high-quality in-depth research results. Summary of the Invention

[0005] This invention provides a language model training method, a language model, and a research report generation method to address the shortcomings of existing technologies, such as system complexity and difficulty in end-to-end optimization due to reliance on multi-model engineering combinations, and the difficulty in simultaneously achieving retrieval and writing capabilities and low report generation quality due to conflicting training objectives and lack of long-text quality supervision signals. It realizes the unified construction of multi-role capabilities within a single model and significantly improves the logical depth and content credibility of in-depth research reports.

[0006] This invention provides a language model training method, comprising: Obtain a cold start training dataset, which contains multiple multi-agent interaction samples. Each multi-agent interaction sample includes a query input and a corresponding reference trajectory. The reference trajectory contains explicit action labels for identifying the behavior of different roles. Using the query input as the model input and the reference trajectory as the supervision label, supervised fine-tuning is performed on the initial large language model to obtain a pre-trained model. Obtain a search task training dataset, which contains multiple question-and-answer samples, each including a question input and a standard answer label; use the question input as the model input and the standard answer label as the first feedback signal to train the pre-trained model to obtain a search optimization model; A writing task training dataset is obtained, which contains multiple complex query samples; the complex query samples are used as model input, and the scoring results determined for the long article report generated by the model are used as the second feedback signal to train the search optimization model to obtain the target language model.

[0007] According to a language model training method provided by the present invention, obtaining the cold start training dataset includes: Retrieve multi-round interaction records for any given query request; Extract the task planning content, search query instructions, search results content, and final reply content from the interaction records; Add a planning tag before the task planning content, a search tag before the search query command, an observation tag before the search results content, and a writing tag before the final response content; Based on the chronological order of the interaction records, the tagged parts are pieced together to form the reference trajectory.

[0008] According to a language model training method provided by the present invention, the step of concatenating the labeled parts into the reference trajectory includes: In the reference trajectory, the planning label is placed before the search label; The search tag is followed immediately by the observation tag to form a search-observation pair; The writing tag is appended after the search observation pair; The reference trajectory contains at least one of the search observation pairs.

[0009] According to a language model training method provided by the present invention, the final response content extracted from the interaction record includes the extraction of an article outline and a draft of the main text; The step of stitching together the tagged parts to form the reference trajectory also includes: Add an outline tag before the article outline; Add a draft tag before the main text draft; In the reference trajectory, the article outline with the outline label is placed before the main text draft with the draft label.

[0010] According to a language model training method provided by the present invention, training the pre-trained model to obtain a search optimization model includes: The problem input is injected into the pre-trained model to generate a search prediction sequence containing search query instructions and search results. Identify the search results content that is wrapped by the observation tag in the search prediction sequence; When calculating the policy gradient used to update model parameters, the marker positions corresponding to the search result content are masked, and the gradient is calculated only based on the portion of the search prediction sequence other than the search result content.

[0011] According to a language model training method provided by the present invention, training the search optimization model further includes: When the model generates the writing tags and subsequent long article prediction results, it detects whether the long article prediction results contain citation anchors. Verify whether the reference anchor points to the search results content within the observed tag; If the long text prediction result does not contain the reference anchor or the reference anchor does not point to the search result content, the value of the second feedback signal is reduced.

[0012] According to a language model training method provided by the present invention, the step of using the standard answer label as the first feedback signal includes: Extract the final answer fragment from the search prediction results generated by the pre-trained model; Calculate the exact match score or F1 score between the final answer fragment and the standard answer label; The exact match score or the F1 score is determined as the first feedback signal.

[0013] According to a language model training method provided by the present invention, the step of using the scoring result determined from the long text report generated by the model as a second feedback signal includes: For the same complex query sample, the search optimization model is used to sample and generate multiple candidate long-text reports; The complex query sample and the candidate long text report are input into the preset evaluation model; Using the evaluation model, a relative advantage score is generated for the candidate long article report based on preset scoring rules; The relative advantage score is used as the second feedback signal.

[0014] According to a language model training method provided by the present invention, generating a relative advantage score for the candidate long text report includes: Select one of all the candidate long article reports as the baseline report, and use the rest as comparison reports; The benchmark report and the comparison report are simultaneously input into the evaluation model for pairwise comparison to determine the probability of the comparison report winning relative to the benchmark report; The winning probability is normalized to obtain the relative advantage score.

[0015] According to a language model training method provided by the present invention, the preset scoring rules cover at least one of the following evaluation dimensions: content comprehensiveness, logical depth, instruction compliance, and text readability; The comprehensiveness of the content is used to characterize whether the candidate long text report covers all sub-questions of the complex query sample; The logical depth is used to characterize whether the candidate long-text report contains inferential analysis or trend prediction. The instruction compliance degree is used to characterize whether the candidate long text report meets the preset format requirements; The text readability is used to characterize the language fluency and paragraph structure of the candidate long text report.

[0016] According to a language model training method provided by the present invention, training the search optimization model includes iteratively executing the following training steps until a preset cutoff condition is reached: For the same complex query sample, the search optimization model is controlled to sample and generate a sequence of long-text prediction results; Calculate the second feedback signal corresponding to each long text prediction result in the long text prediction result sequence; Calculate the average of all the second feedback signals as the baseline; The difference between each of the second feedback signals and the baseline is calculated to obtain the advantage value; The model parameters of the search optimization model are updated using the advantage value.

[0017] The present invention also provides a language model, which is trained based on any of the above-mentioned language model training methods; The language model includes multiple functional role modules that share the same set of model parameters: The planning module is used to receive query input and generate planning tags and task breakdown information; The search interaction module is used to generate search tags and query terms based on the planning tags, and to receive external search results encapsulated in the observation tags; The writing generation module is used to generate a long report containing writing tags and citation anchors based on the external search results within the observation tags. The planning tag, the search tag, the observation tag, and the writing tag together constitute an explicit action tag sequence. The planning module, the search interaction module, and the writing generation module achieve end-to-end data interaction within a single context window through the flow of the explicit action tag sequence.

[0018] According to a language model provided by the present invention, it further includes: The criticism feedback module is configured to review the long article report generated by the writing generation module and generate criticism tags and modification suggestions. The writing generation module is also configured to respond to the criticism tags and regenerate a revised long article report based on the modification suggestions.

[0019] According to a language model provided by the present invention, the search interaction module is further configured as follows: Within the context window, the search tags are generated multiple times consecutively, and multiple observation tags are received accordingly, forming an alternating search observation sequence; The writing generation module is configured to perform a comprehensive analysis based on all search result data in the search observation sequence.

[0020] This invention also provides a method for generating research reports, comprising the following modules: Receive natural language query requests from users; Call the target language model trained by any of the above language model training methods; Respond to the query request in a single context window; Obtain the explicit action tag stream output by the target language model, wherein the explicit action tag stream includes sequentially generated planning tags, search tags, observation tags, and writing tags; In response to the search tag, an external search engine is invoked to obtain the search results, and the search results are encapsulated within the observation tag and fed back to the target language model; Extract the long text report generated by the target language model after the writing tag, and output it as the research report.

[0021] According to a research report generation method provided by the present invention, the long report includes citation anchors, and the research report generation method further includes: Parse the reference anchor and establish a hyperlink mapping between the reference anchor and the source of the search result within the observation tag; When outputting the research report, the hyperlink mapping is provided to support source tracing.

[0022] The present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the language model training method described above.

[0023] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the language model training method as described above.

[0024] The present invention also provides a computer program product, including a computer program that, when executed by a processor, implements the language model training method as described above.

[0025] The language model training method, language model, and research report generation method provided by this invention innovatively internalize the multi-agent interaction logic into the contextual reasoning of a single model. Through a phased course learning strategy, it solves the problems of the separation between retrieval and writing abilities and the lack of effective supervision signals in the generation of long articles in traditional methods. It effectively suppresses model illusion and significantly improves the logical depth, traceability of evidence, and credibility of in-depth research reports. Attached Figure Description

[0026] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0027] Figure 1 This is a flowchart illustrating the language model training method provided in an embodiment of the present invention.

[0028] Figure 2This is a schematic diagram of the training process for the three-stage course learning provided in this embodiment of the invention.

[0029] Figure 3 This is a schematic diagram comparing the architecture of tool-integrated reasoning in existing technologies with the single-model multi-role chain reasoning provided in this embodiment of the invention.

[0030] Figure 4 This is a schematic diagram illustrating the specific process of obtaining a cold start training dataset provided in an embodiment of the present invention.

[0031] Figure 5 This is a schematic diagram of the process of model training using observation masking technology in the search enhancement step provided in the embodiment of the present invention.

[0032] Figure 6 This is a schematic diagram of the feedback adjustment process using the evidence anchor binding mechanism in the writing enhancement step provided in the embodiments of the present invention.

[0033] Figure 7 This is a schematic diagram of the process for generating the second feedback signal in the writing enhancement step provided in an embodiment of the present invention.

[0034] Figure 8 This is a schematic diagram of the process for generating relative advantage scores based on pairwise comparisons provided in an embodiment of the present invention.

[0035] Figure 9 This is a schematic diagram of the process for updating the search optimization model based on the group relative strategy optimization algorithm provided in an embodiment of the present invention.

[0036] Figure 10 This is a schematic diagram illustrating the data flow principle of writing reinforcement training that combines external search and rating feedback, provided in an embodiment of the present invention.

[0037] Figure 11 This is a schematic diagram of the logical architecture of a language model and the interaction of various functional role modules in a single context window, provided by an embodiment of the present invention.

[0038] Figure 12 This is a flowchart illustrating the research report generation method provided in this embodiment of the invention.

[0039] Figure 13 This is a schematic diagram of the structure of the electronic device provided by the present invention. Detailed Implementation

[0040] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0041] It should be noted that, in the description of this invention, the terms "comprising," "including," or any other variations thereof are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element. Those skilled in the art will understand the specific meaning of the above terms in this invention according to the specific circumstances.

[0042] The terms "first," "second," etc., used in this invention are used to distinguish similar objects and not to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that embodiments of the invention can be implemented in orders other than those illustrated or described herein, and the objects distinguished by "first," "second," etc., are generally of the same class and the number of objects is not limited; for example, a first object can be one or more.

[0043] It should be noted that all actions involving the acquisition of information or data in this invention are carried out in accordance with the relevant data protection laws and policies of the country where the invention is located, and with the authorization granted by the owner of the relevant device, such as the construction of various training datasets.

[0044] The following is combined Figures 1-13 This invention describes the language model training method, language model, and research report generation method provided by the present invention.

[0045] The language model training method provided in this embodiment of the invention can be executed by an electronic device, such as a data processing server, model training cluster or high-performance workstation deployed in the cloud. The electronic device typically includes at least one processor, a memory and a communication bus for connecting the two.

[0046] Figure 1 This is a flowchart illustrating the language model training method provided in an embodiment of the present invention, as shown below. Figure 1 As shown, including but not limited to the following steps: Step 1: Obtain the cold start training dataset, which contains multiple multi-agent interaction samples. Each multi-agent interaction sample includes a query input and a corresponding reference trajectory. The reference trajectory contains explicit action labels for identifying the behavior of different roles. Using the query input as the model input and the reference trajectory as the supervision label, the initial large language model is fine-tuned under supervision to obtain the pre-trained model.

[0047] In this embodiment, to enable a single large language model to possess the multiple capabilities required for handling complex tasks, such as planning, interaction, and content generation, a dataset specifically designed for the cold start phase is first constructed. This cold start training dataset refers to the initial teaching data used to impart basic behavioral patterns to the model before it acquires specific domain-specific capabilities. The multi-agent interaction samples in this cold start training dataset are not ordinary question-and-answer pairs, but rather records of real or simulated multi-agent collaborative processes.

[0048] Specifically, each multi-agent interaction sample comprises two parts: query input and reference trajectory. The query input is typically the initial question or task request posed by the user, while the reference trajectory is a complete record of the processing of that query input. Crucially, each reference trajectory not only contains text content but also embeds specific explicit action tags that identify the behaviors of different roles. These explicit action tags act as instructions, logically dividing a continuous stream of text into different functional blocks, such as distinguishing where the model's thinking and planning occurs, where the generated search instructions are, and where the externally returned results are.

[0049] During cold start training, the electronic device uses the query input as the input (Prompt) to the initial large language model (i.e., the base model not trained for this specific task), and the reference trajectory containing explicit action labels as the supervision label. During supervised fine-tuning (SFT), the large language model learns to predict not only the final answer, but also each intermediate step in the processing and its corresponding explicit action labels. The loss function (such as cross-entropy loss) between the final generated content of the large language model and the reference trajectory is calculated, and gradient descent is used to update the model parameters. In this way, the initial large language model gradually learns how to output specific action labels to switch its role or function while generating text. The model obtained after this training stage is the pre-trained model, which has initially acquired the basic ability to simulate multi-role interaction within a single model.

[0050] This invention effectively internalizes the complex interaction logic in a multi-agent system into the context prediction capability of a single model, avoiding the resource consumption and engineering complexity caused by deploying multiple independent models, and laying a structured behavioral foundation for subsequent reinforcement learning.

[0051] Step 2: Obtain the search task training dataset, which contains multiple question-and-answer samples. Each question-and-answer sample includes a question input and a standard answer label. Use the question input as the model input and the standard answer label as the first feedback signal to train the pre-trained model and obtain the search optimization model.

[0052] After the cold start pre-training in step 1, and having acquired basic role-switching capabilities, it is necessary to further improve the accuracy of the information acquired by the pre-trained model. To this end, this embodiment introduces a search task training dataset to enhance the pre-trained model's ability to acquire correct information using tools. The question-and-answer samples contained therein typically contain factually clear questions. Each sample consists of a question input and a corresponding standard answer label (i.e., the correct answer).

[0053] In this search reinforcement training phase, the electronic device injects the question input into the pre-trained model obtained in step 1, allowing the pre-trained model to generate an answer that includes the search action. Unlike the forced imitation in the cold start phase, this phase can adopt the reinforcement learning (RL) paradigm. The answer generated by the pre-trained model is not required to be completely consistent with a fixed trajectory, but rather the focus is on whether its final result is correct.

[0054] This embodiment compares the model-generated answer with the standard answer label and generates a first feedback signal based on the comparison result. This first feedback signal can be a scalar reward value used to measure the accuracy of the content generated by the model. Based on this first feedback signal, the model parameters are updated using a reinforcement learning algorithm, thereby obtaining a search optimization model.

[0055] This embodiment enhances the pre-trained model's ability not only to search, but also to search correctly through feedback training oriented towards result accuracy. It learns to generate more precise search instructions and extract key information from search results, thereby significantly improving the effectiveness of information retrieval.

[0056] Step 3: Obtain the writing task training dataset, which contains multiple complex query samples; use the complex query samples as model input, and use the scoring results determined for the long article report generated by the model as the second feedback signal to train the search optimization model to obtain the target language model.

[0057] After confirming that the search optimization model obtained through search reinforcement training possesses accurate search capabilities, the ultimate goal is to generate high-quality in-depth research reports. To this end, this embodiment uses a writing task training dataset, where complex query samples typically involve open-ended, multi-dimensional analytical questions that often lack a single, definitive answer.

[0058] For such complex query samples, the electronic device inputs the complex query sample into the search optimization model obtained in step 2. The model will comprehensively utilize its planning, search and writing capabilities to generate a long report.

[0059] Since a standard answer is lacking, this embodiment introduces a quality assessment-based feedback mechanism. By performing a multi-dimensional quality assessment on the generated long-form report, a score is determined. This score serves as a second feedback signal, guiding further model optimization. Through this training phase, the model gradually learns how to organize the structure, conduct in-depth analysis, and meet complex user needs when generating long-form reports, ultimately resulting in a comprehensive target language model.

[0060] This invention innovatively internalizes the multi-agent interaction logic into the contextual reasoning of a single model, and solves the problems of the separation between retrieval and writing abilities and the lack of effective supervision signals in the generation of long articles in traditional methods through a phased course learning strategy. It effectively suppresses model illusion and significantly improves the logical depth, traceability of evidence and credibility of in-depth research reports.

[0061] To further illustrate the overall process of the training method and the constructed model architecture in the embodiments of the present invention, the following is combined with... Figure 2 and Figure 3 This paper will take the training of an in-depth research model for the financial industry as an example to illustrate the point in detail.

[0062] Figure 2 This is a schematic diagram of the training process for the three-stage course learning provided in this embodiment of the invention, as shown below. Figure 2 As shown, in step one, the cold start (supervised fine-tuning) phase, a large amount of expert-level multi-agent interaction data is prepared as the cold start training dataset. For example, a certain cold start training dataset records the entire process of how the expert system first plans to "check revenue" and "check profit" for the query input "analyze Company A's financial report," then executes the search separately, and finally summarizes and writes the results, serving as a reference trajectory. These reference trajectories contain clear explicit action labels. By supervising and fine-tuning the initial large language model, it learns this structured behavioral pattern, resulting in a pre-trained model.

[0063] The process then moves to step two, the search reinforcement (search capability training) phase. This involves using a search task training set. For example, one training sample might record the input question, "What was Company A's revenue in 2023?", along with the standard answer label, "10 billion." The pre-trained model attempts to answer these input questions, comparing its generated answers with the standard answer label and calculating the matching degree (e.g., F1 score) as the first feedback signal. Through reinforcement learning, the model maintains its ability to switch roles while significantly improving the accuracy of search commands and the precision of information extraction, evolving into a search optimization model.

[0064] Finally, the third step, writing reinforcement (long-text generation training), is executed. The input is a writing task training set, such as the complex query sample of "writing an investment value analysis report for Company A". Since the long-form report generated by the search optimization model has no standard answer, the system introduces a multi-dimensional scoring mechanism (covering comprehensiveness, depth, etc.) to generate a second feedback signal. Based on this second feedback signal, reinforcement training ultimately yields a target language model capable of writing high-quality reports.

[0065] Figure 3 This is a schematic diagram comparing the architecture of tool-integrated reasoning in existing technologies with the single-model multi-role chain reasoning provided in this embodiment of the invention, as shown in the diagram. Figure 3 As shown, after the above three stages of training, the model internally forms the following... Figure 3 The lower half shows a single-model, multi-role chain architecture.

[0066] Compared with existing technologies ( Figure 3 Unlike the first half, which requires calling multiple independent components such as "thinking model", "search model", "web crawler", "code model" and "question-answering model", the target language model of this invention can independently complete the entire process within a single context window.

[0067] Taking the analysis of Company A's financial statements as an example, combined with Figure 3 As shown, the model first generates a planning label (“Planning” node), outputting “Step 1: Query revenue; Step 2: Query profit…”; then, the model automatically generates a search label (“Tools” node), outputting the query term “Company A 2023 revenue…”; when the external system returns the results, the model encapsulates them in the observation label (feedback after the “Tools / Crawling” node); based on the observed information, the model may reflect (“Reflection” node) or optimize the planning (“Planning: Optimization” node); finally, the model generates a writing label (“Answer” node), outputting a final analysis report containing data support.

[0068] This architecture not only achieves logical interpretability through explicit labeling, but more importantly, because all roles are implemented through token flow within the same model parameters, it can be used... Figure 2 The unified training process shown is jointly optimized end-to-end, thereby significantly improving the efficiency and quality of deep research tasks.

[0069] To further illustrate how to construct a high-quality cold-start training dataset to support supervised fine-tuning of the model, the following combines... Figure 4 The specific implementation process for obtaining the cold start training dataset is described in detail.

[0070] Figure 4 This is a schematic diagram illustrating the specific process of obtaining the cold start training dataset provided in an embodiment of the present invention, as follows: Figure 4 As shown, the main steps include: Step 11: Obtain the multi-round interaction records for any query request.

[0071] First, raw interaction data can be collected from existing multi-agent systems or expert systems. This interaction data originates from the processing of any query request, such as analyzing market trends in the new energy vehicle industry. In this process, multiple specialized agents, such as planning agents, search agents, and analysis agents, collaborate to generate a series of multi-turn interaction records. These interaction records not only contain the final answer but also fully preserve the thought process, tool call history, and intermediate results when responding to the query request. This real-world interaction data serves as the material for training a single model to simulate multi-agent behavior.

[0072] Step 12: Extract the task planning content, search query instructions, search results content, and final reply content from the interaction record.

[0073] Because the original interaction logs may contain a large amount of redundant system logs or unstructured information, they need to be cleaned and structured extracted. Electronic devices parse these interaction logs to separate the core components, which mainly include: (1) Task planning content refers to the list of sub-tasks or execution plan after breaking down a complex problem.

[0074] (2) Search query instructions, which are specific keywords sent to external search engines in order to obtain information.

[0075] (3) Search results content, i.e., webpage summaries or data fragments returned by external search engines.

[0076] (4) The final response content, i.e. the final answer or report generated based on all the information.

[0077] Step 13: Add a planning tag before the task planning content, a search tag before the search query command, an observation tag before the search results content, and a writing tag before the final response content.

[0078] To transform multi-agent interaction logic into a single-model learnable pattern, enabling a single large language model to distinguish different content types and perceive role switching, this embodiment inserts specific explicit action tags before extracting certain content in step 12. Specifically, a planning tag is inserted before the task planning content, for example... <plan>To indicate that this is the planning phase; insert search tags before search query commands, for example... <search>To identify this as a tool invocation action; insert observation tags before the search results content, such as <observation>To indicate that this is feedback from the external environment; insert writing tags before the final response, such as <response>This indicates that this is the content generation stage. These tags act as functional triggers within the context.

[0079] Step 14: According to the time sequence of the interaction records, the content after adding tags is spliced ​​together to form the reference trajectory.

[0080] Finally, based on the chronological order of the actions in the original interaction log, the tagged parts can be pieced together to form a complete text sequence, i.e., the reference trajectory.

[0081] For example, a typical reference trajectory might take the following form: <plan> ... <search> ... <observation> ... <response>..." This reference trajectory not only preserves the logical coherence of the original data, but also enhances the structural information through explicit labels, making it directly usable as a target label for supervised fine-tuning, used to train the model to predict the next action or content.

[0082] This embodiment not only transforms discrete multi-agent interaction logs into structured reference trajectories, but more importantly, it internalizes the complex inter-process communication within a multi-agent system into contextual prediction within a single model by introducing an explicit action label system encompassing planning, searching, observation, and writing labels. This explicit label-based data construction method breaks down the physical barriers of traditional multi-model architectures, enabling a single model to smoothly switch between different roles within a unified parameter space simply by predicting different labels, significantly reducing resource consumption and inference latency in system deployment. Furthermore, these explicit labels serve as strong supervisory signals, guiding the model to learn a clear "think-act-observe" thought chain. This allows the model to move beyond black-box direct generation when facing complex problems, instead exhibiting an interpretable and interventionable step-by-step reasoning process, much like a human expert, thereby significantly improving the success rate and logical rigor of deep research tasks.

[0083] Based on the above embodiments, in order to ensure that the model can learn a deep research paradigm that conforms to human cognitive logic, as an optional embodiment, the specific logic of splicing the labeled parts into the reference trajectory in step 14 is further defined, specifically including: In the reference trajectory, the planning label is placed before the search label; The search tag is followed immediately by the observation tag to form a search-observation pair; The writing tag is appended after the search observation pair; The reference trajectory contains at least one of the search observation pairs.

[0084] First, this embodiment forces the model to output planning labels and task breakdown content before outputting search labels. This ensures that the model has a macro-level understanding and step-by-step breakdown of the overall task before performing specific search actions, thereby avoiding the inefficiency caused by blind searching.

[0085] Secondly, after the model issues a search command, it is followed by the external result feedback corresponding to the search command. By closely binding the two into a search observation pair, the model can learn how to parse and utilize the information returned by external tools and establish a causal relationship between the query terms and the search results.

[0086] Then, to ensure that the model only begins the final summary and writing after fully acquiring and understanding the external observations, this embodiment appends writing tags after the search observation pairs, thereby following the principle of evidence-based generation and effectively suppressing the model's conjecture or illusion when lacking information.

[0087] It is particularly important to note that the reference trajectory contains at least one of the aforementioned search observation pairs, and typically includes multiple consecutive or alternating search observation pairs. This means that a "search-observation-search-observation" cyclic structure is allowed in the reference trajectory. This multi-round iterative data structure enables the model to learn multi-hop reasoning capabilities, that is, to dynamically adjust the query direction of the next round based on the search results of the previous round, thereby gradually approaching the truth of complex problems and achieving a progressive exploration of in-depth research tasks.

[0088] By limiting the above-mentioned label flow order and iteration structure, this embodiment ensures that the model can internalize a scientific and rigorous research methodology during the training process, enabling it to demonstrate logically coherent deep thinking and execution capabilities during reasoning.

[0089] Based on the above embodiments, as an optional embodiment, the final response content extracted from the interaction record includes extracting the article outline and the draft text.

[0090] Specifically, when extracting the final response content, this embodiment does not treat it as a single text block, but rather breaks it down into an outline and a draft of the main text. The outline typically includes the report's hierarchical structure, chapter titles, and summaries of core arguments, while the draft of the main text consists of specific paragraphs, data analysis, and detailed discussions filled in based on the outline.

[0091] Based on this, the following strategy is adopted in this embodiment to stitch together the tagged parts into the reference trajectory: First, add an outline tag before the article outline, for example... <outline>This is used to indicate that the model enters the structural planning mode.

[0092] Secondly, add a draft tag before the main text draft, for example... <draft>This is used to indicate that the model should enter a specific content generation mode.

[0093] Finally, in the reference trajectory, the article outline with the outline label is placed before the main text draft with the draft label.

[0094] The data construction method adopted in this embodiment, which prioritizes an outline before the main text draft, forces the model to output a large outline before starting to write the long article. This effectively gives the model a global planning framework. The model must first clearly define the overall structure and logical flow of the report before it can fill in the details based on this framework.

[0095] This embodiment effectively solves common problems in long text generation, such as inconsistency in logic, topic drift, and loose structure, through this hierarchical generation training strategy. The trained model can build a clear mind map first, just like a professional human writer, and then write in detail, thereby significantly improving the macro-structural quality and micro-content compactness of in-depth research reports.

[0096] To ensure that the model can focus on optimizing its own search strategy during reinforcement learning training, without being disturbed by uncontrollable noise in external search results, this embodiment introduces an observation mask mechanism in search reinforcement training, which is described below. Figure 5 The specific implementation process of this observation masking mechanism is described.

[0097] Figure 5 This is a schematic diagram of the process of model training using observation masking technology in the search enhancement step provided in this embodiment of the invention, as shown below. Figure 5 As shown, the process of training the pre-trained model specifically includes the following steps: Step 21: Inject the question input into the pre-trained model to generate a search prediction sequence containing search query instructions and search results.

[0098] During the actual training, the questions from the search task training set are first input, such as "What city will host the 2024 Olympic Games?", and provided to the pre-trained model. Based on the role capabilities it has learned, the pre-trained model will attempt to generate search query instructions and search results.

[0099] Search query commands are model-generated actions intended to retrieve information. For example... <search>2024 Olympics host city. Search results are generated by a pre-trained model, which calls an external search engine to retrieve and populate the model with webpage summaries or data, such as... <observation> ...Paris...< / observation> The text stream, composed of content generated by the pre-trained model and external feedback, constitutes the search prediction sequence.

[0100] Step 22: Identify the search results content that is wrapped by the observation tag in the search prediction sequence.

[0101] This embodiment will parse the generated search prediction sequence and locate all the observation labels within it, such as... <observation> and< / observation> The text portion enclosed by this pair of tags is identified as the search result content. This part of the search result content is essentially the input of the external environment, rather than the product of the model's own thinking or decision-making.

[0102] Step 23: When calculating the policy gradient used to update the model parameters, the marker position corresponding to the search result content is masked, and the gradient is calculated only based on the part of the search prediction sequence other than the search result content.

[0103] When performing backpropagation to update the model parameters, this embodiment performs masking on the search results identified in step 22. Specifically, the loss function weights or gradient values ​​corresponding to these locations are set to zero. This means that the model parameter updates will only calculate gradients based on the model's own thought process, search instructions, and final answer.

[0104] Since the content returned by external search engines often contains advertisements, irrelevant links, or noisy, formatted content, allowing a pre-trained model to fit or learn from this noise can cause the model's parameter updates to shift in direction and even impair its language capabilities. Through masking, this embodiment forces the pre-trained model to focus solely on how to ask good questions and how to answer them using known information, ignoring the specific form of external information. This significantly improves the stability of search reinforcement training and the model's ability to follow instructions.

[0105] In the writing enhancement stage, in order to solve the illusion problem that easily occurs when the large language model generates long texts, that is, generating seemingly reasonable but unfounded statements, this embodiment further introduces a reference anchor binding mechanism as an additional feedback constraint based on the above embodiments.

[0106] Figure 6 This is a schematic diagram of the feedback adjustment process using the evidence anchor binding mechanism in the writing enhancement step provided by the embodiments of the present invention, as shown in the figure. Figure 6 As shown, training the search optimization model may further include: Step 24: When the model generates the writing tags and subsequent long article prediction results, detect whether the long article prediction results contain citation anchors.

[0107] Step 25: Verify whether the reference anchor points to the search results content within the observation tag.

[0108] Step 26: If the long text prediction result does not contain the reference anchor or the reference anchor does not point to the search result content, then reduce the value of the second feedback signal.

[0109] When the search optimization model starts generating lengthy reports, it enters... <draft>In the content generation stage after the tag, this embodiment will detect the generated text stream in real time or offline. Specifically, it will look for specific anchor reference formats, such as [1], [source-1] or [S1.C2], etc. These anchor references represent the search optimization model's claim that its current argument is based on a certain piece of evidence.

[0110] If a reference anchor is detected, the current context window will be traced back to check whether the number or identifier pointed to by the reference anchor actually exists in the previous observation labels. For example, if the search optimization model outputs "According to reports [1]...", it will check whether there is indeed a search result marked "[1]" in the context window.

[0111] However, if the verification finds that the long text prediction results generated by the model do not cite any anchor points at all, or if the model generates anchor points but the anchor points point to a non-existent source or a fabricated source (i.e., "false references"), the electronic device will apply a negative penalty term when calculating the second feedback signal of this training sample, or directly reduce its value, for example by multiplying it by a coefficient less than 1, or subtracting a fixed score.

[0112] This embodiment utilizes an evidence anchor binding mechanism, enabling the search optimization model to gradually learn through continuous trial and error and feedback: to obtain a high score reward, it must not only write fluently but also correctly cite key facts. This effectively integrates the hard constraint of evidence traceability into the training objective of the search optimization model, fundamentally suppressing the model's tendency to fabricate facts and significantly improving the credibility and rigor of in-depth research reports.

[0113] To provide a clear and objective optimization target for search reinforcement training, this embodiment further elaborates on the specific calculation logic using the standard answer label as the first feedback signal, specifically including: First, extract the final answer fragment from the search prediction results generated by the pre-trained model. The text stream generated by the pre-trained model typically contains a specific identifier, such as "Answer:" or... <answer>To guide the final conclusion, this embodiment will parse and extract the content following the identifier as the predicted answer of the pre-trained model.

[0114] Next, calculate the exact match score or F1 score between the final answer fragment and the standard answer label.

[0115] This embodiment standardizes the predicted answer and the standard answer, such as removing punctuation and unifying capitalization, before comparison. If they are completely identical, the exact match score is 1; otherwise, it is 0.

[0116] The F1 score measures the overlap between predicted and standard answers at the lexical level. It takes into account both recall and precision and can provide positive incentives for partially correct answers.

[0117] Finally, in this embodiment, the exact match score or the F1 score will be determined as the first feedback signal and used as the reward value in reinforcement learning (such as PPO or GRPO algorithms) to guide the model to adjust parameters.

[0118] This embodiment uses hard metrics based on objective fact comparison, such as exact matching score or F1, as feedback signals to guide the pre-trained model to discard redundant information during the search phase and focus on quickly and accurately locating and extracting core factual information. This short and quick training objective complements the long and comprehensive training objective adopted in the subsequent writing reinforcement phase, ensuring that the language model has depth while maintaining its accuracy as a research tool.

[0119] During the writing reinforcement phase, since in-depth research reports are typically long texts and open-ended, with no single correct answer, exact match scores or F1 scores may be inapplicable. Therefore, this embodiment introduces a scoring mechanism based on large model evaluation.

[0120] Figure 7 This is a flowchart illustrating the generation of the second feedback signal in the writing enhancement step provided in an embodiment of the present invention, as shown below. Figure 7 As shown, the step of using the scoring result determined by the long report generated by the model as the second feedback signal specifically includes the following steps: Step 31: For the same complex query sample, use the search optimization model to sample and generate multiple candidate long text reports.

[0121] During training, for each complex query sample, the search optimization model to be trained is controlled to perform multiple independent inference samplings. Since the generation of the search optimization model has randomness, for example by setting temperature parameters, each sampling may produce a candidate long article report with slightly different content, structure, or citations. Therefore, this embodiment will generate a set of candidate reports, such as 4 or 8.

[0122] Step 32: Input the complex query sample and the candidate long text report into the preset evaluation model.

[0123] This embodiment pre-configures a preset evaluation model. The evaluation model can be a large language model with stronger instruction compliance and more accurate logical judgment, such as a teacher model with a larger number of parameters, or a model specifically trained with reward modeling. The original query requirements and the candidate long text report just generated by the search optimization model are provided as input to the evaluation model.

[0124] Step 33: Using the evaluation model, generate a relative advantage score for the candidate long article report based on the preset scoring rules.

[0125] Specifically, the evaluation model acts as a referee, reviewing and judging multiple candidate long-form reports based on pre-defined scoring rules, such as requiring reports to include supporting data and be logically consistent. It should be noted that the evaluation model's output can be either an absolute score (e.g., 1-10) or a relative ranking. In the preferred embodiment, by comparing the merits of different candidate reports, the relative advantage score of each candidate long-form report is calculated.

[0126] Step 34: Use the relative advantage score as the second feedback signal.

[0127] Finally, the relative advantage score output by the evaluation model is directly used as the reward signal for reinforcement learning. If the quality of a candidate long report is considered by the evaluation model to be better than that of other candidate long reports, it will receive high positive feedback, and vice versa.

[0128] This embodiment successfully solves the core problem of lack of supervision signals in complex long text generation tasks. It uses the evaluation model's own ability to identify text quality to guide the evolution of generation capabilities, enabling the language model to be trained to continuously iterate and optimize in the direction of high quality and depth.

[0129] To further improve the accuracy of scoring the relative advantage scores of candidate long reports and eliminate the absolute numerical bias that may exist in direct scoring by a single model (e.g., some models tend to give high scores while others tend to give low scores), this embodiment provides a pairwise comparison scoring strategy.

[0130] Figure 8 This is a schematic diagram of the process for generating relative advantage scores based on pairwise comparisons provided in an embodiment of the present invention, such as... Figure 8 As shown, generating the relative advantage score for the candidate long article report specifically includes the following steps: Step 331: Select one of the candidate long article reports as the baseline report and use the rest as comparison reports.

[0131] From a set of generated candidate long article reports (e.g., candidate long article reports A, B, C, and D), the system will randomly select, or select a candidate long article report of medium quality, as the benchmark report. The remaining reports naturally become comparison reports.

[0132] Step 332: Input the benchmark report and the comparison report into the evaluation model simultaneously for pairwise comparison, and determine the winning probability of the comparison report relative to the benchmark report.

[0133] The system pairs the benchmark report with each comparison report (e.g., "Benchmark vs. Report A", "Benchmark vs. Report B") and inputs each comparison report pair into the evaluation model. The evaluation model is asked to answer either the question, "Which of these two reports is better for this query?" or "What is the win rate of the latter relative to the former?". To eliminate position bias, the system can even swap the order of the two reports, score them twice, and then average the scores. Finally, the evaluation model outputs a probability of victory, representing the likelihood that the comparison report will outperform the benchmark report.

[0134] Step 333: Normalize the winning probability to obtain the relative advantage score.

[0135] Finally, this embodiment transforms these winning probabilities into a relative advantage score required by the reinforcement learning algorithm by statistically and normally processing them (e.g., subtracting the mean and dividing by the standard deviation). This score reflects the relative goodness or badness of a particular report within the current set of generated results.

[0136] By employing a pairwise comparison mechanism, this embodiment effectively avoids the subjective arbitrariness of absolute scoring. The evaluation model only needs to determine which is better. This comparison-based relative feedback signal has lower variance, which can provide a more stable and consistent gradient descent direction for writing reinforcement training, thereby accelerating the convergence of model training and improving the final effect.

[0137] To ensure that the scores given by the evaluation model can comprehensively and objectively reflect the quality of the in-depth research report, this embodiment has refined the preset scoring rules used in the above embodiments in multiple dimensions.

[0138] Specifically, the preset scoring rules cover at least one or more of the following evaluation dimensions: content comprehensiveness, logical depth, instruction compliance, and text readability.

[0139] The comprehensiveness of content is used to evaluate whether the candidate long-text reports generated by the search optimization model fully answer the complex questions raised by users. For example, it evaluates whether the report covers all sub-questions of complex query samples and whether any key information points are missing.

[0140] Logical depth is used to evaluate the depth of thought in the generated candidate long-form reports. Specifically, it checks whether the candidate long-form reports are merely a collection of facts, or whether they include reasoning analysis, causal relationship exploration, or trend prediction.

[0141] Instruction compliance is used to assess the formal compliance of reports, such as checking whether candidate long reports meet the preset format requirements, such as word limits, point-by-point statements, and explicit constraints such as the inclusion of tables.

[0142] Text readability is used to evaluate the language quality of candidate long-form reports, such as examining the fluency of the language, the clarity of paragraph structure, and the professionalism of the vocabulary.

[0143] This embodiment explicitly introduces these fine-grained evaluation dimensions into the training rewards, directly mapping the definition of a high-quality report to the model's optimization objective. This allows for targeted improvement of various capabilities during model training, resulting in in-depth research reports that are not only accurate in content.

[0144] To efficiently utilize the aforementioned relative feedback signals to optimize the model, this embodiment further provides an iterative training method based on Group Relative Policy Optimization (GRPO).

[0145] Figure 9 This is a schematic diagram illustrating the process of updating the search optimization model based on the group relative strategy optimization algorithm provided in an embodiment of the present invention, as shown below. Figure 9 As shown, based on the above embodiments, training the search optimization model includes repeatedly executing the following steps until a preset cutoff condition is reached (such as the number of training steps reaching the upper limit or the model converging): Step 35: For the same complex query sample, control the search optimization model to sample and generate a set of long text prediction result sequences; Step 36: Calculate the second feedback signal corresponding to each long text prediction result in the long text prediction result sequence; Step 37: Calculate the average value of all the second feedback signals as the baseline; Step 38: Calculate the difference between each of the second feedback signals and the baseline to obtain the advantage value; Step 39: Update the model parameters of the search optimization model using the advantage value.

[0146] Figure 10 This is a schematic diagram illustrating the data flow principle of writing reinforcement training that combines external search and scoring feedback, as provided in an embodiment of the present invention. Figure 10 As shown, in the sampling generation phase, this embodiment controls the search optimization model to generate a set of N different long-text prediction sequences in parallel for the same complex query sample (input Q). During this process, the search optimization model may call external searches multiple times, so the search paths and final content contained in each long-text prediction sequence may be different. This makes full use of the randomness of model generation to explore the solution space.

[0147] Subsequently, all generated long-text prediction sequences are input into the evaluation module, which calculates the second feedback signal based on the aforementioned scoring rules or pairwise comparison mechanism. Ultimately, each long-text prediction sequence receives a corresponding quantized value. Figure 10 The values ​​shown in the table (1, 2...N) represent the quality of the generated result.

[0148] Unlike the traditional PPO algorithm, which requires training a large value network to predict the baseline, the GRPO algorithm provided in this embodiment directly calculates the arithmetic mean of the current set of sampled values ​​and uses it as the baseline. This not only saves memory and computing resources, but also allows the baseline to follow the average level of the current policy in real time and dynamically.

[0149] Furthermore, the baseline is subtracted from the quantized value corresponding to each long-text prediction sequence to obtain the advantage value of that long-text prediction sequence. If the score of a long-text prediction sequence is higher than the average score, its advantage value is positive, meaning that it is a generation result of better than average quality; otherwise, it is negative.

[0150] Finally, the policy gradient method is used to update the model parameters of the search optimization model based on the calculated advantage value. The generation probability of long text prediction sequences with a positive advantage value will be increased, while the generation probability of long text prediction sequences with a negative advantage value will be suppressed.

[0151] This embodiment constructs an efficient reinforcement learning paradigm that does not require a value network. While ensuring training effectiveness, it significantly reduces the implementation complexity and computational overhead of the algorithm, making reinforcement learning training on large-scale long text generation tasks more efficient, stable, and easier to implement in engineering.

[0152] To demonstrate the technical details of this invention in greater detail, the following systematically elaborates on the language model training method based on single-model multi-role chain and two-stage reinforcement learning proposed in this invention, strictly following the chronological order of technical implementation.

[0153] This invention first constructs a reasoning architecture based on a single-model, multi-agent chain (CoA). For example... Figure 11 As shown, this inference architecture abandons the complex pattern of traditional multi-agent systems that rely on multi-process communication, and instead solves complex queries through dynamic module orchestration within a single large language model.

[0154] In this architecture, the state transitions of the language model occur within a single context window, which can be formally represented as: ; in, Showing time t The reasoning state (i.e., the current context information); The parameter is θ The model's forward propagation function; Indicates at time t The activated role functions (such as planning, searching, writing, etc.) have a value space of . ; This indicates the observation results from the previous moment, such as the content returned by an external search engine; This represents the probability distribution of choosing the next role in the current state.

[0155] The first stage of language model training is cold start. In this embodiment, knowledge distillation technology is used to transfer the serial decision-making mode of the existing multi-model combination expert multi-agent system to the single language model of the present invention. This is essentially a serial-level knowledge distillation.

[0156] Specifically, this embodiment monitors the execution process of the expert system and extracts its complete execution trajectory: ; in, Represents a complete reference trajectory; This represents the total number of steps in the reference trajectory.

[0157] To ensure the quality of the distillation data, this embodiment employs a test-time scaling strategy to filter reference trajectories. The filtered reference trajectories are then used for supervised fine-tuning of the pre-trained model. The objective function for training is: ; in, This indicates the loss from monitoring and fine-tuning. This represents the policy distribution of the pre-trained model to be trained; Represents the target token of the trajectory at time t; q represents the historical trajectory before time t; q represents the query input.

[0158] In this process, observation masking was specifically introduced, that is, in calculating the supervised fine-tuning loss. At that time, if If it is an external observation result, then its corresponding gradient is set to zero.

[0159] After the model acquires basic role-switching capabilities, it enters the first stage of reinforcement learning, where the core objective is to train the language model's search capabilities.

[0160] Since writing ability only becomes effective when generating excellent search results, this embodiment first focuses on training a language model as an efficient and accurate information acquirer. The core objective is to improve the planning and analysis capabilities of the planning agent and the search performance and execution efficiency of the search agent. To this end, a large amount of verifiable QA data requiring search to answer forms the search task training dataset. This embodiment uses the exact match score (EM) as the first feedback signal. ; in, This indicates the value of the exact match score; This represents a short answer to the model's output. Indicates the standard answer label; For indicator functions, when the parentheses contain The value is 1 when this condition is met, and 0 otherwise.

[0161] Finally, we move to the second stage of reinforcement learning: writing reinforcement based on scoring rules and rewards. This stage primarily involves introducing the GRPO algorithm combined with a search engine, and includes the following implementation process: The first step is to design a reward model based on LLM-as-a-Judge and pairwise comparison mechanisms.

[0162] The expression for calculating the relative advantage score based on the pairwise comparison mechanism is as follows: ; ; in, Indicate the evaluation model; This indicates the pre-defined scoring rules, such as comprehensiveness and depth. Indicates the first to be evaluated i One candidate full-length report; Indicates the selected benchmark report; This indicates the relative advantage score obtained when inputting the baseline report and the comparison report; This indicates the relative advantage score obtained after exchanging the benchmark and comparison reports in the correct order.

[0163] The final relative score, i.e., the second feedback signal The calculation expression is: .

[0164] The second step is to execute the update logic of the GRPO algorithm. This is based on the determined second feedback signal. The objective function of the GRPO algorithm is: ; ; ; in, Indicates the importance ratio; Indicates the first i The sample at the th t The action at each time step; Indicates the first i The sample at the th t The state at each time step; Indicates the current policy network; This represents the old policy network. This is the dominant value; It is the first i The relative scores of each report; b It is the average score of the group's reports; The KL divergence term represents the divergence of a given input. and external search engines Under the given conditions, the current strategy The generated output sequence Reference Strategy Generate Kullback-Leibler divergence between sequences; The KL regularization coefficient is used to control the step size of policy updates and prevent the model from deviating excessively from the reference model. It is the cutoff coefficient, usually taken as a small value (such as 0.2), used to limit the magnitude of policy updates and ensure training stability. It is a mathematical operation that represents the importance ratio. Limited to the interval [ ] within, if Less than Regarding the value ;like Greater than Regarding the value .

[0165] By maximizing this objective function, the model is incentivized to generate candidate long-form reports with relatively higher scores, thereby significantly improving writing quality and the standardization of evidence citations while maintaining search capabilities.

[0166] Based on the model training methods provided in the above embodiments, this invention also provides a language model for in-depth research. This language model is trained using the language model training methods provided in any of the above embodiments. Although this language model may be a parameter-sharing neural network (such as a Transformer) at the physical level, at the logical functional level, its internal architecture can be divided into multiple collaborative role modules: The planning module is used to receive query input and generate planning tags and task breakdown information; The search interaction module is used to generate search tags and query terms based on the planning tags, and to receive external search results encapsulated in the observation tags; The writing generation module is used to generate a long report containing writing tags and citation anchors based on the external search results within the observation tags. The planning tag, the search tag, the observation tag, and the writing tag together constitute an explicit action tag sequence. The planning module, the search interaction module, and the writing generation module achieve end-to-end data interaction within a single context window through the flow of the explicit action tag sequence.

[0167] Figure 11 This is a schematic diagram illustrating the logical architecture of a language model and the interaction of various functional modules within a single context window, as provided in an embodiment of the present invention. Figure 11 As shown, the language model provided in this embodiment of the invention is characterized by maintaining a single context window. This single context window serves as a public display area for all information flow, and stores user query input, explicit action tags generated by the model (such as "tag planning" and "search tags" in the figure), and corresponding text content alternately in chronological order.

[0168] Based on the shared environment provided by this shared single context window, the language model logically includes the following core functional role modules: (1) The planning module is mainly responsible for the top-level design of the task. When it receives complex query input from the user, the planning module is activated to generate planning labels (such as...). <plan>), and then output detailed task breakdown information (such as Figure 11 (The "Label Planning" node in the text). This step breaks down a large research objective into several executable sub-steps.

[0169] (2) The search interaction module is responsible for performing specific information retrieval tasks. Based on the decomposed information generated by the planning module, the search interaction module will generate search tags (such as...). <search>The query terms are then used. An external search engine is invoked, and the returned results are encapsulated in observation tags (such as...). <observation>Fill in the context window within the blank. For example... Figure 11 As shown, the search interaction module supports multi-round search-observation cyclical interaction, which means that the language model can dynamically adjust the direction of the next search based on the result of one search until enough information is obtained.

[0170] (3) The writing generation module is responsible for the final knowledge integration and output. After the search interaction module has collected sufficient information (i.e., multiple observation tags and content have accumulated in the context window), the writing generation module is activated to generate writing tags (such as...). <draft>Based on all observations within the context, this module generates a comprehensive and logically rigorous long report. Specifically, it explicitly inserts citation anchors during the generation process, linking the arguments in the report to specific sources of evidence within the observation tags.

[0171] Although the modules described above have different functions, they share the same set of model parameters. This architectural design enables the language model to achieve a smooth transition from thinker to executor to creator in a single inference flow simply by predicting the next explicit action label, thereby realizing complex multi-agent collaborative logic with extremely low engineering cost.

[0172] To further improve the quality of the generated report and ensure that the evidence cited is accurate and logically consistent, the deep research language model described in this embodiment also includes a criticism feedback module in its logical architecture. This module is configured to review the long report generated by the writing generation module and generate criticism tags and modification suggestions. The writing generation module is also configured to respond to the criticism tags and regenerate a revised long article report based on the modification suggestions.

[0173] like Figure 11 As shown in the bottom right corner, the criticism feedback module is configured as a self-reflection mechanism. Once the writing generation module has initially completed the writing of the long report, the criticism feedback module is activated to generate criticism tags (such as...). <critique>), and conduct a rigorous review of the content of the newly generated long report.

[0174] Specifically, the feedback module checks the room temperature report for factual errors, logical loopholes, missing citations, or inconsistencies with user instructions, and outputs specific modification suggestions. For example: "The source cited in the second paragraph regarding the market size data [3] does not contain the data. It is recommended to verify it."

[0175] Subsequently, the writing generation module responds to these suggestions and enters revision mode. For example, based on the issues pointed out by the criticism feedback module, it regenerates revised long-form report fragments. This closed-loop correction mechanism enables the language model to self-correct, significantly improving the accuracy and reliability of the final output.

[0176] Considering that a single search may not yield sufficient information when dealing with extremely complex in-depth research tasks, this embodiment further enhances the search interaction module.

[0177] Specifically, the search interaction module is further configured as follows: Within the context window, the search tags are generated multiple times consecutively, and multiple observation tags are received accordingly, forming an alternating search observation sequence; The writing generation module is configured to perform a comprehensive analysis based on all search result data in the search observation sequence.

[0178] like Figure 11 As shown, the search interaction module is configured to generate the search tags multiple times consecutively within a single context window. This means that after the language model completes a search and receives the observation results (observation tags), it can determine whether to supplement the search based on the new information.

[0179] For example, when answering the question "Comparison of market share between Company A and Company B", the language model might first generate a search tag query for "Company A's market share", and after receiving the observation results, immediately generate a second search tag query for "Company B's market share". In this way, an alternating sequence of search observations is formed in the context window (i.e., Search 1->Observation 1->Search 2->Observation 2...).

[0180] Accordingly, the writing generation module is configured to perform comprehensive analysis and integration based not only on the last search result, but on all search result data throughout the entire search observation sequence. This mechanism empowers the model to handle multi-hop reasoning problems, enabling it to piece together knowledge across multiple information sources, thereby generating more detailed and comprehensive in-depth research reports.

[0181] Figure 12 This is a flowchart illustrating the research report generation method provided in this embodiment of the invention, such as... Figure 12 As shown, based on the above embodiments, the present invention also provides a research report generation method, which mainly includes, but is not limited to, the following steps: Step 101: Receive the user's natural language query request.

[0182] In this embodiment, the system first receives a natural language query request input by the user through a user interface or an application programming interface (API). This natural language query request is usually an open and complex question, such as "Please help me analyze the global industry landscape of generative AI in 2024".

[0183] Step 102: Using the language model training method provided in any of the above embodiments, the target language model is trained and, in response to the natural language query request in a single context window, the explicit action tag stream output by the target language model is obtained.

[0184] The natural language query request is input into a pre-loaded target language model, which then begins autoregressive generation within a single context window. Unlike ordinary dialogue models that directly generate text, the target language model of this invention outputs an explicit action tag stream containing specially marked tags. This explicit action tag stream sequentially includes planning tags for planning tasks, search tags for obtaining information, observation tags for carrying results, and writing tags for composing content.

[0185] Step 103: In response to the search tag, call an external search engine to obtain the search results, and encapsulate the search results within the observation tag and feed them back to the target language model.

[0186] When the target language model outputs search tags (such as...) <search> query< / search> When the query term is retrieved, the output of the target language model is immediately paused, the query term is extracted, and the API of an external search engine is called. After obtaining the search results returned by the API, they are cleaned and encapsulated in observation tags (such as...). <observation> ...< / observation> The data is appended to the end of the model's context window, which then wakes up the target language model to continue generation. This process may be repeated multiple times.

[0187] Step 104: Extract the long text report generated by the target language model after the writing tag, and output it as the research report.

[0188] When the target language model finally generates writing tags (such as...) <draft>After that, the subsequent text constitutes the formal report content. The system extracts this content and presents it to the user as the final in-depth research report.

[0189] The research report generation method provided in this embodiment not only allows users to obtain a high-quality research report, but also enables them to clearly see how the model plans, searches, and draws conclusions step by step by observing the explicit action tag flow, thereby greatly improving the transparency of research report generation and user trust.

[0190] To further enhance the practicality and verifiability of the research report, this embodiment enhances the interactivity of the output report. Specifically, the long report includes citation anchors automatically generated by the target language model during the writing stage (e.g., [1]).

[0191] Accordingly, the research report generation method provided in this embodiment also includes: First, the system parses the reference anchors and establishes a hyperlink mapping between the reference anchors and the sources of the search results within the observation tags. The system scans the generated report text, identifies all reference markers, and uses their numbers or identifiers to find the corresponding observation tag content, thereby extracting the original URL or source information of the search result.

[0192] Secondly, when outputting the research report, the hyperlink mapping is provided to support attribution. On the user interface, these citation anchors are rendered as clickable hyperlinks. Users can simply click the anchor to jump directly to the original webpage or view detailed evidence of the citation.

[0193] This feature not only makes every claim in the research report verifiable and facilitates fact-checking for users, but also gives static text dynamic interactive value, significantly improving the user experience of in-depth research tools.

[0194] Figure 13 This is a schematic diagram of the structure of the electronic device provided by the present invention, such as... Figure 13 As shown, the electronic device may include: a processor 1310, a communications interface 1320, a memory 1330, and a communications bus 1340, wherein the processor 1310, the communications interface 1320, and the memory 1330 communicate with each other through the communications bus 1340. The processor 1310 can call logical instructions in the memory 1330 to execute a language model training method, which includes: acquiring a cold start training dataset containing multiple multi-agent interaction samples, each multi-agent interaction sample including a query input and a corresponding reference trajectory, the reference trajectory containing explicit action labels for identifying the behavior of different roles; using the query input as model input and the reference trajectory as supervision labels to perform supervised fine-tuning on an initial large language model to obtain a pre-trained model; acquiring a search task training dataset containing multiple question-answering samples, each question-answering sample including a question input and a standard answer label; using the question input as model input and the standard answer label as a first feedback signal to train the pre-trained model to obtain a search optimization model; acquiring a writing task training dataset containing multiple complex query samples; using the complex query samples as model input and the scoring results determined for the long article report generated by the model as a second feedback signal to train the search optimization model to obtain a target language model.

[0195] Furthermore, the logical instructions in the aforementioned memory 1330 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0196] On the other hand, the present invention also provides a computer program product, the computer program product comprising a computer program stored on a non-transitory computer-readable storage medium, the computer program comprising program instructions, wherein when the program instructions are executed by a computer, the computer is able to execute the language model training method provided in the above embodiments, the method comprising: acquiring a cold start training dataset, the cold start training dataset comprising multiple multi-agent interaction samples, each multi-agent interaction sample comprising a query input and a corresponding reference trajectory, the reference trajectory comprising explicit action labels for identifying the behavior of different roles; using the query input as model input, and using the reference trajectory as supervision labels. The initial large language model is subjected to supervised fine-tuning to obtain a pre-trained model. A search task training dataset is obtained, which contains multiple question-answer samples, each including a question input and a standard answer label. The pre-trained model is trained using the question input as the model input and the standard answer label as the first feedback signal to obtain a search optimization model. A writing task training dataset is obtained, which contains multiple complex query samples. The search optimization model is trained using the complex query samples as the model input and the scoring results determined for the long article report generated by the model as the second feedback signal to obtain a target language model.

[0197] In another aspect, the present invention also provides a non-transitory computer-readable storage medium storing a computer program thereon. When executed by a processor, the computer program implements the language model training method provided in the above embodiments. The method includes: acquiring a cold-start training dataset containing multiple multi-agent interaction samples, each multi-agent interaction sample including a query input and a corresponding reference trajectory, the reference trajectory including explicit action labels for identifying the behavior of different roles; using the query input as model input and the reference trajectory as supervision labels to perform supervised fine-tuning on an initial large language model to obtain a pre-trained model; acquiring a search task training dataset containing multiple question-answering samples, each question-answering sample including a question input and a standard answer label; using the question input as model input and the standard answer label as a first feedback signal to train the pre-trained model to obtain a search optimization model; acquiring a writing task training dataset containing multiple complex query samples; using the complex query samples as model input and the scoring results determined for a long article report generated by the model as a second feedback signal to train the search optimization model to obtain a target language model.

[0198] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0199] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0200] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.< / draft> < / critique> < / draft> < / observation> < / search> < / plan> < / answer> < / draft> < / search> < / draft> < / outline> < / response> < / observation> < / search> < / plan> < / response> < / observation> < / search> < / plan>

Claims

1. A language model training method, characterized in that, include: Obtain a cold start training dataset, which contains multiple multi-agent interaction samples. Each multi-agent interaction sample includes a query input and a corresponding reference trajectory. The reference trajectory contains explicit action labels for identifying the behavior of different roles. Using the query input as the model input and the reference trajectory as the supervision label, the initial large language model is fine-tuned under supervision to obtain a pre-trained model. Obtain a search task training dataset, which contains multiple question-and-answer samples, each of which includes a question input and a standard answer label; Using the question input as the model input and the standard answer label as the first feedback signal, the pre-trained model is trained to obtain the search optimization model; Obtain the writing task training dataset, which contains multiple complex query samples; Using the complex query sample as model input and the scoring result determined by the long article report generated by the model as the second feedback signal, the search optimization model is trained to obtain the target language model.

2. The language model training method according to claim 1, characterized in that, The process of obtaining the cold start training dataset includes: Retrieve multi-round interaction records for any given query request; Extract the task planning content, search query instructions, search results content, and final reply content from the interaction records; Add a planning tag before the task planning content, a search tag before the search query command, an observation tag before the search results content, and a writing tag before the final response content; Based on the chronological order of the interaction records, the tagged parts are pieced together to form the reference trajectory.

3. The language model training method according to claim 2, characterized in that, The step of stitching together the tagged parts to form the reference trajectory includes: In the reference trajectory, the planning label is placed before the search label; The search tag is followed immediately by the observation tag to form a search-observation pair; The writing tag is appended after the search observation pair; The reference trajectory contains at least one of the search observation pairs.

4. The language model training method according to claim 2, characterized in that, The final response content extracted from the interaction records includes the extracted article outline and draft text. The step of stitching together the tagged parts to form the reference trajectory also includes: Add an outline tag before the article outline; Add a draft tag before the main text draft; In the reference trajectory, the article outline with the outline label is placed before the main text draft with the draft label.

5. The language model training method according to claim 2, characterized in that, The step of training the pre-trained model to obtain the search optimization model includes: The problem input is injected into the pre-trained model to generate a search prediction sequence containing search query instructions and search results. Identify the search results content that is wrapped by the observation tag in the search prediction sequence; When calculating the policy gradient used to update model parameters, the marker positions corresponding to the search result content are masked, and the gradient is calculated only based on the portion of the search prediction sequence other than the search result content.

6. The language model training method according to claim 2, characterized in that, Training the search optimization model further includes: When the model generates the writing tags and subsequent long article prediction results, it detects whether the long article prediction results contain citation anchors. Verify whether the reference anchor points to the search results content within the observed tag; If the long text prediction result does not contain the reference anchor or the reference anchor does not point to the search result content, the value of the second feedback signal is reduced.

7. The language model training method according to claim 1, characterized in that, The use of the standard answer label as the first feedback signal includes: Extract the final answer fragment from the search prediction results generated by the pre-trained model; Calculate the exact match score or F1 score between the final answer fragment and the standard answer label; The exact match score or the F1 score is determined as the first feedback signal.

8. The language model training method according to claim 1, characterized in that, The second feedback signal is determined by the scoring result of the long report generated by the model, including: For the same complex query sample, the search optimization model is used to sample and generate multiple candidate long-text reports; The complex query sample and the candidate long text report are input into the preset evaluation model; Using the evaluation model, a relative advantage score is generated for the candidate long article report based on preset scoring rules; The relative advantage score is used as the second feedback signal.

9. The language model training method according to claim 8, characterized in that, The generation of the relative advantage score for the candidate long article report includes: Select one of all the candidate long article reports as the baseline report, and use the rest as comparison reports; The benchmark report and the comparison report are simultaneously input into the evaluation model for pairwise comparison to determine the probability of the comparison report winning relative to the benchmark report; The winning probability is normalized to obtain the relative advantage score.

10. The language model training method according to claim 8, characterized in that, The pre-set scoring rules cover at least one of the following evaluation dimensions: comprehensiveness of content, depth of logic, compliance with instructions, and readability of text; The comprehensiveness of the content is used to characterize whether the candidate long text report covers all sub-questions of the complex query sample; The logical depth is used to characterize whether the candidate long-text report contains inferential analysis or trend prediction. The instruction compliance degree is used to characterize whether the candidate long text report meets the preset format requirements; The text readability is used to characterize the language fluency and paragraph structure of the candidate long text report.

11. The language model training method according to claim 1, characterized in that, Training the search optimization model includes iteratively executing the following training steps until a preset cutoff condition is met: For the same complex query sample, the search optimization model is controlled to sample and generate a sequence of long-text prediction results; Calculate the second feedback signal corresponding to each long text prediction result in the long text prediction result sequence; Calculate the average of all the second feedback signals as the baseline; The difference between each of the second feedback signals and the baseline is calculated to obtain the advantage value; The model parameters of the search optimization model are updated using the advantage value.

12. A language model, characterized in that the language model is trained based on the language model training method according to any one of claims 1 to 11; The language model includes multiple functional role modules that share the same set of model parameters: The planning module is used to receive query input and generate planning tags and task breakdown information; The search interaction module is used to generate search tags and query terms based on the planning tags, and to receive external search results encapsulated in the observation tags; The writing generation module is used to generate a long report containing writing tags and citation anchors based on the external search results within the observation tags. The planning tag, the search tag, the observation tag, and the writing tag together constitute an explicit action tag sequence. The planning module, the search interaction module, and the writing generation module achieve end-to-end data interaction within a single context window through the flow of the explicit action tag sequence.

13. The language model according to claim 12, characterized in that, Also includes: The criticism feedback module is configured to review the long article report generated by the writing generation module and generate criticism tags and modification suggestions. The writing generation module is also configured to respond to the criticism tags and regenerate a revised long article report based on the modification suggestions.

14. The language model according to claim 12, characterized in that, The search interaction module is also configured as follows: Within the context window, the search tags are generated multiple times consecutively, and multiple observation tags are received accordingly, forming an alternating search observation sequence; The writing generation module is configured to perform a comprehensive analysis based on all search result data in the search observation sequence.

15. A method for generating a research report, characterized in that, include: Receive natural language query requests from users; Invoke the target language model trained by the language model training method according to any one of claims 1 to 11; In response to the query request within a single context window, an explicit action tag stream output by the target language model is obtained; the explicit action tag stream includes sequentially generated planning tags, search tags, observation tags, and writing tags; In response to the search tag, an external search engine is invoked to obtain the search results, and the search results are encapsulated within the observation tag and fed back to the target language model; Extract the long text report generated by the target language model after the writing tag, and output it as the research report.

16. The research report generation method according to claim 15, characterized in that, The long-form report includes citation anchors, and the research report generation method further includes: Parse the reference anchor and establish a hyperlink mapping between the reference anchor and the source of the search result within the observation tag; When outputting the research report, the hyperlink mapping is provided to support source tracing.

17. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the language model training method as described in any one of claims 1 to 11 or the research report generation method as described in any one of claims 15 to 16.

18. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the language model training method as described in any one of claims 1 to 11 or the research report generation method as described in any one of claims 15 to 16.

19. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by the processor, it implements the language model training method as described in any one of claims 1 to 11 or the research report generation method as described in any one of claims 15 to 16.

Images