A method, device and readable medium for constructing agent task training corpus for large language model

Abstract

The present invention relates to a method, device, and readable medium for constructing an agent task training corpus for a large language model. The method comprises: splitting and redesigning the format and reasoning of the original training corpus, converting it into different agent task training sentences in the form of dialogues by aligning the fine-tuning process with the pre-training domain; assigning different loss weights to different agent task training sentences according to the ability dimension; and constructing a diverse set of negative samples to construct a training corpus dataset. Compared with the prior art, the present invention refines the proxy adjustment process of the language model. On the one hand, the training corpus is deconstructed from a fixed format and aligned with the pre-training domain of the large language model to fully stimulate the capabilities acquired during model pre-training. On the other hand, a data balancing method based on the model's capabilities is used to make the model pay more attention to the aspects of the current weaker capabilities during training, thereby improving training efficiency. The problem of hallucination is also reduced by constructing comprehensive negative samples.

Description

Technical Field

[0001] The present invention relates to the technical field of large language model training, and in particular to a method, device and readable medium for constructing an intelligent agent task training corpus for a large language model. Background Art

[0002] Existing approaches to training large language model (LLM) agents primarily focus on enhancing the model's agent capabilities through pre-training and fine-tuning to address diverse task requirements. However, these approaches face several key challenges: First, the significant distribution discrepancies between pre-training data and agent training tasks make direct transfer learning difficult; second, different agent capabilities learn at varying speeds during training, requiring more refined training strategies; and finally, existing approaches often suffer from the problem of model hallucinations when improving agent capabilities.

[0003] In other words, existing large language model agent training solutions mainly face the following difficulties and challenges:

[0004] 1. Differences in Data Distribution

[0005] A significant challenge in training large language models is the significant discrepancy between pre-training data and proxy task data. Pre-training typically involves collecting data from a wide range of sources covering a wide range of topics and domains, with the goal of training the model to understand a broad range of language concepts. However, when these models are applied to specific proxy tasks, the task-related data often has a narrower distribution, which can differ significantly from the pre-training data. This difference in distribution makes it difficult for the model to effectively transfer its pre-trained knowledge directly to the specific proxy task without significant adjustments.

[0006] 2. Imbalance in Agent Capability Learning

[0007] Another challenge facing large language models when training agent capabilities is the uneven learning rates of different capabilities. Some agent tasks require models to understand complex language structures, reason, and process domain-specific knowledge. The complexity of these capabilities and the required learning depth can vary significantly across different tasks and model architectures. Therefore, agent training needs to be able to flexibly adapt to these diverse learning requirements to ensure balanced and efficient learning across all relevant capabilities.

[0008] 3. Hallucination Problems in Model Training

[0009] Another common problem in improving the proxy capabilities of large language models is model hallucination—generating erroneous or false information without sufficient supporting evidence. This problem is particularly prominent in applications that handle complex reasoning tasks or require high accuracy. Hallucination not only affects the reliability of model output but also poses a challenge to the model's effectiveness in real-world applications.

[0010] Existing techniques train large language models by directly adopting the fixed paradigm of the ReAct format (Think:Action:Observe). However, this approach can easily lead to overfitting the model to a specific format, reducing its generalization and versatility. When fusing datasets, existing techniques only consider the model's distribution across different datasets, without considering the model's inherent learning capabilities. Furthermore, existing techniques only use normal conversation data and general agent training data as training sets, without extensive consideration of hallucinations in real-world scenarios. Although previous research has attempted to address these issues through methods such as data augmentation and model structure optimization, effectively integrating these capabilities into LLMs while avoiding hallucinations and improving the model's performance in proxy tasks remains an unresolved challenge. Summary of the Invention

[0011] The purpose of the present invention is to overcome the defects of the above-mentioned prior art and provide a method, device and readable medium for constructing an intelligent agent task training corpus for a large language model, which effectively refines the agent adjustment process of the language model and greatly reduces the hallucination problem by constructing comprehensive negative samples.

[0012] The object of the present invention can be achieved by the following technical solution: A method for constructing an agent task training corpus for a large language model, comprising the following steps:

[0013] S1. Split and redesign the format and reasoning of the original training corpus, and transform it into training sentences for different agent tasks in the form of dialogues by aligning the fine-tuning process with the pre-training domain;

[0014] S2. According to the ability dimension, different loss weights are given to different agent task training statements;

[0015] S3. Construct diverse negative samples and build a training corpus dataset.

[0016] Furthermore, the step S1 specifically deconstructs different original training corpora from a fixed format and uniformly aligns them to the pre-training domain of the large language model.

[0017] Furthermore, the step S2 specifically includes the following steps:

[0018] S21. Decompose the agent data according to the capabilities required for different tasks;

[0019] S22. According to the learning speed of the model when learning different abilities, the loss weights of the corresponding ability dimensions are allocated, and the decision on whether to learn the current corpus is made by judging the loss value in the corresponding dictionary.

[0020] Furthermore, the step S21 is specifically decomposed according to reasoning ability, retrieval ability, comprehension ability and instruction-following ability.

[0021] Furthermore, the step S22 is specifically implemented by designing a corresponding interface to control the sentences that require loss learning according to the capability dimension of the split corpus.

[0022] Furthermore, the interface is specifically:

[0023] dict(role="assistant", content="xxxxx", loss="True / False)

[0024] When the loss value in the dictionary is True, the current corpus is learned;

[0025] When the loss value in the dictionary is False, no learning is performed on the current corpus.

[0026] Furthermore, the specific process of constructing diverse negative samples in step S3 is as follows:

[0027] Introducing user question data without providing any tools to obtain the first negative sample;

[0028] Based on the first negative sample, tools irrelevant to the current problem are inserted to obtain the second negative sample.

[0029] Furthermore, the actual application scenario corresponding to the first negative sample is: when a user queries and requests a tool, no tool is provided;

[0030] The actual application scenario corresponding to the second negative sample is: a situation where a user queries and requests a regular conversation when providing a tool.

[0031] A computer device comprises: at least one processor; and a memory, wherein the memory stores computer instructions that can be executed on the processor, and when the computer instructions are executed by the processor, the steps of the above-mentioned method for constructing a corpus for training intelligent agent tasks are implemented.

[0032] A computer-readable storage medium stores a computer program, which, when executed by a processor, implements the steps of the above-mentioned method for constructing a corpus for training intelligent agent tasks.

[0033] Compared with the prior art, the present invention has the following advantages:

[0034] The present invention first aligns the agent training corpus to the pre-training domain, that is, by adjusting the agent task to adapt to the pre-training domain. Then, from the perspective of ability, the agent data is decomposed according to the capabilities required by the task, and different loss weights are given to the training sentences in different parts according to the ability division, thereby achieving the purpose of ability decomposition and data balance. In addition, a variety of training negative samples are further designed, which can effectively improve the generalization of the model and reduce its corresponding hallucination phenomenon.

[0035] This invention deconstructs the training corpus from the traditional fixed format, converts the traditional agent corpus into a dialogue form, unifies the training corpus data of different agents into the dialogue domain, and fully stimulates the ability of the large language model as an agent itself by unifying the corpus features of pre-training and downstream agent fine-tuning tasks, thereby improving the performance of the model on downstream tasks.

[0036] The present invention splits the agent corpus according to the model ability dimensions, and matches the corresponding ability dimensions according to the model's learning speed when learning different abilities, thereby efficiently stimulating the model's learning ability. The data balancing solution based on model ability proposed in the present invention enables the model to more effectively focus on aspects with poor current abilities during training and improve training efficiency.

[0037] This invention fully considers the problems of large-model intelligent agents in actual application scenarios. By constructing a variety of negative samples for intelligent agent training, it effectively solves the hallucination problem of large language models when used as intelligent agents, greatly improving the user experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] Figure 1 Schematic diagram of the method flow of the present invention;

[0039] Figure 2 This is a schematic diagram of aligning the original agent corpus to the conversation domain in the embodiment;

[0040] Figure 3 Schematic diagram of intelligent agent application scenario. DETAILED DESCRIPTION

[0041] The present invention will be described in detail below with reference to the accompanying drawings and specific embodiments.

[0042] Example

[0043] like Figure 1 As shown, a method for constructing an agent task training corpus for a large language model includes the following steps:

[0044] S1. Split and redesign the format and reasoning of the original training corpus, and transform it into training sentences for different agent tasks in the form of dialogues by aligning the fine-tuning process with the pre-training domain;

[0045] S2. According to the ability dimension, different loss weights are given to different agent task training statements;

[0046] S3. Construct diverse negative samples and build a training corpus dataset.

[0047] The application of the above technical solution mainly includes:

[0048] 1. Adapting the Agent Task to the Pre-training Domain

[0049] By splitting and redesigning the format and common reasoning of the training corpus, we align the fine-tuning process with the pre-training domain of the language model, thereby stimulating pure agency capabilities in large language models without over-adapting to specific format protocols. By further subdividing the agency tasks into different aspects of the basic capabilities of the large language model, this solution provides training flexibility based on the different learning speeds of each capability.

[0050] Specifically, since the agent training corpus is usually coupled with some fixed formats, such as "thinking: action: behavior input: observation", where the behavior input content is in JSON format, this makes the model training coupled with the agent's reasoning ability and fixed format. At the same time, this training format does not belong to the same language domain as the pre-training corpus of the large language model, which leads to the inconsistency between the downstream fine-tuning of the model and the pre-training. Based on this problem, this solution proposes a coupled training corpus construction method, which greatly alleviates the learning pressure of the model by converting the traditional agent corpus into the form of dialogue and significantly improves the model's capabilities. The specific splitting method is as follows: Figure 2 shown.

[0051] 2. Capability Decomposition and Data Balance

[0052] This approach explores the appropriate mix of various training data sources to improve performance. From a capability perspective, we explicitly decompose agent data into the capabilities required for each task (such as reasoning, retrieval, understanding, and instruction following). Since large language models exhibit different learning rates on these capabilities, it shows that the appropriate combination of these data sources is also crucial to optimizing the final results. Figure 2 As shown in the figure, different loss weights are assigned to different parts of the training sentences according to the ability classification, so that the model can focus on learning the abilities it currently does not fully master, while reducing the problem of overfitting to the related abilities it has already mastered. More specifically, this solution designs the following interface when constructing training data:

[0053] dict(role="assistant",content="xxxxx",loss="True / False)

[0054] The decision to learn the current corpus can be made by judging the loss value in the corresponding dictionary as True / False. Based on this interface, this solution can precisely control the sentences that require loss learning based on the ability dimension of the split corpus, thereby achieving a balanced ability data during the training process of large language models.

[0055] 3. Negative Sample Learning to Eliminate Hallucinations

[0056] In order to comprehensively solve the hallucination problem in agent tasks, this solution designs a variety of training negative samples. The application scenarios of intelligent agents can be divided into the following categories: Figure 3 In the four different cases shown, existing methods typically only cover training data for cases (a) and (d), leaving out cases (b) and (c). This leads to undefined behavior when the model is fed these types of questions, known as hallucinations. To address this issue, this solution proposes a negative sample construction method that introduces user question data from ToolBench without providing any tools, allowing GPT3.5 to provide a response as a negative example (c) (the first negative example). Furthermore, based on this negative example, tools unrelated to the current question are inserted, introducing two types of negative samples:

[0057] (1) When a user queries and requests a tool, no tool is provided;

[0058] (2) When providing tools, users query and request regular dialogue.

[0059] Through explicit supervision, the model is taught not only how to act, but also when to act as an agent. This negative sample learning strategy significantly mitigates the hallucination problem while maintaining excellent performance on relevant downstream tool call evaluation sets.

[0060] When users need to fine-tune large language models for downstream tasks, they often require a systematic agent model training solution. This solution provides users with a rational, efficient, and comprehensive solution for fine-tuning large agent models, ensuring that the models can quickly and effectively adapt to downstream agent tasks. This eliminates the need to worry about the model's inherent performance degradation due to specialized fine-tuning, allowing users to seamlessly apply large language models to downstream applications.

[0061] Taking the open source large language model Llama2-7B as an example, if Llama2-7B is trained directly on the agent corpus using traditional methods, its downstream task performance can only reach 31.7. After adopting the method proposed in this solution, the model performance can be directly improved to 41.7, an improvement of about 30%. At the same time, the performance of the Llama2-7B model itself on basic tasks, including mathematics, code, language reasoning, etc., can achieve good performance.

[0062] In summary, this solution analyzes the existing problems with agent training and proposes a method for aligning agent training data with pre-training domains. It also introduces data-balanced training strategies and negative example construction strategies. This approach can be applied not only to downstream training of large agent models but also to other areas of fine-tuning large models, such as training for downstream tasks like mathematics and coding.

[0063] This solution unifies the training corpus data of different agents into the dialogue domain. By unifying the corpus features of pre-training and downstream agent fine-tuning tasks, it fully unleashes the capabilities of the large language model as an agent itself, thereby improving the model's performance on downstream tasks.

[0064] This solution splits the agent corpus according to the model's ability dimensions and allocates the corresponding ability dimensions according to the model's learning speed when learning different abilities, thereby effectively stimulating the model's learning ability.

[0065] This solution constructs a variety of negative samples for agent training, fully considering the applicability of large-model agents in actual application scenarios, effectively improving the generalization of the model and reducing the corresponding hallucination phenomenon.

[0066] Compared to traditional API (Application Programming Interface)-based models, this approach effectively refines the language model's proxy adjustment process through careful training corpus design and improved methods, and significantly reduces hallucination problems by constructing comprehensive negative samples. This approach outperforms previous methods on a variety of proxy evaluation datasets, consistently improving the model's performance in proxy tasks as the model size increases, while also enhancing the model's generalizability, thereby simultaneously reducing training costs and improving model practicality.

Claims

1. A method for constructing a corpus for agent task training of a large language model, characterized in that: The following steps are involved: S1. Split and redesign the format and reasoning of the original training corpus, and transform it into training sentences for different agent tasks in the form of dialogues by aligning the fine-tuning process with the pre-training domain; S2. According to the ability dimension, different loss weights are given to different agent task training statements; S3. Construct a variety of negative samples to build a training corpus dataset; Step S1 specifically deconstructs the different original training corpora from the fixed format and aligns them uniformly to the pre-training domain of the large language model; Step S2 specifically includes the following steps: S21. Decompose the agent data according to the capabilities required for different tasks; S22. According to the learning speed of the model when learning different abilities, the loss weight ratio is assigned to the corresponding ability dimension, and the loss value in the corresponding dictionary is judged to decide whether to learn the current corpus; The specific process of constructing diverse negative samples in step S3 is: Introducing user question data without providing any tools to obtain the first negative sample; Based on the first negative sample, tools irrelevant to the current problem are inserted to obtain the second negative sample.

2. The method for constructing agent task training corpus for a large language model according to claim 1, characterized in that: The step S21 is specifically decomposed according to reasoning ability, retrieval ability, comprehension ability and instruction-following ability.

3. The method for constructing agent task training corpus for a large language model according to claim 2, characterized in that: The step S22 is specifically to control the sentences that need to be loss-learned according to the capability dimension of the split corpus by designing a corresponding interface.

4. The method for constructing agent task training corpus for a large language model according to claim 3, characterized in that: The interface is specifically: dict(role="assistant", content="xxxxx", loss=True / False) When the loss value in the dictionary is True, the current corpus is learned; When the loss value in the dictionary is False, no learning is performed on the current corpus.

5. The method for constructing agent task training corpus for a large language model according to claim 1, characterized in that: The actual application scenario corresponding to the first negative sample is: when a user queries and requests a tool, no tool is provided; The actual application scenario corresponding to the second negative sample is: a situation where a user queries and requests a regular conversation when providing a tool.

6. A computer device comprising: at least one processor; and a memory storing computer instructions executable on a processor, wherein the computer instructions, when executed by the processor, implement the steps of the method for constructing an intelligent agent task training corpus as described in any one of claims 1 to 5.

7. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the steps of the method for constructing an agent task training corpus as described in any one of claims 1 to 5 are implemented.

Images