Sample augmentation and network iteration training method, device and equipment based on LLM

Abstract

This application relates to a method, apparatus, and device for sample augmentation and iterative network training based on LLM. The method uses LLM for sample augmentation and proposes a feature-based textualization method and a step-by-step prompting method. While using as few prompt words and input words as possible, it can make the output of LLM as consistent as possible with human expectations. Based on the idea of ​​meta-learning, it proposes an iterative network training method, which uses a base network plus iterative training to improve the network's generalization ability on different tasks.

Description

Technical Field

[0001] This invention belongs to the field of artificial intelligence technology and relates to a method, apparatus and device for sample augmentation and network iterative training based on LLM. Background Technology

[0002] As a core branch of artificial intelligence, deep learning's influence has permeated numerous practical applications, including image, natural language, and speech signal processing. Notably, in certain specific domains, models trained with deep learning have demonstrated performance surpassing human capabilities. For example, in image processing, the accuracy of deep learning models in tasks such as image separation, object detection, and face recognition has reached levels that are difficult for humans to match; in natural language processing, deep learning models have outperformed traditional methods in tasks such as text generation, sentiment analysis, and machine translation, even creating articles and poems that are difficult for humans to distinguish from genuine text; and in speech signal processing, deep learning technology has significantly improved the accuracy and speed of speech recognition, making human-computer interaction more natural and efficient.

[0003] In the field of task planning, deep learning methods based on reinforcement learning and imitation learning have made significant progress in areas such as video games, autonomous driving, and robot control. For example, in the gaming field, AlphaGo and MuZero defeated professional players in Go and complex strategy games; in the field of autonomous driving, deep learning models can achieve highly automated driving by mimicking human driving behavior; and in the field of robot control, various robots can perform complex tasks such as assembling parts and transporting objects.

[0004] While deep learning has achieved considerable success in solving a wide range of problems, its application in more real-world scenarios, especially in practical task planning, still presents certain challenges. The two most significant challenges are low sample efficiency and weak model generalization ability: to train a high-level model, most deep learning methods often require a large amount of training data, and models trained with large amounts of data tend to have high domain specificity, meaning they are difficult to reuse in other domains or even other tasks within the same domain. Summary of the Invention

[0005] To address the problems existing in the above-mentioned traditional methods, this invention proposes a sample augmentation and network iterative training method, apparatus and device based on LLM.

[0006] To achieve the above objectives, the embodiments of the present invention adopt the following technical solutions:

[0007] On the one hand, an LLM-based method for sample augmentation and iterative network training is provided, including the following steps:

[0008] Obtain the problem background and corresponding initial training data for several different basic planning tasks.

[0009] The problem background and the corresponding initial training data are processed into text, resulting in a text description of the problem background and a text description of the training data.

[0010] Based on the background text description of the problem, a feature-based stepwise suggestion method is used to provide suggestions for the LLM, generating structured suggestions.

[0011] Structured prompts and training data text descriptions are input into a large language model for specific tasks to generate augmented training data for each basic planning task.

[0012] The basic network is trained using augmented training data for each basic planning task to obtain the basic model.

[0013] When a new planning task is encountered, the basic model is iteratively trained based on the new planning task to obtain a dedicated network for the new planning task. During the iterative training process, the basic model interacts with simulation engines for different new tasks to obtain interaction samples. After LLM enhancement, the interaction sample data is used to continue training the basic model. After several rounds of iterative training, the original basic model is replaced with the iteratively updated network, and it continues to interact with the simulation engine to generate new training data until the iterative network converges to be able to complete the specific task, thus obtaining a dedicated model for the planning task.

[0014] On the other hand, an LLM-based sample augmentation and network iterative training apparatus is also provided, comprising:

[0015] The training data acquisition module is used to acquire the problem background and corresponding initial training data for several different basic planning tasks.

[0016] The text processing module is used to process the problem background and the corresponding initial training data into text, resulting in a text description of the problem background and a text description of the training data.

[0017] The prompt determination module is used to provide prompts to the LLM based on the problem background text description using a feature-based stepwise prompting method, generating structured prompts.

[0018] The sample augmentation module is used to input structured prompts and training data text descriptions into a large language model for specific tasks, generating augmented training data for each basic planning task.

[0019] The basic model training module is used to train the basic network using the augmented training data for each basic planning task to obtain the basic model.

[0020] The network iterative training module is used to iteratively train the base model based on the new planning task when a new planning task is encountered, so as to obtain a dedicated network for the new planning task. During the iterative training process, the base model interacts with the simulation engine of different new tasks to obtain interaction samples. After the interaction sample data is enhanced by LLM, the base model is trained again. After several rounds of iterative training, the original base model is replaced with the iteratively updated network, and the network continues to interact with the simulation engine to generate new training data until the iterative network converges to be able to complete the specific task, thus obtaining a dedicated model for the planning task.

[0021] In another aspect, a computer device is also provided, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps of any of the LLM-based sample augmentation and network iterative training methods described above.

[0022] One of the above technical solutions has the following advantages and beneficial effects:

[0023] The above-mentioned LLM-based sample augmentation and network iterative training method, apparatus, and device employ LLM for sample augmentation and propose a feature-based textualization method and a step-by-step prompting method. While using as few prompt words and input words as possible, the output of LLM can conform to human expectations as much as possible. Based on the meta-learning idea, a network iterative training method is proposed, which uses a base network plus iterative training to improve the network's generalization ability on different tasks. Attached Figure Description

[0024] To more clearly illustrate the technical solutions in the embodiments of this application or the conventional technology, the drawings used in the description of the embodiments or the conventional technology will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0025] Figure 1 This is a flowchart illustrating an LLM-based sample augmentation and network iterative training method in one embodiment.

[0026] Figure 2 This is a schematic diagram of the overall structure of a sample augmentation and network iterative training method based on LLM in one embodiment;

[0027] Figure 3 This is a schematic diagram of a feature-based step-by-step suggestion method in one embodiment;

[0028] Figure 4 This is a schematic diagram of the basic network training process in one embodiment;

[0029] Figure 5 This is a schematic diagram of the network iterative training process in one embodiment;

[0030] Figure 6 This is a schematic diagram of the experimental platform environment and tasks in one embodiment;

[0031] Figure 7 This is a schematic diagram illustrating the sample size required to train an advanced task network based on a base network in one embodiment.

[0032] Figure 8 This is a schematic diagram illustrating the success rate of iterative training for different combinations of basic tasks in one embodiment, wherein... Figure 8 Middle (a) to Figure 8 The diagram in (f) shows the success rate of combined iterative training of the six basic tasks: Synth, SynthLoc, GoToSeq, SynthSeq, GoToImpUnlock, and BossLevel. Detailed Implementation

[0033] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0034] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used in this specification is for the purpose of describing particular embodiments only and is not intended to be limiting of the application.

[0035] It should be noted that, in this document, the reference to "embodiment" means that a particular feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of the invention. The presentation of this phrase in various places throughout the specification does not necessarily refer to the same embodiment, nor is it a separate or alternative embodiment mutually exclusive with other embodiments. Those skilled in the art will understand that the embodiments described herein can be combined with other embodiments. The term "and / or" as used herein refers to any combination of one or more of the associated listed items, and all possible combinations, including such combinations.

[0036] The embodiments of the present invention will now be described in detail with reference to the accompanying drawings.

[0037] In one embodiment, such as Figure 1 As shown, an LLM-based sample augmentation and network iterative training method is provided, which may include the following processing steps 100 to 105:

[0038] Step 100: Obtain the problem background and corresponding initial training data for several different basic planning tasks.

[0039] Specifically, basic planning tasks can be, but are not limited to, planning tasks in the field of autonomous driving, or complex tasks such as assembling parts and moving items that various robots in the field of robot control can perform.

[0040] Step 101: Transform the problem background and the corresponding initial training data into textual descriptions to obtain textual descriptions of the problem background and training data.

[0041] Specifically, textualizing the problem background and data essentially involves creating a textual model of the problem's environment and data information, allowing the LLM to better understand it. While the LLM can directly process information such as images and speech with the intervention of certain plugins, it is essentially a language model, and its ability to process textual information is far superior to that of processing information such as images and speech.

[0042] Textualizing the problem context mainly considers the physical, structural, semantic, and relational features mentioned above. The aim is to describe the background environment and the type of task to be completed using natural language, so that LLM can understand the current environment in which data augmentation work is being carried out.

[0043] Textualizing training data primarily considers the temporal, contextual, and behavioral features mentioned above. The aim is to describe the agent's behavior at a given moment using natural language, allowing the LLM (Low-Level Machine) to determine whether the current behavior meets the requirements and whether the sample needs correction or removal. It's important to note that textualizing training data may include textualized problem context. Without considering the agent's state, it's impossible to evaluate the quality of its actions; therefore, both textualized environmental information and textualized action information must be provided to the LLM simultaneously.

[0044] Step 102: Based on the problem background text description, use a feature-based stepwise prompting method to provide prompts for the LLM and generate structured prompts.

[0045] Specifically, the advantage of LLM lies in its rich collection of expert knowledge from various fields. Therefore, with only a few hints, it can process data in different tasks. The key to using LLM for sample augmentation lies in how to textualize the environment and data, and how to use hint engineering techniques to provide hints to the LLM, making it applicable to different tasks.

[0046] After textualizing the problem context and data, cue engineering techniques are needed to provide prompts to the LLM (Low-Level Machine) so that its answers align with the current problem context, thereby better enhancing the samples. Cue engineering, as a method for guiding and optimizing the emergent capabilities of LLMs, offers lower costs and greater flexibility compared to fine-tuning LLM parameters, and can better improve the LLM's ability to understand complex semantics within a small semantic space. Simply put, cue engineering can be understood as a complete communication process between the user and the LLM. Currently popular cue techniques include zero-shot cueing, one-shot and few-shot cueing, and Chain of Thought (CoT) cueing. The core of Chain of Thought cueing lies in how to write chain of thought samples. A chain of thought sample is a cue word that breaks down a multi-step problem into several steps and solves the final problem step by step through reasoning. Using chain of thought samples as cue words for the LLM allows it to learn the reasoning methods in the samples, thus enabling it to answer questions reasoning-wise, just like humans.

[0047] Drawing inspiration from the concept of thought chains, a feature-based stepwise suggestion method is proposed. This method continuously increases the size of the suggestions, allowing the LLM to gradually understand the problem until its understanding is sufficient to solve it.

[0048] Step 103: Input the structured prompts and training data text descriptions into the large language model for specific tasks to generate augmented training data for each basic planning task.

[0049] Step 104: Train the basic network using the augmented training data for each basic planning task to obtain the basic model.

[0050] Step 105: When a new planning task is encountered, the basic model is iteratively trained based on the new planning task to obtain a dedicated network for the new planning task. During the iterative training process, the basic model interacts with simulation engines for different new tasks to obtain interaction samples. After LLM enhancement, the interaction sample data is used to continue training the basic model. After several rounds of iterative training, the original basic model is replaced with the iteratively updated network, and the network continues to interact with the simulation engine to generate new training data until the iterative network converges to be able to complete the specific task, thus obtaining a dedicated model for the planning task.

[0051] Specifically, during the iterative training of the network, the base network interacts with simulation engines for different tasks to obtain initial training data. After LLM enhancement, the initial training data is used to train the base network again, and different specialized networks can be trained according to different tasks. After several rounds of iterative training, the original base network is replaced with the iteratively updated network, and it continues to interact with the simulation engine to generate new training data until the iteratively updated network converges to be able to complete the specific task.

[0052] The overall structure of the LLM-based sample augmentation and network iterative training method is as follows: Figure 2 As shown, the diagram is divided into two parts: sample augmentation using LLM and network iterative training method design. The former further includes providing hints for LLM and augmenting the data using LLM, represented by green and blue backgrounds respectively. It's important to note that the network iterative training section in the diagram does not include processing of the training data; however, network iterative training is actually based on sample augmentation.

[0053] The above-mentioned LLM-based sample augmentation and network iterative training method uses LLM for sample augmentation and proposes a feature-based textualization method and a step-by-step prompting method. While using as few prompt words and input words as possible, it can make the output of LLM as consistent as possible with human expectations. Based on the meta-learning idea, it proposes a network iterative training method, which uses a base network plus iterative training to improve the network's generalization ability on different tasks.

[0054] In one embodiment, step 101 includes: textualizing the features describing the problem background to obtain a textual description of the problem background, and dividing the textual description of the problem background into physical features, structural features, semantic features, and relational features; physical features are used to describe the physical information of the environment and objects; structural features are used to describe the structural information in the environment; semantic features are used to describe the semantic information and meaning in the environment; relational features are used to describe the relationships and interactions between different features; textualizing the initial training data to obtain a textual description of the training data, and dividing the textual description of the training data into temporal features, contextual features, and behavioral features; temporal features are used to describe time-related information; contextual features are used to describe the contextual information of the environment and objects; and behavioral features are used to describe the behavior of objects and the environment.

[0055] Specifically, physical characteristics are used to describe the physical information of the environment and objects, such as map information, resource information, visibility, the location, speed, and direction of objects in the problem.

[0056] Time features are used to describe characteristics of time-related information, such as the current ambient time, the timestamp set, etc.

[0057] Structural features are used to describe the characteristics of structural information in an environment, such as information about relationships between objects, hierarchical information in the environment, and so on.

[0058] Contextual features are used to describe features of the environment and object context information, such as historical environment information, information about the object's surrounding environment, etc.

[0059] Behavioral characteristics are used to describe the features of the behavior of objects and the environment, such as the actions that an object can perform at a certain moment, the final actions it performs, and the feedback information from the environment at a certain moment.

[0060] Semantic features are used to describe the semantic information and meaning in an environment, such as text content, language expressions, symbolic identifiers, etc.

[0061] Association features are used to describe the relationships and interactions between different features, such as dependencies, influences, and interactions between different features.

[0062] In one embodiment, step 102 includes: selecting the first from the problem background text description. i Class features The process involves constructing a textual feature set, which contains several specific features. These features are used to describe prompt samples. The prompt samples are then used to provide prompts to the LLM, and test questions are used to verify whether the LLM can arrive at the correct answer. If no correct answer is obtained, a feature class is selected from the background text description of the question, and prompt samples are generated to provide further prompts and tests to the LLM. If a correct answer is obtained, a structured prompt is generated.

[0063] Specifically, the steps of the feature-based stepwise suggestion method include: first, textualizing the features used to describe the problem context and classifying them into one of the categories mentioned above. Indicates the first i Class features, using Indicates the first i The first of the class features j Each feature variable is a specific feature. Then, a textual feature set is defined. This set contains several specific features, using The features in the model are used to describe the cue samples. Finally, the cue samples are used to prompt the LLM, and test questions are used to verify whether the LLM can produce the correct answers. Figure 3 This process was demonstrated.

[0064] from Figure 3 It can be seen that at the beginning, The sample data contains only a limited number of features, and the cue samples described by these features may not provide effective cues for LLM. Therefore, it is necessary to continuously add features. Write more cue samples with more features and continue to train the LLM with cue until the LLM can answer the test questions accurately.

[0065] It is well known that the more detailed the cue samples, the more accurate the guidance and optimization of the LLM's emergent capabilities. However, more cue samples require more human effort and can lead the LLM into a highly specialized domain, causing it to lose some of its generalization ability. Therefore, feature-based stepwise cueing methods continuously increase the cue scale, allowing the LLM to continuously understand the problem until its understanding of the problem is just enough to solve it.

[0066] In one embodiment, step 104 includes: selecting several basic tasks, dividing the corresponding augmented training data into a support set and a query set; based on the meta-learning concept, the basic network consists of two parts, and the parameters of the two parts of the network are defined as follows: and ; These are the parameters of the outer network. The parameters of the inner network are defined as follows: The corresponding base network is trained using the support set of each base task. During training, the parameters of each inner network are updated based on the loss of each base task. Similarly, the corresponding base network is trained using the query set of each base task. During training, the parameters of each inner network are updated based on the loss of the base network corresponding to each base task. After the parameters of the inner networks are updated, the corresponding base network is trained again using the query set of each base task. During training, the sum of the losses of the base networks corresponding to all base tasks is used to update the parameters of the outer network of each base network, resulting in a trained base network, which serves as the base model.

[0067] In one embodiment, the process of updating the parameters of the inner network includes: inputting support samples into the base network, calculating the cross-entropy loss between the output and the label, and then backpropagating the loss and updating the parameters of the inner network using gradient descent.

[0068] In one embodiment, the process of updating the parameters of the outer network includes: inputting the query sample into the base network, calculating the cross-entropy loss of the output and label of the base network corresponding to each base task, summing them to obtain the total loss, and then backpropagating the total loss using the gradient descent method to update the parameters of the outer network.

[0069] In one embodiment, step 105 specifically includes:

[0070] Step S10: The basic network interacts with the simulation engine or environment to generate interactive samples. These interactive samples are then enhanced using an LLM algorithm to obtain enhanced samples. The basic network is the... k Network.

[0071] Step S20: Store the enhanced samples in the sample pool, and remove old samples from the sample pool according to the preset removal rules; the preset removal rules are: except for the first k - N Generation basic network to the first k All samples other than those generated by the base network are removed from the sample pool.

[0072] Step S30: Randomly select or select a certain number of samples from the sample pool according to weights to train the target network. Initialize the target network into a base network and update the parameters of its inner network in each training round.

[0073] Step S40: Repeat step S30, each time the loop iterates... M After that, the target network is used to iteratively update the base network, and... k The value is incremented by 1; a preset rejection rule is triggered, and samples in the sample pool are removed; whenever the loop reaches... M After +1, update the outer network of the target network. The parameters.

[0074] Steps S10 and S20 are independent of subsequent steps and will be executed in a loop until the training ends.

[0075] In one embodiment, the base network consists of two parts, and the parameters of the two parts are defined as follows: and ; These are the parameters of the outer network. These are the parameters of the inner network.

[0076] Network iterative training methods are also applicable to reinforcement learning;

[0077] During training, the policy gradient method updates the parameters of the inner network; the update formula is:

[0078]

[0079] in, , These are the parameters of the inner network before and after the update, respectively; To represent the learning rate, N This represents the total number of trajectories used in each training round. Indicates the first The sum of rewards for each trajectory, and They represent the first The first of the trajectories Each state and action, For the outer network, Indicates the length of the trajectory; This is a gradient descent operation.

[0080] During training, the outer network updates its parameters using gradient descent on the sum of the loss values ​​of all subtask samples.

[0081] Specifically, Figure 2 The overall structure of the network iterative training is demonstrated. The iterative training process is mainly divided into two stages: the first stage is training the base network, and the second stage is iterative improvement based on the base network. The iterative training method proposed in this application draws on the idea of ​​meta-learning, that is, using data from some basic tasks to train a base model, so that the model learns some general strategies and rules. When it encounters a new task, it only needs to continue iterative training on the base model to quickly adapt to and learn the new task.

[0082] In the training phase of the base network, several basic tasks need to be selected first. The training samples for these tasks, after LLM augmentation, are divided into support and query sets. Let the set of these basic tasks be:

[0083]

[0084] in Different categories of tasks are represented by pairs:

[0085]

[0086] in and These represent the support set samples and query set samples for this type of task, respectively. i The support samples and query samples for each task are represented as follows: , .

[0087] After the training samples are divided, based on Figure 4 The process involves training the base network. Based on the idea of ​​meta-learning, the base network consists of two parts, and the parameters of the two parts are defined as follows: and , Figure 4 The two sections above and below the red dashed line represent parameter updates performed on the network within the blue box. From a layered perspective, the network parameters are... network This can be understood as the outer network, with the following network parameters: network This is the inner network, the network Its function is to map input samples to the network. The part of the sample that is more suitable can guide the network. To obtain more accurate prediction results.

[0088] from Figure 4 It can be seen that the network The parameter update process is as follows: support samples are input into the base network, the cross-entropy loss between the output and the label is calculated, and then the loss is backpropagated using gradient descent to update the network. parameters ,Right now:

[0089]

[0090] in Represents network learning rate, To support the true label of the sample (if and only if the sample belongs to the first...) Categories are sometimes available ), Based on the network for the first The predicted probability of the class output, i.e.:

[0091]

[0092] network The parameter update process is as follows: update the network using supporting samples. After determining the parameters, the query sample is input into the base network, the cross-entropy loss between the output and the label is calculated, and then the loss is backpropagated using gradient descent to update the network. The parameters. It is important to note that the network in this step... The parameters are the updated parameters. And the loss is the sum of the losses of all different task categories, that is:

[0093]

[0094] in For the network learning rate, For the first The true label of the query sample for the task (if and only if the sample belongs to the first class). Categories are sometimes available ), For the base network after updating parameters, the first The predicted probability of the class output, i.e.:

[0095]

[0096] Similarly, in the iterative improvement phase of the basic network, the training samples for the new task need to be augmented using LLM, the difference being that they do not need to be divided into support and query sets. The network iterative improvement process is as follows: Figure 5 As shown.

[0097] The specific steps for network iterative improvement are as follows:

[0098] Step ① Using the basic network (the first k The network interacts with the engine or environment to generate interaction samples, which are then enhanced using LLM to obtain enhanced samples.

[0099] Step ② adds the augmented samples to the sample pool and removes old samples from the pool according to the rules (see step ④). Steps ① and ② are independent of subsequent steps and will be executed in a loop until training is complete.

[0100] Step ③: Randomly select samples from the sample pool, or select a certain number of samples according to weights, for training the target network (the first step). k +1 generation network). The target network is initialized as the base network, and its inner layers are updated in each training round. The parameters.

[0101] Step ④ repeats step ③, each time the loop continues. M After that (training) M In the first round, the following operations will be performed: iteratively update the base network using the target network, and... k The value is incremented by 1. At this point, the sample removal rule is triggered, except for the first... k - N ( 0) Generation basic network to the first k All samples other than those generated by the base network are removed from the sample pool. Whenever the loop reaches... M +1 times (training to the th) M (+1 round), update the outer layer of the target network. The parameters.

[0102] In practical applications, the above-mentioned iterative network training method is applicable not only to imitation learning based on labeled data, but also to reinforcement learning, where a training sample is a sampling trajectory. , Indicates the length of the trajectory. This represents the sum of rewards for that trajectory. The network can be updated using policy gradient methods. Parameters:

[0103]

[0104] in This represents the learning rate. Since calculating the expectation is difficult, sampling is typically used in practice to calculate it, and the network is updated using the policy gradient method. The formula for the parameters is transformed into:

[0105]

[0106] in N This represents the total number of trajectories used in each training round. Indicates the first The sum of rewards for each trajectory, and They represent the first The first of the trajectories Each state and action.

[0107] Similar to imitation learning, the internet The parameters are also updated using gradient descent on the sum of the loss values ​​of all subtask samples. .

[0108] In a verification embodiment, experiments were conducted on an experimental platform for studying the efficiency of basic language learning samples to verify the effectiveness of the proposed method. This embodiment first briefly introduces the experimental platform, then provides details of the experimental setup and implementation, and finally analyzes the experimental results.

[0109] (I) Introduction to the Experimental Platform

[0110] This experimental platform is an intelligent agent training experimental platform built on the basis of the MiniGrid world. Its elements are the same as those in the MiniGrid. The difference is that it can randomly generate basic environments and items for tasks of different levels, and comes with a complete evaluation system to evaluate whether the task has been completed.

[0111] The experimental platform contains five environmental elements: walls, doors, keys, balls, and boxes. Items in the platform come in six colors: red, green, blue, yellow, purple, and gray. The walls are a fixed gray; the colors of the balls, boxes, and keys are purely for completing the task and have no special function; the door and key must be the same color for the key to open the door. The agent in the experimental platform has six actions: turn left, turn right, take a step forward, pick up, discard, and switch. Switching is an operation performed on doors. When a door is locked, the switching action can use the key to close the door; when a door is closed, the switching action can directly open the door.

[0112] Figure 6The demonstration showcased a task example from the experimental platform. A 22x22 map was used to generate numerous small rooms containing items such as keys, balls, and boxes, each with its own color. The rooms were connected by doors. A red triangle represented the agent's initial position, facing down. The agent's task was to "place the purple ball from the top-left room onto the gray box in the bottom-right room."

[0113] One of the features of the experimental platform is its own natural language syntax, which allows users to describe a task in natural language and determine whether the agent has completed the task. The platform divides tasks into 19 levels, each requiring different capabilities from the agent, as shown in Table 1.

[0114] Table 1. Correspondence between tasks and capabilities of the experimental platform

[0115]

[0116] Note in Table 1: The names of the first-level tasks are: GoToObj level tasks; the second-level tasks are: GoToRedBallGrey level tasks; the third-level tasks are: GoToRedBall level tasks; the fourth-level tasks are: GoToLocal level tasks; the fifth-level tasks are: PutNextLocal level tasks; the sixth-level tasks are: PickUpLoc level tasks; the seventh-level tasks are: GoToObjMaze level tasks; the eighth-level tasks are: GoTo level tasks; the ninth-level tasks are: Pickup level tasks; and the tenth-level tasks... The tasks are named as follows: Level 11: UnlockPickup; Level 12: Open; Level 13: Unlock; Level 14: PutNext; Level 15: Synth; Level 16: GoToSeq; Level 17: SynthSeq; Level 18: GoToImpUnlock; and Level 19: BossLevel.

[0117] As shown in Table 1, there are a total of 13 different capabilities, each used to test different aspects of the agent's performance. The actual content corresponding to the capabilities with different numbers is shown below:

[0118] Capability 1: Room navigation, i.e., navigating within a 6x6 room.

[0119] Ability 2: Navigation with boxes as obstacles, i.e., navigating in a room with multiple distracting gray boxes.

[0120] Ability 3: Navigation with any object as an obstacle, the same as Ability 2, but the obstacle is not limited to gray boxes, it can be any type of object.

[0121] Ability 4: Navigation in a maze, specifically a 3x3 maze consisting of 6x6 rooms connected by doors.

[0122] Ability 5: Navigation to clear obstacles, which involves moving objects that obstruct the path during navigation.

[0123] Ability 6: Unlocking doors, meaning being able to find the key and unlock the door when the instructions are clear.

[0124] Capability 7: Hidden door unlocking, meaning it can automatically determine which door needs to be unlocked and find the corresponding key to unlock it.

[0125] Ability 8: Understanding the instruction "go to", such as "go to the red ball".

[0126] Skill 9: Understanding the instruction "open", such as "open the door on your left".

[0127] Skill 10: Understand the instruction "pick up", such as "pick up a box".

[0128] Skill 11: Understanding the instruction "place", such as "place the ball next to the blue key".

[0129] Skill 12: Understanding relative positions, such as "go to the red ball in front of you".

[0130] Ability 13: Understanding sequential instructions, such as "After opening the door, put the red ball next to the green box".

[0131] With different levels of tasks and corresponding required capabilities, an agent's level can be tested. For example, when an agent can complete all (or most) of the tasks at its current level, it means that the agent has learned the capabilities corresponding to that level. Therefore, experimental platforms are well-suited for training agents with planning capabilities and for judging the agent's performance by testing the levels it reaches.

[0132] (II) Experimental setup and procedure

[0133] The experimental platform's built-in rule-based agent generated a large amount of adversarial data at various levels, which was used as initial training data. DThis indicates that the initial training data is augmented using LLM, and the augmented training data is then used... DL express.

[0134] Three experiments were conducted to verify that the LLM-based sample augmentation and network iterative training method can alleviate the two problems of low sample efficiency and weak model generalization ability in deep learning.

[0135] The first experiment compares datasets. D and dataset DL The performance of the trained agents is used to demonstrate that LLM can improve sample efficiency by augmenting the samples. This is mainly achieved through comparisons from two perspectives: horizontally, comparing the number of samples required for training when two types of agents can solve the corresponding task; and vertically, comparing the success rate of the two types of agents in completing the corresponding task with the same number of samples.

[0136] The second experiment aims to demonstrate that LLM can improve sample efficiency through sample augmentation, primarily achieved through the learning examples provided in the experimental platform's built-in courses. This is achieved using the dataset... DL Alternative datasets D To train basic and target level agents and compare them with the provided examples.

[0137] The purpose of the third experiment is to demonstrate that the proposed iterative training method has a certain generalization ability. This is mainly achieved through the following process: First, select several tasks as base tasks and use the dataset... D and dataset DL The basic network is trained using samples corresponding to the basic task. Then, several higher-level tasks are selected, and the basic network is iteratively trained on different higher-level tasks. Finally, the final performance (success rate on the corresponding higher-level tasks) of the two types of networks is compared.

[0138] For network training, imitation learning was used instead of reinforcement learning, meaning labeled data was used for training, with samples in the form of state-action pair sequences. The network architecture used was the same as the imitation learning network provided by the experimental platform: a gated recurrent unit network (GRU) was used to encode task instructions; a convolutional network with two batch-normalized feature-wise linear modulation (FiLM) layers was used to process state and action features simultaneously; and a long short-term memory network (LSTM) was used to integrate the representation vectors generated by the FiLM module. The LSTM had 128 units, and the GRU was unidirectional with 256 units, also including an attention mechanism. Each network was trained using a Tesla V100 GPU.

[0139] For sample augmentation, Baidu's LLM—ERNIE-4.0-8k—was used, which allows for iterative interaction via an API interface. In the experiment, the number of tokens used in each query was limited to 8000, and this process was completed on a server equipped with a Tesla V100.

[0140] (III) Analysis of Experimental Results

[0141] To verify the effectiveness of the proposed LLM-based sample augmentation and network iterative training method, experiments were conducted to verify it from three aspects: agent performance comparison, curriculum learning comparison, and analysis of the generalization ability of the basic network.

[0142] (1) Comparison of agent performance

[0143] To demonstrate that LLM can improve sample efficiency through sample augmentation, this experiment compares the use of datasets from both horizontal and vertical perspectives. D and dataset DL The performance of the trained agent.

[0144] Table 2 shows the sample size required to train agents at different levels to solve the corresponding tasks. D The training sample size represents the final number of samples required to train the corresponding level of agent using the original samples without LLM enhancement. DL The training sample size represents the final number of samples required to train the corresponding level of agent using samples enhanced by LLM. The final sample size starts from 500,000 samples. The success rate of the agent completing the corresponding task decreases exponentially, i.e., when testing with sample sizes of 500,000, 353,600, 250,000, etc., the agent is considered capable of solving the task at that level when the success rate reaches 99% or higher. Therefore, the sample size required to train different agents is a range. Using the sample size on the left side of the range, the agent cannot solve the corresponding task; using the sample size on the right side, the agent can solve the corresponding task. If the agent still cannot solve the task using 500,000 samples, the success rate at this point will be specifically noted, as shown in Table 2 under the "Unlock" level. D The training sample size requirement is ">500 (96.29%)", indicating that 500,000 samples are required. D When using samples from the dataset to train an agent at the "Unlock" level, it cannot completely solve tasks at that level; it can only achieve a success rate of 96.29%.

[0145] Table 2. Results of Sample Size Required for Training Agents at Different Levels (Unit: Thousands)

[0146]

[0147] As can be seen from Table 2, at the same level, using datasets... D The number of samples needed to train the agent is greater than the number of datasets used. DL This indicates that the sample quality is improved after LLM augmentation, enabling the network to converge faster, thus proving that LLM-based sample augmentation methods can effectively improve sample efficiency. It's important to note that the number of samples required for training does not strictly increase with increasing level. This is because the capabilities required for high and low levels may not correspond, or the training samples for higher-level tasks may contain more information, allowing for faster learning of lower-level capabilities.

[0148] The above experiments compared the number of samples required for training when two types of agents could solve their corresponding tasks. To better illustrate the point, Table 3 provides a longitudinal comparison (only comparing agents at certain levels), showing the success rate of the two types of agents in completing their corresponding tasks with the same number of samples. Here, 10K, 20K, etc., represent datasets of 10,000 and 20,000 samples, respectively. D The success rate of an agent trained on a dataset of 10,000, 20,000, etc., in completing a corresponding task; 10K-L, 20K-L, etc., represent the success rate of an agent trained on a dataset of 10,000, 20,000, etc. DL The success rate of the agent trained on the sample in the dataset to complete the corresponding task.

[0149] Table 3. Performance Comparison of Agents Trained with the Same Sample Size

[0150]

[0151] Table 3 shows that when training two types of agents using the same sample size, the dataset used... DL Agents trained on samples in the dataset have a higher success rate on their corresponding tasks than those trained on the dataset. D The success rate of the agent trained on the samples in the dataset on its corresponding task is shown. This indicates that the sample quality is improved after LLM augmentation, which can enable the network to converge faster, thus further demonstrating that the LLM-based sample augmentation method effectively improves sample efficiency.

[0152] (2) Comparison of course learning

[0153] To further demonstrate that LLM can improve sample efficiency through sample augmentation, this experiment is based on the course learning cases provided by the experimental platform and uses a dataset. DL Alternative datasets D Train basic and target level agents and compare them with the provided examples.

[0154] Course learning, by gradually increasing the difficulty of training, enables neural networks to progressively learn the ability to perform complex tasks. Unlike traditional training models, course learning requires defining a list of "courses," that is, a series of "courses" with gradually increasing difficulty. The 19 levels of tasks provided by the experimental platform can be naturally regarded as a list of "courses," which meets the requirements of course learning.

[0155] Table 4 shows the learning outcomes in the dataset. D and dataset DL The training results are shown above. The base level can be considered a pre-trained network, which needs to be trained in advance using training samples of the corresponding level. Based on the neural network represented by the base level, the network is further trained using training samples of the target level until the agent represented by the network can solve the target level task. The meanings of the last four columns of data in Table 4 are as follows: and These respectively represent the use of datasets D and dataset DL The samples in the dataset represent the number of samples required to train a base-level network to the target-level network. D and DL These respectively indicate the use of datasets D or dataset DL The sample size required to directly train the target-level network is shown in Table 2. The data results are represented in the same format as those in Table 2.

[0156] Table 4 Comparison of Data Required for Course Learning and Training (Unit: Thousands)

[0157]

[0158] As can be seen from Table 4, regardless of whether a dataset is used... D Or dataset DL The course-learning method can reduce the amount of data required to reach the target level. The only exception is when training from the basic level "Level 7" to the target level "Level 8". In this case, the course-learning method did not reduce the amount of training data required, which may be due to certain behaviors of the agent at the basic level being inconsistent with those at the target level. column sum The data results in the column show that using the dataset DL The training samples in the dataset allow the network to improve from the basic level to the target level more quickly, which further proves that LLM can incorporate its extensive knowledge into the data, thereby effectively improving sample efficiency.

[0159] (3) Analysis of the generalization ability of the basic network

[0160] The experiments in the course-learning comparison, analyzed from the perspective of LLM-enhanced samples, demonstrate that agents at the same base level can generalize to different target levels based on the course-learning method. To further verify the generalization ability of the proposed iterative training method, this section selects several tasks and combines them as the base task, using a dataset... D and dataset DL The basic network is trained using samples corresponding to these tasks. Then, several higher-level tasks are selected, and the basic network is iteratively trained on different higher-level tasks. Finally, the sample size required to train the two types of networks is compared.

[0161] Using GoToObjMaze, Pickup, and Unlock as the three levels of tasks as the basic tasks, and with different datasets... D and dataset DL The base network is trained using samples corresponding to three levels. Base_D and Base_DL (The sample size for each level is 50,000). The six levels of tasks—Synth, SynthLoc, GoToSeq, SynthSeq, GoToImpUnlock, and BossLevel—are treated as high-level tasks, and each is implemented using a different dataset. D and dataset DL The base network is trained iteratively using samples corresponding to six levels. Base_ D and Base_DL The number of samples used when iteratively training the network can solve the corresponding level of task is recorded, and the results are as follows: Figure 7 As shown, the black vertical lines indicate the use of the dataset. DL Iterative training of the base network Base_DLThe required sample size is indicated by the blue vertical line, which represents the dataset used. D Iterative training of the base network Base_D The required sample size is indicated by each vertical line representing a data interval, with red dots indicating boundary points and green circles indicating no boundary points. If a high-level task cannot be solved with a sample size of 500,000, the success rate will be marked above the green circle.

[0162] from Figure 7 It can be seen that iterative training methods can generalize the same base network to different levels of advanced tasks. Furthermore, comparing Table 2 shows that agents using the base network require significantly fewer samples to reach higher levels of performance compared to those without, especially when using larger datasets. DL The trained network. This demonstrates that the proposed LLM-based sample augmentation and network iterative training method can improve sample efficiency while increasing the model's generalization ability. Furthermore, even without using LLM to augment the samples, the model trained by the network iterative method still possesses generalization ability to some extent.

[0163] To more intuitively illustrate the differences in generalization ability among different base models, Figure 8 This diagram illustrates the success rates of models iteratively trained with a limited number of samples on six high-level tasks. Each subplot represents a high-level task, and the four histograms within each subplot represent the success rates of agents trained with 10,000, 20,000, 30,000, and 40,000 samples (represented as 10,000, 20,000, 30,000, and 40,000 samples on the horizontal axis, respectively) on the corresponding high-level task. Each histogram cluster contains six bars, with the first three bars representing combinations of the three basic task datasets. D The task success rate of the agent trained iteratively; the last three bars represent the combined use of the dataset for the three basic tasks. DL The task success rate of the agent trained iteratively (indicated by adding L to the legend, i.e., combination 1-L, combination 2-L, combination 3-L). The basic tasks used in the three types of basic task combinations are as follows: Combination 1 uses three levels of tasks as basic tasks: GoToObj, PutNextLocal, and PickupLoc; Combination 2 uses three levels of tasks as basic tasks: GoToObjMaze, Pickup, and Unlock; Combination 3 uses three levels of tasks as basic tasks: GoTo, UnlockPickup, and Open.

[0164] A diagram illustrating the success rate of iterative training with different combinations of basic tasks is shown below. Figure 8 As shown, where Figure 8 Middle (a) to Figure 8The diagram in (f) shows the success rate of combined iterative training of the six basic tasks: Synth, SynthLoc, GoToSeq, SynthSeq, GoToImpUnlock, and BossLevel.

[0165] While a large sample size may be required for a model to fully solve the task, from Figure 8 It can be seen that using a relatively small number of samples for iterative training on the basic network can achieve above-average model performance in a short time, which verifies the rapid generalization ability of the basic model on various advanced tasks. Figure 8 It can also be seen that using datasets DL The iterative training of the base network represented by combination 2 achieved the highest task success rate across various advanced tasks. This indicates that task combinations such as GoToObjMaze, Pickup, and Unlock are the most suitable for training as base tasks. This combination is also... Figure 8 The basic task combination used in the experiment aligns with common sense; that is, in the experimental platform, if an agent learns basic skills such as "maze navigation," "picking up," and "lock picking," it can learn advanced skills relatively quickly. This experimental result indirectly demonstrates that as long as the most basic experience and rules are learned, the basic network can generalize on networks based on these experience and rules, further proving the correctness of the proposed method's generalization ability.

[0166] It should be understood that, although the above Figure 1 The steps are shown sequentially as indicated by the arrows, but these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise explicitly stated in this document, there is no strict order in which these steps are executed; they can be performed in other orders. Furthermore, the above... Figure 1 At least some of the steps may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily executed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be executed in turn or alternately with other steps or at least some of the sub-steps or stages of other steps.

[0167] In one embodiment, an LLM-based sample augmentation and network iterative training apparatus is also provided, comprising:

[0168] The training data acquisition module is used to acquire the problem background and corresponding initial training data for several different basic planning tasks.

[0169] The text processing module is used to process the problem background and the corresponding initial training data into text, resulting in a text description of the problem background and a text description of the training data.

[0170] The prompt determination module is used to provide prompts to the LLM based on the problem background text description using a feature-based stepwise prompting method, generating structured prompts.

[0171] The sample augmentation module is used to input structured prompts and training data text descriptions into a large language model for specific tasks, generating augmented training data for each basic planning task.

[0172] The basic model training module is used to train the basic network using the augmented training data for each basic planning task to obtain the basic model.

[0173] The network iterative training module is used to iteratively train the base model based on the new planning task when a new planning task is encountered, so as to obtain a dedicated network for the new planning task. During the iterative training process, the base model interacts with the simulation engine of different new tasks to obtain interaction samples. After the interaction sample data is enhanced by LLM, the base model is trained again. After several rounds of iterative training, the original base model is replaced with the iteratively updated network, and the network continues to interact with the simulation engine to generate new training data until the iterative network converges to be able to complete the specific task, thus obtaining a dedicated model for the planning task.

[0174] In one embodiment, the text processing module is further configured to textify the features describing the problem background to obtain a problem background text description, and to divide the problem background text description into physical features, structural features, semantic features, and relational features; physical features are used to describe the physical information of the environment and objects; structural features are used to describe the structural information of the environment; semantic features are used to describe the semantic information and meaning of the environment; relational features are used to describe the relationships and interactions between different features; and the initial training data is text-processed to obtain a training data text description, which is divided into temporal features, contextual features, and behavioral features; temporal features are used to describe time-related information; contextual features are used to describe the contextual information of the environment and objects; and behavioral features are used to describe the behavior of objects and the environment.

[0175] In one embodiment, the prompting module is further configured to select the first from the problem background text description. i Class features The process involves constructing a textual feature set, which contains several specific features. These features are used to describe prompt samples. The prompt samples are then used to provide prompts to the LLM, and test questions are used to verify whether the LLM can arrive at the correct answer. If no correct answer is obtained, a feature class is selected from the background text description of the question, and prompt samples are generated to provide further prompts and tests to the LLM. If a correct answer is obtained, a structured prompt is generated.

[0176] In one embodiment, the base model training module is further configured to select several base tasks and divide the corresponding augmented training data into a support set and a query set; based on the meta-learning concept, the base network consists of two parts, and the parameters of the two parts are defined as follows: and ; These are the parameters of the outer network. The parameters of the inner network are defined as follows: The corresponding base network is trained using the support set of each base task. During training, the parameters of each inner network are updated based on the loss of each base task. Similarly, the corresponding base network is trained using the query set of each base task. During training, the parameters of each inner network are updated based on the loss of the base network corresponding to each base task. After the parameters of the inner networks are updated, the corresponding base network is trained again using the query set of each base task. During training, the sum of the losses of the base networks corresponding to all base tasks is used to update the parameters of the outer network of each base network, resulting in a trained base network, which serves as the base model.

[0177] In one embodiment, the process of updating the parameters of the inner network in the base model training module includes: inputting support samples into the base network, calculating the cross-entropy loss between the output and the label, and then backpropagating the loss and updating the parameters of the inner network using gradient descent.

[0178] In one embodiment, the process of updating the parameters of the outer network in the base model training module includes: inputting the query sample into the base network, calculating the cross-entropy loss of the output and label of the base network corresponding to each base task, summing them to obtain the total loss, and then backpropagating the total loss using the gradient descent method to update the parameters of the outer network.

[0179] In one embodiment, the network iterative training module further includes: an interaction submodule, a sample pool update submodule, an inner network parameter update submodule, and an outer network parameter update submodule; wherein:

[0180] The interaction submodule is used to interact with the simulation engine or environment using the base network, generate interaction samples, and then enhance these interaction samples using LLM to obtain enhanced samples; the base network is the first... k Network.

[0181] The sample pool update submodule is used to store enhanced samples into the sample pool and remove old samples from the sample pool according to a preset removal rule; the preset removal rule is: except for the first k - N Generation basic network to the first k All samples other than those generated by the base network are removed from the sample pool.

[0182] The inner network parameter update submodule is used to randomly select or select a certain number of samples from the sample pool according to weights for training the target network. The target network is initialized as the base network, and its inner network parameters are updated in each training round.

[0183] The outer network parameter update submodule is used to repeatedly execute the inner network parameter update submodule. Each time the loop iterates... M After that, the target network is used to iteratively update the base network, and... k The value is incremented by 1; a preset rejection rule is triggered, and samples in the sample pool are removed; whenever the loop reaches... M After +1, update the inner network of the target network. The parameters.

[0184] The interaction submodule and the sample pool update submodule are independent of the inner network parameter update submodule and the outer network parameter update submodule, and will be executed in a loop until the training ends.

[0185] In one embodiment, the base network consists of two parts, and the parameters of the two parts are defined as follows: and ; These are the parameters of the outer network. These are the parameters of the inner network; the iterative training method is also applicable to reinforcement learning; during training, the policy gradient method updates the parameters of the inner network; the update formula is:

[0186]

[0187] in, , These are the parameters of the inner network before and after the update, respectively; Indicates the learning rate. N This represents the total number of trajectories used in each training round. Indicates the first The sum of rewards for each trajectory, and They represent the first The first of the trajectories Each state and action, For the outer network, Indicates the length of the trajectory; This is a gradient descent operation.

[0188] During training, the outer network updates its parameters using gradient descent on the sum of the loss values ​​of all subtask samples.

[0189] It is understood that for detailed explanations of the LLM-based sample augmentation and network iterative training apparatus, please refer to the corresponding explanations of the various embodiments of the LLM-based sample augmentation and network iterative training method above, and will not be repeated here. Each module in the above-described LLM-based sample augmentation and network iterative training apparatus can be implemented entirely or partially through software, hardware, or a combination thereof. Each module can be embedded in hardware or independently of a device with data processing capabilities, or stored in software in the memory of the aforementioned device, so that the processor can call and execute the operations corresponding to each module. The aforementioned device can be, but is not limited to, various types of data processing computer devices already existing in the art.

[0190] In one embodiment, a computer device is also provided, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps in the above method embodiment.

[0191] It is understood that, in addition to the memory and processor mentioned above, the computer equipment described above also includes other hardware and software components not listed in this specification. The specific components can be determined according to the model of the image processing computer in different application scenarios, and will not be listed and described in detail in this specification.

[0192] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0193] The above embodiments are merely illustrative of several implementation methods of this application, and their descriptions are relatively specific and detailed. However, they should not be construed as limiting the scope of protection of this application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and all such modifications and improvements fall within the scope of protection of this application.

Claims

1. A method for sample augmentation and iterative network training based on LLM, characterized in that, Including the following steps: Obtain the problem background and corresponding initial training data for several different basic planning tasks; The problem background and the corresponding initial training data are processed into text to obtain a text description of the problem background and a text description of the training data. Based on the background text description of the problem, a feature-based stepwise suggestion method is used to provide suggestions for the LLM, generating structured suggestions; Specifically, this includes: selecting the first from the background text description of the problem. i Class features A textual feature set is constructed, which contains several specific features. The features in the feature set are used to describe the prompt samples. The prompt samples are used to prompt the LLM, and a test question is used to verify whether the LLM can get the correct answer. If the correct answer is not obtained, a feature is selected from the background text description of the question, and prompt samples are generated to prompt and test the LLM. If the correct answer is obtained, a structured prompt is obtained. The structured prompts and the text descriptions of the training data are input into a large language model for specific tasks to generate enhanced training data for each basic planning task. The basic network is trained using augmented training data for each basic planning task to obtain the basic model; When a new planning task is encountered, the basic model is iteratively trained based on the new planning task to obtain a dedicated network for the new planning task. During the iterative training process, the basic model interacts with simulation engines for different new tasks to obtain interaction samples. After LLM enhancement, the interaction samples are used to further train the basic model. After several rounds of iterative training, the original basic model is replaced with the iteratively updated network, and the network continues to interact with the simulation engine to generate new training data until the iteratively updated network converges to the point where it can complete the task, thus obtaining a dedicated model for the planning task. The tasks include planning tasks in the field of autonomous driving and complex tasks such as assembling parts and transporting goods performed by various robots in the field of robot control. Specifically, step S10: The basic network interacts with the simulation engine or environment to generate interaction samples, and the interaction samples are enhanced by LLM to obtain enhanced samples. The basic network is the k-th generation network. Step S20: Store the enhanced samples in the sample pool, and remove old samples from the sample pool according to a preset removal rule; the preset removal rule is: except for the first k - N Generation basic network to the first k All samples other than those generated by the base network are removed from the sample pool; Step S30: Randomly select or select a certain number of samples from the sample pool according to weights to train the target network. Initialize the target network into a base network and update the parameters of its inner network in each training round. Step S40: Repeat step S30, each time the loop iterates... M After that, the target network is used to iteratively update the base network, and... k The value is incremented by 1; a preset rejection rule is triggered, and samples in the sample pool are removed; whenever the loop reaches... M After +1, update the parameters of the outer network of the target network; Steps S10 and S20 are independent of subsequent steps and will be executed in a loop until the training ends. The basic network consists of two parts, and the parameters of the two parts are defined as follows: and ; These are the parameters of the outer network. These are the parameters of the inner network; Network iterative training methods are also applicable to reinforcement learning; During training, the policy gradient method updates the parameters of the inner network; the update formula is: in, , These are the parameters of the inner network before and after the update, respectively; To represent the learning rate, N This represents the total number of trajectories used in each training round. Indicates the first The sum of rewards for each trajectory, and They represent the first The first of the trajectories Each state and action, For the outer network, Indicates the length of the trajectory; This is a gradient descent operation; During training, the outer network updates its parameters using gradient descent on the sum of the loss values ​​of all subtask samples.

2. The LLM-based sample augmentation and network iterative training method according to claim 1, characterized in that, The problem background and the corresponding initial training data are processed into text to obtain a text description of the problem background and a text description of the training data, including: The features describing the problem background are textualized to obtain a problem background text description, which is then divided into physical features, structural features, semantic features, and relational features. The physical features are used to describe the physical information of the environment and objects; the structural features are used to describe the structural information in the environment; the semantic features are used to describe the semantic information and meaning in the environment; and the relational features are used to describe the relationships and interactions between different features. The initial training data is processed into text to obtain a text description of the training data. The text description of the training data is divided into time features, context features, and behavioral features. The time features are used to describe the features of time-related information. The context features are used to describe the features of environmental and object context information. The behavioral features are used to describe the features of object and environmental behavior.

3. The LLM-based sample augmentation and network iterative training method according to claim 1, characterized in that, The base network is trained using augmented training data from each basic planning task to obtain the base model, including: Select several basic tasks and divide the corresponding augmented training data into support sets and query sets; Based on the meta-learning concept, the underlying network consists of two parts, and the parameters of the two parts are defined as follows: and ; These are the parameters of the outer network. These are the parameters of the inner network; The corresponding basic network is trained using the support set of each basic task, and the parameters of each inner network are updated according to the loss of each basic task during the training process. The corresponding basic network is trained using the query set of each basic task, and the parameters of each inner network are updated according to the loss of the basic network corresponding to each basic task during the training process. After the parameters of the inner network are updated, the corresponding basic network is trained using the query set of each basic task. During the training process, the sum of the losses of the basic networks corresponding to all basic tasks is used to update the parameters of the outer network of each basic network to obtain the trained basic network, which serves as the basic model.

4. The LLM-based sample augmentation and network iterative training method according to claim 3, characterized in that, The process of updating the parameters of the inner network includes: inputting support samples into the base network, calculating the cross-entropy loss between the output and the label, and then backpropagating the loss and updating the parameters of the inner network using gradient descent.

5. The LLM-based sample augmentation and network iterative training method according to claim 3, characterized in that, The process of updating the parameters of the outer network includes: inputting the query sample into the base network, calculating the cross-entropy loss of the output of the base network and the label for each base task, summing them to obtain the total loss, and then backpropagating the total loss using the gradient descent method to update the parameters of the outer network.

6. A sample augmentation and network iterative training device based on LLM, characterized in that, include: The training data acquisition module is used to acquire the problem background and corresponding initial training data for several different basic planning tasks. The text processing module is used to process the problem background and the corresponding initial training data into text, so as to obtain a text description of the problem background and a text description of the training data. The prompt determination module is used to provide prompts to the LLM based on the problem background text description using a feature-based stepwise prompting method, and generate structured prompts; Specifically, this includes: selecting the first from the background text description of the problem. i Class features A textual feature set is constructed, which contains several specific features. The features in the feature set are used to describe the prompt samples. The prompt samples are used to prompt the LLM, and a test question is used to verify whether the LLM can get the correct answer. If the correct answer is not obtained, a feature is selected from the background text description of the question, and prompt samples are generated to prompt and test the LLM. If the correct answer is obtained, a structured prompt is obtained. The sample augmentation module is used to input the structured prompts and the text descriptions of the training data into a large language model for specific tasks, and generate augmented training data for each basic planning task. The basic model training module is used to train the basic network using augmented training data from each basic planning task to obtain the basic model; it employs a network iteration method. The network iterative training module is used to iteratively train the base model based on the new planning task when a new planning task is encountered, thereby obtaining a dedicated network for the new planning task. During the iterative training process, the base model interacts with simulation engines for different new tasks to obtain interaction samples. These interaction samples are then enhanced using LLM (Liquidity Module) and the base model continues to be trained. After several rounds of iterative training, the updated network replaces the original base model, and the network continues to interact with the simulation engine to generate new training data until the iteratively updated network converges to the point where it can complete the task, thus obtaining a dedicated model for the planning task. The tasks include planning tasks in the field of autonomous driving and complex tasks such as assembling parts and transporting goods performed by various robots in the field of robot control. Specifically, step S10 involves: using the base network to interact with the simulation engine or environment to generate interaction samples, and then enhancing these interaction samples using LLM to obtain enhanced samples. The base network is a k-th generation network. Step S20: Store the enhanced samples in the sample pool, and remove old samples from the sample pool according to a preset removal rule; the preset removal rule is: except for the first k - N Generation basic network to the first k All samples other than those generated by the base network are removed from the sample pool; Step S30: Randomly select or select a certain number of samples from the sample pool according to weights to train the target network. Initialize the target network into a base network and update the parameters of its inner network in each training round. Step S40: Repeat step S30, each time the loop iterates... M After that, the target network is used to iteratively update the base network, and... k The value is incremented by 1; a preset rejection rule is triggered, and samples in the sample pool are removed; whenever the loop reaches... M After +1, update the parameters of the outer network of the target network; Steps S10 and S20 are independent of subsequent steps and will be executed in a loop until the training ends. The basic network consists of two parts, and the parameters of the two parts are defined as follows: and ; These are the parameters of the outer network. These are the parameters of the inner network; Network iterative training devices are also suitable for reinforcement learning; During training, the policy gradient method updates the parameters of the inner network; the update formula is: in, , These are the parameters of the inner network before and after the update, respectively; To represent the learning rate, N This represents the total number of trajectories used in each training round. Indicates the first The sum of rewards for each trajectory, and They represent the first The first of the trajectories Each state and action, For the outer network, Indicates the length of the trajectory; This is a gradient descent operation; During training, the outer network updates its parameters using gradient descent on the sum of the loss values ​​of all subtask samples.

7. A computer device, comprising a memory and a processor, characterized in that, The memory stores a computer program, and when the processor executes the computer program, it implements the steps of the LLM-based sample augmentation and network iterative training method according to any one of claims 1 to 5.

Images