A language model based multi-agent multi-task sequential decision method and system

Abstract

The present application relates to a kind of multi-agent multi-task sequence decision-making method and system based on language model, by reconstructing multi-agent, multi-task decision-making as dialog sequence modeling, solve the challenge of input representation mismatch when reinforcement learning method is extended to multi-task scene, and the robustness of distribution outside behavior is improved without reward supervision or online interaction, enhanced the generalization ability of model.The training of decision language model is divided into two stages: supervised fine-tuning stage and group relative strategy optimization stage.Compared with prior art, the present application carries out simulation verification and physical verification in SMAC benchmark test environment, and the experimental evaluation results show that DLM reaches SOTA performance using only offline data set, has strong zero sample generalization ability to unseen scene, and can efficiently adapt to diversified agent configuration, to realize the most advanced performance with single unified model.

Description

Technical Field

[0001] This invention relates to the field of multi-agent collaborative decision-making technology, and in particular to a multi-agent multi-task sequence decision-making method and system based on a language model. Background Technology

[0002] In recent years, multi-agent reinforcement learning has attracted widespread attention for its ability to model tasks and interact with the environment in real-world sequential decision-making problems, particularly in practical applications such as robot swarm planning, vehicle-road-cloud integration, and distributed sensor networks. Offline reinforcement learning (RL) has become a research focus due to its ability to learn from static datasets and train without real-time interaction, demonstrating certain performance advantages in policy learning on fixed datasets. However, existing offline RL methods primarily focus on improving single-task performance, neglecting the challenge of cross-task generalization: most models employ static architectures, encoding states and actions in fixed formats, thus tightly coupling them to specific task definitions and making it difficult to transfer between environments with different input-output structures.

[0003] To improve the general performance of models across different tasks, more attention has been paid to modeling decisions as sequence generation. More flexible data-driven modeling approaches have been achieved through methods such as Decision Transformer and Trajectory Transformer. However, existing methods primarily focus on single-agent scenarios, lacking collaborative and scalable capabilities in multi-agent scenarios, and facing three main problems. First, existing methods based on Centralized Training with Decentralized Execution (CTDE) architectures generally lack compatibility, failing to balance global context and agent autonomy in a single model. Second, the lack of individual rewards reduces the feasibility of these methods. In multi-agent environments, only global rewards are typically provided, not individual rewards, making it difficult to apply sequence modeling methods that rely on individual rewards and leading to credit assignment problems. Furthermore, while offline datasets readily provide centralized observations and actions, they often lack single-step reward signals, further limiting the practicality of reward-conditional models.

[0004] Therefore, there is currently a lack of a method that can achieve multi-agent, multi-task sequence decision-making to solve these problems. Summary of the Invention

[0005] The purpose of this invention is to overcome the shortcomings of the existing technology and provide a multi-agent multi-task sequence decision-making method and system based on a language model, so as to solve or partially solve the problems that the failure to balance global context and agent autonomy in a single model and the lack of individual rewards reduce the feasibility of the method.

[0006] The objective of this invention can be achieved through the following technical solutions:

[0007] One aspect of the present invention provides a multi-agent multi-task sequence decision-making method based on a language model. Based on input multi-agent local observation information, a language model is used to generate corresponding multi-task strategies to achieve multi-agent multi-task sequence decision-making. The process of constructing the language model includes the following steps:

[0008] Raw data generated by the interaction between multiple agents and the environment are collected online, the decision trajectories of the multiple agents are constructed, and the decision trajectories are divided into two offline subsets.

[0009] Based on the first offline subset of data, the pre-trained language model is supervised fine-tuned using a centralized training and distributed execution method.

[0010] Based on the second offline subset, the language model after supervised fine-tuning is used to generate corresponding policies, and actions that are different from the expert actions in the subset or violate the executable constraints in the environment are selected to form a filtered subset.

[0011] Based on the selected subset, a lightweight preference reward function is used to optimize the language model after supervised fine-tuning, resulting in a language model with optimized group relative policy, which is the final language model.

[0012] As a preferred technical solution, the process of obtaining the offline subset data includes the following steps:

[0013] By training a high-performance graph collapse network model, the interaction history between the model and the environment is preserved and trajectories are generated. Offline data is collected in accordance with the data construction method of deep data-driven reinforcement learning datasets.

[0014] Environmental information, agent observations, and actions are mapped to a predefined language format, which is then transformed into a dialogue format of the language model, forming an offline subset dataset.

[0015] As a preferred technical solution, the supervised fine-tuning process includes the following steps:

[0016] The language model is trained to predict response actions based on dialogue history.

[0017] Determine whether the predicted action is valid and allowed. If it does not meet the requirements, re-extract actions from the set of actions whose probability exceeds the top-k threshold in the distribution predicted by the language model.

[0018] As a preferred technical solution, the objectives of the preference optimization include:

[0019] Ensure that the action can be performed under environmental constraints;

[0020] The actions generated by the language model are consistent with the actions in the selected subset.

[0021] As a preferred technical solution, the lightweight preference reward function is as follows:

[0022]

[0023] Where r(o) i ,a i ) for observing o i and action a i The following preference reward function, To observe o i The actions of pairing within the filtered subset. Represents local observation o based on agent i i The corresponding set of available actions.

[0024] As a preferred technical solution, the loss function during supervised fine-tuning is:

[0025]

[0026] in, The loss function is used to supervise fine-tuning, where M is the number of trajectories and T is the number of trajectories. (k) Let N represent the length of the k-th trajectory, and N represent the number of agents. Indicates the observations up to the current time. The history of dialogue to date, Indicates in Probability estimates of previous historical trajectories.

[0027] As a preferred technical solution, the loss function for optimizing the relative strategy of the population is:

[0028]

[0029] in, Let be the loss function for group relative policy optimization. This indicates the calculation of expectation. For observation o i From monitoring and fine-tuning strategies The G candidate actions sampled in the middle, Indicates the observations up to the current time. The history of dialogue to date, The strategy currently being learned. This represents the normalized advantage value calculated based on the relative reward within each sampled action group. is the sequence length of the j-th sampling action, ∈ is the pruning threshold used to ensure training stability, and β controls the strength of KL divergence regularization. Let π be the KL divergence operator. ref This is a strategy for reference.

[0030] Another aspect of the present invention provides a multi-agent, multi-task sequence decision-making system based on a language model, used to implement the aforementioned multi-agent, multi-task sequence decision-making method, wherein the multi-agent, multi-task sequence decision-making system includes:

[0031] A conversational data modeler is used to collect raw data generated by the interaction between multiple agents and the environment online, construct the decision trajectories of the multiple agents, and divide the decision trajectories into two offline subsets.

[0032] The supervised fine-tuning training module is used to supervise the fine-tuning of the pre-trained language model based on the first offline subset dataset, using a centralized training and distributed execution method.

[0033] The out-of-distribution sample filter is used to generate corresponding policies based on the second offline subset of the dataset using a supervised fine-tuned language model, and to filter out actions that are different from the expert actions in the subset or violate the executable constraints in the environment, thus forming a filtered subset.

[0034] The group relative policy optimization module is used to optimize the language model after supervision and fine-tuning based on the selected subset using a lightweight preference reward function, so as to obtain the language model optimized by the group relative policy, which is the final language model.

[0035] In another aspect, an electronic device is provided, comprising: one or more processors and a memory, wherein the memory stores one or more programs, the one or more programs including instructions for executing the aforementioned language model-based multi-agent multi-task sequence decision-making method.

[0036] In another aspect, the present invention provides a computer-readable storage medium including one or more programs executable by one or more processors of an electronic device, the one or more programs including instructions for performing the aforementioned language model-based multi-agent multi-task sequence decision-making method.

[0037] Compared with the prior art, the present invention has at least one of the following beneficial effects:

[0038] (1) It abandons reward data and avoids training the model from scratch. Instead, it uses a temporal attention mechanism to fine-tune the pre-trained language model for implicit credit allocation, which solves the problem of individual reward scarcity common in multi-agent environments. In the GRPO stage, it introduces a simple reward mechanism based on executability to reduce the risk of OOD by punishing invalid actions.

[0039] (2) In multi-agent reinforcement learning, the decision-making process is reconstructed into conversational sequence modeling. This invention can utilize the word segmentation technology of large language models to flexibly encode multiple tasks and embed them into a shared semantic space, thereby achieving flexible representation and solving the problem of input representation mismatch when most reinforcement learning methods are extended to multi-task scenarios.

[0040] (3) Achieving group relative strategy optimization improves the robustness of the system: In the process of group relative strategy optimization, this invention optimizes the preferences by screening samples that differ significantly from the actions obtained by the model, so that the model and the expert strategy are consistent, thereby improving the robustness of the system to different problems.

[0041] (4) Implement language sequence training based on large language model to enhance the system’s generalization ability to complex environments: The data conversion module converts the dataset composed of basic information into natural language data with fixed specifications, thereby improving the large language model’s ability to understand the environment, helping the system to solve problems in unfamiliar and complex environments, and improving the system’s generalization ability. Attached Figure Description

[0042] Figure 1 This is a flowchart of the multi-agent multi-task sequence decision-making method based on a language model in the embodiment;

[0043] Figure 2 This is a schematic diagram of a multi-agent, multi-task sequence decision-making system based on a language model, as shown in the embodiment.

[0044] Figure 3 This is a schematic diagram of the multi-task sequence decision-making process in the four-wheeled unmanned vehicle cluster scenario in the embodiment;

[0045] Figure 4 This is a schematic diagram of the electronic device in the embodiment. Detailed Implementation

[0046] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0047] Example 1

[0048] To address the problems existing in the aforementioned prior art, this embodiment provides a multi-agent multi-task sequence decision-making method based on a language model. The method aims to align the pre-trained language model with the decision-making environment by performing supervised fine-tuning on a conversational dataset, so that the generated actions meet the environmental constraints. Furthermore, the robustness of the model to out-of-distribution actions is enhanced through group relative policy optimization, thereby demonstrating a strong zero-shot generalization ability in unfamiliar scenarios.

[0049] See Figure 1 The method includes the following steps:

[0050] Step S1: Collect data from online multi-agent reinforcement learning algorithms, convert it into textualized conversational sequences, and divide it into two offline subsets.

[0051] The pre-trained Decision Language Model (DLM) interacts with a benchmark environment used to test multi-agent algorithms, and the pre-trained model demonstrates over 80% accuracy when facing the task. The pre-trained model interacts with the environment and generates trajectories; this trajectory information is pre-collected as initial data and then processed.

[0052] The initial dataset's environmental information, agent observations, and actions are mapped to a fixed cue format that conforms to the natural language format standards required by large language models. This facilitates more flexible representation learning within a shared semantic space. In this embodiment, the multi-agent decision trajectory is represented as a structured sequence of observation-action pairs. Formally, each trajectory is defined as:

[0053]

[0054] Where N represents the number of agents, T (k) Let represent the length of the k-th trajectory. Here, Let represent the observation of agent i at time t in the k-th trajectory. This represents the corresponding action. Since each observation-action pair has been converted into a conversational format, the raw data is stacked and constructed in the order of formula (1), and the data is divided into two subsets.

[0055] Specifically, the process of transforming into a conversational sequence involves mapping environmental information, agent observations, and actions to a fixed language format, and further transforming it into a large language model's conversational format.

[0056] Specifically, the offline subset includes offline data collected by training a high-performance Transformer-Based Graph Coarsening Network (TGCNet) model, preserving the interaction history between the model and the environment and generating trajectories, and following the data construction method of Datasets for Deep Data-Driven Reinforcement Learning (D4RL).

[0057] Step S2, supervised fine-tuning stage: Based on the dialogue history of an offline subset of data from step S1, a pre-trained model is trained to predict the assistant's response actions; based on the model, decentralized decision-making and reasoning are implemented for each agent.

[0058] The DLM-SFT model is obtained by supervising the pre-trained model with Supervised Fine-Tuning (SFT). In Supervised Fine-Tuning, the pre-trained model uses the generated dialogue history to predict the agent's decisions and continuously fits the actions in the example data. Furthermore, during training, each agent makes decisions based on local observations, achieving a decentralized decision-making process while retaining the advantages of centralized training. Supervised fine-tuning ensures that the model's output is consistent with the multi-agent decision examples.

[0059] This step provides a representation method of centralized training and distributed execution. That is, during supervised fine-tuning, the dialogue trajectory information between agents is used for centralized training, and decentralization is achieved in the inference stage, that is, each agent generates actions using its local observations.

[0060] Specifically, the supervised fine-tuning process involves training the model to predict response actions based on the dialogue history, determining whether the predicted actions are valid and allowed, and if not, re-extracting actions from the distribution predicted by the model.

[0061] Specifically, during the supervised fine-tuning process, the loss function is shown in formula (2):

[0062]

[0063] Where M is the number of trajectories, T (k) Let N represent the length of the k-th trajectory, and N represent the number of agents. Indicates the observations up to the current time. The history of dialogue up to this point.

[0064] Specifically, during the inference process, each agent generates actions in a decentralized manner. For each agent i at time step t, the model predicts the action using formula (3):

[0065]

[0066] in, Indicates in Probability estimates of previous historical trajectories.

[0067] If the action is not allowed or is invalid, it will be resampled in the predicted distribution after the stage. At this time, only tokens with a probability exceeding the top-k threshold will be sampled. The sampling formula is shown in formula (4):

[0068]

[0069] Step S3: Based on the second offline subset, generate the corresponding strategy, compare it with the original dataset, and filter out out-of-distribution samples;

[0070] This step, based on the output of the DLM-SFT model and the constructed conversational sequence dataset (i.e., the second offline subset), filters out samples whose predicted actions differ significantly from expert actions or violate environmental constraints. Specifically, after obtaining the DLM-SFT through supervised fine-tuning training, due to the inherent limitations of the dataset coverage, it may still occasionally generate invalid or out-of-distribution actions. To address this issue, this step performs sample filtering on the second offline subset to prepare for subsequent preference optimization training.

[0071] Specifically, the screening process includes retaining two types of action samples: actions that differ from expert actions in the dataset or violate executable constraints in the environment.

[0072] During the data screening process, due to the excessively large multi-agent observation-action space in some tasks, the offline dataset can only occupy a sparse subset of the space. Therefore, the dataset is screened by predicting actions, expert actions, and approximate environmental actions. The screened dataset is formally defined as Equation (5):

[0073]

[0074] in, Represents local observation o based on agent i i The corresponding set of available actions.

[0075] Step S4: Based on the filtered subset, preference optimization is introduced. A set of candidate actions is sampled based on the SFT policy, and the model is further trained using a lightweight reward signal.

[0076] This step, based on the new subset constructed in step S3, introduces biased training to penalize out-of-distribution (OOD) behavior, which helps improve robustness to OOD behavior and reduces the model training load. Finally, the group relative policy optimization module optimizes the decision consistency based on two benchmarks by manually designing a simple objective, resulting in the DLM-GRPO model after Group Relative Policy Optimization (GRPO), which improves the model's generalization performance and accuracy.

[0077] Specifically, the preference optimization process includes penalizing out-of-distribution behavior with a lightweight reward objective, and further, scoring the agent's actions to encourage the model to generate actions consistent with those in the filtered dataset.

[0078] This step optimizes preferences based on two criteria: ensuring actions are executable under environmental constraints; and encouraging actions generated by the model to be consistent with... The actions within remain consistent.

[0079] A lightweight preference reward function r(o) is defined. i ,a i ), used for observing o i and action a i Scoring, r(o) i ,a i Specifically, this is expressed as formula (6):

[0080]

[0081] in, Representation and observation o i exist The action of pairing data in the dataset. Based on And the corresponding preference reward, this invention performs GRPO on the DLM-SFT model to further optimize its decision consistency. For each observation o i From the SFT strategy A set of G candidate actions is sampled. The loss function of GRPO is defined by formula (7):

[0082]

[0083] in, This represents the normalized advantage value calculated based on the relative reward within each sampled action group. is the sequence length of the j-th sampling action, ∈ is the pruning threshold used to ensure training stability, and β controls the strength of KL divergence regularization.

[0084] By training with GRPO, the DLM model significantly improves its robustness to out-of-distribution actions while maintaining consistency with the in-distribution behavior established in the SFT stage.

[0085] To verify the effectiveness of this method, simulation and physical verification were conducted in the SMAC (StarCraft Multi-Agent Challenge) benchmark environment. Experimental evaluation results show that DLM achieves state-of-the-art (SOTA) performance using only offline datasets, has strong zero-shot generalization ability for unseen scenarios, and can efficiently adapt to diverse agent configurations, achieving state-of-the-art performance with a single unified model.

[0086] The implementation process of this method is illustrated below using a practical application scenario as an example. (See [link to relevant documentation]). Figure 3 In this example, four-wheeled autonomous vehicles are used as intelligent agents. Through the DLM framework, the four-wheeled autonomous vehicle swarm can efficiently complete multi-task sequence decision-making. The specific implementation steps are as follows:

[0087] Step S01: Build an integrated equipment platform that includes a four-wheeled unmanned vehicle control system, deploy sensors and integrate DLM modules.

[0088] Step S02: Collect collaborative task data of the four-wheeled unmanned vehicle cluster, including environmental information, observations and actions, and convert them into conversational sequences.

[0089] Step S03: Based on the collected offline data subset, the pre-trained language model is fine-tuned in the supervised fine-tuning stage to generate actions that conform to the task constraints of the four-wheeled unmanned vehicle.

[0090] Step S04: Further optimize the model through the group relative strategy optimization stage to enhance its robustness to out-of-distribution actions.

[0091] Step S05: After completing the training of the DLM-GRPO model adapted to the specific scenario, during the centralized training phase, the four-wheeled autonomous vehicle cluster receives global task instructions. The DLM module analyzes sensor data and generates a global action value function to guide the cluster's collaborative behavior. During the distributed execution phase, each four-wheeled autonomous vehicle independently generates actions based on local observation information through the DLM module, achieving decentralized decision-making. If the actions do not meet the task requirements, the control system sends instructions to regenerate the actions until the task requirements are met.

[0092] This method has the following beneficial effects:

[0093] (1) The general language model for multi-agent multi-task sequence decision-making in this method reconstructs the decision-making process into conversational sequence modeling in multi-agent reinforcement learning. This invention can utilize the word segmentation technology of a large language model to flexibly encode multiple tasks and embed them into a shared semantic space, thereby achieving flexible representation and solving the problem of input representation mismatch when most reinforcement learning methods are extended to multi-task scenarios.

[0094] (2) The general language model for multi-agent multi-task sequence decision-making in this method, and the conversational trajectory representation designed for each agent, enable this method to be compatible with the CTDE structure. During centralized training, the context between agents can be preserved, and at the same time, it supports fractional execution based on local observation, effectively balancing global collaboration and individual autonomy.

[0095] (3) The general language model for multi-agent multi-task sequence decision-making in this method abandons reward data and avoids training the model from scratch. Instead, it uses a temporal attention mechanism to fine-tune the pre-trained language model for implicit credit allocation, which solves the problem of individual reward scarcity common in multi-agent environments. Furthermore, a simple reward mechanism based on executability is introduced in the GRPO stage to reduce the risk of OOD by punishing invalid actions.

[0096] Example 2

[0097] Based on Example 1, see Figure 2 This embodiment provides a multi-agent multi-task sequence decision-making system based on a language model, used to implement the multi-agent multi-task sequence decision-making method of Embodiment 1. The multi-agent multi-task sequence decision-making system 1 includes a conversational data modeler 2, a supervised fine-tuning training module 3, an out-of-distribution sample filter 4, a group relative policy optimization module 5, and an agent 6.

[0098] The conversational data modeler 2 accepts raw data generated from the interaction between the pre-trained model and the environment, constructs the data into a dialogue sequence pattern adapted to a large language model, and divides the modeled conversational language sequence data into two subsets for subsequent use. The first subset is output to the supervised fine-tuning training module 3.

[0099] The supervised fine-tuning training module 3 accepts the conversational dataset (subset one) constructed by the conversational data modeler 2 and performs supervised fine-tuning on the pre-trained model to obtain the DLM-SFT model, ensuring that the model's output is consistent with the multi-agent decision examples. Specifically, in the supervised fine-tuning training module, the pre-trained model uses the generated dialogue history to predict the agents' decisions and continuously fits the actions in the example data. Furthermore, during training, each agent makes decisions based on local observations, achieving a decentralized decision-making process while retaining the advantages of centralized training.

[0100] The out-of-distribution sample filter 4 accepts the output of the DLM-SFT model and the conversational sequence dataset (subset two) constructed by the conversational data modeler, and filters out samples whose predicted actions differ significantly from expert actions or violate environmental constraints, outputting them to the group relative policy optimization module 5. Specifically, after the DLM-SFT is trained by the supervised fine-tuning training module 3, it may still occasionally generate invalid or out-of-distribution actions due to the inherent limitations of the dataset coverage. To address this problem, this invention performs sample filtering on the conversational sequence dataset (subset two) for further preference optimization training.

[0101] The group relative policy optimization module 5 accepts a new subset constructed by the out-of-distribution sample filter 4 and introduces biased training to penalize OOD behavior, which helps improve the robustness of OOD behavior and reduces the model training load. Finally, the group relative policy optimization module optimizes the decision consistency based on two benchmarks by manually designing a simple objective, resulting in the DLM-GRPO model, which improves the model's generalization performance and accuracy.

[0102] The present invention has the following beneficial effects:

[0103] (1) Implement language sequence training based on large language model to enhance the system’s generalization ability to complex environments: The data conversion module converts the dataset composed of basic information into natural language data with fixed specifications, thereby improving the large language model’s ability to understand the environment, helping the system to solve problems in unfamiliar and complex environments, and improving the system’s generalization ability.

[0104] (2) Achieving group relative strategy optimization improves the robustness of the system: In the process of group relative strategy optimization, this invention selects samples that differ significantly from the actions obtained by the model for preference optimization, so that the model and the expert strategy are consistent, thereby improving the robustness of the system to different problems.

[0105] Example 3

[0106] This embodiment provides an electronic device, including: one or more processors and a memory, wherein the memory stores one or more programs, the one or more programs including instructions for executing the language model-based multi-agent multi-task sequence decision-making method as described in Embodiment 1.

[0107] like Figure 4 At the hardware level, the electronic device includes a processor, internal bus, network interface, memory, and non-volatile memory, and may also include other hardware required for the business operations. The processor reads the corresponding computer program from the non-volatile memory into memory and then runs it to achieve the above-mentioned functions. Figure 1The method described herein. Of course, in addition to software implementation, this invention does not exclude other implementation methods, such as logic devices or a combination of hardware and software, etc. That is to say, the execution subject of the following processing flow is not limited to each logic unit, but can also be hardware or logic devices.

[0108] Memory may include non-persistent storage in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM. Memory is an example of computer-readable media.

[0109] Computer-readable media includes both permanent and non-permanent, removable and non-removable media that can store information using any method or technology. Information can be computer-readable instructions, data structures, modules of programs, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, disk storage or other magnetic storage devices, or any other non-transferable medium that can be used to store information accessible by a computing device. As defined herein, computer-readable media does not include transient computer-readable media, such as modulated data signals and carrier waves.

[0110] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any person skilled in the art can easily conceive of various equivalent modifications or substitutions within the technical scope disclosed in the present invention, and these modifications or substitutions should all be covered within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A multi-agent, multi-task sequence decision-making method based on a language model, characterized in that, Based on the input multi-agent local observation information, a language model is used to generate corresponding multi-task policies to achieve multi-agent multi-task sequence decision-making. The construction process of the language model includes the following steps: The raw data generated by the interaction between multi-agent and the environment is collected online, transformed into text-based conversational sequences, and the decision trajectories of multi-agent are constructed. The decision trajectories are then divided into two offline subsets. Based on the first offline subset of data, the pre-trained language model is supervised fine-tuned using a centralized training and distributed execution method. Based on the second offline subset, the language model after supervised fine-tuning is used to generate corresponding policies, and actions that are different from the expert actions in the subset or violate the executable constraints in the environment are selected to form a filtered subset. Based on the selected subset, a lightweight preference reward function is used to optimize the language model after supervised fine-tuning, resulting in a language model optimized by the group relative policy, which is used as the final language model. The lightweight preference reward function is as follows: in, In order to observe and actions The following preference reward function, In order to observe The actions of pairing within the filtered subset. Indicates based on intelligent agent Local observation The corresponding set of available actions; The loss function for optimizing the relative strategy of the population is: in, Let be the loss function for group relative policy optimization. This indicates the calculation of expectation. For observation From monitoring and fine-tuning strategies Mid-sampling One candidate action, Indicates the observations up to the current time. The history of dialogue to date, The strategy currently being learned. This represents the normalized advantage value calculated based on the relative reward within each sampled action group. It is the first The sequence length of each sampling action. It is a pruning threshold used to ensure training stability. Controlling the strength of KL divergence regularization, Let KL divergence operator be used. This is a strategy for reference.

2. The multi-agent, multi-task sequence decision-making method based on a language model according to claim 1, characterized in that, The process of obtaining the offline subset dataset includes the following steps: By training a high-performance graph collapse network model, the interaction history between the model and the environment is preserved and trajectories are generated. Offline data is collected in accordance with the data construction method of deep data-driven reinforcement learning datasets. Environmental information, agent observation information, and actions are mapped to a preset language format, which is then transformed into the dialogue format of the language model to form an offline subset dataset.

3. The multi-agent, multi-task sequence decision-making method based on a language model according to claim 1, characterized in that, The aforementioned supervised fine-tuning process includes the following steps: The language model is trained to predict response actions based on dialogue history. Determine whether the predicted action is valid and allowed. If it does not meet the requirements, re-extract actions from the set of actions whose probability exceeds the top-k threshold in the distribution predicted by the language model.

4. The multi-agent, multi-task sequence decision-making method based on a language model according to claim 1, characterized in that, The objectives of the preference optimization include: Ensure that the action can be performed under environmental constraints; The actions generated by the language model are consistent with the actions in the selected subset.

5. The multi-agent, multi-task sequence decision-making method based on a language model according to claim 1, characterized in that, The loss function for supervised fine-tuning is: in, The loss function is used to supervise fine-tuning. It is the number of trajectories. Indicates the first The length of the trajectory, Indicates the number of agents. Indicates the observations up to the current time. The history of dialogue to date, Indicates in Probability estimates of previous historical trajectories.

6. A multi-agent, multi-task sequence decision-making system based on a language model, characterized in that, For implementing the multi-agent multi-task sequence decision-making method as described in any one of claims 1-5, the multi-agent multi-task sequence decision-making system comprises: A conversational data modeler is used to collect raw data generated by the interaction between multiple agents and the environment online, construct the decision trajectories of the multiple agents, and divide the decision trajectories into two offline subsets. The supervised fine-tuning training module is used to supervise the fine-tuning of the pre-trained language model based on the first offline subset dataset, using a centralized training and distributed execution method. The out-of-distribution sample filter is used to generate corresponding policies based on the second offline subset of the dataset using a supervised fine-tuned language model, and to filter out actions that are different from the expert actions in the subset or violate the executable constraints in the environment, thus forming a filtered subset. The group relative policy optimization module is used to optimize the language model after supervision and fine-tuning based on the selected subset using a lightweight preference reward function, so as to obtain the language model optimized by the group relative policy, which is the final language model.

7. An electronic device, characterized in that, include: One or more processors and a memory, wherein the memory stores one or more programs, the one or more programs including instructions for executing the language model-based multi-agent multi-task sequence decision-making method as described in any one of claims 1-5.

8. A computer-readable storage medium, characterized in that, It includes one or more programs that are executed by one or more processors of an electronic device, the one or more programs including instructions for performing the language model-based multi-agent multi-task sequence decision-making method as described in any one of claims 1-5.

Images