An efficient human-machine collaboration strategy deployment method based on natural language instructions

Abstract

The present invention discloses an efficient human-machine collaboration strategy deployment method based on natural language instructions. The method compresses the optimal collaboration strategy parameters of diverse teammates into a single conditional diffusion model by means of the powerful representation ability of the conditional diffusion model guided by natural language, and generates customized collaboration strategies for teammates based on the natural language instructions provided by human teammates during the deployment phase. The method can realize the generation and deployment of collaboration strategies with only a small number of natural language instructions in actual scenarios with high trial and error costs, and can ensure the efficiency of human-machine collaboration strategy deployment.

Description

Technical Field

[0001] The present invention relates to an efficient human-machine collaboration strategy deployment method, and in particular to an efficient human-machine collaboration strategy deployment method based on natural language instructions. Background Art

[0002] Human-machine collaboration focuses on the comprehensive quality and efficiency of the overall work results of a team composed of human workers and artificial intelligence systems, in order to seamlessly integrate artificial intelligence systems into existing human workflows. In the process of collaborative work, it is necessary to reasonably integrate the experience, knowledge and rational judgment ability of human workers with the high-speed computing and pattern recognition capabilities of artificial intelligence, break the information barriers between humans and artificial intelligence systems, and realize knowledge transfer between humans and machines, so as to give full play to their respective advantages and improve the quality and efficiency of collaborative work results. Traditional human-machine collaboration mainly adopts automated control methods, which require modeling for the behaviors of different human workers, and rely on experts in related fields to formulate collaboration strategies and design human-machine collaboration strategies that meet actual usage scenarios. This type of human-machine collaboration method has poor generalization ability and lacks the ability to adapt to human collaborators with different behavioral styles. In addition, when facing different usage scenarios, it often needs to rely on domain experts to redesign collaboration strategies, which is costly.

[0003] In recent years, with the vigorous development of machine learning, especially deep learning, reinforcement learning technology has provided new solutions for human-machine collaboration. Reinforcement learning enables intelligent agents to learn in the process of interacting with the environment. Intelligent agents obtain rewards or penalties by interacting with the environment, thereby learning strategies that can adapt to the environment and maximize rewards. In cooperative multi-agent reinforcement learning, multiple agents interact with the environment at the same time and obtain common rewards or penalties, so that multiple agents learn to adapt to the behavior of teammates to complete collaborative tasks together. Due to the high time cost and trial-and-error cost of human collaborators participating in training, the current mainstream method models human-machine collaboration as a zero-shot collaboration problem, and models the strategy of human collaborators as an agent in the multi-agent system, so as to use collaborative multi-agent reinforcement learning technology for training. This type of method usually requires the construction of a diverse teammate population as a human agent strategy, and then uses multi-agent reinforcement learning technology to train a single general collaboration strategy, or trains multiple collaboration strategies for teammates of different styles, and selects the optimal collaboration strategy in the deployment phase.

[0004] Although multi-agent reinforcement learning technology has achieved certain results in the field of human-machine collaboration and has achieved more robust results than traditional methods in areas such as game theory, in many real application scenarios such as intelligent collaborative confrontation scenarios, the behavioral styles of different human collaborators are usually quite different. A single general collaboration strategy trained based on reinforcement learning will be affected by the behavior of multimodal teammates and cannot accurately identify collaborative teammates, thus affecting the coordination between agents and humans. Therefore, when facing human teammates with changeable behavioral styles, it is usually necessary to train multiple collaboration strategies for teammates with different styles to cope with the challenges brought by multimodal teammates' behavior to human-machine collaboration. However, after training multiple collaboration strategies, in the strategy deployment stage, existing methods often need to interact with the environment in a small amount to obtain task features and then select the optimal collaboration strategy, which is costly and inefficient. Summary of the invention

[0005] Purpose of the invention: Based on the current problems with human-machine collaboration, how to achieve efficient human-machine collaboration strategy deployment in an environment with high trial and error costs is the problem that the present invention aims to solve. The present invention proposes an efficient human-machine collaboration strategy deployment method based on natural language instructions. In actual scenarios with high trial and error costs, the generation and deployment of collaboration strategies are achieved through only a small number of natural language instructions, which can ensure the efficiency of human-machine collaboration strategy deployment. With the help of a conditional diffusion model guided by natural language, this method generates customized collaboration strategies for teammates based on the natural language instructions provided by human teammates.

[0006] Technical solution: An efficient human-machine collaboration strategy deployment method based on natural language instructions, comprising the following steps:

[0007] Step 1: Based on the collaboration scenario, define task-related events to construct task language representation;

[0008] Step 2: Build a diverse teammate population through population training methods or behavior cloning as a human proxy strategy, and collect the task language representation dataset and natural language instruction dataset corresponding to the teammates;

[0009] Step 3: Train the optimal cooperation strategy for the diverse teammate population. First, initialize a cooperation strategy for each teammate strategy, then fix the teammate strategy parameters, and train the cooperation strategy to the optimal cooperation strategy through the PPO algorithm.

[0010] Step 4: Train the strategy parameter encoding module and the strategy parameter decoding module. The strategy parameter encoding module and the strategy parameter decoding module constitute a variational autoencoder on the strategy parameter space, and are jointly trained by minimizing the loss function.

[0011] Step 5: Fine-tune the Bert model; add a fully connected layer at the output of the Bert model, encapsulate the Bert model as a natural language instruction classifier, and use the natural language instruction dataset to fine-tune it by minimizing the cross entropy loss function;

[0012] Step 6: Train the language conversion module; fix the parameters of the fine-tuned BERT model and only train the variational autoencoder in the language conversion module so that it can convert the natural language instructions encoded by the BERT model into the corresponding task language representation;

[0013] Step 7: compress the optimal collaboration strategy; with the help of the trained strategy parameter encoding module, compress the optimal collaboration strategy of the teammate population into a strategy parameter representation dataset;

[0014] Step 8: Train the conditional diffusion model; use the task language representation dataset and the strategy parameter representation dataset to train the conditional diffusion model by minimizing the loss function, so that the conditional diffusion model generates the corresponding collaborative strategy parameter representation according to the task language representation;

[0015] Step 9: Human teammates use natural language instructions to express their behavioral preferences or task requirements;

[0016] Step 10: The language conversion module converts the natural language instructions into task language representation;

[0017] Step 11: Randomly sample Gaussian noise and use the conditional diffusion model to perform reverse denoising;

[0018] Step 12: The strategy parameter decoding module predicts the strategy network parameters based on the collaborative strategy parameter representation, initializes the collaborative strategy network and loads the strategy network parameters to obtain the reconstructed optimal collaborative strategy.

[0019] If the behavioral preferences of human teammates change or the task requirements change during the collaboration process, the above steps 9 to 12 can be repeated to generate and deploy the optimal collaboration strategy based on the new natural language instructions.

[0020] In step 2, a diverse teammate population is constructed through population training methods or behavior cloning. , as a human proxy strategy, and collect the corresponding task language representation dataset of teammates and natural language instruction datasets ; Each strategy Corresponding to the unique task language representation and natural language instructions with similar semantics, is the size of the teammate population; during the training process, the frequency of teammate strategies executing predefined task-related events within the trajectory is recorded and recorded as a vector ; According to the following formula for the vector The various dimensions of are discretized to obtain the task language representation :

[0021]

[0022] in, is a constant, and Respectively represent L T and the i-th component of v, d is the number of task-related events.

[0023] In step 3, the optimal collaboration strategy is trained for the diverse teammate population; first, for each teammate strategy Initialize a collaboration strategy Then fix the teammate strategy parameters and train the collaboration strategy through the PPO algorithm Get the optimal collaboration strategy Specifically, the collaboration strategy during training should match the behavior style of teammates to maximize the cumulative reward:

[0024]

[0025] During the training process, the state s of each step is t Save to an experience replay array D for step 4 to train the strategy parameter encoding module and strategy parameter decoding module.

[0026] In step 4, the strategy parameter encoding module and the strategy parameter decoding module are trained; the strategy parameter encoding module and policy parameter decoding module Construct a variational autoencoder on the policy parameter space and jointly train by minimizing the following loss function:

[0027]

[0028] Among them, the strategy parameter encoding module Input strategy network weights at each layer and bias , respectively get the weight representation and bias characterization , splicing and And then output the strategy parameter representation z after passing through the fully connected layer; strategy parameter decoding module Predict the policy network parameters based on the policy parameter representation z and the policy structure representation g, where g is a vector containing the number of neurons in each layer of the policy network; is the collaborative strategy reconstructed by the parameter decoding module, is the Kullback-Leibler divergence function.

[0029] In step 5, fine-tuning is performed by minimizing the following cross entropy loss function so that the Bert model can more accurately capture the semantic information in the natural language instructions:

[0030] .

[0031] In step 8, the conditional diffusion model is trained by minimizing the following loss function:

[0032]

[0033] in, ; ; is the representation obtained in the i-th step of the forward noise addition process, β i is the variance of the Gaussian noise added in the i-th step during the forward diffusion process, α i =1-β i , ; Diffusion Model according to , the current time step i and the noise added to the i-th step predicted by the task language encoding.

[0034] In step 11, randomly sampled Gaussian noise , using the conditional diffusion model , based on the reverse denoising process described by the following formula For denoising, i takes values ​​from N to 1, which means N steps of denoising:

[0035]

[0036] in, , Diffusion Model according to , the noise added to the i-th step predicted by the current time step i and the task language encoding, β i is the variance of the Gaussian noise added in the i-th step during the forward diffusion process, that is, , , α i =1-β i , , It is the task language representation, which is used to guide the diffusion model to denoise Gaussian noise and obtain the collaborative strategy parameter representation .

[0037] A system for implementing the above-mentioned efficient human-machine collaborative strategy deployment method based on natural language instructions, comprising a language conversion module, a strategy parameter encoding module, a conditional diffusion model and a strategy parameter decoding module;

[0038] The language conversion module is used to convert the natural language instructions of human teammates into task language representation; it is composed of a pre-trained Bert model and a variational autoencoder, and can convert the natural language instructions of human teammates into task language encoding; specifically, the pre-trained Bert model first encodes the natural language instructions, and then the variational autoencoder converts the encoded natural language instructions into the corresponding task language encoding;

[0039] The strategy parameter encoding module is used to compress the parameters of the collaborative strategy into a low-dimensional latent space; the strategy parameter decoding module is used to reconstruct the parameters of the collaborative strategy from the strategy parameter representation; the strategy parameter encoding module and the strategy parameter decoding module constitute a variational autoencoder, and the strategy parameter encoding module and the strategy parameter decoding module are trained simultaneously in an end-to-end manner. The strategy parameter encoding module is composed of a convolutional neural network and a fully connected neural network. First, the weights and biases of the strategy network are encoded by the convolutional neural network and the fully connected neural network respectively, and then the weight representation and the bias representation are combined, and finally the parameter representation is obtained through the fully connected layer;

[0040] The conditional diffusion model is used to generate the strategy parameter representation in the latent space from the task language representation. The conditional diffusion model uses a U-net as the main architecture, which consists of a down-sampling encoder, an up-sampling decoder, and a bottleneck layer connecting the two. The output representation of each up-sampling layer of the decoder is concatenated with the output representation of the corresponding layer of the encoder as the input of the next up-sampling layer. The diffusion model is called cyclically. Can be based on , time step i and task language representation are used to denoise the randomly initialized Gaussian noise to obtain the collaborative strategy parameter representation corresponding to the task language encoding; for the strategy parameter representation, the forward diffusion process gradually adds Gaussian noise to the parameter representation, and the reverse process gradually denoises the Gaussian noise to obtain the strategy parameter representation;

[0041] The policy parameter decoding module includes a graph hypernetwork, which can predict the parameters of the policy network according to the policy parameter representation and the policy structure representation; in the training phase, the policy parameter representation is generated by the policy parameter encoding part; in the deployment phase, the policy parameter representation is generated by the conditional diffusion model; given the policy network structure, the policy structure representation is a fixed vector, whose components represent the number of neurons in each layer of the policy network;

[0042] The language conversion module and the conditional diffusion model are trained independently.

[0043] A computer device comprises a memory, a processor and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the steps of the efficient human-computer collaboration strategy deployment method based on natural language instructions as described above are implemented.

[0044] A computer-readable storage medium stores a computer program for executing the above-mentioned efficient human-machine collaboration strategy deployment method based on natural language instructions.

[0045] Beneficial effects: The present invention provides an efficient human-machine collaborative strategy deployment method based on natural language instructions. With the powerful expression ability of the conditional diffusion model, when collaborating with human teammates, only a small amount of natural language instructions from teammates can generate customized collaborative strategies and collaboratively complete tasks, which can effectively avoid the high cost and inefficiency caused by interacting with the environment during the strategy deployment stage. The present invention can be applied to many fields such as game games, intelligent collaborative confrontation, and household robots, and has broad application prospects. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] Figure 1 is a method flow chart of an embodiment of the present invention;

[0047] Figure 2 is a framework diagram of a method according to an embodiment of the present invention;

[0048] Figure 3 Schematic diagram of a language conversion module described in an embodiment of the present invention;

[0049] Figure 4 This is a schematic diagram of a strategy parameter encoding module described in an embodiment of the present invention;

[0050] Figure 5 Schematic diagram of a policy parameter decoding module according to an embodiment of the present invention;

[0051] Figure 6 Schematic diagram of the conditional diffusion model module described in an embodiment of the present invention. DETAILED DESCRIPTION

[0052] The present invention is further explained below in conjunction with specific embodiments. It should be understood that these embodiments are only used to illustrate the present invention and are not used to limit the scope of the present invention. After reading the present invention, various equivalent forms of modifications to the present invention by those skilled in the art all fall within the scope defined by the claims attached to this application.

[0053] The human-machine collaboration problem can be modeled as a two-player cooperative Markov decision process ,in, is the state space, and are the action spaces of the two agents respectively. is the state transfer function. The joint action of the two agents will result in state transfer and Given a shared reward. Assuming human teammates use natural language instructions Express your behavioral preferences, and your agent strategy is recorded as , the corresponding optimal collaboration strategy is recorded as Both the human agent strategy and the optimal collaboration strategy are represented using a fully connected neural network, which can obtain the optimal action under the single-step agent control process based on the given observation information.

[0054] The efficient natural language-based collaborative strategy deployment method proposed in the present invention uses the powerful expressive power of the conditional diffusion model to generate customized collaborative strategies for teammates based on the natural language instructions provided by human teammates. Since natural language exhibits variability and redundancy, natural language instructions with similar semantics may have multiple ways of expression, and there are many task-independent grammatical components in natural language instructions. In order to more accurately describe the behavioral preferences of human teammates, the present invention proposes to use task language representation to describe the behavioral style of human strategies. The details of the task language representation will be described in the specific steps.

[0055] like Figure 1 As shown, the efficient collaborative strategy deployment method based on natural language instructions specifically includes the following steps:

[0056] Step 1: Based on the specific collaboration scenario and combining domain-related knowledge, define a series of task-related events to construct the task language representation.

[0057] Step 2: Build a diverse teammate population through population training methods or behavioral cloning , as a human proxy strategy, and collect the corresponding task language representation dataset of teammates and natural language instruction datasets . Each strategy Corresponding to a unique task language representation and natural language instructions with similar semantics, is the size of the teammate population. Specifically, during the training process, the frequency of teammate strategies executing predefined task-related events within the trajectory is recorded and recorded as the vector ,in is the number of task-related events. The various dimensions of are discretized to obtain the task language representation :

[0058]

[0059] in, It is a small constant with a value in the interval (0,1), such as 0.1. and Respectively and the i-th component of v, clip represents the truncation function, which limits the first value in the brackets to 0 and 1. Between, more than 1- When taking 1- , It means taking the maximum component value in vector v, and j represents the component index.

[0060] Step 3: Train the optimal collaboration strategy for a diverse group of teammates. First, for each teammate strategy Initialize a collaboration strategy Then fix the teammate strategy parameters and train the collaboration strategy through the PPO algorithm Get the optimal collaboration strategy , and finally get the optimal collaboration strategy set Specifically, the collaboration strategy during training should match the behavior style of teammates to maximize the cumulative reward:

[0061]

[0062] During the training process, the state of each step Save to an experience playback array D, which is used for step 4 to train the strategy parameter encoding module and strategy parameter decoding module. t represents the time step in the Markov decision process, represents the action output by teammate 1 at time step t, represents the state of the environment at time step t, represents the action output by teammate 2 at time step t. R represents the reward function of the Markov decision process, It means that the environment gives reward feedback according to the state at time step t and the actions of the two teammates.

[0063] Step 4: Train the strategy parameter encoding module and strategy parameter decoding module. Strategy parameter encoding module and policy parameter decoding module Construct a variational autoencoder on the policy parameter space and jointly train by minimizing the following loss function:

[0064]

[0065] Among them, the strategy parameter encoding module Input strategy network layer weights and bias , respectively get the weight representation and bias characterization , splicing and And then output the strategy parameter representation z after passing through the fully connected layer; strategy parameter decoding module Predict the policy network parameters based on the policy parameter representation z and the policy structure representation g, where g is a vector containing the number of neurons in each layer of the policy network; , is the collaborative strategy reconstructed by the parameter decoding module, is the Kullback-Leibler divergence function.

[0066] Step 5: Fine-tune the Bert model. Add a fully connected layer at the output end. After adding a fully connected layer at the output end of the Bert model, a natural language instruction classifier is formed. , using the natural language instruction dataset collected in step 2 , fine-tuned by minimizing the following cross entropy loss function, so that the BERT model can more accurately capture the semantic information in natural language instructions:

[0067]

[0068] K represents the number of possible natural language instructions for each teammate style type. The base of log is usually taken as the constant 2 or the base of natural logarithm e, abbreviated as log. P represents probability, represents the aforementioned natural language instruction classifier. The left side of the equation in square brackets represents the classification result (label) of the natural language classifier on the natural language instruction. The equation in square brackets represents that the natural language instruction classifier will Correct classification.

[0069] Step 6: Train the language conversion module. Fixed fine-tuned BERT model Parameters of only training the variational autoencoder in the language conversion module , so that it can convert the natural language instructions encoded by the Bert model into the corresponding task language representation. Specifically, the task language representation dataset and the natural language instruction dataset collected in step 2 are used to train by minimizing the following variational autoencoder loss function:

[0070]

[0071] in, is the variational autoencoder latent space representation corresponding to the natural language instruction, The task language representation reconstructed by the language conversion module according to the natural language instructions, represents the KL divergence function, represents the standard normal distribution, Represents the latent space representation distribution of the variational autoencoder.

[0072] Step 7: Compress the optimal collaborative strategy. Using the strategy parameter encoding module trained in step 4 , the optimal collaboration strategy of the teammate population trained in step 3 Compression into a policy parameter representation dataset , used in step eight to train the conditional diffusion model.

[0073] Step 8: Train the conditional diffusion model. Able to generate corresponding collaborative strategy parameter representation based on task language representation, using the task language representation dataset collected in step 2 And the strategy parameter characterization dataset obtained in step 7 , the conditional diffusion model is trained by minimizing the following loss function:

[0074]

[0075] in, ; ; is the representation obtained in the i-th step of the forward noise addition process, β i is the variance of the Gaussian noise added in the i-th step during the forward diffusion process, α i =1-β i , ; Diffusion Model according to , the current time step i and the noise added to the i-th step predicted by the task language encoding.

[0076] At this point, the training of all modules of the efficient human-machine collaboration strategy deployment method is completed. The specific deployment process is described below.

[0077] Step 9: Human teammates use natural language instructions to express their behavioral preferences or task requirements.

[0078] Step 10: The language conversion module converts natural language instructions Transformed into task language representation .

[0079] Step 11: Randomly sample Gaussian noise , using the conditional diffusion model , based on the reverse denoising process described by the following formula To denoise:

[0080]

[0081] in, After N steps of denoising, the predicted collaborative strategy parameter representation is obtained .

[0082] Step 12: Strategy parameter decoding module based on collaborative strategy parameter representation Predict policy network parameters and initialize collaborative policy network And load the policy network parameters to obtain the reconstructed optimal collaboration strategy .

[0083] If the behavioral preferences of human teammates or the task requirements change during the collaboration process, the above steps nine to twelve can be repeated to generate and deploy the optimal collaboration strategy based on the new natural language instructions.

[0084] The system for implementing the above-mentioned efficient collaborative strategy deployment method based on natural language instructions includes four modules:

[0085] The language conversion module is used to convert teammates’ natural language instructions into task language representation;

[0086] Strategy parameter encoding module, used to compress the parameters of the collaborative strategy into a low-dimensional latent space;

[0087] The conditional diffusion model is used to generate the policy parameter representation in the latent space from the task language representation. It consists of a down-sampling encoder, an up-sampling decoder, and a bottleneck layer connecting the two. The output representation of each up-sampling layer of the decoder is concatenated with the output representation of the corresponding layer of the encoder as the input of the next up-sampling layer. The diffusion model is called cyclically. Can be based on , time step The randomly initialized Gaussian noise is denoised with the task language representation to obtain the collaborative strategy parameter representation corresponding to the task language encoding;

[0088] The strategy parameter decoding module is used to reconstruct the parameters of the collaboration strategy from the strategy parameter representation.

[0089] The policy parameter encoding module and the policy parameter decoding module constitute a variational autoencoder. The policy parameter encoding module and the policy parameter decoding module are trained simultaneously in an end-to-end manner, while the language conversion module and the conditional diffusion model are trained independently.

[0090] The language conversion module consists of a pre-trained Bert model and a variational autoencoder, which can convert the natural language instructions of human teammates into task language encoding. Specifically, the pre-trained Bert model first encodes the natural language instructions, and then the variational autoencoder converts the encoded natural language instructions into task language encoding. Convert to the corresponding task language code .

[0091] The strategy parameter encoding module consists of a convolutional neural network and a fully connected neural network, which can compress the parameters of the collaborative strategy into a representation in a low-dimensional latent space. Specifically, the weights and biases of the strategy network are first encoded through a convolutional neural network and a fully connected neural network, and then the weight representation and bias representation are combined, and finally the parameter representation is obtained through a fully connected layer.

[0092] The conditional diffusion model uses a U-shaped network as the main architecture, which can generate corresponding collaborative strategy parameter representations based on the teammate's task language representations. , the forward diffusion process gradually adds Gaussian noise to the parameter representation. The process of gradually adding Gaussian noise can be expressed as a Markov chain ,in , , is the predefined noise variance, is the identity matrix, Indicates that the noise is total step, for The representation obtained after adding noise can be approximated as pure Gaussian noise, so the first step of the reverse denoising process is to sample from the Gaussian noise The reverse process is based on the following equation for Gaussian noise Step-by-step denoising results in the strategy parameter representation:

[0093]

[0094] in, , In the forward diffusion process The variance of the Gaussian noise added in the first step is ; ; , , conditional diffusion model according to , current time step and the noise added by the forward diffusion process of task language encoding prediction .

[0095] The policy parameter decoding module contains a graph hypernetwork that can predict the parameters of the policy network based on the policy parameter representation and the policy structure representation. In the training phase, the policy parameter representation is generated by the policy parameter encoding part; in the deployment phase, the policy parameter representation is generated by the conditional diffusion model. Given the policy network structure, the policy structure representation is a fixed vector whose components represent the number of neurons in each layer of the policy network.

[0096] Obviously, those skilled in the art should understand that the various steps of the above-mentioned efficient collaborative strategy deployment method based on natural language instructions or the various modules of the efficient collaborative strategy deployment system based on natural language instructions of the embodiments of the present invention can be implemented by a general-purpose computing device, they can be concentrated on a single computing device, or distributed on a network composed of multiple computing devices, optionally, they can be implemented with executable program codes of the computing device, so that they can be stored in a storage device and executed by the computing device, and in some cases, the steps shown or described can be executed in a different order than herein, or they can be made into individual integrated circuit modules respectively, or multiple modules or steps therein can be made into a single integrated circuit module for implementation. In this way, the embodiments of the present invention are not limited to any specific combination of hardware and software.

Claims

1. An efficient human-machine collaboration strategy deployment method based on natural language instructions, characterized in that: The steps include: Step 1: Based on the collaboration scenario, define task-related events to construct task language representation; Step 2: Build a diverse teammate population through population training methods or behavior cloning as a human proxy strategy, and collect the task language representation dataset and natural language instruction dataset corresponding to the teammates; Step 3: Train the optimal cooperation strategy for the diverse teammate population. First, initialize a cooperation strategy for each teammate strategy, then fix the teammate strategy parameters, and train the cooperation strategy to the optimal cooperation strategy through the PPO algorithm. Step 4: Train the strategy parameter encoding module and the strategy parameter decoding module. The strategy parameter encoding module and the strategy parameter decoding module constitute a variational autoencoder on the strategy parameter space, and are jointly trained by minimizing the loss function. Step 5: Fine-tune the Bert model; add a fully connected layer at the output of the Bert model, encapsulate the Bert model as a natural language instruction classifier, and use the natural language instruction dataset to fine-tune it by minimizing the cross entropy loss function; Step 6: Train the language conversion module; fix the parameters of the fine-tuned BERT model and only train the variational autoencoder in the language conversion module so that it can convert the natural language instructions encoded by the BERT model into the corresponding task language representation; Step 7: compress the optimal collaboration strategy; with the help of the trained strategy parameter encoding module, compress the optimal collaboration strategy of the teammate population into a strategy parameter representation dataset; Step 8: Training the conditional diffusion model; Using the task language representation dataset and the strategy parameter representation dataset, the conditional diffusion model is trained by minimizing the loss function, so that the conditional diffusion model generates the corresponding collaboration strategy parameter representation according to the task language representation; Step 9: Human teammates use natural language instructions to express their behavioral preferences or task requirements; Step 10: The language conversion module converts the natural language instructions into task language representation; Step 11: Randomly sample Gaussian noise and use the conditional diffusion model to perform reverse denoising; Step 12: The strategy parameter decoding module predicts the strategy network parameters based on the collaborative strategy parameter representation, initializes the collaborative strategy network and loads the strategy network parameters to obtain the reconstructed optimal collaborative strategy.

2. The method for deploying efficient human-machine collaboration strategies based on natural language instructions according to claim 1, characterized in that: If the behavioral preferences of human teammates change or the task requirements change during the collaboration process, repeat steps 9 to 12 to generate and deploy the optimal collaboration strategy based on the new natural language instructions.

3. The efficient human-machine collaboration strategy deployment method based on natural language instructions according to claim 1 is characterized in that: In step 2, a diverse teammate population is constructed through population training methods or behavior cloning as a human agent strategy, and a task language representation dataset and a natural language instruction dataset corresponding to the teammates are collected; during the training process, the frequency of the teammate strategy executing predefined task-related events within the trajectory is recorded, recorded as a vector v; each dimension of the vector v is discretized to obtain the task language representation L T .

4. The method for deploying efficient human-machine collaboration strategies based on natural language instructions according to claim 1, characterized in that: In step 3, the optimal collaboration strategy is trained for the diverse teammate population. First, for each teammate strategy π H Initialize a collaboration strategy π E , then fix the teammate strategy parameters and train the collaborative strategy π through the PPO algorithm E Get the optimal collaboration strategy π BR ; During the training process, the state s of each step t Save to an experience replay array D for step 4 to train the strategy parameter encoding module and strategy parameter decoding module.

5. The method for deploying efficient human-machine collaboration strategies based on natural language instructions according to claim 1, characterized in that: In step 11, the randomly sampled Gaussian noise x N ~N(0,I) with the help of conditional diffusion model , based on the reverse denoising process described by the following formula for Gaussian noise x i To denoise: , in, , Diffusion Model According to x i , the noise added to the i-th step predicted by the current time step i and the task language encoding; β i is the variance of the Gaussian noise added in the i-th step during the forward diffusion process, that is, , , α i =1-β i , ; L T It is the task language representation, which is used to guide the diffusion model to denoise Gaussian noise and obtain the collaborative strategy parameter representation .

6. A system for implementing the efficient human-machine collaboration strategy deployment method based on natural language instructions as described in any one of claims 1 to 5, characterized in that: It includes language conversion module, strategy parameter encoding module, conditional diffusion model and strategy parameter decoding module; The language conversion module is used to convert the natural language instructions of human teammates into task language representation; it is composed of a pre-trained Bert model and a variational autoencoder, and can convert the natural language instructions of human teammates into task language encoding; the pre-trained Bert model first encodes the natural language instructions, and then the variational autoencoder converts the encoded natural language instructions into the corresponding task language encoding; The strategy parameter encoding module is used to compress the parameters of the collaborative strategy into a low-dimensional latent space; the strategy parameter decoding module is used to reconstruct the parameters of the collaborative strategy from the strategy parameter representation; the strategy parameter encoding module and the strategy parameter decoding module constitute a variational autoencoder, and the strategy parameter encoding module and the strategy parameter decoding module are trained simultaneously in an end-to-end manner. The strategy parameter encoding module is composed of a convolutional neural network and a fully connected neural network. First, the weights and biases of the strategy network are encoded by the convolutional neural network and the fully connected neural network respectively, and then the weight representation and the bias representation are combined, and finally the parameter representation is obtained through the fully connected layer; The conditional diffusion model is used to represent the strategy parameters in the latent space generated by the task language representation; The conditional diffusion model uses a U-shaped network as the main architecture, which can generate corresponding collaborative strategy parameter representations based on teammates' task language representations; For the strategy parameter representation, the forward diffusion process gradually adds Gaussian noise to the parameter representation, and the reverse process gradually removes the Gaussian noise to obtain the strategy parameter representation; The policy parameter decoding module includes a graph hypernetwork, which can predict the parameters of the policy network based on the policy parameter representation and the policy structure representation; during the training phase, the policy parameter representation is generated by the policy parameter encoding part; In the deployment phase, the policy parameter representation is generated by the conditional diffusion model; given the policy network structure, the policy structure is represented as a fixed vector, whose components represent the number of neurons in each layer of the policy network; The language conversion module and the conditional diffusion model are trained independently.

7. A computer device, characterized in that: The computer device includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the steps of the efficient human-computer collaboration strategy deployment method based on natural language instructions as described in any one of claims 1 to 5 are implemented.

8. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program for executing the efficient human-machine collaboration strategy deployment method based on natural language instructions as described in any one of claims 1-5.

Images