Skill prediction model training method

Abstract

The embodiment of the invention discloses a skill prediction model training method, and the method comprises the steps: obtaining offline sample data and a preliminary skill prediction model; the preliminary skill prediction model comprises a skill discovery model and an agent diffusion model; inputting the offline sample data into a skill discovery model to obtain an agent action prediction result, and determining a first loss function based on the agent action prediction result; inputting the offline sample data into the agent diffusion model to obtain an agent trajectory prediction result, and determining a second loss function based on the agent trajectory prediction result; and adjusting the preliminary skill prediction model based on the first loss function and the second loss function to obtain a target skill prediction model. According to the technical scheme provided by the embodiment of the invention, a more powerful model is provided for skill discovery by combining the intelligent agent diffusion model, so that the precision and diversity of skill discovery are improved.

Description

Technical Field

[0001] This invention relates to the field of offline multi-agent reinforcement learning technology, and in particular to a method for training a skill prediction model. Background Technology

[0002] Offline multi-agent reinforcement learning is a data-driven paradigm that learns effective policies from a set of behavioral policy choices. It provides a bridge between reinforcement learning and data-driven paradigms. It is trained using offline data, avoiding the limitations of real-time interaction, and is an ideal method for solving tasks in life where data collection is costly or dangerous.

[0003] Most existing skill discovery methods employ unsupervised reinforcement learning, such as variational autoencoder structures based on Transformers. However, these methods may be limited by the assumption of an overly smooth normal distribution, resulting in insufficient spatial representation capabilities and an inability to fully capture differences between tasks. For complex skill transfer tasks, they may fail to adequately capture the complexity of the task, leading to poor transfer performance. Furthermore, in the process of transferring skills across multiple tasks, they may be biased towards generating common skill patterns, making it difficult to flexibly adapt to various changes or complex skills in new tasks. Summary of the Invention

[0004] This invention provides a method for training a skill prediction model, which can combine an agent diffusion model to provide a more powerful model for skill discovery, thereby improving the accuracy and diversity of skill discovery.

[0005] In a first aspect, embodiments of the present invention provide a skill prediction model training method, the method comprising:

[0006] The process involves acquiring offline sample data and a preliminary skill prediction model. The offline sample data includes at least one of global state information, action information, and observation information of multiple agents. The preliminary skill prediction model includes a skill discovery model and an agent diffusion model. The offline sample data is input into the skill discovery model to obtain agent action prediction results, and a first loss function is determined based on these results. The offline sample data is then input into the agent diffusion model to obtain agent trajectory prediction results, and a second loss function is determined based on these results. The preliminary skill prediction model is adjusted based on the first and second loss functions to obtain a target skill prediction model.

[0007] Secondly, embodiments of the present invention provide a skill prediction model training apparatus, the apparatus comprising:

[0008] A sample data acquisition module is used to acquire offline sample data and a preliminary skill prediction model; wherein, the offline sample data includes at least one of global state information, action information, and observation information of multiple agents; the preliminary skill prediction model includes a skill discovery model and an agent diffusion model; a first loss function determination module is used to input the offline sample data into the skill discovery model to obtain agent action prediction results, and determine a first loss function based on the agent action prediction results; a second loss function determination module is used to input the offline sample data into the agent diffusion model to obtain agent trajectory prediction results, and determine a second loss function based on the agent trajectory prediction results; a model adjustment module is used to adjust the preliminary skill prediction model based on the first loss function and the second loss function to obtain a target skill prediction model.

[0009] Thirdly, embodiments of the present invention provide a computer device, the computer device comprising:

[0010] One or more processors;

[0011] Memory, used to store one or more programs;

[0012] When the one or more programs are executed by the one or more processors, the one or more processors implement the skill prediction model training method described in any embodiment.

[0013] Fourthly, embodiments of the present invention provide a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the skill prediction model training method described in any embodiment.

[0014] The technical solution provided by this invention involves acquiring offline sample data and a preliminary skill prediction model. The offline sample data includes at least one of global state information, action information, and observation information of multiple agents. The preliminary skill prediction model includes a skill discovery model and an agent diffusion model. The offline sample data is input into the skill discovery model to obtain agent action prediction results, and a first loss function is determined based on these results. The offline sample data is then input into the agent diffusion model to obtain agent trajectory prediction results, and a second loss function is determined based on these results. The preliminary skill prediction model is adjusted based on the first and second loss functions to obtain a target skill prediction model. This invention solves the problem of insufficient latent space representation capability in existing skill discovery methods. By combining the agent diffusion model with its powerful latent space representation capability, excellent data generation capability, and adaptability to diverse tasks, a more powerful model is provided for skill discovery, thereby improving the accuracy and diversity of skill discovery. Attached Figure Description

[0015] Figure 1 This is a flowchart of a skill prediction model training method provided in an embodiment of the present invention;

[0016] Figure 2 This is a flowchart of another skill prediction model training method provided in an embodiment of the present invention;

[0017] Figure 3 This is a schematic diagram of the structure of a state encoder provided in an embodiment of the present invention;

[0018] Figure 4 This is a schematic diagram of the structure of an action decoder provided in an embodiment of the present invention;

[0019] Figure 5 This is a schematic diagram of the structure of an agent diffusion model provided in an embodiment of the present invention;

[0020] Figure 6 This is a schematic diagram of the structure of a skill prediction model training device provided in an embodiment of the present invention;

[0021] Figure 7 This is a schematic diagram of the structure of a computer device provided in an embodiment of the present invention. Detailed Implementation

[0022] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention. The acquisition, storage, use, and processing of data in the technical solutions of the embodiments of the present invention all comply with the relevant provisions of national laws and regulations.

[0023] Figure 1 This is a flowchart of a skill prediction model training method provided by an embodiment of the present invention. The embodiment of the present invention can be applied to scenarios where a skill prediction model about an agent is trained. The method can be executed by a skill prediction model training device, which can be implemented by software and / or hardware.

[0024] like Figure 1 As shown, the skill prediction model training method includes the following steps:

[0025] S110. Obtain offline sample data and a preliminary skill prediction model.

[0026] The offline sample data can be data used as training samples for the model. Specifically, the offline sample data can include at least one of the following: global state information, action information, and observation information of multiple agents. The preliminary skill prediction model can be an untrained skill prediction model. The technical solution of this embodiment of the invention can train the preliminary skill prediction model based on the offline sample data, and then predict the skill that the agent will release in the next moment based on the trained model. Specifically, the preliminary skill prediction model includes: a skill discovery model and an agent diffusion model.

[0027] S120. Input the offline sample data into the skill discovery model to obtain the agent action prediction result, and determine the first loss function based on the agent action prediction result.

[0028] The agent action prediction result can be the action prediction result of the agent at a future time from offline sample data. Specifically, the offline sample data corresponding to time t can be input into the skill discovery model to obtain the agent action prediction result at time t+1. The first loss function can be the error function corresponding to the agent action prediction by the skill discovery model. Specifically, the first loss function can be determined by the error between the agent action prediction result at time t+1 and the actual action of the agent at time t+1 in the action information.

[0029] S130. Input the offline sample data into the agent diffusion model to obtain the agent trajectory prediction result, and determine the second loss function based on the agent trajectory prediction result.

[0030] The agent trajectory prediction result can be a prediction of the agent's movement trajectory at future times. For example, offline sample data at time t can be input into the agent diffusion model to obtain the agent's trajectory prediction result at time t+1. The second loss function can be the error function corresponding to the agent diffusion model's trajectory prediction. Specifically, the trajectory prediction result at time t+1 and the offline sample data at time t can be substituted into the formula for calculating the second loss function to obtain the second loss function.

[0031] S140. The preliminary skill prediction model is adjusted based on the first loss function and the second loss function to obtain the target skill prediction model.

[0032] The target skill prediction model can be the final trained skill prediction model. Specifically, the model parameters in the skill discovery model can be adjusted based on the first loss function, and the model parameters in the agent diffusion model can be adjusted based on the second loss function to obtain the target skill prediction model.

[0033] The technical solution provided by this invention involves acquiring offline sample data and a preliminary skill prediction model. The offline sample data includes at least one of global state information, action information, and observation information of multiple agents. The preliminary skill prediction model includes a skill discovery model and an agent diffusion model. The offline sample data is input into the skill discovery model to obtain agent action prediction results, and a first loss function is determined based on these results. The offline sample data is then input into the agent diffusion model to obtain agent trajectory prediction results, and a second loss function is determined based on these results. The preliminary skill prediction model is adjusted based on the first and second loss functions to obtain a target skill prediction model. This invention solves the problem of insufficient latent space representation capability in existing skill discovery methods. By combining the agent diffusion model with its powerful latent space representation capability, excellent data generation capability, and adaptability to diverse tasks, a more powerful model is provided for skill discovery, thereby improving the accuracy and diversity of skill discovery.

[0034] Figure 2This is a flowchart of another skill prediction model training method provided by an embodiment of the present invention. This embodiment of the present invention can be applied to scenarios where a skill prediction model for an agent is trained. Based on the above embodiments, this embodiment further explains how to input offline sample data into a skill discovery model to obtain agent action prediction results, and determine a first loss function based on the agent action prediction results; how to input offline sample data into an agent diffusion model to obtain agent trajectory prediction results, and determine a second loss function based on the agent trajectory prediction results; and how to adjust the preliminary skill prediction model based on the first loss function and the second loss function to obtain a target skill prediction model. This device can be implemented by software and / or hardware and integrated into a computer device with application development capabilities.

[0035] like Figure 2 As shown, the skill prediction model training method includes the following steps:

[0036] S210. Obtain offline sample data and a preliminary skill prediction model.

[0037] Offline sample data can be data used as training samples for the model. Specifically, offline sample data can include at least one of the following: global state information, action information, and observation information of multiple agents. The global state information s can be decomposed into environmental state. Friendly status and enemy status Action information 'a' can be decomposed into the actions of each agent. The observed information can be broken down into the observed information of one's own side (health, etc.). The observation information obtained by the local observation agent j of the i side. .

[0038] Furthermore, the preliminary skill prediction model can be an untrained skill prediction model. The technical solution of this invention can train the preliminary skill prediction model based on offline sample data, and then predict the skill the agent will release in the next time step based on the trained model. Specifically, the preliminary skill prediction model includes: a skill discovery model and an agent diffusion model.

[0039] Skill discovery models can discover diverse skills through unsupervised reinforcement learning (RL). These models can generate exploration strategies for different skills, thereby accelerating the learning efficiency of downstream tasks (such as image recognition and decision-making). For example, by maximizing the difference in state density between skills, cross-skill state diversity can be promoted. Skill discovery models consist of a state encoder and an action decoder. The state encoder extracts latent variables representing skills, which are then processed by the action decoder to predict the actions the agent might take.

[0040] Optionally, the underlying architecture of the state encoder and action decoder can be a Transformer structure, which handles the variable dimensions of state, action, and observation information in a multi-task setting. The key technology is to decompose the input into sequential data and project each part into an embedding with fixed dimensions. The state encoder includes fully connected layers (FC layers), self-attention layers, and FC & Softmax layers; the action decoder includes fully connected layers (FC layers), self-attention layers, and multilayer perceptron layers (MLP layers).

[0041] Furthermore, the agent diffusion model is a novel model that combines diffusion models with AI agent technology. Diffusion models can solve the problem of accumulated error in long-sequence planning using traditional autoregressive methods by simultaneously generating multi-step planning sequences. For example, in model-based reinforcement learning (MBRL), diffusion models can generate complete trajectory plans in one go, rather than simulating them step by step, thus more closely resembling the overall decision-making pattern of humans. Agent diffusion models include: conditional diffusion models and inverse dynamics models. Conditional diffusion models can be used to predict the agent's movement trajectory, while inverse dynamics models are used to predict the agent's next action based on the predicted movement trajectory.

[0042] S220. Input the global state information and action information from the offline sample data into the state encoder to obtain collaborative skill information.

[0043] Collaborative skill information refers to the ability of multiple agents to jointly complete complex tasks or solve problems through collaborative work, information sharing, and decision-making cooperation. Collaborative skills can achieve efficient resolution of complex tasks through role specialization, interaction optimization, decision-making coordination, and system controllability. For example, a state encoder... The collaborative skills of an agent can be discovered from the sampled global state information s and action information a. Using global state information and action information as input to the state encoder ensures that the state encoder discovers appropriate collaborative skills from a global perspective.

[0044] S230. Input the collaborative skill information and the observation information in the offline sample data into the action decoder to obtain the agent action prediction result, and determine the first loss function based on the agent action prediction result and the agent's actual action in the action information.

[0045] The agent action prediction result can be the prediction of the agent's actions at future times from offline sample data. Specifically, the collaborative skill information and observation information at time t can be input into the action decoder to obtain the agent action prediction result at time t+1. The first loss function can be the error function corresponding to the skill discovery model's agent action prediction. Specifically, the first loss function can be determined by the error between the agent action prediction result at time t+1 and the agent's actual action at time t+1 in the action information.

[0046] Optionally, as mentioned above, the training objective of the action decoder is to minimize the reconstructed action. With real action The differences. Meanwhile, to prevent the state encoder from selecting similar or identical collaborative skills for all input offline data, and unified prior The KL divergence between skills is used as a constraint, which is beneficial for discovering different collaborative skills. Therefore, the reconstruction loss function can also be used as the first loss function. The reconstruction loss function of the skill discovery model is as follows:

[0047]

[0048] Among them, parameters and parameters These are the parameters for the state encoder and the action decoder, respectively. This represents the normalization factor.

[0049] S240. Determine the diffusion loss function based on the offline sample data and the preset action discard conditions.

[0050] The preset action discard condition represents the discard condition of the conditional diffusion model for the analyzed collaborative skill information. For example, the preset action discard condition can be to discard the analyzed collaborative skill information with a preset probability. The diffusion loss function can be a loss function determined from the perspective of input data analysis of the conditional diffusion model. Optionally, the initial trajectory information of the agent can be determined based on offline sample data, and conditional noise processing can be applied to the initial trajectory information to obtain processed trajectory information; the diffusion loss function is determined based on the processed trajectory information, the preset action discard condition, and the hyperparameters of the conditional diffusion model.

[0051] The initial trajectory information can be the agent's movement trajectory information from offline sample data. Optionally, the initial trajectory information can also include the agent's state information during movement (e.g., changes in blood volume). Specifically, the initial trajectory information can be determined based on global state information and action information from offline sample data. Further, the processed trajectory information can be the trajectory information obtained after adding noise to the initial trajectory information. This step simulates the noise addition process of the conditional diffusion model on the input data. Finally, the processed trajectory information, preset action discard conditions, and hyperparameters of the conditional diffusion model can be substituted into a preset formula for calculating the diffusion loss function to determine the diffusion loss function.

[0052] For example, one could first, for the given initial trajectory Conditional noise addition:

[0053]

[0054] in , It is a hyperparameter, determined by the variance table.

[0055] Then, regarding the given information Application condition discarding mechanism:

[0056]

[0057] Finally, the diffusion loss function is calculated:

[0058]

[0059] in, A unified noise model representing all intelligent agents. diffusion time step

[0060] S250. Input the offline sample data into the conditional diffusion model to obtain the trajectory prediction result of the agent, and determine the inverse dynamics loss function based on the trajectory prediction result.

[0061] The agent trajectory prediction result can be a prediction of the agent's movement trajectory at future times. For example, offline sample data at time t can be input into the conditional diffusion model to obtain the agent's trajectory prediction result at time t+1. The inverse dynamics loss function can be a loss function derived from the analysis of the conditional diffusion model's prediction results from an inverse dynamics perspective.

[0062] Optionally, offline sample data is input into a conditional diffusion model to obtain trajectory prediction results for the agent. Based on these trajectory prediction results, an inverse dynamics loss function is determined. This includes: inputting the initial trajectory information corresponding to the offline sample data into the conditional diffusion model to obtain trajectory prediction results for the agent; and then, based on the trajectory prediction results, action information, and observation information from the offline sample data, determining the inverse dynamics loss function. For example, the initial trajectory information at time t can be input into the conditional diffusion model to obtain the agent's trajectory prediction results at time t+1, and then the inverse dynamics loss function can be determined based on the trajectory prediction results at time t+1, the observation information at time t, and the action information at time t.

[0063] For example, the formula for calculating the inverse dynamics loss function is shown below:

[0064]

[0065] in, This represents the observation information of agent i at time t. This represents the action of agent i at time t. This indicates the observation information of agent i at the next time step generated by the conditional diffusion model. This represents the inverse dynamics model of agent i.

[0066] S260. Determine the second loss function based on the diffusion loss function and the inverse dynamics loss function.

[0067] The second loss function can be the error function corresponding to the agent's trajectory prediction by the agent diffusion model. Specifically, the diffusion loss function and the inverse dynamics loss function can be summed to obtain the second loss function.

[0068] For example, diffusion loss functions can be combined. and inverse dynamic loss function To obtain the second loss function And minimize the final loss using stochastic gradient descent. To train the entire agent diffusion model:

[0069]

[0070] S270. Adjust the skill discovery model based on the first loss function, and adjust the agent diffusion model based on the second loss function to obtain the target skill prediction model.

[0071] The target skill prediction model can be the final trained skill prediction model. Specifically, the model parameters in the skill discovery model can be adjusted based on the first loss function, and the model parameters in the agent diffusion model can be adjusted based on the second loss function to obtain the target skill prediction model.

[0072] Optionally, during the use of the target skill prediction model, data to be analyzed can be acquired; wherein, the data to be analyzed includes at least one of the following: global state information, action information and observation information of multiple agents; the data to be analyzed is input into the target skill prediction model to obtain the skill prediction result corresponding to the data to be analyzed.

[0073] The beneficial effects of this invention are:

[0074] 1) Considering the insufficient latent space representation capability of existing skill discovery methods, this paper combines a diffusion model with its powerful latent space representation capability, excellent data generation capability, and ability to adapt to diverse tasks, thus providing a more powerful tool for skill discovery and improving the accuracy and diversity of skill discovery.

[0075] 2) Considering the similarity between multiple tasks, the model trained on multiple source tasks is distilled into a target model through knowledge distillation, so that a model with stronger generalization can be obtained, thereby improving its transfer performance in the target task.

[0076] For example, in order to better understand the technical solution provided by the present invention, specific embodiments are described below:

[0077] The experimental environment for this example is StarCraft II. The source missions were set to 3m, 5m, 6m, 9m, and 10m, and the target missions to 4m, 5m, 10m, 12m, 7m, 8m, 8m, 9m, 10m, 11m, 10m, 12m, 13m, and 15m. Both the source and target missions were different maps in StarCraft II; for example, a 5m or 6m mission represents a battle between 5 players on our side and 6 players on the enemy side.

[0078] Based on this, we will use an offline dataset containing source task data as an example to elaborate on the solution of this invention in detail. The specific implementation process is as follows:

[0079] Step 1: Preprocess the offline dataset. Offline dataset preprocessing specifically includes four sub-steps: dataset interface initialization, hierarchical dataset reading, constructing data association structures, and data sampling.

[0080] Step 11: Dataset Interface Initialization. The original dataset is an H5 file containing attributes such as actions (action space), actions_onehot (one-hot encoded action space), available_action (executable action space, generally in one-hot encoded format), filled (filled state), obs (observation space), reward (reward), state (state space), and terminated (terminated marker). To initialize the H5 file interface, it is necessary to call an HDF5 data reading library (such as the h5py module in a Python environment), create a file object by specifying the file path, identify the root-level data group of the HDF5 file, and verify the existence of necessary attributes (such as action space, observation space, etc.).

[0081] Step 12: Layered Dataset Reading. Attributes are read layer by layer based on their structural characteristics. For continuous time-series data (such as reward, teminated, etc.), they are loaded into memory as one-dimensional arrays; for high-dimensional data (such as obs, state, etc.), their original dimensional structure is preserved before loading them into memory; when processing one-hot encoding of actions, discrete action indices and one-hot encoded representations are read simultaneously; mask parsing is performed on executable actions, and dynamic spatial constraint identifiers are loaded; state filling processing is performed, and data validity representations are read to distinguish between real data and filled data.

[0082] Step 13: Construct the data association structure. Align the data read hierarchically according to time steps to construct a structured data object (such as a dictionary or dictionary-like object). After the structured data object is constructed, perform resource release and integrity verification, i.e., close the HDF5 file handle and verify data consistency.

[0083] Step 14: Data Sampling. Randomly sample the global state information s, action information a, and observation information o contained in the structured data object obtained in Step 13, and decompose them according to different tasks, focusing on the attribute dimensions. Specifically, the global state information s is decomposed into environmental state... Friendly status and enemy status Decompose action information 'a' into the actions of each agent. The observed information o is decomposed into the observed information (health, etc.) of the friendly agent i. The observation information obtained by agent i and local agent j .

[0084] Specifically, the method for dividing dimensions is as follows: each agent has a basic feature vector, regardless of whether it is an enemy or an ally. For a specific task i, the basic feature vectors of friendly units and enemy units are nf_al and nf_en, respectively. For the global state information s, it can be decomposed into enemy state (enemy_state), friendly state (ally_state), the state of the last action (last_action_state), and the time step state (time_state). The formula for calculating enemy_state is the product of the number of enemy agents and nf_en; the formula for calculating ally_state is the product of the number of friendly agents (including itself) and nf_al; the formula for calculating the state of the last action is the product of the total number of agents and the actions of each agent; the time step state is set to 1 by default. For the observation information o, it can be decomposed into movement features (move_feats), enemy features (enemy_feats), friendly features (ally_feats), and individual features (own_feats). The default value for move_feats is 4; the formula for enemy_feats is the product of the number of enemy agents and nf_en; the formula for ally_feats is the product of the number of friendly agents (excluding itself) and nf_al; and the formula for own_feats is the sum of individual unit information and shield information, which defaults to 0.

[0085] Step 2, Collaborative Skill Discovery and Action Prediction, specifically includes two sub-steps: collaborative skill discovery and action prediction.

[0086] Step 21, Collaborative Skill Discovery. The discovery of collaborative skills primarily relies on a state encoder. State Encoder Discovering the agent's collaborative skills from sampled global state information s and action information a By using state and action information as input to the state encoder, it can be ensured that the state encoder discovers appropriate collaborative skills from a global perspective.

[0087] The neural network structure of the state encoder is as follows Figure 3 As shown, its basic structure is a Transformer structure, containing fully connected layers (FC layers), self-attention layers, and FC & Softmax layers. The global state information processed in step 14 passes through the FC layer and reaches the self-attention layer, where the self-attention mechanism is applied to the global state information to reflect the environment state. Friendly status Enemy status and actions Embeddings of Q, K, and V (intermediate parameters) are generated, and the attention output is calculated based on the attention mechanism. Finally, the result is passed through FC and Softmax layers. Core collaborative representations, i.e., latent variables of collaborative skills, are extracted from the global state information.

[0088]

[0089]

[0090]

[0091]

[0092] Step 22, Action Prediction. Action prediction relies on an action decoder. Action Decoder Utilizing observational information and collaborative skills output from step 21 To predict skill-related actions , where τ=( ,..., ), Represents the trajectory of agent i ( , ,..., , , Its training objective is to minimize the reconstruction action. With real action The differences.

[0093] The neural network structure of the action decoder is as follows: Figure 4 As shown, similar to the state encoder structure, its basic structure is a Transformer structure, containing a fully connected layer (FC layer), a self-attention layer, and a multilayer perceptron layer (MLP layer). The observation information processed in step 14 passes through the FC layer and reaches the self-attention layer, where the self-attention mechanism is applied to the observation information, using the observation information (health level, etc.) of the agent i. The observation information obtained by agent i and local agent j The embeddings of Q, K, and V are generated, and the attention output is calculated based on the attention mechanism. Finally, these, along with the collaborative skills, are input into the MLP layer to reconstruct the action sequences of all agents. .

[0094]

[0095]

[0096]

[0097]

[0098] Attention Output , It can be further used to derive the Q value, where This indicates an action that does not require interaction with other entities. This represents the actions that agent i can use to interact with agent j. To address partial observability, historical embeddings from the previous time step are appended to the input sequence of the action decoder. And obtain the historical embedding for the current time step. .

[0099] Step 3: Calculate the reconstruction loss. To minimize the action decoder's reconstruction loss... With real action To prevent the state encoder from selecting similar or identical collaborative skills for all input offline data, the differences will be addressed. and unified prior The KL divergence between skills is used as a constraint, which helps to discover different collaborative skills. Therefore, the loss function of the skill discovery model is as follows:

[0100]

[0101] Among them, parameters and parameters These are the parameters for the state encoder and the action decoder, respectively. This represents the normalization factor.

[0102] Step 4: The neural network structure diagram of the agent diffusion model is shown below. Figure 4 As shown, it consists of two parts: a conditional diffusion model and a reverse dynamics model.

[0103] Step 41, Conditional Diffusion Model. The basic structure of the Conditional Diffusion model is the U-net network. The U-net network adopts an encoder-decoder architecture, fusing low-level features (containing more spatial information) from the encoder and high-level features (containing more semantic information) from the decoder through skip connections. In its specific implementation, it consists of repeated one-dimensional convolutional residual blocks (Conv1D + GroupNorm + Mish activation). The convolution is performed in the time step dimension, while the observation information feature dimension is treated as the channel dimension.

[0104] Since the experimental environment is a multi-agent environment, it needs to be modeled as a conditional generation model. Conditions are used as constraints to encourage the generation of the overall optimal trajectory, rather than the optimal trajectory for a single agent. Therefore, a classifier-free approach is adopted to incorporate conditions into the diffusion model. The core idea is to leverage the difference between implicit learning and unconditional generation, enhancing conditional control through the mixing of prediction results without the need for an external classifier. Specifically, during the training phase, random probability is used... Discarding conditions allows the model to learn condition generation simultaneously. and unconditional generation In the inference phase, mixed-condition prediction... and unconditional prediction Its expression is:

[0105]

[0106] in, Representing the noise model, Indicates the initial trajectory ( (These are hyperparameters, determined by the variance table). Indicates the number of diffusion steps. As a condition, Indicates the guiding scale, when When, it enhances the conditional effect; when At that time, it degenerates into a normal condition generation.

[0107] In summary, the input to the conditional diffusion model is the initial noise trajectory. With conditions (Rewards and Skills) are processed through a three-layer downsampling module followed by a three-layer upsampling module, ultimately producing a trajectory generated under the guidance of the generation conditions:

[0108]

[0109] The whole process from Begin, Diffusion Ultimately, this gradually diffuses to generate a pure observation trajectory. .in This represents the denoised trajectory of a single agent over K diffusion steps. This represents the denoising observation under the diffusion step of K. .

[0110] Step 42, Inverse dynamics model.

[0111] The observation trajectory generated via step 41 Since specific actions to be performed cannot be directly provided for online evaluation, training an inverse dynamics model is necessary. Used to predict the current state Transition to the next state The corresponding actions .

[0112] The corresponding expression is:

[0113]

[0114] Step 5: Cross-agent multi-head attention mechanism.

[0115] To facilitate information exchange and improve collaboration among multiple agents, an attention mechanism is added before all decoding layers of the U-net network for each agent i. The input to the l-th layer decoder consists of two parts: (1) skip connection features from the symmetric l-th layer encoder. (2) Embedded features generated by the previous layer decoder Encoder features, having undergone multiple convolutions / downsampling, contain richer semantic information; therefore, the attention mechanism is applied to... Instead And with enhanced features after attention aggregation replace Input decoder. Therefore, the final input of the l-th layer decoder is: (1) the enhanced skip connection features from the symmetric l-th layer encoder after attention aggregation. (2) Embedded features generated by the previous layer decoder .

[0116] The calculation expression for the cross-agent attention mechanism is as follows:

[0117]

[0118]

[0119]

[0120] in, express Dimensions.

[0121] Step 6: Calculate the loss function of the agent diffusion model. The calculation of the loss function of the agent diffusion model includes two sub-steps: calculating the conditional diffusion loss and calculating the inverse dynamics loss.

[0122] Step 61: Calculate the conditional diffusion loss:

[0123] First, for the given initial trajectory Conditional noise addition:

[0124]

[0125] in , It is a hyperparameter, determined by the variance table.

[0126] Then, regarding the given information Application condition discarding mechanism:

[0127]

[0128] Finally, calculate the diffusion loss:

[0129]

[0130] in, A unified noise model representing all intelligent agents. This represents the diffusion time step.

[0131] Step 62: Calculate the reverse dynamic loss:

[0132]

[0133] in, This represents the observation information of agent i at time t. This represents the action of agent i at time t. This indicates the observation information of agent i at the next time step generated by the conditional diffusion model. This represents the inverse dynamics model of agent i.

[0134] Step 63, Combined Diffusion Loss and inverse dynamic loss Obtain the final loss of the model. And minimize the final loss using stochastic gradient descent. To train the entire multi-agent diffusion network:

[0135]

[0136] Step 7: Distill the target model using knowledge distillation. The agent diffusion models trained on several source tasks are used as teacher models to distill a unified student model (target model) applicable to several target tasks. Teacher skills are weighted and integrated via an attention mechanism. The target model simultaneously aligns teacher features (U-net output) and action distributions (KL divergence), and a skill reconstruction loss is added, forming a three-level supervision system of features, output, and skills. Based on the methods described above, knowledge distillation is used to distill multiple agent diffusion models trained on several source tasks into a unified target model applicable to several target tasks.

[0137] The technical solution provided by this invention involves: acquiring offline sample data and a preliminary skill prediction model; inputting global state information and action information from the offline sample data into a state encoder to obtain collaborative skill information; inputting the collaborative skill information and observation information from the offline sample data into an action decoder to obtain agent action prediction results; determining a first loss function based on the agent action prediction results and the agent's actual actions in the action information; determining a diffusion loss function based on the offline sample data and preset action discard conditions; inputting the offline sample data into a conditional diffusion model to obtain trajectory prediction results for the agent; determining an inverse dynamics loss function based on the trajectory prediction results; determining a second loss function based on the diffusion loss function and the inverse dynamics loss function; adjusting the skill discovery model based on the first loss function; and adjusting the agent diffusion model based on the second loss function to obtain a target skill prediction model. This invention solves the problem of insufficient latent space representation capability in existing skill discovery methods. By combining the agent diffusion model with its powerful latent space representation capability, excellent data generation capability, and adaptability to diverse tasks, it provides a more powerful model for skill discovery, thereby improving the accuracy and diversity of skill discovery.

[0138] Figure 6 This is a schematic diagram of a skill prediction model training device provided in an embodiment of the present invention. The embodiment of the present invention can be applied to scenarios where a skill prediction model for an agent is trained. The device can be implemented by software and / or hardware and integrated into a computer device with application development capabilities.

[0139] like Figure 6 As shown, the skill prediction model training device includes: a sample data acquisition module 310, a first loss function determination module 320, a second loss function determination module 330, and a model adjustment module.

[0140] The system includes a sample data acquisition module 310, used to acquire offline sample data and a preliminary skill prediction model; wherein the offline sample data includes at least one of global state information, action information, and observation information of multiple agents; the preliminary skill prediction model includes a skill discovery model and an agent diffusion model; a first loss function determination module 320, used to input the offline sample data into the skill discovery model to obtain agent action prediction results, and determine a first loss function based on the agent action prediction results; a second loss function determination module 330, used to input the offline sample data into the agent diffusion model to obtain agent trajectory prediction results, and determine a second loss function based on the agent trajectory prediction results; and a model adjustment module 340, used to adjust the preliminary skill prediction model based on the first loss function and the second loss function to obtain a target skill prediction model.

[0141] The technical solution provided by this invention involves acquiring offline sample data and a preliminary skill prediction model. The offline sample data includes at least one of global state information, action information, and observation information of multiple agents. The preliminary skill prediction model includes a skill discovery model and an agent diffusion model. The offline sample data is input into the skill discovery model to obtain agent action prediction results, and a first loss function is determined based on these results. The offline sample data is then input into the agent diffusion model to obtain agent trajectory prediction results, and a second loss function is determined based on these results. The preliminary skill prediction model is adjusted based on the first and second loss functions to obtain a target skill prediction model. This invention solves the problem of insufficient latent space representation capability in existing skill discovery methods. By combining the agent diffusion model with its powerful latent space representation capability, excellent data generation capability, and adaptability to diverse tasks, a more powerful model is provided for skill discovery, thereby improving the accuracy and diversity of skill discovery.

[0142] In one optional implementation, the skill discovery model includes a state encoder and an action decoder. The first loss function determination module 320 is specifically configured to: input global state information and action information from the offline sample data into the state encoder to obtain collaborative skill information; input the collaborative skill information and observation information from the offline sample data into the action decoder to obtain the agent's action prediction result; and determine the first loss function based on the agent's action prediction result and the agent's actual action in the action information.

[0143] In one optional implementation, the agent diffusion model includes a conditional diffusion model and an inverse dynamics model. The second loss function determination module 330 is specifically used to: determine a diffusion loss function based on the offline sample data and preset action discarding conditions; wherein, the preset action discarding conditions represent the discarding conditions of the conditional diffusion model for the analyzed collaborative skill information; input the offline sample data into the conditional diffusion model to obtain the agent trajectory prediction result, determine the inverse dynamics loss function based on the trajectory prediction result; and determine the second loss function according to the diffusion loss function and the inverse dynamics loss function.

[0144] In one optional implementation, the second loss function determination module 330 includes: a diffusion loss function determination unit, configured to: determine the initial trajectory information of the agent based on the offline sample data; perform conditional noise addition processing on the initial trajectory information to obtain processed trajectory information; and determine the diffusion loss function based on the processed trajectory information, preset action discard conditions, and the hyperparameters of the conditional diffusion model.

[0145] In one optional implementation, the second loss function determination module 330 includes: an inverse dynamics loss function determination unit, configured to: input the initial trajectory information corresponding to the offline sample data into the conditional diffusion model to obtain the agent trajectory prediction result; and determine the inverse dynamics loss function based on the trajectory prediction result, the action information and observation information in the offline sample data.

[0146] In one optional implementation, the model adjustment module 340 is specifically used to: adjust the skill discovery model based on the first loss function, and adjust the agent diffusion model based on the second loss function to obtain the target skill prediction model.

[0147] In one optional implementation, the skill prediction model training device further includes: a model usage module, configured to: acquire data to be analyzed; wherein the data to be analyzed includes at least one of global state information, action information, and observation information of multiple agents; and input the data to be analyzed into the target skill prediction model to obtain a skill prediction result corresponding to the data to be analyzed.

[0148] The skill prediction model training device provided in this embodiment of the invention can execute the skill prediction model training method provided in any embodiment of the invention, and has the corresponding functional modules and beneficial effects of the execution method.

[0149] Figure 7 This is a schematic diagram of the structure of a computer device provided in an embodiment of the present invention. Figure 7A block diagram of an exemplary computer device 12 suitable for implementing embodiments of the present invention is shown. Figure 7 The computer device 12 shown is merely an example and should not be construed as limiting the functionality or scope of the embodiments of the present invention. The computer device 12 can be any terminal device with computing capabilities and can be configured in a skill prediction model training device.

[0150] like Figure 7 As shown, the computer device 12 is represented in the form of a general-purpose computing device. The components of the computer device 12 may include, but are not limited to: one or more processors or processing units 16, system memory 28, and bus 18 connecting different system components (including system memory 28 and processing unit 16).

[0151] Bus 18 can be one or more of several bus architectures, including a memory bus or memory controller, a peripheral bus, a graphics acceleration port, a processor, or a local bus using any of the various bus architectures. For example, these architectures include, but are not limited to, the Industry Standard Architecture (ISA) bus, the Micro Channel Architecture (MAC) bus, the Enhanced ISA bus, the Video Electronics Standards Association (VESA) local bus, and the Peripheral Component Interconnect (PCI) bus.

[0152] Computer device 12 typically includes a variety of computer system readable media. These media can be any available media that can be accessed by computer device 12, including volatile and non-volatile media, removable and non-removable media.

[0153] System memory 28 may include computer system readable media in the form of volatile memory, such as random access memory (RAM) 30 and / or cache 32. Computer device 12 may further include other removable / non-removable, volatile / non-volatile computer system storage media. By way of example only, storage system 34 may be used to read and write non-removable, non-volatile magnetic media (… Figure 7 Not shown; usually referred to as a "hard drive"). Although Figure 7 Not shown, a disk drive for reading and writing to a removable non-volatile disk (e.g., a "floppy disk") and an optical disk drive for reading and writing to a removable non-volatile optical disk (e.g., a CD-ROM, DVD-ROM, or other optical media) may be provided. In these cases, each drive may be connected to bus 18 via one or more data media interfaces. System memory 28 may include at least one program product having a set (e.g., at least one) of program modules configured to perform the functions of the embodiments of the present invention.

[0154] A program / utility 40 having a set (at least one) of program modules 42 may be stored, for example, in system memory 28. Such program modules 42 include, but are not limited to, an operating system, one or more application programs, other program modules, and program data. Each or some combination of these examples may include an implementation of a model environment. Program modules 42 typically perform the functions and / or methods described in the embodiments of the present invention.

[0155] Computer device 12 can also communicate with one or more external devices 14 (e.g., keyboard, pointing device, display 24, etc.), and with one or more devices that enable a user to interact with the computer device 12, and / or with any device that enables the computer device 12 to communicate with one or more other computing devices (e.g., network card, modem, etc.). This communication can be performed via input / output (I / O) interface 22. Furthermore, computer device 12 can also communicate with one or more models (e.g., local area network (LAN), wide area network (WAN), and / or public models, such as the Internet) via model adapter 20. Figure 7 As shown, model adapter 20 communicates with other modules of computer device 12 via bus 18. It should be understood that, although... Figure 7 As not shown, it can be used in conjunction with computer device 12 with other hardware and / or software modules, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems.

[0156] Processing unit 16 executes various functional applications and data processing by running programs stored in system memory 28, such as implementing the skill prediction model training method provided in this embodiment of the invention, which includes:

[0157] The process involves acquiring offline sample data and a preliminary skill prediction model. The offline sample data includes at least one of global state information, action information, and observation information of multiple agents. The preliminary skill prediction model includes a skill discovery model and an agent diffusion model. The offline sample data is input into the skill discovery model to obtain agent action prediction results, and a first loss function is determined based on these results. The offline sample data is then input into the agent diffusion model to obtain agent trajectory prediction results, and a second loss function is determined based on these results. The preliminary skill prediction model is adjusted based on the first and second loss functions to obtain a target skill prediction model.

[0158] This embodiment provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the skill prediction model training method as provided in any embodiment of the present invention, including:

[0159] The process involves acquiring offline sample data and a preliminary skill prediction model. The offline sample data includes at least one of global state information, action information, and observation information of multiple agents. The preliminary skill prediction model includes a skill discovery model and an agent diffusion model. The offline sample data is input into the skill discovery model to obtain agent action prediction results, and a first loss function is determined based on these results. The offline sample data is then input into the agent diffusion model to obtain agent trajectory prediction results, and a second loss function is determined based on these results. The preliminary skill prediction model is adjusted based on the first and second loss functions to obtain a target skill prediction model.

[0160] The computer storage medium of this invention can be any combination of one or more computer-readable media. A computer-readable medium can be a computer-readable signal medium or a computer-readable storage medium. A computer-readable storage medium can be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of computer-readable storage media (a non-exhaustive list) include: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In this document, a computer-readable storage medium can be any tangible medium that contains or stores a program that can be used by or in conjunction with an instruction execution system, apparatus, or device.

[0161] Computer-readable signal media may include data signals propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. Computer-readable signal media may also be any computer-readable medium other than computer-readable storage media, capable of sending, propagating, or transmitting programs for use by or in connection with an instruction execution system, apparatus, or device.

[0162] Program code contained on a computer-readable medium may be transmitted using any suitable medium, including but not limited to: wireless, wire, optical fiber, RF, etc., or any suitable combination thereof.

[0163] Computer program code for performing the operations of this invention can be written in one or more programming languages ​​or a combination thereof. Programming languages ​​include object-oriented programming languages ​​such as C, Java, Smalltalk, C++, C#, and Python, as well as conventional procedural programming languages ​​such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of model, including a local area network (LAN) or a wide area network (WAN), or it can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0164] Those skilled in the art will understand that the modules or steps of the present invention described above can be implemented using general-purpose computing devices. They can be centralized on a single computing device or distributed across a model composed of multiple computing devices. Optionally, they can be implemented using computer-executable program code, thereby allowing them to be stored in a storage device for execution by a computing device, or they can be fabricated as separate integrated circuit modules, or multiple modules or steps can be fabricated as a single integrated circuit module. Thus, the present invention is not limited to any particular combination of hardware and software.

[0165] Note that the above description is merely a preferred embodiment of the present invention and the technical principles employed. Those skilled in the art will understand that the present invention is not limited to the specific embodiments described herein, and various obvious changes, readjustments, and substitutions can be made without departing from the scope of protection of the present invention. Therefore, although the present invention has been described in detail through the above embodiments, the present invention is not limited to the above embodiments, and may include many other equivalent embodiments without departing from the concept of the present invention, the scope of which is determined by the scope of the appended claims.

Claims

1. A method for training a skill prediction model, characterized in that, include: Acquire offline sample data and a preliminary skill prediction model; wherein, the offline sample data includes at least one of: global state information, action information, and observation information of multiple agents; the preliminary skill prediction model includes: a skill discovery model and an agent diffusion model; The offline sample data is input into the skill discovery model to obtain the agent action prediction result, and a first loss function is determined based on the agent action prediction result. The offline sample data is input into the agent diffusion model to obtain the agent trajectory prediction result, and a second loss function is determined based on the agent trajectory prediction result. The preliminary skill prediction model is adjusted based on the first loss function and the second loss function to obtain the target skill prediction model.

2. The method according to claim 1, characterized in that, The skill discovery model includes a state encoder and an action decoder. The step of inputting the offline sample data into the skill discovery model to obtain agent action prediction results, and determining a first loss function based on the agent action prediction results, includes: The global state information and action information in the offline sample data are input into the state encoder to obtain collaborative skill information; The collaborative skill information and the observation information in the offline sample data are input into the action decoder to obtain the agent's action prediction result. The first loss function is determined based on the agent's action prediction result and the agent's actual action in the action information.

3. The method according to claim 1, characterized in that, The agent diffusion model includes a conditional diffusion model and an inverse dynamics model. The offline sample data is input into the agent diffusion model to obtain agent trajectory prediction results. Based on the agent trajectory prediction results, a second loss function is determined, including: The diffusion loss function is determined based on the offline sample data and the preset action discarding conditions; wherein, the preset action discarding conditions are used to represent the discarding conditions of the conditional diffusion model for the analyzed collaborative skill information; The offline sample data is input into the conditional diffusion model to obtain the trajectory prediction result of the agent, and the inverse dynamics loss function is determined based on the trajectory prediction result; The second loss function is determined based on the diffusion loss function and the inverse dynamics loss function.

4. The method according to claim 3, characterized in that, The step of determining the diffusion loss function based on the agent's action prediction results, the offline sample data, and preset action discard conditions includes: The initial trajectory information of the agent is determined based on the offline sample data, and the initial trajectory information is subjected to conditional noise addition to obtain the processed trajectory information. Based on the processed trajectory information, the preset action discard conditions, and the hyperparameters of the conditional diffusion model, the diffusion loss function is determined.

5. The method according to claim 3, characterized in that, The step of inputting the offline sample data into the conditional diffusion model to obtain the agent trajectory prediction result, and determining the inverse dynamics loss function based on the trajectory prediction result, includes: The initial trajectory information corresponding to the offline sample data is input into the conditional diffusion model to obtain the trajectory prediction result of the agent. Based on the trajectory prediction results, the action information and observation information in the offline sample data, the inverse dynamics loss function is determined.

6. The method according to claim 1, characterized in that, The step of adjusting the preliminary skill prediction model based on the first loss function and the second loss function to obtain the target skill prediction model includes: The skill discovery model is adjusted based on the first loss function, and the agent diffusion model is adjusted based on the second loss function to obtain the target skill prediction model.

7. The method according to claim 1, characterized in that, The method further includes: Acquire the data to be analyzed; wherein the data to be analyzed includes at least one of the following: global state information, action information, and observation information of multiple agents; The data to be analyzed is input into the target skill prediction model to obtain the skill prediction result corresponding to the data to be analyzed.

8. A skill prediction model training device, characterized in that, The device includes: The sample data acquisition module is used to acquire offline sample data and a preliminary skill prediction model; wherein, the offline sample data includes at least one of global state information, action information and observation information of multiple agents; the preliminary skill prediction model includes a skill discovery model and an agent diffusion model; The first loss function determination module is used to input the offline sample data into the skill discovery model to obtain the agent action prediction result, and determine the first loss function based on the agent action prediction result. The second loss function determination module is used to input the offline sample data into the agent diffusion model to obtain the agent trajectory prediction result, and determine the second loss function based on the agent trajectory prediction result. The model adjustment module is used to adjust the preliminary skill prediction model based on the first loss function and the second loss function to obtain the target skill prediction model.

9. A computer device, characterized in that, The computer device includes: One or more processors; Memory, used to store one or more programs; When the one or more programs are executed by the one or more processors, the one or more processors implement the skill prediction model training method as described in any one of claims 1-7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it implements the skill prediction model training method as described in any one of claims 1-7.

Images