Intelligent agent action prediction method based on permutation bias and contrast decoding

Abstract

The invention discloses an intelligent agent action prediction method based on permutation bias and contrast decoding, and the method comprises the steps: constructing a strategy generation model which comprises an Encoder module and a Decoder module, obtaining a local observation vector and an action sequence, inputting the local observation vector into the Encoder module for processing, obtaining the observation representation of an intelligent agent, transmitting the observation representation to the Decoder module, and obtaining the action sequence of the intelligent agent. Obtaining an observed agent state value function according to the observation representation; inputting the action sequence into a Decoder module to be decoded together with the observation representation to obtain an implicit vector, converting the implicit vector to obtain an agent action set with action replaceability, and performing comparison, exploration and decoding on the agent action set with action replaceability to obtain an execution action; and calculating a total loss function according to the agent state value function and the execution action to obtain a target model, and predicting the behavior of the target agent by using the target model. According to the invention, the prediction precision of the target agent action is improved.

Description

Technical Field

[0001] The present invention relates to the technical field of multi-agent reinforcement learning, and in particular to an agent action prediction method, system, terminal and computer-readable storage medium based on permutation bias and contrast decoding. Background Art

[0002] The field of Multi-Agent Reinforcement Learning (MARL) often uses value decomposition methods and centralized training with decentralized execution (CTDE) frameworks, and has made some progress in solving the exponential growth of observation space and action space problems and difficult exploration problems in MARL tasks. Under the CTDE framework, the global state that encodes information about all agents and enemies can be used in the training phase, while the test phase completely relies on the local observations of the agent itself, which leads to the problem of information asymmetry between the training phase and the test phase.

[0003] The Multi-Agent Transformer (MAT) method introduces the Transformer architecture into the multi-agent system and models the multi-agent system as a multi-agent action sequence prediction task. Although it solves the information asymmetry problem in the training and testing phases faced by CTDE and the information interaction problem of the value decomposition method, it still faces the multi-agent dimensionality curse problem. In addition, there is the LLM (Large Language Model) hallucination problem in the large model field. This phenomenon is transferred to the multi-agent task by analogy. In the multi-agent task, since the size of the joint action space may be larger than the vocabulary space in the LLM, when using the Transformer structure to generate actions, the model is likely to face similar hallucination problems.

[0004] Therefore, the prior art still needs to be improved and developed. Summary of the invention

[0005] The main purpose of the present invention is to provide an agent action prediction method, system, terminal and computer-readable storage medium based on permutation bias and contrast decoding, aiming to solve the problems in the prior art of using MAT for multi-agent reinforcement learning, such as the multi-agent dimensionality disaster and the model facing the LLM-like illusion, which leads to inaccurate information generated by the model and insufficient prediction accuracy of the target agent action.

[0006] To achieve the above object, the present invention provides an intelligent agent action prediction method based on permutation bias and contrast decoding, and the intelligent agent action prediction method based on permutation bias and contrast decoding comprises the following steps:

[0007] Constructing a strategy generation model and acquiring training data, wherein the training data includes a local observation vector and an action sequence of the agent, wherein the strategy generation model includes an Encoder module and a Decoder module;

[0008] Input the local observation vector into the Encoder module for embedding to obtain an embedded representation of the local observation vector, sort and transform the embedded representation of the local observation vector to obtain an agent observation set with observation permutation invariance, encode the agent observation set with observation permutation invariance to obtain an observation representation of the agent, obtain an observed agent state value function according to the observation representation, and pass the observation representation to the Decoder module;

[0009] The action sequence is input into the Decoder module for embedding to obtain an embedded representation of the action sequence, the embedded representation of the action sequence is sorted, the embedded representation of the sorted action sequence and the observation representation transmitted by the Encoder module are decoded to obtain a latent vector, the latent vector is transformed to obtain an intelligent agent action set with action permutation, the intelligent agent action set with action permutation is compared, explored and decoded to obtain the execution action of the intelligent agent;

[0010] Calculating a total loss function according to the agent state value function and the execution action, and updating the strategy generation model according to the total loss function until the total loss function converges to obtain a target model;

[0011] A target intelligent agent in an e-sports game is obtained, a local observation vector of the target intelligent agent is input into the target model for calculation, an action of the target intelligent agent is predicted, and the behavior of the target intelligent agent is controlled according to the action of the target intelligent agent.

[0012] Optionally, in the agent action prediction method based on permutation bias and contrast decoding, the Encoder module comprises: a first embedding layer, a first sorting layer, an input permutation invariant network, an encoding layer and a multi-layer perceptron layer;

[0013] The Decoder module includes: a second embedding layer, a second sorting layer, a decoding layer, an output permutation equivariant network and a contrast decoding network.

[0014] Optionally, the agent action prediction method based on permutation bias and contrast decoding, wherein the local observation vector is input into the Encoder module for embedding to obtain an embedded representation of the local observation vector, and the embedded representation of the local observation vector is sorted and transformed to obtain an agent observation set with observation permutation invariance, specifically includes:

[0015] Embed the dimensions of the local observation vectors corresponding to the multiple agents into the first embedding layer to unify the dimensions, so as to obtain embedded representations of the multiple local observation vectors;

[0016] Inputting the embedded representations of multiple local observation vectors into the first sorting layer to sort them in the agent dimension to obtain multiple sorted observations;

[0017] Construct a first hypernetwork, input each of the sorted observations into the input permutation invariant network, generate a network layer weight for each entity of the sorted observation through the first hypernetwork, and accumulate and sum the network layer weights of multiple entities of the sorted observations to obtain an intelligent agent observation set with observation permutation invariance.

[0018] Optionally, the agent action prediction method based on permutation bias and contrast decoding, wherein the agent observation set with observation permutation invariance is encoded to obtain the agent observation representation, the observed agent state value function is obtained according to the observation representation, and the observation representation is passed to the Decoder module, specifically comprising:

[0019] Inputting the agent observation set with observation permutation invariance into the five-layer TransformerEncoder module of the encoding layer for encoding to obtain observation representations of multiple agents, and passing the multiple observation representations to the Decoder module to calculate the latent vector;

[0020] The observation representation is calculated through the multi-layer perceptron layer to obtain the observed agent state value function.

[0021] Optionally, the agent action prediction method based on permutation bias and contrast decoding, wherein the action sequence is input into the Decoder module for embedding to obtain an embedded representation of the action sequence, the embedded representation of the action sequence is sorted, and the embedded representation of the sorted action sequence and the observation representation transmitted by the Encoder module are decoded to obtain a latent vector, specifically comprising:

[0022] Embed the action sequences corresponding to the multiple agents into the second embedding layer to unify the dimensions, so as to obtain embedding representations of the multiple action sequences;

[0023] Inputting the embedded representations of the plurality of action sequences into the second sorting layer for sorting in the agent dimension to obtain embedded representations of the plurality of sorted action sequences;

[0024] The plurality of observation representations transmitted from the Encoder module are obtained, and the embedded representations of the plurality of sorted action sequences and the plurality of observation representations transmitted from the Encoder module are input into the five-layer Transformer Encoder module of the decoding layer for decoding to obtain a plurality of latent vectors.

[0025] Optionally, the agent action prediction method based on permutation bias and contrast decoding, wherein the latent vector is transformed to obtain an agent action set with action permutation, and the agent action set with action permutation is subjected to contrast exploration decoding to obtain the agent's execution action, specifically includes:

[0026] Constructing a second hypernetwork, inputting each latent vector into the output permutation equivariant network, generating a network layer weight for each entity of the latent vector through the second hypernetwork, and accumulating and summing the network layer weights of the entities of multiple latent vectors to obtain an intelligent agent action set with action permutation;

[0027] The intermediate outputs of the first four layers of the Transformer Encoder modules in the five layers of the decoding layer are obtained, and the intermediate outputs are compared with the latent vectors output by the fifth layer of the Transformer Encoder module for exploration and decoding to obtain the execution action of the intelligent agent.

[0028] Optionally, the agent action prediction method based on permutation bias and contrast decoding, wherein the total loss function is calculated according to the agent state value function and the execution action, specifically includes:

[0029] Calculate the loss function of the Encoder module based on the agent state value function

[0030]

[0031] Among them, φ represents the network parameters of the Encoder module, represents the updated network parameters of the Encoder module, n represents the number of agents, T represents the maximum length of the training data, m represents the ordinal number of the agent, t represents the ordinal number of the training data, R(o t ,a t ) indicates that the agent is observing o t Take action at The reward obtained, γ represents the discount factor that controls the weight of the reward signal, represents the state value function of the mth agent in the t+1th training data, represents the state value function of the mth agent of the tth training data, represents the observation representation of the mth agent of the tth training data, represents the observation representation of the mth agent in the t+1th training data;

[0032] Calculate the loss function of the Decoder module according to the execution action

[0033] in, A t represents the global advantage function at time step t, θ represents the network parameters of the Decoder module, θ' represents the updated network parameters of the Decoder module, clip(·) represents the clip algorithm, ∈ represents the clipping coefficient of the clip algorithm, represents the probability distribution ratio of the new and old agent strategies, represents the new strategy of the mth agent, represents the old strategy of the mth agent, Indicates that all agents’ observations are accepted. Represents the execution actions of agents from 1 to m-1. It means that based on the observations of all agents and the actions of agents 1 to m-1, the mth agent selects action a through a new strategy. t The probability distribution of It means that based on the observations of all agents and the actions of agents 1 to m-1, the mth agent selects action a through the old strategy. t The probability distribution of

[0034] According to the loss function of the Encoder module And the loss function of the Decoder module Calculate the total loss function

[0035]

[0036] In addition, to achieve the above-mentioned purpose, the present invention also provides an intelligent agent action prediction system based on permutation bias and contrast decoding, wherein the intelligent agent action prediction system based on permutation bias and contrast decoding comprises:

[0037] A model building module is used to build a strategy generation model and obtain training data, wherein the training data includes a local observation vector and an action sequence of the agent, wherein the strategy generation model includes an Encoder module and a Decoder module;

[0038] The first training module is used to input the local observation vector into the Encoder module for embedding to obtain an embedded representation of the local observation vector, sort and transform the embedded representation of the local observation vector to obtain an intelligent agent observation set with observation permutation invariance, encode the intelligent agent observation set with observation permutation invariance to obtain an observation representation of the intelligent agent, obtain an observed intelligent agent state value function according to the observation representation, and pass the observation representation to the Decoder module;

[0039] The second training module is used to input the action sequence into the Decoder module for embedding to obtain an embedded representation of the action sequence, sort the embedded representation of the action sequence, decode the sorted embedded representation of the action sequence and the observation representation transmitted by the Encoder module to obtain a latent vector, transform the latent vector to obtain an intelligent agent action set with action permutation, and perform comparative exploration and decoding on the intelligent agent action set with action permutation to obtain the execution action of the intelligent agent;

[0040] A loss function calculation module, used to calculate a total loss function according to the agent state value function and the execution action, and update the strategy generation model according to the total loss function until the total loss function converges to obtain a target model;

[0041] The target agent behavior prediction module is used to obtain the target agent in the e-sports game, input the local observation vector of the target agent into the target model for calculation, predict the action of the target agent, and control the behavior of the target agent according to the action of the target agent.

[0042] In addition, to achieve the above-mentioned purpose, the present invention also provides a terminal, wherein the terminal includes: a memory, a processor, and an intelligent agent action prediction program based on permutation bias and contrast decoding stored in the memory and executable on the processor, wherein the intelligent agent action prediction program based on permutation bias and contrast decoding, when executed by the processor, implements the steps of the intelligent agent action prediction method based on permutation bias and contrast decoding as described above.

[0043] In addition, to achieve the above-mentioned purpose, the present invention also provides a computer-readable storage medium, wherein the computer-readable storage medium stores an intelligent agent action prediction program based on permutation bias and contrast decoding, and when the intelligent agent action prediction program based on permutation bias and contrast decoding is executed by a processor, the steps of the intelligent agent action prediction method based on permutation bias and contrast decoding as described above are implemented.

[0044] In the present invention, a strategy generation model is constructed, including an Encoder module and a Decoder module, a local observation vector and an action sequence are obtained, the local observation vector is input into the Encoder module for processing, an observation representation of the agent is obtained, and the observation representation is passed to the Decoder module, and the observed agent state value function is obtained according to the observation representation; the action sequence is input into the Decoder module and decoded together with the observation representation to obtain a latent vector, the latent vector is transformed to obtain an agent action set with action permutation, the agent action set with action permutation is compared, explored and decoded to obtain an execution action, the total loss function is calculated according to the agent state value function and the execution action, the target model is obtained, and the behavior of the target agent is predicted using the target model. The present invention improves the prediction accuracy of the target agent action. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] Figure 1 It is a flow chart of a preferred embodiment of the intelligent agent action prediction method based on displacement bias and contrast decoding of the present invention;

[0046] Figure 2 It is a schematic diagram of the interaction process of MARL in the agent action prediction method based on permutation bias and contrast decoding of the present invention;

[0047] Figure 3 It is a schematic diagram of multi-agent action decision modeling in the agent action prediction method based on permutation bias and contrast decoding of the present invention;

[0048] Figure 4 It is an overall network architecture diagram of the target model in the intelligent agent action prediction method based on displacement bias and contrast decoding of the present invention;

[0049] Figure 5 It is an overall architecture diagram of a permutation bias super network in an intelligent agent action prediction method based on permutation bias and contrast decoding of the present invention;

[0050] Figure 6 It is a diagram of the architecture of the Transformer in the intelligent agent action prediction method based on permutation bias and contrast decoding of the present invention;

[0051] Figure 7It is a flow diagram of comparative exploration decoding in the intelligent agent action prediction method based on permutation bias and comparative decoding of the present invention;

[0052] Figure 8 It is a performance comparison chart of the PBCE method and other methods in the first experimental scenario of the intelligent agent action prediction method based on permutation bias and contrast decoding of the present invention;

[0053] Fig. 9 It is a performance comparison chart of the PBCE method and other methods in the second experimental scenario of the intelligent agent action prediction method based on permutation bias and contrast decoding of the present invention;

[0054] Fig.10 It is a performance comparison chart of the PBCE method and other methods in the third experimental scenario of the intelligent agent action prediction method based on permutation bias and contrast decoding of the present invention;

[0055] Fig.11 It is a performance comparison chart of the PBCE method and other methods in the fourth experimental scenario of the intelligent agent action prediction method based on displacement bias and contrast decoding of the present invention;

[0056] Fig.12 It is a schematic diagram of the principle of a preferred embodiment of the intelligent agent action prediction system based on displacement bias and contrast decoding of the present invention;

[0057] Fig.13 Schematic diagram of the operating environment of a preferred embodiment of the terminal of the present invention. DETAILED DESCRIPTION

[0058] The present application provides an intelligent agent action prediction method and related equipment based on permutation bias and contrast decoding. In order to make the purpose, technical solution and effect of the present application clearer and more specific, the present application is further described in detail with reference to the accompanying drawings and examples. It should be understood that the specific embodiments described here are only used to explain the present application and are not used to limit the present application.

[0059] It will be understood by those skilled in the art that, unless otherwise defined, all terms (including technical and scientific terms) used herein have the same meaning as those generally understood by those skilled in the art to which this application belongs. It should also be understood that terms such as those defined in general dictionaries should be understood to have meanings consistent with those in the context of the prior art, and will not be interpreted with idealized or overly formal meanings unless specifically defined as here.

[0060] In addition, if there are descriptions involving "first", "second", etc. in the embodiments of the present invention, the descriptions of "first", "second", etc. are only used for descriptive purposes and cannot be understood as indicating or suggesting their relative importance or implicitly indicating the number of the indicated technical features. Therefore, the features defined as "first" and "second" may explicitly or implicitly include at least one of the features. In addition, the technical solutions between the various embodiments can be combined with each other, but they must be based on the ability of ordinary technicians in the field to implement them. When the combination of technical solutions is contradictory or cannot be implemented, it should be deemed that such a combination of technical solutions does not exist and is not within the scope of protection required by the present invention.

[0061] The agent action prediction method based on permutation bias and contrast decoding described in the preferred embodiment of the present invention is as follows: Figure 1 As shown, the agent action prediction method based on permutation bias and contrast decoding includes the following steps:

[0062] Step S10: construct a strategy generation model and obtain training data, wherein the training data includes a local observation vector and an action sequence of the agent, wherein the strategy generation model includes an Encoder module and a Decoder module.

[0063] The interactive process of multi-agent reinforcement learning is as follows Figure 2 As shown in the figure, for the agent itself, each agent still optimizes its strategy through continuous interaction with the environment. For the entire multi-agent system, the local actions made by multiple agents at the same time will produce joint actions and act on the environment. This is different from the single-agent reinforcement learning method (SARL). The environment will not feedback immediate rewards for the local actions of any agent alone, but will feedback the overall reward (Team Reward) based on the joint actions of the multi-agent system. However, as the number of agents grows exponentially, the joint observation space and joint action space make the exploration of the multi-agent environment particularly difficult, and the traditional SARL method is difficult to solve the multi-agent problem.

[0064] MAT (Multi-Agent Transformer) draws on the idea of ​​sequence generation tasks in the field of natural language processing and the above-mentioned multi-agent advantage function decomposition theory, and models the action decision of multi-agents as a sequence generation task, in which each element of the sequence corresponds to an action made by an agent, as follows: Figure 3As shown in the figure, since the multi-agent action decision is modeled as a sequence generation task, the elements in the sequence have a sequential generation order. This is reflected in the multi-agent system, that is, agent 1 first makes action 1, and then agent 2 makes action 2 based on the action information of agent 1, and so on. Therefore, the premise for the subsequent agent to make an action decision is to know the action decision information of all the previous agents. In this way, information exchange is achieved between multiple agents. Instead of making actions at the same time, the information of other agents is considered when deciding actions.

[0065] Although MAT solves the problem of information asymmetry in the training and testing phases and the information interaction problem of the value decomposition method faced by CTDE, it still faces the problem of multi-agent dimensionality disaster. In addition, there is the LLM hallucination problem in the large model field. This phenomenon is transferred to multi-agent tasks by analogy. In multi-agent tasks, since the size of the joint action space may be larger than the vocabulary space in LLM, when using the Transformer structure to generate actions, the model is likely to face similar hallucination problems.

[0066] Therefore, this paper proposes an innovative solution to the severe challenges faced by the field of multi-agent reinforcement learning (MARL), namely the multi-agent dimensionality curse and the difficulty of exploration: a multi-agent collaborative exploration reinforcement learning method (PBCE) based on permutation bias and contrastive decoding to achieve the prediction of agent actions. This method is based on the well-known method MAT in the field of multi-agent reinforcement learning, and innovatively designs the permutation bias hypernetwork (HPBN) and contrastive decoding mechanism (DCLE).

[0067] Specifically, construct a strategy generation model (such as Figure 4 As shown), the strategy generation model includes an Encoder module and a Decoder module; wherein the Encoder module includes: a first embedding layer, a first sorting layer, an input permutation invariant network, an encoding layer and a multi-layer perceptron layer; the Decoder module includes: a second embedding layer, a second sorting layer, a decoding layer, an output permutation equivariant network and a contrast decoding network.

[0068] The training data is obtained, wherein the training data includes a local observation vector and an action sequence of the agent, and the Encoder module and the Decoder module are trained using the local observation vector and the action sequence to obtain a target model.

[0069] It is understandable that when the environment is completely knowable, that is, the agent has a global perspective and can know all the state information of the environment, then MDP is used to model the environment. However, when there is more than one agent in the environment, each agent only has its own partial observation of the entire environment, rather than the global state. At this time, Dec-POMDP (Partially Observable Markov Decision Process) is used to model the multi-agent environment.

[0070] Dec-POMDP can usually be formally described as a tuple containing eight elements: Where n represents the number of agents in the environment. Describes the true global state information about the environment, represents the set of actions of a single agent, and at each time step, each agent i∈{1,...,n} chooses an action Finally, they are combined to form a joint action is the state transition equation of the environment, It represents the transition of the environment from the current state s to a new state s' after taking a joint action a.

[0071] All agents share a global scalar reward signal Where γ is a discount factor used to control the weight of reward signals at different times, and γ∈[0,1); each agent can only Obtain local observation z∈Z, and only based on its own local observation action history Learn an independent random policy The joint goal of all agents is to maximize the cumulative discounted reward

[0072] Step S20, input the local observation vector into the Encoder module for embedding to obtain an embedded representation of the local observation vector, sort and transform the embedded representation of the local observation vector to obtain an agent observation set with observation permutation invariance, encode the agent observation set with observation permutation invariance to obtain the agent observation representation, obtain the observed agent state value function according to the observation representation, and pass the observation representation to the Decoder module.

[0073] The inputting the local observation vector into the Encoder module for embedding to obtain an embedded representation of the local observation vector, sorting and transforming the embedded representation of the local observation vector to obtain an intelligent agent observation set with observation permutation invariance, specifically includes:

[0074] Embed the dimensions of the local observation vectors corresponding to the multiple agents into the first embedding layer to unify the dimensions, so as to obtain embedded representations of the multiple local observation vectors;

[0075] Inputting the embedded representations of multiple local observation vectors into the first sorting layer to sort them in the agent dimension to obtain multiple sorted observations;

[0076] Construct a first hypernetwork, input each of the sorted observations into the input permutation invariant network, generate a network layer weight for each entity of the sorted observation through the first hypernetwork, and accumulate and sum the network layer weights of multiple entities of the sorted observations to obtain an intelligent agent observation set with observation permutation invariance.

[0077] It can be understood that the first embedding layer is used for vector embedding, the input is the dimension of the local observation vector, and the output is a uniform dimension (such as 64 dimensions); the first sorting layer (Permutation layer) is used to sort the vector embedding representation of uniform dimension on the agent dimension to obtain sorted observations.

[0078] Furthermore, the present invention innovatively designs a permutation biased hypernetwork (HPBN), such as Figure 5 As shown in the figure, the core of HPBN is to use permutation invariance and permutation equivariance to reduce the dimensionality of the joint observation and action space of multiple agents. Specifically, HPBN first ensures the insensitivity of the model to the decision order of agents through a specific network structure and operation rules. No matter how the observation sequence is disrupted, the model can correctly learn and model the interaction relationship between agents. Secondly, HPBN captures the dependency relationship between the actions of agents through a specific encoding method, achieving effective compression of the high-dimensional action space. In practical applications, HPBN can reduce the learning burden of the model and help solve the problem of multi-agent dimensionality disaster.

[0079] The permutation biased hypernetwork in the present invention includes an input permutation invariant network (PIN) and an output permutation equivariant network (PEN) to respectively achieve permutation invariance of the action of the agent itself and permutation equivariance of the action of other agents. For the input permutation invariant network, the present invention arranges the observation sequence of each time step provided by the environment, rather than assuming that the first observation is the observation of agent 1, the second observation is the observation of agent 2, and so on. Based on this theoretical basis, the observation sequence of environmental feedback is transformed as follows: Among them, G is the set of all permutations of the observation sequence, g is the permutation of any observation sequence, o' is the transformed observation sequence, and o is the original observation sequence.

[0080] Specifically, the input permutation invariant network is a special input layer that receives the original observation o i As input, output the transformed permutation invariant observation o' i For o i Each entity o in i [j], for example, teammate 1, enemy 2, etc., construct the first super network to generate the matching network layer weights for them. The specific formula is as follows:

[0081]

[0082] Where G represents the observation o i The set of all permutations of entities in , g represents the observation o i Any arrangement of Net in represents the first hypernetwork, j represents the ordinal number of the entity, i represents the ordinal number of the observation, p represents the total number of observations, and m represents the total number of entities.

[0083] Based on the above formula, different entities o i [j] The weight generated after entering the PIN Different, no matter i How to arrange, the same entity o i [j] will always output the same weight Finally, after the entity matrix multiplication, the sum is accumulated. i How to arrange, get o' i are always the same, so permutation invariance is satisfied.

[0084] Furthermore, encoding the agent observation set with observation permutation invariance to obtain the agent observation representation, obtaining the observed agent state value function according to the observation representation, and passing the observation representation to the Decoder module specifically includes:

[0085] Inputting the agent observation set with observation permutation invariance into the five-layer TransformerEncoder module of the encoding layer for encoding to obtain observation representations of multiple agents, and passing the multiple observation representations to the Decoder module to calculate the latent vector;

[0086] The observation representation is calculated through the multi-layer perceptron layer to obtain the observed agent state value function.

[0087] In this embodiment, the encoding layer includes five layers of Transformer modules, each of which contains a self-attention layer, and the input and output dimensions are the same. Due to the autoregression property of the Transformer structure, it is theoretically very suitable for multi-agent sequence generation tasks. Figure 6 As shown in the figure, it mainly includes an encoder (Encoder) for encoding input embedding into a high-level latent space representation vector and a decoder (Decoder) for autoregressive output. The role of the encoder is to encode the input embedding representation into a semantic embedding representation of a higher dimensional space, and input the semantic embedding representation output by the top encoder to the decoder for decoding. The role of the decoder is to generate the target sequence based on the autoregressive property. In Transformer, the self-attention mechanism (SelfAttention) is one of the most important concepts, in which the attention in the encoder is bidirectional, while the attention in the decoder is unidirectional. As shown in the figure, the self-attention mechanism (SelfAttention) is one of the most important concepts, in which the attention in the encoder is bidirectional, while the attention in the decoder is unidirectional. Figure 4 As described above, at each time step, the encoder accepts a series of agent observations and encodes them into a series of latent representations, which are then passed to the decoder. The decoder generates the optimal action for each agent in a sequential and autoregressive manner. The Maskattention block ensures that the agent can only access the actions of its previous agent during training.

[0088] Step S30, input the action sequence into the Decoder module for embedding to obtain an embedded representation of the action sequence, sort the embedded representation of the action sequence, decode the embedded representation of the sorted action sequence and the observation representation transmitted by the Encoder module to obtain a latent vector, transform the latent vector to obtain an intelligent agent action set with action substitutability, compare, explore and decode the intelligent agent action set with action substitutability to obtain the execution action of the agent.

[0089] The step of inputting the action sequence into the Decoder module for embedding to obtain an embedded representation of the action sequence, sorting the embedded representation of the action sequence, and decoding the sorted embedded representation of the action sequence and the observation representation transmitted by the Encoder module to obtain a latent vector specifically includes:

[0090] Embed the action sequences corresponding to the multiple agents into the second embedding layer to unify the dimensions, so as to obtain embedding representations of the multiple action sequences;

[0091] Inputting the embedded representations of the plurality of action sequences into the second sorting layer for sorting in the agent dimension to obtain embedded representations of the plurality of sorted action sequences;

[0092] The plurality of observation representations transmitted from the Encoder module are obtained, and the embedded representations of the plurality of sorted action sequences and the plurality of observation representations transmitted from the Encoder module are input into the five-layer Transformer Encoder module of the decoding layer for decoding to obtain a plurality of latent vectors.

[0093] Further, the transforming of the latent vector to obtain an agent action set with action substitution, and the comparative exploration decoding of the agent action set with action substitution to obtain the agent execution action specifically includes:

[0094] Constructing a second hypernetwork, inputting each latent vector into the output permutation equivariant network, generating a network layer weight for each entity of the latent vector through the second hypernetwork, and accumulating and summing the network layer weights of the entities of multiple latent vectors to obtain an intelligent agent action set with action permutation;

[0095] The intermediate outputs of the first four layers of the Transformer Encoder modules in the five layers of the decoding layer are obtained, and the intermediate outputs are compared with the latent vectors output by the fifth layer of the Transformer Encoder module for exploration and decoding to obtain the execution action of the intelligent agent.

[0096] It can be understood that the permutation biased super network mentioned above includes an input permutation invariant network (PIN) and an output permutation equivariant network (PEN) to respectively achieve permutation invariance of the action of the agent itself and permutation equivariance of the action of other agents. In addition to the permutation invariance property for observations, the multi-agent system should have a permutation property for joint actions.

[0097] For the output permutation equivariance network, taking e-sports as an example, multi-agent collaborative tasks can be roughly divided into two categories due to the presence of teammates (allies, other agents) and enemies (AI-controlled agents). On the one hand, actions that only involve the current agent itself (such as moving up, down, left, right, staying in place, etc.) should have action permutation invariance, because these actions only involve the current agent and are not affected by other agents. On the other hand, for actions involving teammates and enemies (such as adding shields to teammates, releasing skills to attack enemies, etc.), the system needs to maintain the action permutation equivariance property.

[0098] The output permutation equivariant network (PEN) is a special output layer that receives the output of the backbone network as input and outputs the final strategy π. Similar to PIN, a second super network is constructed to observe o i Each entity o in i [j] Generate the network layer weights that match it, and observe o i Corresponding changes in the permutations will cause the output to change in the same way, so permutation equivariance is satisfied.

[0099] Therefore, according to HPBN, after satisfying the observation order permutation invariance and action permutation (self-related action permutation invariance and other agent-related action permutation equivariance), the joint observation space and joint action space of the multi-agent system can be effectively reduced in dimension, and the existence of the hypernetwork can expand the dimension of each entity o i [j] corresponds to the parameter space of the network weights, which makes the fitting effect better and helps the convergence stability of the multi-agent advantage function decomposition method.

[0100] Furthermore, based on the above-mentioned hallucination problem existing in the field of large models, the present invention transfers this phenomenon by analogy to multi-agent tasks. Based on the MAT method, this paper proposes a contrast layer decoding exploration mechanism (DCLE). This mechanism aims to perform exploratory decoding on action generation when the multi-agent advantage decomposition target is adopted and the Transformer structure is used as the action decoding network. The core idea is to maximize the retention of real actions and reduce hallucinations, and by emphasizing high-level semantic knowledge and downplaying low-level knowledge, thereby expanding the exploratory nature of the decoding stage and generating more diverse and higher-quality samples. The samples will be used for subsequent rounds of training, forming a positive cycle iteration of sample data and algorithm training. The introduction of DCLE aims to break the hallucination cycle in multi-agent tasks, enhance the exploratory nature of action decoding, thereby improving sample diversity and sample quality, injecting more effective information into the training process, and promoting further optimization of the algorithm.

[0101] When the Transformer structure is usually used as an action decoding network, it consists of an embedding layer (Embedding), N stacked Transformer layers (Block) and an output layer (Head). Given an input action sequence First, all actions are converted into latent vectors of the same dimension through the embedding layer The latent vector will then be transformed and compressed through the Transformer layer by layer, and finally the action probability will be output after passing through the output layer. In this case, the output of the model is completely dependent on the top-level latent vector, i.e., h N, which may induce motion hallucinations. Figure 7 As shown in Figure 1, to address this problem, DCLE implements contrastive decoding by applying the output layer to the latent vector output by the intermediate Transformer layer and then comparing it with the top layer output, and finally obtains the execution action of the agent.

[0102] A total loss function is calculated according to the agent state value function and the execution action, and the strategy generation model is updated according to the total loss function until the total loss function converges to obtain a target model.

[0103] Step S40, calculating a total loss function according to the agent state value function and the execution action, and updating the strategy generation model according to the total loss function until the total loss function converges to obtain a target model.

[0104] Specifically, the loss function of the Encoder module is calculated according to the agent state value function

[0105]

[0106] Among them, φ represents the network parameters of the Encoder module, represents the updated network parameters of the Encoder module, n represents the number of agents, T represents the maximum length of the training data, m represents the ordinal number of the agent, t represents the ordinal number of the training data, R(o t ,a t ) indicates that the agent is observing o t Take action a t The reward obtained, γ represents the discount factor that controls the weight of the reward signal, represents the state value function of the mth agent in the t+1th training data, represents the state value function of the mth agent of the tth training data, represents the observation representation of the mth agent of the tth training data, represents the observation representation of the mth agent in the t+1th training data;

[0107] Furthermore, the loss function of the Decoder module is calculated according to the execution action

[0108]

[0109] in, A trepresents the global advantage function at time step t, θ represents the network parameters of the Decoder module, θ' represents the updated network parameters of the Decoder module, clip(·) represents the clip algorithm, ∈ represents the clipping coefficient of the clip algorithm, represents the probability distribution ratio of the new and old agent strategies, represents the new strategy of the mth agent, represents the old strategy of the mth agent, Indicates that all agents’ observations are accepted. Represents the execution actions of agents from 1 to m-1. It means that based on the observations of all agents and the actions of agents 1 to m-1, the mth agent selects action a through a new strategy. t The probability distribution of It means that based on the observations of all agents and the actions of agents 1 to m-1, the mth agent selects action a through the old strategy. t The probability distribution of

[0110] Furthermore, according to the loss function of the Encoder module And the loss function of the Decoder module Calculate the total loss function

[0111] Step S50, obtaining a target intelligent agent in the e-sports game, inputting a local observation vector of the target intelligent agent into the target model for calculation, predicting the action of the target intelligent agent, and controlling the behavior of the target intelligent agent according to the action of the target intelligent agent.

[0112] It is understandable that the PBCE method is applicable to various complex decision-making scenarios that require efficient multi-agent collaboration, such as multiplayer e-sports, drone formation flying, network routing optimization, intelligent traffic scheduling, etc. In these practical applications, the PBCE method exhibits the following characteristics of efficient collaborative decision-making: through the permutation bias hypernetwork and contrast decoding mechanism, the agent can actively and purposefully explore in a high-dimensional complex environment and discover a better collaborative strategy, thereby significantly improving the collaborative efficiency and task completion quality of the entire multi-agent system.

[0113] In this embodiment, taking e-sports games as an example, the PBCE method can provide decision suggestions for each player through the autoregressive module in the Transformer. The model continuously optimizes these strategies during the training process, allowing players to make optimal or near-optimal decisions based on the behavior of teammates and expected enemy reactions. At the same time, the PBCE method uses the self-attention mechanism to directly model the interactions between players without explicit message passing or state aggregation. In e-sports scenarios, this mechanism can accurately capture complex interaction modes such as coordinated attacks, support cooperation, and information sharing between players, thereby improving the overall combat effectiveness of the team.

[0114] Specifically, the observation vectors of the five heroes that need to be controlled are obtained from an e-sports game environment, and are input into the Encoder module of the target model, and the Value vector is obtained through the first embedding layer, the first sorting layer, the input permutation invariant network, the encoding layer, and the multi-layer perceptron layer in sequence. It should be noted that the present embodiment does not process only a single observation vector at a time, but processes a batch of observation vectors. A batch of data contains observation vectors of five heroes in multiple time steps, so that a batch of Value values ​​will be obtained, and then the loss function of the Encoder module is used to update the Encoder module network.

[0115] The action control of the five heroes, such as attack, movement, skill release, etc., will be handled by the action of the reinforcement learning agent. The action generation here is the responsibility of the Decoder module of the PBCE method. Since the basic architecture of the Decoder module is transformer, the Decoder module adopts the autoregressive method. The autoregressive method decides the actions of the five agents in sequence. First, the input given to the Decoder module is a full 0 vector or a randomly initialized vector. After the second embedding layer, the second sorting layer, the decoding layer, the output permutation equivariant network and the contrast decoding network, the action of the first agent is obtained. Then this action is used as the input of the next decoder end, and then the action of the second agent is generated. Then this action is spliced ​​to the end of the first agent action as the input of the third decoder end, and so on until all the actions of the agents are generated.

[0116] It can be seen that the PBCE method combined with the CTDE framework forms a device that can solve multi-agent decision-making problems. It accepts agent observations as input and outputs agent actions. It learns successful team collaboration patterns and strategies based on a large amount of historical data, and continuously improves its guidance ability in actual game AI combat.

[0117] Furthermore, the present invention selects Random-SMAC, an upgraded version of SMAC, as the verification platform of the PBCE method. Random-SMAC has better robustness than SMAC and can more comprehensively evaluate the performance of the PBCE method.

[0118] like Figure 8-Figure 11 As shown in Figure 2, the performance of all experimental methods on four super hard experimental scenarios is shown. Figure 8 As shown in the figure, in the MMM2 scenario (the first experimental scenario), PBCE is the only method with a 100% winning rate after convergence, while the other comparison methods have a winning rate of around 90% after convergence, and MAT has a very large performance fluctuation. Fig. 9 As shown in the figure, in the 6h_vs_8z scenario (the second experimental scenario), after PBCE converges quickly, there is a large performance drop in the middle period (about 5 million total steps), and then gradually climbs and finally converges. MAT and MAIC have achieved better performance than other comparison methods, and the other comparison methods have no more than 20% winning rate. Fig.10 As shown in Figure 3, in the 3s5z_vs_3s6z scenario (the third experimental scenario), MAPPO has a very large performance fluctuation. PBCE and MAT eventually converge to a 100% win rate, but PBCE converges faster. Fig.11 As shown in the figure, the 27m_vs_30m scenario (the fourth experimental scenario) has the largest number of agents in all experimental scenarios. The experiments on this scenario are also the slowest and have large fluctuations. This is reflected in the fact that the performance fluctuations of each method are larger than those of the experiments in other scenarios. It can be seen that PBCE is superior to MAT in terms of convergence speed and final peak win rate, and the final peak win rates of other experimental methods are lower than those of the first two.

[0119] Furthermore, if Fig.12 As shown, based on the above-mentioned intelligent agent action prediction method based on permutation bias and contrast decoding, the present invention also provides an intelligent agent action prediction system based on permutation bias and contrast decoding, wherein the intelligent agent action prediction system based on permutation bias and contrast decoding includes:

[0120] A model building module 61 is used to build a strategy generation model and obtain training data, wherein the training data includes a local observation vector and an action sequence of the agent, wherein the strategy generation model includes an Encoder module and a Decoder module;

[0121] The first training module 62 is used to input the local observation vector into the Encoder module for embedding to obtain an embedded representation of the local observation vector, sort and transform the embedded representation of the local observation vector to obtain an agent observation set with observation permutation invariance, encode the agent observation set with observation permutation invariance to obtain an observation representation of the agent, obtain an observed agent state value function according to the observation representation, and pass the observation representation to the Decoder module;

[0122] The second training module 63 is used to input the action sequence into the Decoder module for embedding to obtain an embedded representation of the action sequence, sort the embedded representation of the action sequence, decode the sorted embedded representation of the action sequence and the observation representation transmitted by the Encoder module to obtain a latent vector, transform the latent vector to obtain an intelligent agent action set with action permutation, and perform comparative exploration and decoding on the intelligent agent action set with action permutation to obtain the execution action of the intelligent agent;

[0123] A loss function calculation module 64, used to calculate a total loss function according to the agent state value function and the execution action, and update the strategy generation model according to the total loss function until the total loss function converges to obtain a target model;

[0124] The target agent behavior prediction module 65 is used to obtain the target agent in the e-sports game, input the local observation vector of the target agent into the target model for calculation, predict the action of the target agent, and control the behavior of the target agent according to the action of the target agent.

[0125] Furthermore, if Fig.13 As shown, based on the above-mentioned intelligent agent action prediction method and system based on replacement bias and contrast decoding, the present invention also provides a terminal accordingly, and the terminal includes a processor 10, a memory 20 and a display 30. Fig.13 Only some components of the terminal are shown, but it should be understood that it is not required to implement all of the components shown, and more or fewer components may be implemented instead.

[0126] In some embodiments, the memory 20 may be an internal storage unit of the terminal, such as a hard disk or memory of the terminal. In other embodiments, the memory 20 may also be an external storage device of the terminal, such as a plug-in hard disk, a smart memory card (SmartMedia Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), etc. equipped on the terminal. Further, the memory 20 may also include both an internal storage unit of the terminal and an external storage device. The memory 20 is used to store application software and various types of data installed on the terminal, such as the program code of the installation terminal. The memory 20 may also be used to temporarily store data that has been output or is to be output. In one embodiment, an intelligent agent action prediction program 40 based on permutation bias and contrast decoding is stored on the memory 20, and the intelligent agent action prediction program 40 based on permutation bias and contrast decoding can be executed by the processor 10, thereby realizing the intelligent agent action prediction method based on permutation bias and contrast decoding in the present application.

[0127] In some embodiments, the processor 10 may be a central processing unit (CPU), a microprocessor or other data processing chip, used to run the program code or process data stored in the memory 20, such as executing the intelligent agent action prediction method based on substitution bias and contrast decoding.

[0128] In some embodiments, the display 30 may be an LED display, a liquid crystal display, a touch-sensitive liquid crystal display, an OLED (Organic Light-Emitting Diode) touch device, etc. The display 30 is used to display information on the terminal and to display a visual user interface. The components 10-30 of the terminal communicate with each other via a system bus.

[0129] In one embodiment, when the processor 10 executes the agent action prediction program 40 based on permutation bias and contrast decoding in the memory 20, the following steps are implemented:

[0130] Constructing a strategy generation model and acquiring training data, wherein the training data includes a local observation vector and an action sequence of the agent, wherein the strategy generation model includes an Encoder module and a Decoder module;

[0131] Input the local observation vector into the Encoder module for embedding to obtain an embedded representation of the local observation vector, sort and transform the embedded representation of the local observation vector to obtain an agent observation set with observation permutation invariance, encode the agent observation set with observation permutation invariance to obtain an observation representation of the agent, obtain an observed agent state value function according to the observation representation, and pass the observation representation to the Decoder module;

[0132] Input the action sequence into the Decoder module for embedding to obtain an embedded representation of the action sequence, sort the embedded representation of the action sequence, decode the sorted embedded representation of the action sequence and the observation representation transmitted by the Encoder module to obtain a latent vector, transform the latent vector to obtain an intelligent agent action set with action permutation, perform comparative exploration and decoding on the intelligent agent action set with action permutation to obtain the execution action of the intelligent agent;

[0133] Calculating a total loss function according to the agent state value function and the execution action, and updating the strategy generation model according to the total loss function until the total loss function converges to obtain a target model;

[0134] A target intelligent agent in an e-sports game is obtained, a local observation vector of the target intelligent agent is input into the target model for calculation, an action of the target intelligent agent is predicted, and the behavior of the target intelligent agent is controlled according to the action of the target intelligent agent.

[0135] The encoder module includes: a first embedding layer, a first sorting layer, an input permutation invariant network, a coding layer and a multi-layer perceptron layer;

[0136] The Decoder module includes: a second embedding layer, a second sorting layer, a decoding layer, an output permutation equivariant network and a contrast decoding network.

[0137] The step of inputting the local observation vector into the Encoder module for embedding to obtain an embedded representation of the local observation vector, sorting and transforming the embedded representation of the local observation vector to obtain an agent observation set with observation permutation invariance specifically includes:

[0138] Embed the dimensions of the local observation vectors corresponding to the multiple agents into the first embedding layer to unify the dimensions, so as to obtain embedded representations of the multiple local observation vectors;

[0139] Inputting the embedded representations of multiple local observation vectors into the first sorting layer to sort them in the agent dimension to obtain multiple sorted observations;

[0140] Construct a first hypernetwork, input each of the sorted observations into the input permutation invariant network, generate a network layer weight for each entity of the sorted observation through the first hypernetwork, and accumulate and sum the network layer weights of multiple entities of the sorted observations to obtain an intelligent agent observation set with observation permutation invariance.

[0141] The step of encoding the agent observation set with observation permutation invariance to obtain the agent observation representation, obtaining the observed agent state value function according to the observation representation, and passing the observation representation to the Decoder module specifically includes:

[0142] Inputting the agent observation set with observation permutation invariance into the five-layer TransformerEncoder module of the encoding layer for encoding to obtain observation representations of multiple agents, and passing the multiple observation representations to the Decoder module to calculate the latent vector;

[0143] The observation representation is calculated through the multi-layer perceptron layer to obtain the observed agent state value function.

[0144] The step of inputting the action sequence into the Decoder module for embedding to obtain an embedded representation of the action sequence, sorting the embedded representation of the action sequence, and decoding the sorted embedded representation of the action sequence and the observation representation transferred from the Encoder module to obtain a latent vector specifically includes:

[0145] Embed the action sequences corresponding to the multiple agents into the second embedding layer to unify the dimensions, so as to obtain embedding representations of the multiple action sequences;

[0146] Inputting the embedded representations of the plurality of action sequences into the second sorting layer for sorting in the agent dimension to obtain embedded representations of the plurality of sorted action sequences;

[0147] The plurality of observation representations transmitted from the Encoder module are obtained, and the embedded representations of the plurality of sorted action sequences and the plurality of observation representations transmitted from the Encoder module are input into the five-layer Transformer Encoder module of the decoding layer for decoding to obtain a plurality of latent vectors.

[0148] The transforming of the latent vector to obtain an agent action set with action substitution, and the comparison, exploration and decoding of the agent action set with action substitution to obtain the agent execution action specifically include:

[0149] Constructing a second hypernetwork, inputting each latent vector into the output permutation equivariant network, generating a network layer weight for each entity of the latent vector through the second hypernetwork, and accumulating and summing the network layer weights of the entities of multiple latent vectors to obtain an intelligent agent action set with action permutation;

[0150] The intermediate outputs of the first four layers of the Transformer Encoder modules in the five layers of the decoding layer are obtained, and the intermediate outputs are compared with the latent vectors output by the fifth layer of the Transformer Encoder module for exploration and decoding to obtain the execution action of the intelligent agent.

[0151] The step of calculating the total loss function according to the agent state value function and the execution action specifically includes:

[0152] Calculate the loss function of the Encoder module based on the agent state value function

[0153]

[0154] Among them, φ represents the network parameters of the Encoder module, represents the updated network parameters of the Encoder module, n represents the number of agents, T represents the maximum length of the training data, m represents the ordinal number of the agent, t represents the ordinal number of the training data, R(o t ,a t ) indicates that the agent is observing o t Take action a t The reward obtained, γ represents the discount factor that controls the weight of the reward signal, represents the state value function of the mth agent in the t+1th training data, represents the state value function of the mth agent of the tth training data, represents the observation representation of the mth agent of the tth training data, represents the observation representation of the mth agent in the t+1th training data;

[0155] Calculate the loss function of the Decoder module according to the execution action

[0156] in, A trepresents the global advantage function at time step t, θ represents the network parameters of the Decoder module, θ' represents the updated network parameters of the Decoder module, clip(·) represents the clip algorithm, ∈ represents the clipping coefficient of the clip algorithm, represents the probability distribution ratio of the new and old agent strategies, represents the new strategy of the mth agent, represents the old strategy of the mth agent, Indicates that all agents’ observations are accepted. Represents the execution actions of agents from 1 to m-1. It means that based on the observations of all agents and the actions of agents 1 to m-1, the mth agent selects action a through a new strategy. t The probability distribution of It means that based on the observations of all agents and the actions of agents 1 to m-1, the mth agent selects action a through the old strategy. t The probability distribution of

[0157] According to the loss function of the Encoder module And the loss function of the Decoder module Calculate the total loss function

[0158]

[0159] In summary, the present invention provides a method, system and terminal for predicting agent actions based on permutation bias and contrast decoding, the method comprising: constructing a strategy generation model, including an encoder module and a decoder module, obtaining a local observation vector and an action sequence, inputting the local observation vector into the encoder module for processing, obtaining an observation representation of the agent, and passing the observation representation to the decoder module, obtaining an observed agent state value function according to the observation representation; inputting the action sequence into the decoder module for decoding together with the observation representation to obtain a latent vector, transforming the latent vector to obtain an agent action set with action permutation, performing contrast exploration decoding on the agent action set with action permutation to obtain an execution action, calculating a total loss function according to the agent state value function and the execution action, obtaining a target model, and using the target model to predict the behavior of the target agent. The present invention proposes a permutation bias super network and a contrast decoding mechanism, which greatly improves the MAT network structure, increases its robustness in processing multi-agent reinforcement learning problems, and improves the prediction accuracy of the target agent action.

[0160] It should be noted that, in this article, the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article or terminal including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or terminal. In the absence of further restrictions, an element defined by the sentence "comprises a ..." does not exclude the presence of other identical elements in the process, method, article or terminal including the element.

[0161] Of course, those skilled in the art can understand that all or part of the processes in the above-mentioned embodiments can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided by the present invention can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. As an illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).

[0162] It should be understood that the application of the present invention is not limited to the above examples. For ordinary technicians in this field, improvements or changes can be made based on the above description. All these improvements and changes should fall within the scope of protection of the claims attached to the present invention.

Claims

1. An agent action prediction method based on permutation bias and contrast decoding, characterized in that: The agent action prediction method based on permutation bias and contrast decoding includes: Constructing a strategy generation model and acquiring training data, wherein the training data includes a local observation vector and an action sequence of the agent, wherein the strategy generation model includes an Encoder module and a Decoder module; Input the local observation vector into the Encoder module for embedding to obtain an embedded representation of the local observation vector, sort and transform the embedded representation of the local observation vector to obtain an agent observation set with observation permutation invariance, encode the agent observation set with observation permutation invariance to obtain an observation representation of the agent, obtain an observed agent state value function according to the observation representation, and pass the observation representation to the Decoder module; Input the action sequence into the Decoder module for embedding to obtain an embedded representation of the action sequence, sort the embedded representation of the action sequence, decode the sorted embedded representation of the action sequence and the observation representation transmitted by the Encoder module to obtain a latent vector, transform the latent vector to obtain an intelligent agent action set with action permutation, compare and explore the intelligent agent action set with action permutation to obtain the execution action of the intelligent agent; Calculating a total loss function according to the agent state value function and the execution action, and updating the strategy generation model according to the total loss function until the total loss function converges to obtain a target model; A target intelligent agent in an e-sports game is obtained, a local observation vector of the target intelligent agent is input into the target model for calculation, an action of the target intelligent agent is predicted, and the behavior of the target intelligent agent is controlled according to the action of the target intelligent agent.

2. The method for predicting intelligent agent actions based on permutation bias and contrast decoding according to claim 1, characterized in that: The Encoder module includes: a first embedding layer, a first sorting layer, an input permutation invariant network, a coding layer and a multi-layer perceptron layer; The Decoder module includes: a second embedding layer, a second sorting layer, a decoding layer, an output permutation equivariant network and a contrast decoding network.

3. The method for predicting intelligent agent actions based on permutation bias and contrast decoding according to claim 2, characterized in that: The inputting the local observation vector into the Encoder module for embedding to obtain an embedded representation of the local observation vector, sorting and transforming the embedded representation of the local observation vector to obtain an intelligent agent observation set with observation permutation invariance, specifically includes: Embed the dimensions of the local observation vectors corresponding to the multiple agents into the first embedding layer to unify the dimensions, so as to obtain embedded representations of the multiple local observation vectors; Inputting the embedded representations of multiple local observation vectors into the first sorting layer to sort them in the agent dimension to obtain multiple sorted observations; Construct a first hypernetwork, input each of the sorted observations into the input permutation invariant network, generate a network layer weight for each entity of the sorted observation through the first hypernetwork, and accumulate and sum the network layer weights of multiple entities of the sorted observations to obtain an intelligent agent observation set with observation permutation invariance.

4. The method for predicting intelligent agent actions based on permutation bias and contrast decoding according to claim 3, characterized in that: The step of encoding the agent observation set with observation permutation invariance to obtain the agent observation representation, obtaining the observed agent state value function according to the observation representation, and passing the observation representation to the Decoder module specifically includes: Inputting the agent observation set with observation permutation invariance into the five-layer TransformerEncoder module of the encoding layer for encoding to obtain observation representations of multiple agents, and passing the multiple observation representations to the Decoder module to calculate the latent vector; The observation representation is calculated through the multi-layer perceptron layer to obtain the observed agent state value function.

5. The method for predicting intelligent agent actions based on permutation bias and contrast decoding according to claim 4, characterized in that: The step of inputting the action sequence into the Decoder module for embedding to obtain an embedded representation of the action sequence, sorting the embedded representation of the action sequence, and decoding the sorted embedded representation of the action sequence and the observation representation transmitted by the Encoder module to obtain a latent vector specifically includes: Embed the action sequences corresponding to the multiple agents into the second embedding layer to unify the dimensions, so as to obtain embedding representations of the multiple action sequences; Inputting the embedded representations of the plurality of action sequences into the second sorting layer for sorting in the agent dimension to obtain embedded representations of the plurality of sorted action sequences; The plurality of observation representations transmitted by the Encoder module are obtained, and the embedded representations of the plurality of sorted action sequences and the plurality of observation representations transmitted by the Encoder module are input into the five-layer Transformer Encoder module of the decoding layer for decoding to obtain a plurality of latent vectors.

6. The method for predicting intelligent agent actions based on permutation bias and contrast decoding according to claim 5, characterized in that: The transforming of the latent vector to obtain an agent action set with action substitution, and performing comparative exploration decoding on the agent action set with action substitution to obtain the agent execution action specifically includes: Constructing a second hypernetwork, inputting each latent vector into the output permutation equivariant network, generating a network layer weight for each entity of the latent vector through the second hypernetwork, and accumulating and summing the network layer weights of the entities of multiple latent vectors to obtain an intelligent agent action set with action permutation; The intermediate outputs of the first four layers of the five-layer Transformer Encoder modules of the decoding layer are obtained, and the intermediate outputs are compared with the latent vector output by the fifth layer of the Transformer Encoder module to explore and decode, so as to obtain the execution action of the intelligent agent.

7. The method for predicting intelligent agent actions based on permutation bias and contrast decoding according to claim 6, characterized in that: The calculating of the total loss function according to the agent state value function and the execution action specifically includes: Calculate the loss function of the Encoder module based on the agent state value function Among them, φ represents the network parameters of the Encoder module, represents the updated network parameters of the Encoder module, n represents the number of agents, T represents the maximum length of the training data, m represents the ordinal number of the agent, t represents the ordinal number of the training data, R(o t ,a t ) indicates that the agent is observing o t Take action a t The reward obtained, γ represents the discount factor that controls the weight of the reward signal, represents the state value function of the mth agent in the t+1th training data, represents the state value function of the mth agent of the tth training data, represents the observation representation of the mth agent of the tth training data, represents the observation representation of the mth agent in the t+1th training data; Calculate the loss function of the Decoder module according to the execution action in, A t represents the global advantage function at time step t, θ represents the network parameters of the Decoder module, θ' represents the updated network parameters of the Decoder module, clip(·) represents the clip algorithm, ∈ represents the clipping coefficient of the clip algorithm, represents the probability distribution ratio of the new and old agent strategies, represents the new strategy of the mth agent, represents the old strategy of the mth agent, Indicates that all agents’ observations are accepted. Represents the execution actions of agents from 1 to m-1. It means that based on the observations of all agents and the actions of agents 1 to m-1, the mth agent selects action a through a new strategy. t The probability distribution of It means that based on the observations of all agents and the actions of agents 1 to m-1, the mth agent selects action a through the old strategy. t The probability distribution of According to the loss function of the Encoder module And the loss function of the Decoder module Calculate the total loss function 8. An intelligent agent action prediction system based on permutation bias and contrast decoding, characterized in that: The agent action prediction system based on permutation bias and contrast decoding includes: A model building module is used to build a strategy generation model and obtain training data, wherein the training data includes a local observation vector and an action sequence of the agent, wherein the strategy generation model includes an Encoder module and a Decoder module; The first training module is used to input the local observation vector into the Encoder module for embedding to obtain an embedded representation of the local observation vector, sort and transform the embedded representation of the local observation vector to obtain an intelligent agent observation set with observation permutation invariance, encode the intelligent agent observation set with observation permutation invariance to obtain an observation representation of the intelligent agent, obtain an observed intelligent agent state value function according to the observation representation, and pass the observation representation to the Decoder module; The second training module is used to input the action sequence into the Decoder module for embedding to obtain an embedded representation of the action sequence, sort the embedded representation of the action sequence, decode the sorted embedded representation of the action sequence and the observation representation transmitted by the Encoder module to obtain a latent vector, transform the latent vector to obtain an intelligent agent action set with action permutation, and perform comparative exploration and decoding on the intelligent agent action set with action permutation to obtain the execution action of the intelligent agent; A loss function calculation module, used to calculate a total loss function according to the agent state value function and the execution action, and update the strategy generation model according to the total loss function until the total loss function converges to obtain a target model; The target agent behavior prediction module is used to obtain the target agent in the e-sports game, input the local observation vector of the target agent into the target model for calculation, predict the action of the target agent, and control the behavior of the target agent according to the action of the target agent.

9. A terminal, characterized in that: The terminal includes: a memory, a processor, and an intelligent agent action prediction program based on permutation bias and contrast decoding stored in the memory and executable on the processor. When the intelligent agent action prediction program based on permutation bias and contrast decoding is executed by the processor, the steps of the intelligent agent action prediction method based on permutation bias and contrast decoding as described in any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores an intelligent agent action prediction program based on permutation bias and contrast decoding, and when the intelligent agent action prediction program based on permutation bias and contrast decoding is executed by a processor, the steps of the intelligent agent action prediction method based on permutation bias and contrast decoding as described in any one of claims 1-7 are implemented.

Images