Adversary modeling method and device based on uncertainty characteristics

Abstract

The present invention provides an adversary modeling method and apparatus based on uncertainty characteristics, comprising: obtaining interaction data during the interaction between an intelligent agent and an environment; the interaction data including current self-observation data, current self-action data, current opponent observation data, current opponent action data, first environment reward data, and next self-observation data; inputting the current self-observation data and current self-action data into an adversary modeling model, modeling the deterministic and uncertain characteristics of the adversary to obtain a model output; calculating a loss function using the current opponent observation data, current opponent action data, first environment reward data, next self-observation data, and the model output; optimizing the model based on the loss function to obtain a trained model; and performing adversary modeling based on the trained model to obtain an adversary modeling result. The present invention introduces mixed deterministic and uncertain characteristics, thereby effectively improving the ability to model adversaries with stochastic behavior.

Description

Technical Field

[0001] The present invention relates to the technical fields of opponent modeling and deep reinforcement learning in intelligent games, and in particular to an opponent modeling method and device based on uncertainty characteristics. Background Art

[0002] Opponent modeling is crucial in multi-agent environments, especially in highly dynamic and complex game-playing tasks. In recent years, deep reinforcement learning has made significant progress and has become a core approach for addressing complex tasks. However, the vast majority of research focuses on optimizing the autonomous agent, while insufficiently considering the interaction between the agent and the opponent. To improve the performance of agents in complex environments, research on opponent modeling provides an important complement to deep reinforcement learning. Deep reinforcement learning based on opponent modeling aims to enhance the decision-making ability of agents by analyzing and predicting the opponent's strategy. Traditional reinforcement learning methods typically assume a static environment and ignore the dynamics of the opponent's behavior. However, in real-world scenarios, opponents often adjust their behavior based on their own interests, and this dynamic behavior has a profound impact on the agent's behavior. Agents that fail to adapt to the opponent's changes may not be able to learn effectively, resulting in policy optimization failures. To address this, many researchers have begun exploring the use of opponent models to assist the agent's learning process, enabling it to predict and respond to the opponent's strategy adjustments. To this end, a number of researchers have recently proposed deep reinforcement learning methods based on opponent modeling, but these methods struggle to achieve good results when facing opponents with stochastic decision-making behaviors. Summary of the Invention

[0003] The present invention provides an adversary modeling method and apparatus based on uncertainty characteristics, which is used to address the drawback of existing adversary modeling methods that cannot effectively model adversaries with random behavior, thereby enabling effective modeling of adversaries with random behavior. The technical solutions proposed by the present invention are as follows:

[0004] In a first aspect, the present invention provides an adversary modeling method based on uncertainty characteristics, comprising:

[0005] Obtaining a pre-established opponent modeling model and interaction data during the interaction between the agent and the environment; wherein the interaction data includes current own observation data, current own action data, current opponent observation data, current opponent action data, first environment reward data, and next own observation data;

[0006] Inputting the current own observation data and the current own action data into the opponent modeling model, modeling the deterministic characteristics and uncertain characteristics of the opponent, and obtaining a model output;

[0007] Calculating a loss function using the current opponent observation data, the current opponent action data, the first environment reward data, the next step own observation data, and the model output;

[0008] Optimizing the opponent modeling model based on the loss function until a preset condition is met to obtain a trained opponent modeling model;

[0009] Perform opponent modeling based on the trained opponent modeling model to obtain an opponent modeling result.

[0010] Optionally, the model output includes first action prediction data, first observation prediction data, second action prediction data, second observation prediction data, own state prediction data and second environment reward data; the opponent modeling model includes a deterministic feature encoder, an uncertain feature encoder, a deterministic feature decoder, an uncertain feature decoder, an environment feature encoder and an environment feature decoder;

[0011] The step of inputting the current own observation data and the current own action data into the opponent modeling model, modeling the deterministic characteristics and the uncertain characteristics of the opponent, and obtaining a model output includes:

[0012] Taking the current own observation data and the current own action data as input data;

[0013] Inputting the input data into the deterministic feature encoder and the uncertain feature encoder to obtain a deterministic feature vector and an uncertain feature vector of the opponent respectively;

[0014] Inputting the deterministic feature vector into the deterministic feature decoder to obtain the opponent's first action prediction data and first observation prediction data;

[0015] Inputting the opponent's uncertainty feature vector into the uncertainty feature decoder to obtain the opponent's second action prediction data and second observation prediction data;

[0016] Inputting the input data into the environmental feature encoder to obtain an environmental feature vector;

[0017] The environmental feature vector is input into the environmental feature decoder to obtain own state prediction data and second environmental reward data.

[0018] Optionally, the adversary's uncertainty feature vector is determined by:

[0019]

[0020] in, is the uncertainty feature vector of the opponent, is the mean vector, is the covariance matrix, is the uncertainty feature encoder, is the current observation data of one's own side, is the current own action data, are the deep neural network parameters of the uncertainty feature encoder.

[0021] Optionally, inputting the opponent's uncertainty feature vector into the uncertainty feature decoder to obtain the opponent's second action prediction data and second observation prediction data includes:

[0022] Obtain a mean vector and a covariance matrix according to the opponent's uncertainty feature vector, and based on the mean vector and the covariance matrix Joint Gaussian distribution;

[0023] The joint Gaussian distribution is sampled to obtain the opponent's second action prediction data and second observation prediction data.

[0024] Optionally, calculating a loss function using the current opponent observation data, the current opponent action data, the first environment reward data, the next own observation data, and the model output includes:

[0025] fusing the first action prediction data and the second action prediction data of the opponent to obtain action prediction data of the opponent, and fusing the first observation prediction data and the second observation prediction data to obtain observation prediction data of the opponent;

[0026] Based on the current opponent observation data, the current opponent action data, the first environment reward data and the next own observation data, the opponent's action prediction data and observation prediction data, and the own state prediction data and the second environment reward data, the loss function is calculated by the following formula:

[0027]

[0028] in, is the loss function, , is the total number of time steps, is the current opponent observation data, Predict data for the opponent's observations, is the probability function, is the current opponent action data, Predict data for the opponent's actions, For the next step of our own observation data, Predict data for your own status, Reward data for the first environment, Reward data for the second environment.

[0029] Optionally, optimizing the opponent modeling model based on the loss function until a preset condition is met to obtain a trained opponent modeling model includes:

[0030] Obtain an optimizer, and optimize the deep neural network parameters of the deterministic feature encoder, the uncertain feature encoder, the deterministic feature decoder, the uncertain feature decoder, the environmental feature encoder, and the environmental feature decoder based on the loss function and the optimizer until preset conditions are met, thereby obtaining a trained opponent modeling model.

[0031] In a second aspect, the present invention further provides an opponent modeling device based on uncertainty characteristics, comprising the following modules:

[0032] A data acquisition module is used to acquire a pre-established opponent modeling model and interaction data during the interaction between the agent and the environment; wherein the interaction data includes current own observation data, current own action data, current opponent observation data, current opponent action data, first environment reward data, and next self observation data;

[0033] a feature modeling module, configured to input the current own observation data and the current own action data into the opponent modeling model, model the deterministic features and uncertain features of the opponent, and obtain a model output;

[0034] a loss calculation module, configured to calculate a loss function using the current opponent observation data, the current opponent action data, the first environment reward data, the next step own observation data, and the model output;

[0035] A model optimization module, configured to optimize the opponent modeling model based on the loss function until a preset condition is met, thereby obtaining a trained opponent modeling model;

[0036] The opponent modeling module is used to perform opponent modeling based on the trained opponent modeling model to obtain an opponent modeling result.

[0037] In a third aspect, the present invention also provides an electronic device comprising a memory, a processor, and a computer program stored in the memory and running on the processor, wherein when the processor executes the computer program, the opponent modeling method based on uncertainty characteristics as described in the first aspect above is implemented.

[0038] In a fourth aspect, the present invention further provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the opponent modeling method based on uncertainty characteristics as described in the first aspect above.

[0039] In a fifth aspect, the present invention further provides a computer program product, comprising a computer program, which, when executed by a processor, implements the opponent modeling method based on uncertainty characteristics as described in the first aspect above.

[0040] Based on the above technical solution, the present invention has the following beneficial effects compared with the prior art:

[0041] The uncertainty-based opponent modeling method and apparatus provided by the present invention fully utilizes the data generated during the interaction between the intelligent agent and the environment. This data encompasses a rich source of information, including observation data, action data, and environmental reward data for both the agent and the opponent. This data provides the necessary input for the model, helping to accurately capture the adversary's stochastic behavior. The present invention not only incorporates deterministic characteristics of the adversary into the adversary model, but also innovatively introduces uncertain characteristics, such as random variables and noise terms, to fully reflect the adversary's stochastic behavior. This approach helps the model more accurately grasp the volatility and uncertainty of the adversary's behavior, enabling effective modeling of adversaries with stochastic behavior. As the interaction between the agent and the environment continues to deepen, new interaction data is continuously collected and used to update the adversary modeling model. This process enhances the model's adaptability to changes and uncertainties in adversary behavior, improving the real-time and accuracy of modeling. During the modeling process, this method quantifies the uncertainty of the adversary's behavior, providing the agent with more comprehensive information about the likelihood of adversary behavior, thereby more realistically reflecting the adversary's behavioral patterns. Furthermore, by calculating a loss function and optimizing based on it, the method enables continuous iteration and improvement of the model, ensuring a better fit to actual observed data. This optimization mechanism helps to reduce model errors and further improve the accuracy of modeling.

[0042] Other features and advantages of the present invention will be described in the following description, and in part will become apparent from the description, or will be understood through implementation of the present invention. The purposes and other advantages of the present invention are realized and obtained by the structures particularly pointed out in the description, claims and drawings.

[0043] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, preferred embodiments are given below and described in detail with reference to the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] In order to more clearly illustrate the technical solutions in the present invention or the prior art, a brief introduction is given below to the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0045] Figure 1 This is one of the flow charts of the opponent modeling method based on uncertainty characteristics provided by the present invention.

[0046] Figure 2 This is the second flow chart of the opponent modeling method based on uncertainty characteristics provided by the present invention.

[0047] Figure 3 It is a schematic diagram of quantitative analysis of the final effects of the traditional method and the method of the present application in a simulation environment.

[0048] Figure 4 It is a diagram showing the quantitative analysis of the final effects of the traditional method and the method of the present application on the human-computer interaction environment.

[0049] Figure 5 This is a diagram showing the visual analysis results of the traditional method and the method of this application in terms of opponent differentiation.

[0050] Figure 6 It is a structural diagram of the opponent modeling device based on uncertainty characteristics provided by the present invention.

[0051] Figure 7 It is a structural schematic diagram of the electronic device provided by the present invention. DETAILED DESCRIPTION

[0052] To make the objectives, technical solutions, and advantages of the present invention more clear, the technical solutions of the present invention will be clearly and completely described below in conjunction with the accompanying drawings. Obviously, the embodiments described are only some of the embodiments of the present invention, not all of them. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts shall fall within the scope of protection of the present invention.

[0053] The following combination Figures 1-6 The present invention describes an opponent modeling method and device based on uncertainty characteristics.

[0054] Reference Figure 1 As shown, the adversary modeling method based on uncertainty features includes the following:

[0055] Step S110: Acquire a pre-established opponent modeling model and interaction data during the interaction between the agent and the environment; wherein the interaction data includes current own observation data, current own action data, current opponent observation data, current opponent action data, first environment reward data, and next own observation data;

[0056] In this step, first, obtain a pre-designed and preliminarily constructed opponent model. Next, initialize the model and simulation environment. Specifically, initialize the parameters of the own decision-making model and the opponent model; initialize the data collection list; and initialize the multi-agent simulation environment.

[0057] The own decision model refers to the part of the own agent that makes decisions. This can be a simple set of rules, a lookup table, a deep neural network, or another type of machine learning model. The own decision model determines what action the own agent should take in a given state. The opponent modeling model is a model used to predict the opponent's likely actions in a given state and observation data. This model aims to capture the opponent's behavioral characteristics during the interaction between the agent and the environment, including both deterministic and uncertain characteristics. Similarly, it can be a deep neural network, whose parameters are optimized during training using actual opponent behavior data. The initialization data collection list is a list used to store interaction data generated by agents (including the own agent and the opponent) during their interaction with the environment. This interaction data includes observation data, action data, and environment reward data for both the own and opponent. This data is used to train and optimize the opponent modeling model.

[0058] A simulation environment is a platform that mimics the real world and contains multiple agents (both friendly and adversary) and the environment in which they interact. Initializing the environment returns initial state data and observation data for both friendly and adversary agents. These observations form the basis for the agents' decision-making within the environment.

[0059] At each time step, the friendly agent first receives current observation data from the environment (such as position, velocity, obstacle information, etc.). The friendly agent inputs this observation data into its decision-making model, which outputs an action decision (such as movement direction, attack strategy, etc.). The friendly agent executes the action and sends the action data to the environment. The environment updates its state based on the friendly agent's action and any actions of the opponent agent. The environment returns the new state data and the corresponding observation data for the friendly and opponent agents (i.e., the next observation data). Simultaneously, the environment calculates and returns reward data to the friendly agent based on the state changes and possible rules. After receiving the new observation data and reward data, the friendly agent uses this data, along with previously stored interaction data, to update the opponent model.

[0060] Second, interaction data is collected during the agent's interaction with the environment. The agent receives observations from the simulated environment, takes an action based on these observations and possible internal states, and then receives the next observation and reward from the environment. This process may repeat multiple times until a termination condition is reached. In the simulated environment, the opponent agent is another entity interacting with the agent. It also receives observations and influences the state of the environment, thereby affecting the agent's observations and rewards.

[0061] The above interaction data specifically includes the current observation data of the party , Current own action data , Current opponent observation data , current opponent action data , First Environment Reward Data And the next step of our own observation data . Record the current step data and save it in data format .

[0062] Current friendly observation data is the agent's own agent at the time step Observation data received from the environment, including information such as the agent's current position, the characteristics of the surrounding environment, and the positions or states of other agents. The specific content of the observation data depends on the settings of the simulation environment. It is the time step of the agent according to the current observation data and its own strategy. The specific content of the action depends on the agent's strategy or decision-making process, and can include moving, attacking, collecting resources, etc.

[0063] Current opponent observation data is the opponent agent at time step Observation data received from the environment. Current opponent action data is the opponent agent at time step Action taken.

[0064] First environment reward data is the agent's time step The reward obtained from the environment after executing an action at the current time step reflects the effect of the agent's behavior at the current time step. The specific value and meaning of the reward depends on the setting of the simulation environment and is used to evaluate the effect or performance of the agent's behavior. It is the time step after the agent performs the current action. New observation data is received from the environment at the same time. It reflects the change of the environment state and can include information such as the new position of the agent, new environmental features, and the new position or state of other agents.

[0065] After each time step, the current observation data , Current own action data , Current opponent observation data , current opponent action data , First Environment Reward Data And the next step of our own observation data Stored in the data collection list.

[0066] Step S120: Input the current own observation data and the current own action data into the opponent modeling model, model the deterministic characteristics and uncertain characteristics of the opponent, and obtain a model output.

[0067] The current observation and action data from the collected data list is used as input to the opponent modeling model. The model then uses this data to model the opponent's deterministic and uncertain characteristics. Deterministic characteristics include relatively stable features such as the opponent's intelligence level and strategic preferences; whereas uncertain characteristics include unpredictable features such as the opponent's random behavior and noise interference. The model output includes predictions of the opponent's possible actions and the next observation data.

[0068] Step S130: Calculate a loss function using the current opponent observation data, the current opponent action data, the first environment reward data, the next self observation data, and the model output. The loss function measures the difference between the model's predicted results and the actual results, reflecting the model's performance on the current dataset.

[0069] Step S140: Optimize the opponent modeling model based on the loss function until a preset condition is met to obtain a trained opponent modeling model.

[0070] The loss function provides a clear direction for model training. Based on the calculated loss function, optimization algorithms (such as gradient descent and stochastic gradient descent) are used to optimize the parameters of the adversary model. The optimization goal is to minimize the loss function, thereby improving the accuracy of the model's predictions. This process requires multiple iterations, repeating steps S110-S140 to continuously update the model parameters until a preset condition is met (such as the loss function converges below a certain threshold or the maximum number of iterations is reached). By minimizing the loss function, the model's predictions can be made closer to the actual results. The loss function is also an important metric for evaluating model performance. During model training, changes in the loss function can be used to assess the effectiveness of the model training. During model training, certain hyperparameters (such as the learning rate and batch size) need to be adjusted to optimize model performance. The loss function can serve as a basis for adjusting these hyperparameters.

[0071] Step S150: Perform opponent modeling based on the trained opponent modeling model to obtain an opponent modeling result.

[0072] After model optimization is complete, a trained adversary model is obtained. This model is used to perform adversary modeling, which involves predicting the adversary's likely behavior and next observation based on the current observation and action data of the agent. The adversary modeling results are output and used in the subsequent decision-making and planning process of the agent.

[0073] The uncertainty-based opponent modeling method provided by this invention fully utilizes the data generated during the interaction between the agent and the environment. This data includes rich information sources such as observation data, action data, and environmental reward data for both the agent and the opponent. This data provides the necessary input for the model, helping to accurately capture the stochastic behavior characteristics of the opponent. This invention not only incorporates deterministic characteristics of the opponent into the opponent modeling model, but also innovatively introduces uncertain characteristics, such as random variables and noise terms, to fully reflect the opponent's stochastic behavior. This approach helps the model more accurately grasp the volatility and uncertainty of the opponent's behavior, enabling effective modeling of opponents with stochastic behavior. As the interaction between the agent and the environment continues to deepen, new interaction data is continuously collected and used to update the opponent modeling model. This process enhances the model's adaptability to changes and uncertainties in opponent behavior, improving the real-time and accuracy of modeling. This method quantifies the uncertainty of the opponent's behavior during the modeling process, providing the agent with more comprehensive information about the likelihood of the opponent's behavior, thereby more realistically reflecting the opponent's behavioral patterns. Furthermore, by calculating a loss function and performing optimization based on it, this method enables continuous iteration and improvement of the model, ensuring that it better fits actual observed data. This optimization mechanism helps to reduce model errors and further improve the accuracy of modeling.

[0074] In an optional embodiment, the model output in S120 includes the opponent's first action prediction data , first observation prediction data , second action prediction data , second observation prediction data , and own status prediction data and second environment reward data ;

[0075] Opponent's first move prediction data Represents the model prediction, at time step The actions that the opponent may take are based on the prediction of deterministic features. It represents the model's prediction based on the current input data (the current observation data of the opponent). , Current own action data ) predicts a possible opponent action. The opponent's first observation prediction data This means that the model predicts If the opponent takes the first action prediction data The environmental states that may be observed when taking actions in .

[0076] Opponent's second move prediction data Also represents the model prediction, at time step The action that the opponent may take is predicted based on uncertainty. Here, it refers to the state of the environment that the opponent sees from his own perspective. The opponent's second observation prediction data represents the model's prediction at the time step If the opponent takes the first action prediction data The environmental states that may be observed when taking actions in .

[0077] Team status prediction data Represents the model prediction, at time step The environment state that your agent may observe also refers to the environment state seen from your perspective. Unlike the opponent's observation and prediction data, this is directly related to the future state of your agent.

[0078] Second Environment Reward Data Represents the model prediction, at time step The environment may give rewards to the agent. Rewards are the environment's feedback on the agent's behavior and are used to guide the agent's learning and optimization process.

[0079] Reference Figure 2As shown, the opponent modeling model includes a deterministic feature encoder, an uncertain feature encoder, a deterministic feature decoder, an uncertain feature decoder, an environment feature encoder, and an environment feature decoder. The present invention collects data on the interaction between the intelligent agent and the game simulation environment; uses the data and the deterministic feature encoder to calculate the opponent's deterministic feature vector; uses the data and the uncertain feature encoder to calculate the opponent's distribution characteristics; uses the deterministic feature decoder and the deterministic feature vector to calculate the opponent's possible actions and observations; uses the uncertain feature decoder and the uncertain feature vector to calculate the opponent's possible actions and observations; uses the data and the environment encoder to calculate the environment's feature vector; uses the data and the environment encoder to calculate the environment's feature vector; uses the environment feature vector and the environment feature decoder to calculate the environment's possible next state and the current step reward; and uses real data to update the parameters of each encoder and decoder.

[0080] The step S120 described above, wherein the current own observation data and the current own action data are input into the opponent modeling model, modeling the deterministic characteristics and the uncertain characteristics of the opponent, and obtaining the model output, includes:

[0081] S1201. Use the current own observation data and the current own action data as input data, input the input data into the deterministic feature encoder and the uncertain feature encoder, and obtain the opponent's deterministic feature vector and uncertain feature vector respectively.

[0082] The input data is fed into a deterministic feature encoder and an uncertain feature encoder, respectively. The deterministic feature encoder extracts deterministic features from the input data. These features are typically predictable and directly related to the adversary's behavior, and outputs a deterministic feature vector. The uncertain feature encoder extracts uncertain features from the input data. These features reflect the randomness and unpredictability of the adversary's behavior, and outputs an uncertain feature vector for the adversary.

[0083] Specifically, A deterministic feature encoder composed of a deep neural network with parameters Input Data . Record the neural network output vector As the deterministic feature vector of the opponent. The calculation process formula is: . Uncertain feature encoder composed of deep neural networks with parameters Incoming Data . Record the neural network output vector As the opponent's uncertainty feature vector. The calculation process formula is: .

[0084] S1202: Input the deterministic feature vector into the deterministic feature decoder to obtain the first action prediction data. and the first observation prediction data These data reflect the possible actions that the adversary may take and the possible observation results under deterministic features.

[0085] First, the deterministic feature vector is input into the deterministic feature decoder. This feature vector is obtained from the above step S1201 and contains deterministic information about the opponent's behavior, such as the opponent's common strategy, preferred actions, etc. The deterministic feature decoder is a deep neural network that can convert the input deterministic feature vector into specific prediction data. In this step, the decoder outputs two sets of data: the first action prediction data and the first observation prediction data .

[0086] First action prediction data Represents the most likely action taken by the adversary under deterministic characteristics. It is inferred based on the adversary’s historical behavior and the current state of the environment. First Observation Prediction Data The first action prediction data reflects the environment state that the opponent might observe if they take the action in the first action prediction data. It is a prediction of the future environment state from the opponent's perspective. These two sets of data provide the agent with direct information about the opponent's possible behavior, helping the agent to consider the opponent's reaction when making decisions.

[0087] Specifically, A deterministic feature decoder consisting of a deep neural network with parameters Input the deterministic feature vector of the adversary . Record the neural network output As the opponent's observation and action prediction.

[0088] S1203: Input the opponent's uncertainty feature vector into the uncertainty feature decoder to obtain the second action prediction data. and the second observation prediction data These data reflect the probability distribution of possible actions taken by the adversary and possible observation results under uncertainty characteristics.

[0089] The adversary’s uncertainty feature vector is input into the uncertainty feature decoder. This feature is obtained by the uncertainty feature encoder and contains information about the uncertainty of the adversary’s behavior, such as the probability distribution of different actions the adversary may take.

[0090] The uncertainty feature decoder is also a deep neural network, but it can process uncertainty features and output prediction data in the form of probability distribution. In this step, the decoder outputs two sets of data: the second action prediction data and the second observation prediction data .

[0091] Second action prediction data It is a probability distribution that represents the probability of different actions that the opponent may take under uncertainty. It provides the agent with more comprehensive information about the opponent's possible behavior.

[0092] Second observation prediction data is also a probability distribution that represents the probability distribution of the environment states that the adversary might observe if they take an action in the second action prediction data. These two sets of data provide the agent with a probabilistic description of the adversary's possible behavior, helping it make more robust decisions in uncertain environments.

[0093] S1204: Input the input data into the environment feature encoder to obtain an environment feature vector, which contains state information of the current environment.

[0094] The input data is fed into the environment feature encoder. This encoder is a deep neural network that extracts information about the current environment from the input data. The output of the encoder is the environment feature vector, which contains key information about the current environment, such as the location of obstacles and the relative positions of the player and the opponent. This vector serves as an important basis for environmental prediction and decision-making in subsequent steps.

[0095] Specifically, The environmental feature encoder is composed of a deep neural network with parameters Incoming Data . Record the neural network output vector As the opponent's environmental feature vector. The calculation process formula is: .

[0096] S1205: Input the environmental feature vector into the environmental feature decoder to obtain own state prediction data and second environment reward data .

[0097] The environment feature vector is input into the environment feature decoder. This decoder is a deep neural network that can convert the environment feature vector into specific prediction data. Reflects the possible state of the team at a certain point in the future. It is predicted based on the current state of the environment and the team's action strategy. Second environment reward data The reward is the potential reward for the agent under the current environment. It is calculated based on the environment's rules and the agent's action strategy. These two sets of data provide the agent with predictive information about possible future states and rewards, helping it consider long-term benefits when making decisions and optimize its strategy.

[0098] Specifically, The environmental feature decoder composed of a deep neural network with parameters Input the opponent's environment feature vector . Record the neural network output As the predicted output of the next observation of the environment and the reward of this step. The calculation process formula is: .

[0099] By extracting feature vectors of the opponent and environment and using a decoder to make predictions, the present invention provides the intelligent agent with comprehensive information about the opponent's likely behavior and future environmental states. This information is a crucial basis for the agent's decision-making and strategy optimization. Specifically, by separately processing deterministic and uncertain features, the model can more accurately capture the opponent's behavioral patterns. Deterministic features provide the predictable aspects of the opponent's behavior, while uncertain features account for the randomness and unpredictability of the opponent's behavior. This processing approach allows the model to simultaneously consider both deterministic and uncertain factors when predicting opponent actions and observations, thereby improving prediction accuracy and robustness. The deterministic and uncertain feature encoders extract different types of features, respectively, enabling the model to more flexibly adapt to different environments and opponents. When faced with a new environment or opponent, the model can readjust the feature encoding and decoding processes based on the new input data, allowing it to quickly adapt to the new situation. By outputting multiple types of predicted data (including opponent action predictions, opponent observation predictions, own state predictions, and environmental reward data), the model provides decision makers with rich information support. This data can help decision makers better understand the current environment and opponent situation, leading to more informed decisions.

[0100] Compared to existing technologies, this invention introduces a hybrid deterministic and uncertain feature encoding and decoding technology, effectively improving the ability to model adversaries with stochastic behavior. Compared to existing adversary modeling methods that only utilize deterministic encoding and decoding, the introduction of an uncertain encoder can effectively improve modeling accuracy and downstream decision-making performance.

[0101] In an optional embodiment, in the process of adversary modeling, in order to capture the uncertainty of adversary behavior, an uncertainty feature encoder is introduced. The purpose of this encoder is to use the current observation data of the party and action data , to infer the adversary’s uncertainty feature vector. Uncertainty Feature Encoder is a deep neural network with parameters The input of the network is the current observation data and action data of the player, and the output is the uncertainty feature vector of the opponent.

[0102] The uncertainty feature vector of the opponent in step S1201 is determined by the following formula:

[0103]

[0104] in, is the uncertainty feature vector of the opponent, is the mean vector, is the covariance matrix, is the uncertainty feature encoder, is the current observation data of one's own side, is the current own action data, are the deep neural network parameters of the uncertainty feature encoder.

[0105] is the output of the uncertainty feature encoder, which represents the uncertainty feature vector of the opponent's behavior. This distribution can be used to generate possible opponent actions or observation predictions, thereby providing a basis for the agent's decision-making. It consists of two parts: the mean vector μ and the covariance matrix The mean vector describes the central tendency of the adversary’s behavior in the uncertainty distribution, while the covariance matrix describes the correlations between dimensions and the shape of the uncertainty distribution.

[0106] By introducing an uncertainty feature encoder, the present invention enables the model to better handle the randomness and uncertainty of the opponent's behavior. This helps the intelligent agent make more robust decisions in complex and changing environments. As the intelligent agent interacts with the environment, the uncertainty feature encoder can continuously update its parameters based on new observation data and action data, thereby adapting to changes in the opponent's behavior. This enables the model to continuously learn and optimize its opponent modeling capabilities. By modeling the uncertainty of the opponent's behavior, the intelligent agent can have a more comprehensive understanding of the opponent's potential behavior patterns. This helps the intelligent agent consider more possibilities when making decisions and make more reasonable choices. In a multi-agent environment, the opponent's uncertainty feature vector can serve as an important basis for optimizing the agent's strategy. By continuously learning and adjusting the strategy, the intelligent agent can gradually reduce the uncertainty when interacting with the opponent, thereby improving its overall performance.

[0107] In an optional embodiment, the step of inputting the opponent's uncertainty feature vector into the uncertainty feature decoder to obtain the opponent's second action prediction data and second observation prediction data in S1203 includes:

[0108] S12031, obtaining a mean vector and a covariance matrix according to the opponent's uncertainty feature vector, and based on the mean vector and the covariance matrix Joint Gaussian distribution.

[0109] In the opponent uncertainty feature vector After inputting the uncertainty feature decoder, the first task of the decoder is to parse these features. These features include a mean vector ( ) and a covariance matrix ( The mean vector describes the central tendency of the adversary’s behavior in the uncertainty distribution, i.e., the most likely action or observation. The covariance matrix describes the correlations between dimensions and the shape of the uncertainty distribution, i.e., the correlations between actions or observations and their degree of dispersion.

[0110] After obtaining the mean vector and covariance matrix, a joint Gaussian distribution can be reconstructed based on this information The joint Gaussian distribution is a multivariate probability distribution that describes the joint probability between multiple random variables. In this scenario, it is used to represent the probabilistic relationship between the adversary's possible actions and possible observations. By reconstructing the joint Gaussian distribution, a more comprehensive and accurate model of the adversary's behavior can be obtained.

[0111] S12032: Sampling the joint Gaussian distribution to obtain second action prediction data and the second observation prediction data .

[0112] After reconstructing the joint Gaussian distribution, we can sample this distribution. The sampling process is to generate a set of random data points based on the characteristics of the distribution. These data points represent the possible actions and possible observation results of the opponent. Through sampling, we can get the second action prediction data and the second observation prediction data These data not only contain the adversary’s possible behavior patterns, but also reflect the probabilistic relationships between these behavior patterns.

[0113] By introducing an uncertainty feature decoder and a joint Gaussian distribution, the present invention can more comprehensively capture the randomness and uncertainty of the opponent's behavior. This makes the prediction results more accurate and can better reflect the opponent's true behavior. Because the uncertainty of the opponent's behavior is taken into account, the method can demonstrate greater robustness when faced with complex and changing environments. Even if the opponent's behavior changes or there is noise interference, the model can maintain stable prediction performance. By providing uncertainty information about the opponent's behavior, the method provides strong support for the strategy optimization of the intelligent agent. The intelligent agent can adjust its own strategy based on this information to better deal with the opponent's uncertain behavior. As the interaction between the intelligent agent and the environment deepens, new interaction data is continuously collected and used to update the opponent modeling model. This process promotes the learning and adaptation of the intelligent agent, enabling it to gradually improve its ability to predict the opponent's behavior.

[0114] In an optional embodiment, the step S130 described above of calculating the loss function using the current opponent observation data, the current opponent action data, the first environment reward data, the next step own observation data, and the model output includes:

[0115] S1301, the opponent's first action prediction data and the second action prediction data Fusion to obtain the opponent's action prediction data , and the first observation prediction data and the second observation prediction data Fusion to obtain the opponent's observation and prediction data .

[0116] Using the first action prediction data output by the deterministic feature decoder and the second action prediction data output by the uncertain feature decoder, the final opponent's action prediction data is obtained through a fusion method (such as weighted average, maximum likelihood estimation, etc.) .

[0117] Similarly, the first observation prediction data output by the deterministic feature decoder and the second observation prediction data output by the uncertain feature decoder are used to obtain the opponent's observation prediction data through a fusion method. .

[0118] S1302. Calculate the loss function using the following formula based on the real data (i.e., the current opponent observation data, the current opponent action data, the first environment reward data, and the next own observation data) and the multiple prediction data output by the model (the opponent's action prediction data and observation prediction data, as well as the own state prediction data and the second environment reward data):

[0119]

[0120] in, is the loss function, , is the total number of time steps, is the current opponent observation data, Predict data for the opponent's observations, is the probability function, is the current opponent action data, Predict data for the opponent's actions, For the next step of our own observation data, Predict data for your own status, Reward data for the first environment, Reward data for the second environment.

[0121] When calculating the loss function, 、 The accuracy of the opponent's observation predictions and the accuracy of the environment's reward predictions are taken into account separately. These terms are added together and divided by the total number of time steps , to get an average loss value.

[0122] By comprehensively considering information from multiple aspects, the loss function of the present invention can more comprehensively evaluate the model's predictive performance. This helps the model continuously optimize its predictive capabilities during training, thereby improving the accuracy of predictions of opponent behavior and its own state. Because the loss function takes into account a variety of predictive data and environmental factors, the model needs to simultaneously consider multiple constraints during training. This helps the model demonstrate greater robustness and adaptability in the face of complex and changing interactive environments. Accurate predictive capabilities provide strong support for the agent's strategy optimization. By predicting the opponent's possible behavior and its own future state, the agent can develop more effective strategies to respond to the opponent's challenges and environmental changes.

[0123] In an optional embodiment, the step S140 of optimizing the opponent modeling model based on the loss function until a preset condition is met to obtain a trained opponent modeling model includes:

[0124] Obtain an optimizer, and optimize the deep neural network parameters of the deterministic feature encoder, the uncertain feature encoder, the deterministic feature decoder, the uncertain feature decoder, the environmental feature encoder, and the environmental feature decoder based on the loss function and the optimizer until preset conditions are met, thereby obtaining a trained opponent modeling model.

[0125] First, select an optimizer that converges quickly and is stable and reliable, such as stochastic gradient descent (SGD) or Adam. The choice of optimizer depends on the specific application scenario and model architecture. Next, initialize the parameters of the deep neural network, including the deterministic feature encoder, uncertain feature encoder, deterministic feature decoder, uncertain feature decoder, environment feature encoder, and environment feature decoder. These parameters are key to model learning, determining how the model processes input data and generates output. Next, use the model with the current parameters to calculate the loss function. This involves inputting the current opponent observation data and action data into the model, generating predictions, and then calculating the difference between the predictions and the actual observations (i.e., the loss). The optimizer uses the gradient information of the loss function to calculate how to adjust the parameters to minimize the loss. This can be done using the backpropagation algorithm, which calculates the gradient of the loss function with respect to each parameter and updates the parameters based on these gradients. Repeat these steps until a predetermined condition is met. These conditions may include reaching a certain number of iterations, the loss function value being less than a threshold, or the model's performance on the validation set no longer improving. When the predetermined condition is met, the iteration stops and the model with the current parameters is saved as the trained opponent model.

[0126] During the training phase, refer to Figure 2 As shown, the current observation data of one's own side and current own action data Input the deterministic feature encoder and uncertainty feature encoder of the opponent behavior modeling module, use the deterministic feature encoder to output a deterministic feature vector, and the uncertainty feature encoder to output an uncertainty feature vector. Deterministic feature decoder and uncertainty feature decoder are used to decode the features respectively. Combine the outputs of the deterministic feature decoder and the uncertainty feature decoder to obtain the opponent's action prediction data. and observational forecast data . The current observation data of your own side and current own action data The environment feature encoder of the input environment modeling module outputs the environment feature vector, and the environment feature decoder decodes the features and outputs the prediction data of the own state. and second environment reward data , calculate the loss function using the loss function formula in step S1302 above .

[0127] In the test phase, the deterministic feature vectors output by the deterministic feature encoder, the uncertainty feature encoder, and the environment feature encoder in the training phase are converted into , uncertainty eigenvector , environmental feature vector These features are spliced together, specifically by splicing the deterministic feature vector and the uncertain feature vector into Then, and the environmental feature vector The spliced features are obtained by splicing, and the spliced features are used for downstream reinforcement learning tasks.

[0128] The present invention can minimize the loss function by iteratively optimizing the parameters of the deep neural network, thereby improving the model's predictive accuracy. This means that the model can better understand the opponent's behavior and the dynamic changes in the environment, thereby making more intelligent decisions. During the training process, the model will learn the inherent patterns and characteristics of the data, rather than just memorizing the training data. This helps the model still perform well when faced with unseen data, that is, enhances the model's generalization ability. Selecting the right optimizer and adjusting parameters can accelerate the model's convergence and reduce training time. This is especially important for large-scale datasets and complex models, which require longer training times to achieve satisfactory performance. By monitoring changes in the loss function and the model's performance on the validation set, model problems can be discovered and debugged in a timely manner. In addition, different hyperparameters such as optimizers, learning rates, and batch sizes can be tried to further optimize model performance.

[0129] Reference Figure 3 Figure 2 shows a quantitative analysis of the final performance of traditional methods (LIAM, NAM, CBAM, and GR2) and our method (Ours) in a simulation environment. Our method was tested in multiple validation environments and achieved higher scores against a baseline random behavior simulated adversary. In particular, our method significantly outperformed other methods in some environments. Figure 3 The vertical axis, Rewards, represents the score, and the horizontal axis, Steps, represents the time step. The first column, titled Double Speaker Listener (hard, easy), represents the two-person navigation task (difficult, easy). The second column, titled Level-Based Foraging (large, scarcity), represents the two-person carrying task (large, small). The third column, titled Predator Prey (complex, simple), represents the pursuit-escape task (complex, simple).

[0130] LIAM can refer to the literature: Papoudakis G, Christianos F, Albrecht S. Agentmodelling under partial observability for deep reinforcement learning[J]. Advances in Neural Information Processing Systems, 2021, 34: 19210-19222. NAM can refer to the literature: Yan Duan, John Schulman, Xi Chen, Peter L Bartlett, IlyaSutskever, and Pieter Abbeel. 2016. RL2: Fast reinforcement learning via slowreinforcement learning. arXiv preprint arXiv:1611.02779 (2016). CBAM can refer to the literature: Yuheng Jing, Kai Li, Bingyun Liu, Yifan Zang, Haobo Fu, QIANG FU, JunliangXing, and Jian Cheng. 2023. Towards Offline Opponent Modeling with In-contextLearning. In The Twelfth International Conference on LearningRepresentations. GR2 can refer to the literature: Ying Wen, Yaodong Yang, Rui Luo, and Jun Wang. 2019. Modeling bounded rationality in multi-agent interactions by generalized recursive reasoning. arXiv preprint arXiv:1901.09216 (2019).

[0131] Reference Figure 4 Figure 2 shows a quantitative analysis of the final performance of traditional methods (LIAM, NAM, and CBAM) and our method (Ours) in a human-computer interaction environment. Our method was validated in a human-computer interaction environment and demonstrated superiority over other methods when interacting with human opponents.

[0132] Overcook is a policy environment, and Cramped Room, Asymmetric Advantages Environment, and Coordination Ring are the three test tasks within this environment. Cramped Room refers to Overcook (a narrow room), Asymmetric Advantages Environment refers to Overcook (an asymmetric room), and Coordination Ring refers to a circular room. For more information about the Overcook policy environment, please refer to the following literature: Micah Carroll, Rohin Shah, Mark K Ho, Tom Griffiths, Sanjit Seshia, Pieter Abbeel, and Anca Dragan. 2019. On the utility of learning about humans for human-AI coordination. Advances in neural information processing systems 32 (2019).

[0133] Reference Figure 5 Figure 2 shows a visualization of the discriminability of adversaries (i.e., Agent 1 and Agent 2 in the figure) using a traditional method (LIAM) and our method (Ours). During adversary modeling, our method extracts latent space features of the adversary model and finds that our method exhibits stronger adversary discriminability than the baseline method, demonstrating the effectiveness of our method for adversary modeling.

[0134] The opponent modeling device based on uncertainty features provided by the present invention is described below. The opponent modeling device based on uncertainty features described below and the opponent modeling method based on uncertainty features described above can refer to each other.

[0135] The opponent modeling device based on uncertainty characteristics provided by the present invention refers to Figure 6 Shown, including:

[0136] Data acquisition module 210, for acquiring a pre-established opponent modeling model and interaction data during the interaction between the agent and the environment; wherein the interaction data includes current own observation data, current own action data, current opponent observation data, current opponent action data, first environment reward data, and next self observation data;

[0137] A feature modeling module 220 is configured to input the current own observation data and the current own action data into the opponent modeling model, model the deterministic features and uncertain features of the opponent, and obtain a model output;

[0138] a loss calculation module 230 for calculating a loss function using the current opponent observation data, the current opponent action data, the first environment reward data, the next step own observation data, and the model output;

[0139] A model optimization module 240 is configured to optimize the opponent modeling model based on the loss function until a preset condition is met to obtain a trained opponent modeling model;

[0140] The opponent modeling module 250 is configured to perform opponent modeling based on the trained opponent modeling model to obtain an opponent modeling result.

[0141] Figure 7 An example of a physical structure diagram of an electronic device is shown below. Figure 7 As shown, the electronic device may include: a processor 310, a communications interface 320, a memory 330, and a communication bus 340. The processor 310, the communications interface 320, and the memory 330 communicate with each other via the communication bus 340. The processor 310 may call logic instructions in the memory 330 to execute the adversary modeling method based on uncertainty characteristics.

[0142] Furthermore, the logic instructions in the aforementioned memory 330 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the portion that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product, stored in a storage medium, includes several instructions for enabling a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as a USB flash drive, a mobile hard drive, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk.

[0143] On the other hand, the present invention also provides a computer program product, which includes a computer program, which can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the opponent modeling method based on uncertainty characteristics provided by the above methods.

[0144] On the other hand, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, is implemented to execute the opponent modeling method based on uncertainty characteristics provided by the above methods.

[0145] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, i.e., they may be located in one location or distributed across multiple network units. Some or all of the modules may be selected based on actual needs to achieve the objectives of the present embodiment. Persons of ordinary skill in the art will be able to understand and implement the present invention without inventive effort.

[0146] Through the above description of the embodiments, those skilled in the art will clearly understand that each embodiment can be implemented using software plus a necessary general-purpose hardware platform, or of course, hardware. Based on this understanding, the essence of the above technical solution, or the portion that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a magnetic disk, or an optical disk, and includes a number of instructions for causing a computer device (such as a personal computer, server, or network device) to execute the methods described in each embodiment or certain portions of the embodiments.

[0147] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the various embodiments of the present invention.

Claims

1. An adversary modeling method based on uncertainty characteristics, characterized in that: include: Obtaining a pre-established opponent modeling model and interaction data during the interaction between the agent and the environment; wherein the interaction data includes current own observation data, current own action data, current opponent observation data, current opponent action data, first environment reward data, and next own observation data; Inputting the current own observation data and the current own action data into the opponent modeling model, modeling the opponent's deterministic characteristics and uncertain characteristics, and obtaining a model output; the deterministic characteristics are relatively stable characteristics, including the opponent's intelligence level and strategic preferences; the uncertain characteristics are difficult to predict, including the opponent's random behavior and noise interference; the deterministic characteristics are the predictable parts directly related to the opponent's behavior, and the uncertain characteristics reflect the randomness and unpredictability of the opponent's behavior; Calculating a loss function using the current opponent observation data, the current opponent action data, the first environment reward data, the next step own observation data, and the model output; Optimizing the opponent modeling model based on the loss function until a preset condition is met to obtain a trained opponent modeling model; Performing opponent modeling based on the trained opponent modeling model to obtain an opponent modeling result; The model output includes first action prediction data, first observation prediction data, second action prediction data, second observation prediction data, own state prediction data and second environment reward data; the opponent modeling model includes a deterministic feature encoder, an uncertain feature encoder, a deterministic feature decoder, an uncertain feature decoder, an environment feature encoder and an environment feature decoder; The step of inputting the current own observation data and the current own action data into the opponent modeling model, modeling the deterministic characteristics and the uncertain characteristics of the opponent, and obtaining a model output includes: Taking the current own observation data and the current own action data as input data; Inputting the input data into the deterministic feature encoder and the uncertain feature encoder to obtain a deterministic feature vector and an uncertain feature vector of the opponent respectively; Inputting the deterministic feature vector into the deterministic feature decoder to obtain the opponent's first action prediction data and first observation prediction data; Inputting the opponent's uncertainty feature vector into the uncertainty feature decoder to obtain the opponent's second action prediction data and second observation prediction data; Inputting the input data into the environmental feature encoder to obtain an environmental feature vector; The environmental feature vector is input into the environmental feature decoder to obtain own state prediction data and second environmental reward data.

2. The adversary modeling method based on uncertainty characteristics according to claim 1, characterized in that: The adversary's uncertainty eigenvector is determined by: Among them, (μ, Σ) is the uncertainty feature vector of the opponent, μ is the mean vector, Σ is the covariance matrix, is the uncertainty feature encoder, o t is the current observation data of one's own side, a t is the current own action data, are the deep neural network parameters of the uncertainty feature encoder.

3. The adversary modeling method based on uncertainty characteristics according to claim 1, characterized in that: The step of inputting the uncertainty feature vector of the opponent into the uncertainty feature decoder to obtain the second action prediction data and the second observation prediction data of the opponent comprises: Obtaining a mean vector and a covariance matrix according to the uncertainty feature vector of the opponent, and reconstructing a joint Gaussian distribution based on the mean vector and the covariance matrix; The joint Gaussian distribution is sampled to obtain the opponent's second action prediction data and second observation prediction data.

4. The adversary modeling method based on uncertainty characteristics according to claim 1, characterized in that: The calculating of the loss function using the current opponent observation data, the current opponent action data, the first environment reward data, the next step own observation data, and the model output includes: fusing the first action prediction data and the second action prediction data of the opponent to obtain action prediction data of the opponent, and fusing the first observation prediction data and the second observation prediction data to obtain observation prediction data of the opponent; Based on the current opponent observation data, the current opponent action data, the first environment reward data and the next own observation data, the opponent's action prediction data and observation prediction data, and the own state prediction data and the second environment reward data, the loss function is calculated by the following formula: Where L is the loss function, t = 1, 2, ..., T, T is the total number of time steps, is the current opponent observation data, is the opponent’s observed prediction data, P is the probability function, is the current opponent action data, Prediction data for the opponent's actions, o t+1 For the next step of our own observation data, is the prediction data of one's own state, r t Reward data for the first environment, Reward data for the second environment.

5. The adversary modeling method based on uncertainty characteristics according to claim 1, characterized in that: Optimizing the opponent modeling model based on the loss function until a preset condition is met to obtain a trained opponent modeling model includes: Obtain an optimizer, and optimize the deep neural network parameters of the deterministic feature encoder, the uncertain feature encoder, the deterministic feature decoder, the uncertain feature decoder, the environmental feature encoder, and the environmental feature decoder based on the loss function and the optimizer until preset conditions are met, thereby obtaining a trained opponent modeling model.

6. An opponent modeling device based on uncertainty characteristics, characterized in that: include: A data acquisition module is used to acquire a pre-established opponent modeling model and interaction data during the interaction between the agent and the environment; wherein the interaction data includes current own observation data, current own action data, current opponent observation data, current opponent action data, first environment reward data, and next self observation data; A feature modeling module is used to input the current own observation data and the current own action data into the opponent modeling model, model the deterministic features and uncertain features of the opponent, and obtain a model output; the deterministic features are relatively stable features, including the opponent's intelligence level and strategy preferences; the uncertain features are difficult to predict features, including the opponent's random behavior and noise interference; the deterministic features are the predictable parts directly related to the opponent's behavior, and the uncertain features reflect the randomness and unpredictability of the opponent's behavior; the model output includes first action prediction data, first observation prediction data, second action prediction data, second observation prediction data, own state prediction data and second environment reward data; the opponent modeling model includes a deterministic feature encoder, an uncertain feature encoder, a deterministic feature decoder, an uncertain feature decoder, an environment feature encoder and an environment feature decoder; the The current own observation data and the current own action data are input into the opponent modeling model to model the deterministic features and uncertainty features of the opponent to obtain a model output, including: taking the current own observation data and the current own action data as input data; inputting the input data into the deterministic feature encoder and the uncertainty feature encoder to obtain the opponent's deterministic feature vector and uncertainty feature vector respectively; inputting the deterministic feature vector into the deterministic feature decoder to obtain the opponent's first action prediction data and first observation prediction data; inputting the opponent's uncertainty feature vector into the uncertainty feature decoder to obtain the opponent's second action prediction data and second observation prediction data; inputting the input data into the environment feature encoder to obtain the environment feature vector; inputting the environment feature vector into the environment feature decoder to obtain the own state prediction data and second environment reward data; a loss calculation module, configured to calculate a loss function using the current opponent observation data, the current opponent action data, the first environment reward data, the next step own observation data, and the model output; A model optimization module, configured to optimize the opponent modeling model based on the loss function until a preset condition is met, thereby obtaining a trained opponent modeling model; The opponent modeling module is used to perform opponent modeling based on the trained opponent modeling model to obtain an opponent modeling result.

7. An electronic device comprising a memory, a processor, and a computer program stored in the memory and running on the processor, characterized in that: When the processor executes the computer program, the opponent modeling method based on uncertainty characteristics as described in any one of claims 1 to 5 is implemented.

8. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the opponent modeling method based on uncertainty features as described in any one of claims 1 to 5 is implemented.

9. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the opponent modeling method based on uncertainty features as described in any one of claims 1 to 5 is implemented.

Images