Method, device, electronic device and medium for constructing target game decision model

Abstract

The present invention discloses a method, device, electronic device and medium for constructing a target game decision model, including: obtaining an offline game data set corresponding to a target intelligent agent, and constructing a virtual environment simulation model corresponding to the target intelligent agent based on the offline game data set; determining the current offline game data corresponding to the current data collection time to determine the situation information to be used corresponding to the target intelligent agent at the current data collection time for the offline game data at each data collection time; inputting the situation information to be used into a pre-constructed virtual environment simulation model to obtain the state information to be determined of the target intelligent agent at the next data collection time; iteratively updating the model parameters of the initial game model corresponding to the target intelligent agent based on the state information to be determined corresponding to at least one data collection time, and obtaining the target game decision model corresponding to the target intelligent agent. The effect of constructing a more accurate game model corresponding to the target intelligent agent is achieved.

Description

Technical Field

[0001] The present invention relates to the field of data processing technology, and in particular to a method, device, electronic equipment and medium for constructing a target game decision model. Background Art

[0002] In cross-domain confrontational game scenarios, there are characteristics such as high environmental complexity, strong confrontational games, high real-time response and incomplete information. Therefore, it is urgent to introduce advanced artificial intelligence technology in confrontational game scenarios based on unmanned system clusters.

[0003] At present, a high-fidelity simulation environment is mostly required to determine the execution strategy of each intelligent agent in the adversarial game process. However, existing modeling technologies make it difficult to construct simulation data that fully matches the actual adversarial game scenario. Secondly, existing technologies require real-time data interaction with the modeling and simulation model to adjust the decision-making model corresponding to the intelligent agent. In this process, a large amount of data is required and the computing resources consumed are high, which makes it difficult to meet the control requirements of the intelligent agent in practical applications.

[0004] In order to solve the above problems, it is necessary to construct a game model applied to the adversarial game scenario to determine the execution strategy corresponding to the intelligent agent based on the game model. Summary of the invention

[0005] The present invention provides a method, device, electronic device and medium for constructing a target game decision model to solve the problem that it is difficult for existing modeling technologies to construct simulation data that fully matches the actual competitive game scenario, and that the existing technology requires real-time data interaction with the modeling simulation model, resulting in a large amount of data and high consumption of computing resources.

[0006] In a first aspect, an embodiment of the present invention provides a method for constructing a target game decision model, comprising:

[0007] Acquire an offline game data set corresponding to a target intelligent agent, and construct a virtual environment simulation model corresponding to the target intelligent agent based on the offline game data set; wherein the offline game data set includes offline game data of the target intelligent agent at at least one data collection time;

[0008] For the offline game data at each data collection moment, determine the current offline game data corresponding to the current data collection moment, and determine the to-be-used situation information corresponding to the target intelligent agent at the current data collection moment; wherein, the current state information corresponding to the target intelligent agent in the to-be-used situation information;

[0009] Inputting the situation information to be used into a pre-built virtual environment simulation model to obtain the state information to be determined of the target agent at the next data collection moment;

[0010] Based on the to-be-determined state information corresponding to at least one data collection moment, the model parameters of the initial game model corresponding to the target intelligent agent are iteratively updated to obtain a target game decision model corresponding to the target intelligent agent.

[0011] In a second aspect, an embodiment of the present invention further provides a device for constructing a target game decision model, comprising:

[0012] A simulation model construction module, used to obtain an offline game data set corresponding to a target intelligent agent, and to construct a virtual environment simulation model corresponding to the target intelligent agent based on the offline game data set; wherein the offline game data set includes offline game data of the target intelligent agent at at least one data collection moment;

[0013] A situation information determination module is used to determine the current offline game data corresponding to the current data collection time for the offline game data at each data collection time, and determine the situation information to be used corresponding to the target intelligent agent at the current data collection time; wherein, the current state information corresponding to the target intelligent agent in the situation information to be used;

[0014] A state information determination module, used to input the situation information to be used into a pre-built virtual environment simulation model to obtain the state information to be determined of the target intelligent agent at the next data collection moment;

[0015] The target game decision model determination module is used to iteratively update the model parameters of the initial game model corresponding to the target intelligent agent based on the to-be-determined state information corresponding to at least one data collection moment, so as to obtain the target game decision model corresponding to the target intelligent agent.

[0016] In a third aspect, an embodiment of the present invention further provides an electronic device, including:

[0017] at least one processor; and

[0018] a memory communicatively connected to the at least one processor; wherein,

[0019] The memory stores a computer program that can be executed by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can execute the method for constructing the target game decision model described in any embodiment of the present invention.

[0020] In a fourth aspect, an embodiment of the present invention further provides a computer-readable storage medium, wherein the computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a processor to implement the method for constructing a target game decision model described in any embodiment of the present invention when executed.

[0021] The technical solution of the embodiment of the present invention obtains an offline game data set corresponding to the target intelligent agent, and constructs a virtual environment simulation model corresponding to the target intelligent agent based on the offline game data set. In order to ensure the high fidelity of the game data corresponding to the target intelligent agent in this technical solution, the real game data of the target intelligent agent in the historical process can be obtained as offline game data, so as to construct a virtual environment simulation model corresponding to the target intelligent agent based on the offline game data, and then construct a target game decision model corresponding to the target intelligent agent based on the virtual environment simulation model and the offline game data. Furthermore, for the offline game data at each data collection moment, the current offline game data corresponding to the current data collection moment is determined to determine the state information to be used corresponding to the target intelligent agent at the current data collection moment; the state information to be used is input into a pre-constructed virtual environment simulation model to obtain the state information to be determined of the target intelligent agent at the next data collection moment. In the present technical solution, after determining the state information corresponding to the target intelligent agent at the current data collection moment, the state information corresponding to the data collection moment and the initial strategy information corresponding to the current data collection moment are input into the pre-constructed virtual environment simulation model, so that the state information of the target intelligent agent at the next data collection moment can be predicted according to the state information of the target intelligent agent at the current data collection moment. Furthermore, based on the to-be-determined state information corresponding to at least one data collection moment, the model parameters of the initial game model corresponding to the target intelligent agent are iteratively updated to obtain a target game decision model corresponding to the target intelligent agent. In practical applications, the reward value corresponding to the to-be-determined state information at each data collection moment is determined based on the reward function deployed in the initial game model, so that the model parameters of the initial game model are updated based on the reward value corresponding to each data collection moment, until the reward value corresponding to the initial game model at each data collection moment tends to be stable, and the output adversarial game strategy corresponding to the target intelligent agent enables the target intelligent agent to have a winning rate greater than a preset winning rate in the adversarial game with the opponent intelligent agent, and the initial game model at this time is used as the target game decision model. The problem that it is difficult for existing modeling technologies to construct simulation data that fully matches the real confrontation game scenario is solved, and the existing technology requires real-time data interaction with the modeling simulation model, resulting in large data volume and high computing resource consumption. In this technical solution, the real game data corresponding to the target intelligent agent is used as offline game data, and a virtual environment simulation model is constructed based on the offline game data, and then a target game decision model is constructed based on the virtual environment simulation model. This method realizes the construction of a more accurate game model corresponding to the target intelligent agent, and then the game model can be used to make decisions for the target intelligent agent in the real confrontation game process, thereby improving the winning rate of the target intelligent agent in the confrontation game.

[0022] It should be understood that the contents described in this section are not intended to identify the key or important features of the embodiments of the present invention, nor are they intended to limit the scope of the present invention. Other features of the present invention will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0024] Figure 1 is a flow chart of a method for constructing a target game decision model provided according to Embodiment 1 of the present invention;

[0025] Figure 2 is a flow chart of a method for constructing a target game decision model provided in accordance with Embodiment 2 of the present invention;

[0026] Figure 3 is a structural schematic diagram of a device for constructing a target game decision model provided according to Embodiment 3 of the present invention;

[0027] Figure 4 It is a structural schematic diagram of an electronic device for implementing a method for constructing a target game decision model according to an embodiment of the present invention. DETAILED DESCRIPTION

[0028] In order to enable those skilled in the art to better understand the scheme of the present invention, the technical scheme in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.

[0029] It should be noted that the terms "first", "second", etc. in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the numbers used in this way can be interchanged where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein.

[0030] Embodiment 1

[0031] Figure 1A flowchart of a method for constructing a target game decision model is provided for the first embodiment of the present invention. This embodiment can be applied to the case where real game data corresponding to a target intelligent agent is obtained as offline game data, and a virtual environment simulation model corresponding to the target intelligent agent is constructed based on the offline game data, and then a target game decision model corresponding to the target intelligent agent is constructed based on the virtual environment simulation model. The method can be executed by a target game decision model construction device, which can be implemented in the form of hardware and / or software, and can be configured in a computing device that can execute the target game decision model construction method.

[0032] like Figure 1 As shown, the method includes:

[0033] S110, obtaining an offline game data set corresponding to the target intelligent agent, and constructing a virtual environment simulation model corresponding to the target intelligent agent based on the offline game data set.

[0034] Among them, the target agent can be understood as an entity with intelligence. In practical applications, the target agent can be an intelligent system with three-dimensional perception, global coordination, accurate judgment, continuous evolution and openness. For example, the target agent can be an airplane with teamwork function. The offline game data set can be understood as an offline data set generated based on the adversarial game data generated by the target agent in the real adversarial game process. In this technical solution, the offline game data set includes the offline game data of the target agent at at least one data collection moment. The so-called offline game data set can be directly retrieved and used when the network or simulation system and other software are not connected. The virtual environment simulation model can be understood as a model for determining the state information corresponding to the target agent at the next data collection moment based on the offline game data at any data collection moment in the offline game data set, and the state information of the target agent represented by the offline game data at the current data collection moment. For example, based on the offline game data of the target intelligent agent at the current data collection moment, it is determined that the target intelligent agent is at the origin. After analyzing the offline game data at the current data collection moment through the virtual environment simulation model, it is determined that the state information corresponding to the target intelligent agent at the next data collection moment is a position 1 meter to the left of the origin.

[0035] This technical solution is mainly aimed at the aerial confrontation game scenario. Taking the target intelligent agent as an airplane as an example, in order to ensure that the target intelligent agent can cooperate with other intelligent agents in the team in the actual confrontation game, so as to improve the winning rate of the intelligent agent team in the aerial confrontation game process, it is necessary to ensure that each intelligent agent in the intelligent agent team reaches the best confrontation game state. Therefore, in practical applications, each intelligent agent needs to be controlled. Taking one of the intelligent agents as the target intelligent agent as an example, the offline game data set corresponding to the target intelligent agent is retrieved from the offline database, and a virtual environment simulation model corresponding to the target intelligent agent is constructed based on the offline data set.

[0036] It should be noted that in the prior art, a high-fidelity simulation environment is required when simulating intelligent agent confrontation game scenarios. However, it is difficult to construct simulation scenarios that are completely equivalent to real scenarios through modeling for most existing confrontation game scenarios, and building simulation models is time-consuming and labor-intensive. Secondly, in the prior art, it is necessary to interact with the simulation model online in real time, and train the confrontation game decision model through continuous trial and error. The scenario scale and search space required are larger, the data volume required is high, and the amount of calculation is large. Therefore, the adaptability to confrontation game scenarios is relatively poor.

[0037] Based on this, this technical solution uses offline game data corresponding to the intelligent agent as simulation data for model training to ensure the authenticity and accuracy of the simulation data. At the same time, in the process of training the adversarial game decision model, when there is no need to interact with the simulation model online in real time, the accuracy of the adversarial decision model can still be guaranteed.

[0038] S120. For the offline game data at each data collection time, determine the current offline game data corresponding to the current data collection time to determine the to-be-used situation information corresponding to the target intelligent agent at the current data collection time.

[0039] Among them, the current state information corresponding to the target intelligent agent in the situation information to be used may include the position information, speed information and current action information corresponding to the target intelligent agent.

[0040] In this technical solution, when the target agent is simulated in a confrontation game, an offline game data set corresponding to the target agent is used, wherein the offline game data set includes offline game data corresponding to at least one data collection moment. It is understandable that in an actual confrontation game, the target agent is in a continuous confrontation process, so it is necessary to accurately adjust and control the state information and action information of the target agent at each moment.

[0041] Specifically, for at least one offline game data corresponding to the target intelligent agent, the to-be-used situation information corresponding to the target intelligent agent at the current data collection moment can be directly determined, and the to-be-used situation information can be processed based on the virtual environment simulation model to obtain the state information corresponding to the target intelligent agent at the next data collection moment. On this basis, by comparing the state information corresponding to each data collection moment output by the virtual environment simulation model with the real state information corresponding to the target intelligent agent, the model accuracy of the virtual environment simulation model can be determined, and the model parameters can be adjusted according to the model accuracy until the state information at each data collection moment output by the virtual environment simulation model reaches the expected result, and the model training is completed, and then the state information corresponding to the intelligent agent at each moment can be determined based on the virtual environment simulation model during the real confrontation game.

[0042] S130, inputting the situation information to be used into the pre-built virtual environment simulation model to obtain the state information to be determined of the target intelligent agent at the next data collection moment.

[0043] The state information to be determined refers to the state information corresponding to the target value agent at the next data collection moment.

[0044] In the present technical solution, the virtual environment can be simulated based on the imitation learning algorithm to obtain a virtual environment simulation model. For example, the imitation learning algorithm can be a behavior cloning algorithm. After obtaining the virtual environment simulation model, the to-be-used situation information corresponding to the target intelligent agent at the current data collection moment is input into the virtual environment simulation model, and the virtual environment model can determine the to-be-determined state information corresponding to the target intelligent agent at the next data collection moment according to the action execution strategy information corresponding to the current data collection moment. Among them, in order to more clearly introduce the present technical solution, the specific determination process of the to-be-determined state information will be described in detail in the following introduction.

[0045] S140, iteratively updating the model parameters of the initial game model corresponding to the target intelligent agent based on the to-be-determined state information corresponding to at least one data collection moment, to obtain a target game decision model corresponding to the target intelligent agent.

[0046] The initial game model can be understood as a game model whose model parameters are randomly generated and not debugged. The initial game model can be trained by a reinforcement learning algorithm, and the model parameters can be neural network setting parameters corresponding to the initial game model.

[0047] In this technical solution, the game strategy of the target intelligent agent in the actual confrontation game includes two parts: one is to determine the state information of the target intelligent agent at the next data collection moment based on the virtual environment simulation model, and the other is to determine the game strategy information of the target intelligent agent at each data collection moment based on the target game decision model. In other words, in this technical solution, it is necessary to combine the virtual environment simulation model with the target game decision model to jointly decide the game strategy corresponding to the target intelligent agent.

[0048] In practical applications, an initial game model is obtained, and the to-be-determined state information corresponding to at least one data collection moment output by the virtual simulation environment model is input into the initial game model, and the reward value of the to-be-executed action information corresponding to the target intelligent agent at each data collection moment is determined based on the reward function in the initial game model, and the model parameters of the initial game model are updated according to the reward value, until the reward value corresponding to the initial game model at at least one data collection moment tends to be stable, and the winning rate of the target intelligent agent in the simulated confrontation game process exceeds the preset winning rate, and the initial game model at this time is used as the target game decision model.

[0049] Optionally, the situation information to be used is input into a pre-built virtual environment simulation model to obtain the state information to be determined of the target intelligent agent at the next data collection moment, including: determining the action information to be executed of the target intelligent agent at the next data collection moment according to the situation information to be used and the initial strategy information corresponding to the current data collection moment; inputting the action information to be executed into the virtual environment simulation model to obtain the state information to be determined of the target intelligent agent at the next data collection moment.

[0050] The initial strategy information includes the action information that the target agent needs to perform at the current moment, such as the target agent performs a detection action after reaching the target position. The information to be executed can be understood as the action information that the target agent needs to perform at the next data collection moment based on the initial strategy information, such as controlling the target agent to fly 1 km to the left to reach the target position and perform a detection action at the target position.

[0051] In practical applications, the offline game data set includes the state information corresponding to the target intelligent agent at at least one data collection moment. When the offline game data set is used as a simulation data set for training the game model, the situation information to be used and the initial strategy information corresponding to each data collection moment can be pre-set, so as to adjust the situation information to be used based on the initial strategy information at the current data collection moment, and obtain the action information to be executed corresponding to the target intelligent agent at the next data collection moment. Furthermore, the virtual environment simulation model in the technical solution can determine the state information to be determined corresponding to the next data collection moment according to the action information to be executed corresponding to the current data collection moment of the target intelligent agent.

[0052] Optionally, based on the to-be-determined state information corresponding to at least one data collection moment, the model parameters of the initial game model corresponding to the target intelligent agent are iteratively updated to obtain the target game decision model corresponding to the target intelligent agent, including: determining the reward value corresponding to the to-be-determined state information at at least one data collection moment based on the reward function pre-deployed in the initial game model; iteratively updating the model parameters of the initial game model according to the reward value corresponding to at least one to-be-determined state information to obtain the target game decision model corresponding to the target intelligent agent.

[0053] Among them, the target game decision model can be understood as a model used to generate the game strategy of the target intelligent agent in the confrontation game with the opponent intelligent agent.

[0054] In practical applications, in order to ensure the accuracy of model training, a reward function can be deployed in the initial game model in advance, and the reward value of the action information to be executed corresponding to each data collection moment can be calculated based on the reward function to obtain the reward value corresponding to the state information to be determined corresponding to each data collection moment. For example, the state information (s, s') corresponding to the target agent at the current data collection moment and the next data collection moment can be determined and analyzed to obtain the reward value r corresponding to the state information to be determined at the current data collection moment. Furthermore, after each reward value is obtained, the accuracy of the state information to be determined output by the initial game model is characterized according to the size of the reward value, and the model parameters of the initial game model are iteratively updated according to the reward value until the reward value corresponding to the initial game model at all data collection moments tends to be stable, and the target game decision model corresponding to the target agent is obtained.

[0055] It should be noted that, in the present technical solution, when training the initial game model, it is also necessary to determine whether the adversarial game strategy corresponding to the target intelligent agent determined by the initial game model can enable the target intelligent agent to win the adversarial game with the other intelligent agent, and the higher the winning rate, the higher the model accuracy of the target game decision model.

[0056] The technical solution of the embodiment of the present invention obtains an offline game data set corresponding to the target intelligent agent, and constructs a virtual environment simulation model corresponding to the target intelligent agent based on the offline game data set. In order to ensure the high fidelity of the game data corresponding to the target intelligent agent in this technical solution, the real game data of the target intelligent agent in the historical process can be obtained as offline game data, so as to construct a virtual environment simulation model corresponding to the target intelligent agent based on the offline game data, and then construct a target game decision model corresponding to the target intelligent agent based on the virtual environment simulation model and the offline game data. Furthermore, for the offline game data at each data collection moment, the current offline game data corresponding to the current data collection moment is determined to determine the state information to be used corresponding to the target intelligent agent at the current data collection moment; the state information to be used is input into a pre-constructed virtual environment simulation model to obtain the state information to be determined of the target intelligent agent at the next data collection moment. In the present technical solution, after determining the state information corresponding to the target intelligent agent at the current data collection moment, the state information corresponding to the data collection moment and the initial strategy information corresponding to the current data collection moment are input into the pre-constructed virtual environment simulation model, so that the state information of the target intelligent agent at the next data collection moment can be predicted according to the state information of the target intelligent agent at the current data collection moment. Furthermore, based on the to-be-determined state information corresponding to at least one data collection moment, the model parameters of the initial game model corresponding to the target intelligent agent are iteratively updated to obtain a target game decision model corresponding to the target intelligent agent. In practical applications, the reward value corresponding to the to-be-determined state information at each data collection moment is determined based on the reward function deployed in the initial game model, so that the model parameters of the initial game model are updated based on the reward value corresponding to each data collection moment, until the reward value corresponding to the initial game model at each data collection moment tends to be stable, and the output adversarial game strategy corresponding to the target intelligent agent enables the target intelligent agent to have a winning rate greater than a preset winning rate in the adversarial game with the opponent intelligent agent, and the initial game model at this time is used as the target game decision model. The problem that it is difficult for existing modeling technologies to construct simulation data that fully matches the real confrontation game scenario is solved, and the existing technology requires real-time data interaction with the modeling simulation model, resulting in large data volume and high computing resource consumption. In this technical solution, the real game data corresponding to the target intelligent agent is used as offline game data, and a virtual environment simulation model is constructed based on the offline game data, and then a target game decision model is constructed based on the virtual environment simulation model. This method realizes the construction of a more accurate game model corresponding to the target intelligent agent, and then the game model can be used to make decisions for the target intelligent agent in the real confrontation game process, thereby improving the winning rate of the target intelligent agent in the confrontation game.

[0057] Embodiment 2

[0058] Figure 2A flowchart of a method for constructing a target game decision model provided in Embodiment 2 of the present invention, optionally, a virtual environment simulation model corresponding to the target intelligent agent based on an offline game data set is refined.

[0059] like Figure 2 As shown, the method includes:

[0060] S210: Obtain an offline game data set and initial strategy information corresponding to the target intelligent agent, and update the offline game data set based on the initial strategy information to obtain a data set to be used.

[0061] In this technical solution, it is necessary to pre-build a virtual environment simulation model corresponding to the target intelligent agent, so as to determine the state information of the target intelligent agent at the next data collection moment based on the virtual environment simulation model and the state information of the target intelligent agent at the current moment.

[0062] The initial strategy information includes the action information to be executed corresponding to the offline game data at each data collection moment. The dataset to be used can be understood as a dataset obtained by updating the offline game data corresponding to the offline game dataset based on the initial strategy information corresponding to each data collection moment.

[0063] S220 , calling an initial environment simulation model, and inputting the data set to be used into the initial environment simulation model to obtain a function to be optimized corresponding to the initial environment simulation model.

[0064] The initial environment simulation model can be understood as an environment simulation model whose model parameters are randomly generated and not set. The function to be optimized refers to the model optimization function deployed in the initial environment simulation model.

[0065] In this technical solution, the function to be optimized can be expressed by the following formula:

[0066] min θ ∑ (s，a，s′)∈D log(P θ (s′|s,a))

[0067] Among them, P θ represents the experience transfer function based on the output of the initial environment simulation model, θ represents the state transfer parameter of the experience transfer function, s represents the state information corresponding to the target agent at the current data collection time, a represents the action information corresponding to the target agent at the current data collection time, and s′ represents the state information corresponding to the target agent at the next data collection time.

[0068] S230, determining a to-be-determined mean and a to-be-determined variance corresponding to the to-be-determined function, and updating model parameters of the initial environment simulation model according to the to-be-determined mean and the to-be-determined variance, to obtain a virtual environment simulation model corresponding to the target intelligent agent.

[0069] In the present technical solution, the to-be-determined mean and to-be-determined variance corresponding to the function to be optimized are determined, including: based on the initial environment simulation model, determining the optimal solution set of the data set to be used in the Markov decision process; determining the Gaussian distribution corresponding to the optimal solution set, and determining the to-be-determined mean and to-be-determined variance corresponding to the optimal solution set based on the Gaussian distribution.

[0070] Specifically, the to-be-determined mean and to-be-determined variance corresponding to the function to be optimized can be determined based on the following formula:

[0071]

[0072] Based on the above formula, for the state (s, a) of the target agent at each data collection moment, the experience transfer function P θ (s′|s, a) is represented by Gaussian distribution N(μ θ (s, a), σ θ (s, a)).

[0073] Among them, P θ (s′|s, a) represents the optimal solution set corresponding to the function to be optimized, μ θ (s, a) represents the mean to be determined, σ θ (s, a) represents the variance to be determined.

[0074] On this basis, according to the variance to be determined and the mean to be determined, the regression problem based on the mean and variance is solved. Specifically, the regression problem corresponding to the function to be optimized can be solved based on the following formula:

[0075] min θ ∑ (s，a，s′)∈D (s′-μ θ (s,a)) 2

[0076] Among them, θ represents the state transfer parameter of the experience transfer function, s represents the state information corresponding to the target agent at the current data collection time, a represents the action information corresponding to the target agent at the current data collection time, s′ represents the state information corresponding to the target agent at the next data collection time, and μ θ (s, a) represents the mean to be determined.

[0077] Optionally, the model parameters of the initial environment simulation model are updated according to the to-be-determined mean and the to-be-determined variance to obtain a virtual environment simulation model corresponding to the target intelligent agent, including: based on the to-be-determined mean and the to-be-determined variance, the parameters of the initial environment model are updated to obtain the simulation model to be optimized; the estimated value and the true value corresponding to the simulation model to be optimized are determined, and the model evaluation of the simulation model to be optimized is performed according to the estimated value and the true value to obtain the virtual environment simulation model corresponding to the target intelligent agent.

[0078] The simulation model to be optimized can be understood as the simulation model iteratively updated during the training of the initial environment simulation model, which is in the intermediate state between the initial environment simulation model and the virtual environment simulation model. The estimated value can be understood as the state information corresponding to the target intelligent agent at each data collection moment based on the output of the simulation model to be optimized, and the true value can be understood as the state information corresponding to the target intelligent agent at each data collection moment generated in advance based on the expert strategy.

[0079] Optionally, a model evaluation is performed on the simulation model to be optimized according to the estimated value and the true value to obtain a virtual environment simulation model corresponding to the target intelligent agent, including: calling a model evaluation function corresponding to the simulation model to be optimized, and determining a model evaluation attribute corresponding to the simulation model to be optimized based on the model evaluation function; when the model evaluation attribute is greater than a preset evaluation attribute, the simulation model to be optimized is used as the virtual environment simulation model corresponding to the target intelligent agent.

[0080] The model evaluation function can be understood as an evaluation function for determining the model training completion of the simulation model to be optimized. The model evaluation attribute can be used to characterize the training completion of the simulation model to be optimized. The higher the model evaluation attribute, the higher the training completion of the simulation model to be optimized. When the model evaluation attribute is greater than the preset evaluation attribute, the simulation model to be optimized at this time can be used as the virtual environment simulation model corresponding to the target intelligent agent. The preset evaluation attribute can be understood as the preset minimum model evaluation attribute corresponding to the simulation model to be optimized.

[0081] In this technical solution, the model evaluation function can be expressed by the following formula:

[0082]

[0083] in, Indicates the dispersion between the estimated value and the true value, (P θ |s, a) represents the estimated value, Represents the true value.

[0084] It should be noted that the smaller the KL discreteness between the estimated value and the true value, the better, indicating that the prediction ability of the simulation model to be optimized for the target agent at the next data collection moment is closer to the actual situation.

[0085] S240. For the offline game data at each data collection moment, determine the current offline game data corresponding to the current data collection moment and determine the to-be-used situation information corresponding to the target intelligent agent at the current data collection moment.

[0086] S250: Input the situation information to be used into the pre-built virtual environment simulation model to obtain the state information to be determined of the target intelligent agent at the next data collection moment.

[0087] S260. Iteratively update the model parameters of the initial game model corresponding to the target intelligent agent based on the to-be-determined state information corresponding to at least one data collection moment, to obtain a target game decision model corresponding to the target intelligent agent.

[0088] Based on the above examples, in a specific example, taking the air confrontation game scenario as an example, the aircraft can be regarded as an intelligent agent, and the intelligent agent corresponding to one of the aircraft is taken as the target intelligent agent. The maneuvers, detections, and strikes performed by the target intelligent agent can be regarded as the state information of the target intelligent agent. The environment is a virtual environment model trained based on the offline game data corresponding to the target intelligent agent, and the indicators such as advantage, combat damage, and energy consumption used to evaluate the results can be regarded as the reward returned to the target intelligent agent by the target game decision model through the reward function. Exemplarily, the specific reward includes the individual intelligent agent benefits and team intelligent agent benefits of the target intelligent agent. For example, the reward value r specifically includes: when a single intelligent agent is damaged, the reward value is negative; a larger positive reward is given for shooting down the opponent's intelligent agent; a smaller reward value is set based on the azimuth angle difference and relative distance between our intelligent agent and the opponent's intelligent agent.

[0089] In the target game decision model, the transfer of the state information of the target agent at different data collection moments depends on the behavior combination U of all agents. k =U 1k , U 2k , U 3k ,......,U nk , U k ∈U. In this technical solution, a different-strategy reinforcement learning algorithm is used to solve the adversarial game problem based on offline game data. Each agent makes independent decisions, and the collaboration between agents is completed through information sharing. Each agent uses the incomplete situation information obtained to interact with the environment, optimizes its own goals, and finally achieves local or global optimization. All agents have the same goal, cooperate to complete the task and avoid repeated scheduling of resources to maximize common benefits.

[0090] Based on this, the reward expectation of the target agent can be determined based on the following formula:

[0091]

[0092] in, represents the reward expectation corresponding to the target agent, h represents the strategy information combination corresponding to the target agent at at least one data collection moment, and r k represents the reward factor corresponding to the target agent at the current data collection moment, r i，k+1 Represents the reward factor corresponding to the target agent at the next data collection moment.

[0093] Based on the above examples, in the actual application of the reinforcement learning algorithm, this scheme will separately learn the virtual environment model, and use the offline data set to train a virtual environment simulation model, and combine the virtual environment simulation model based on the offline game data set, and construct the corresponding target game decision model based only on the offline game data corresponding to the target intelligent agent.

[0094] The technical solution of the embodiment of the present invention is to obtain an offline game data set corresponding to the target intelligent agent, and construct a virtual environment simulation model corresponding to the target intelligent agent based on the offline game data set; determine the current offline game data corresponding to the current data collection moment to determine the to-be-used situation information of the target intelligent agent at the current data collection moment for the offline game data at each data collection moment; input the to-be-used situation information into a pre-constructed virtual environment simulation model to obtain the to-be-determined state information of the target intelligent agent at the next data collection moment; iteratively update the model parameters of the initial game model corresponding to the target intelligent agent based on the to-be-determined state information corresponding to at least one data collection moment to obtain the target game decision model corresponding to the target intelligent agent. The problem that it is difficult for existing modeling technologies to construct simulation data that fully matches the real confrontation game scenario is solved, and the existing technology requires real-time data interaction with the modeling simulation model, resulting in large data volume and high computing resource consumption. In this technical solution, the real game data corresponding to the target intelligent agent is used as offline game data, and a virtual environment simulation model is constructed based on the offline game data, and then a target game decision model is constructed based on the virtual environment simulation model. This method realizes the construction of a more accurate game model corresponding to the target intelligent agent, and then the game model can be used to make decisions for the target intelligent agent in the real confrontation game process, thereby improving the winning rate of the target intelligent agent in the confrontation game.

[0095] Embodiment 3

[0096] Figure 3 This is a schematic diagram of the structure of a device for constructing a target game decision model provided by Embodiment 3 of the present invention. Figure 3 As shown, the device includes: a simulation model construction module 310, a situation information determination module 320, a state information determination module 330 and a target game decision model determination module 340.

[0097] The simulation model construction module 310 is used to obtain an offline game data set corresponding to the target intelligent agent, and to construct a virtual environment simulation model corresponding to the target intelligent agent based on the offline game data set; wherein the offline game data set includes offline game data of the target intelligent agent at at least one data collection time;

[0098] The situation information determination module 320 is used to determine the current offline game data corresponding to the current data collection time and determine the situation information to be used corresponding to the target intelligent agent at the current data collection time for the offline game data at each data collection time; wherein, the current state information corresponding to the target intelligent agent in the situation information to be used;

[0099] The state information determination module 330 is used to input the situation information to be used into the pre-built virtual environment simulation model to obtain the state information to be determined of the target intelligent agent at the next data collection moment;

[0100] The target game decision model determination module 340 is used to iteratively update the model parameters of the initial game model corresponding to the target intelligent agent based on the to-be-determined state information corresponding to at least one data collection moment, so as to obtain the target game decision model corresponding to the target intelligent agent.

[0101] The technical solution of the embodiment of the present invention is to obtain an offline game data set corresponding to the target intelligent agent, and construct a virtual environment simulation model corresponding to the target intelligent agent based on the offline game data set; determine the current offline game data corresponding to the current data collection moment to determine the to-be-used situation information of the target intelligent agent at the current data collection moment for the offline game data at each data collection moment; input the to-be-used situation information into a pre-constructed virtual environment simulation model to obtain the to-be-determined state information of the target intelligent agent at the next data collection moment; iteratively update the model parameters of the initial game model corresponding to the target intelligent agent based on the to-be-determined state information corresponding to at least one data collection moment to obtain the target game decision model corresponding to the target intelligent agent. The problem that it is difficult for existing modeling technologies to construct simulation data that fully matches the real confrontation game scenario is solved, and the existing technology requires real-time data interaction with the modeling simulation model, resulting in large data volume and high computing resource consumption. In this technical solution, the real game data corresponding to the target intelligent agent is used as offline game data, and a virtual environment simulation model is constructed based on the offline game data, and then a target game decision model is constructed based on the virtual environment simulation model. This method realizes the construction of a more accurate game model corresponding to the target intelligent agent, and then the game model can be used to make decisions for the target intelligent agent in the real confrontation game process, thereby improving the winning rate of the target intelligent agent in the confrontation game.

[0102] Optionally, the simulation model construction module includes: a submodule for determining a data set to be used, which is used to obtain an offline game data set and initial strategy information corresponding to the target intelligent agent, and update the offline game data set based on the initial strategy information to obtain the data set to be used; wherein the initial strategy information includes information on actions to be executed corresponding to the offline game data at each data collection moment;

[0103] The submodule for determining the function to be optimized is used to call the initial environment simulation model and input the data set to be used into the initial environment simulation model to obtain the function to be optimized corresponding to the initial environment simulation model;

[0104] The simulation environment model determination submodule is used to determine the to-be-determined mean and to-be-determined variance corresponding to the function to be optimized, and to update the model parameters of the initial environment simulation model according to the to-be-determined mean and to-be-determined variance to obtain a virtual environment simulation model corresponding to the target intelligent agent.

[0105] Optionally, the simulation environment model determination submodule includes: an optimal solution set determination unit, used to determine the optimal solution set of the data set to be used in the Markov decision process based on the initial environment simulation model;

[0106] The mean and variance determination unit is used to determine the Gaussian distribution corresponding to the optimal solution set, and determine the to-be-determined mean and to-be-determined variance corresponding to the optimal solution set according to the Gaussian distribution.

[0107] Optionally, the simulation environment model determination submodule includes: a simulation model to be optimized determination unit, used to update the parameters of the initial environment model based on the to-be-determined mean and the to-be-determined variance to obtain the simulation model to be optimized;

[0108] The simulation environment model determination unit is used to determine the estimated value and the true value corresponding to the simulation model to be optimized, and to perform model evaluation on the simulation model to be optimized based on the estimated value and the true value to obtain a virtual environment simulation model corresponding to the target intelligent agent.

[0109] Optionally, the simulation environment model determination unit includes: a model evaluation attribute determination subunit, which is used to call a model evaluation function corresponding to the simulation model to be optimized, and determine a model evaluation attribute corresponding to the simulation model to be optimized based on the model evaluation function;

[0110] The simulation environment model determination subunit is used to use the simulation model to be optimized as the virtual environment simulation model corresponding to the target intelligent agent when the model evaluation attribute is greater than the preset evaluation attribute.

[0111] Optionally, the state information determination module includes: an action information determination submodule, which is used to determine the action information to be executed by the target agent at the next data collection moment according to the to-be-used situation information and the initial strategy information corresponding to the current data collection moment;

[0112] The state information determination submodule is used to input the action information to be executed into the virtual environment simulation model to obtain the state information to be determined of the target intelligent agent at the next data collection moment.

[0113] Optionally, the target game decision model determination module includes: a reward value determination submodule, which is used to determine the reward value corresponding to the state information to be determined at least one data collection moment based on the reward function pre-deployed in the initial game model;

[0114] The target game decision model determination submodule is used to iteratively update the model parameters of the initial game model according to the reward value corresponding to at least one state information to be determined, so as to obtain the target game decision model corresponding to the target intelligent agent.

[0115] The target game decision model construction device provided in the embodiment of the present invention can execute the target game decision model construction method provided in any embodiment of the present invention, and has the corresponding functional modules and beneficial effects of the execution method.

[0116] Embodiment 4

[0117] Figure 4A schematic diagram of the structure of an electronic device 10 of an embodiment of the present invention is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices (such as helmets, glasses, watches, etc.) and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present invention described and / or required herein.

[0118] like Figure 4 As shown, the electronic device 10 includes at least one processor 11, and a memory connected to the at least one processor 11, such as a read-only memory (ROM) 12, a random access memory (RAM) 13, etc., wherein the memory stores a computer program that can be executed by at least one processor, and the processor 11 can perform various appropriate actions and processes according to the computer program stored in the read-only memory (ROM) 12 or the computer program loaded from the storage unit 18 to the random access memory (RAM) 13. In the RAM 13, various programs and data required for the operation of the electronic device 10 can also be stored. The processor 11, the ROM 12, and the RAM 13 are connected to each other through a bus 14. An input / output (I / O) interface 15 is also connected to the bus 14.

[0119] A number of components in the electronic device 10 are connected to the I / O interface 15, including: an input unit 16, such as a keyboard, a mouse, etc.; an output unit 17, such as various types of displays, speakers, etc.; a storage unit 18, such as a disk, an optical disk, etc.; and a communication unit 19, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 19 allows the electronic device 10 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0120] The processor 11 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the processor 11 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, a digital signal processor (DSP), and any appropriate processor, controller, microcontroller, etc. The processor 11 executes the various methods and processes described above, such as a method for constructing a target game decision model.

[0121] In some embodiments, the method for constructing the target game decision model may be implemented as a computer program, which is tangibly contained in a computer-readable storage medium, such as a storage unit 18. In some embodiments, part or all of the computer program may be loaded and / or installed on the electronic device 10 via the ROM 12 and / or the communication unit 19. When the computer program is loaded into the RAM 13 and executed by the processor 11, one or more steps of the method for constructing the target game decision model described above may be performed. Alternatively, in other embodiments, the processor 11 may be configured to execute the method for constructing the target game decision model in any other appropriate manner (e.g., by means of firmware).

[0122] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), load programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0123] The computer program for implementing the method for constructing the target game decision model of the present invention can be written in any combination of one or more programming languages. These computer programs can be provided to a processor of a general-purpose computer, a special-purpose computer or other programmable data processing device, so that when the computer program is executed by the processor, the functions / operations specified in the flow chart and / or the block diagram are implemented. The computer program can be executed entirely on the machine, partially on the machine, partially on the machine as an independent software package and partially on a remote machine, or completely on a remote machine or server.

[0124] In the context of the present invention, a computer-readable storage medium may be a tangible medium that may contain or store a computer program for use by or in combination with an instruction execution system, device or equipment. A computer-readable storage medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices or equipment, or any suitable combination of the foregoing. Alternatively, a computer-readable storage medium may be a machine-readable signal medium. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0125] To provide interaction with a user, the systems and techniques described herein may be implemented on an electronic device having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and a pointing device (e.g., a mouse or trackball) through which the user can provide input to the electronic device. Other types of devices may also be used to provide interaction with the user; for example, the feedback provided to the user may be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user may be received in any form (including acoustic input, voice input, or tactile input).

[0126] The systems and techniques described herein may be implemented in a computing system that includes backend components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes frontend components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such backend components, middleware components, or frontend components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), a blockchain network, and the Internet.

[0127] A computing system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The client and server relationship is generated by computer programs running on the corresponding computers and having a client-server relationship with each other. The server may be a cloud server, also known as a cloud computing server or cloud host, which is a host product in the cloud computing service system to solve the defects of difficult management and weak business scalability in traditional physical hosts and VPS services.

[0128] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps described in the present invention can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solution of the present invention can be achieved, and this document does not limit this.

[0129] The above specific implementations do not constitute a limitation on the protection scope of the present invention. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. A method for constructing a target game decision model, characterized in that: include: Applied to the aerial confrontation game scenario, the aircraft is used as an intelligent agent, and the intelligent agent corresponding to one of the aircraft is used as the target intelligent agent. The maneuvers, detections, and strikes performed by the target intelligent agent are the state information of the target intelligent agent. Acquire an offline game data set corresponding to a target intelligent agent, and construct a virtual environment simulation model corresponding to the target intelligent agent based on the offline game data set; wherein the offline game data set includes offline game data of the target intelligent agent at at least one data collection time; For the offline game data at each data collection moment, determine the current offline game data corresponding to the current data collection moment, and determine the to-be-used situation information corresponding to the target intelligent agent at the current data collection moment; wherein, the current state information corresponding to the target intelligent agent in the to-be-used situation information; Inputting the situation information to be used into a pre-built virtual environment simulation model to obtain the state information to be determined of the target agent at the next data collection moment; Iteratively updating the model parameters of the initial game model corresponding to the target intelligent agent based on the to-be-determined state information corresponding to at least one data collection moment, to obtain a target game decision model corresponding to the target intelligent agent; The step of constructing a virtual environment simulation model corresponding to the target intelligent agent based on the offline game data set includes: Acquire an offline game data set and initial strategy information corresponding to the target agent, and update the offline game data set based on the initial strategy information to obtain a data set to be used; wherein the initial strategy information includes information on actions to be executed corresponding to the offline game data at each data collection moment; Retrieving an initial environment simulation model, and inputting the data set to be used into the initial environment simulation model, to obtain a function to be optimized corresponding to the initial environment simulation model; wherein the initial environment simulation model is an environment simulation model whose model parameters are randomly generated and not set, and the function to be optimized is a model optimization function deployed in the initial environment simulation model; Determine the to-be-determined mean and the to-be-determined variance corresponding to the to-be-determined function, and update the model parameters of the initial environment simulation model according to the to-be-determined mean and the to-be-determined variance to obtain a virtual environment simulation model corresponding to the target intelligent agent.

2. The method according to claim 1, characterized in that: The determining the to-be-determined mean and the to-be-determined variance corresponding to the to-be-optimized function includes: Based on the initial environment simulation model, determining an optimal solution set of the data set to be used in a Markov decision process; A Gaussian distribution corresponding to the optimal solution set is determined, and a to-be-determined mean and a to-be-determined variance corresponding to the optimal solution set are determined according to the Gaussian distribution.

3. The method according to claim 1, characterized in that The updating of the model parameters of the initial environment simulation model according to the to-be-determined mean and the to-be-determined variance to obtain a virtual environment simulation model corresponding to the target intelligent agent includes: Based on the to-be-determined mean and the to-be-determined variance, updating the parameters of the initial environment simulation model to obtain a simulation model to be optimized; Determine an estimated value and a true value corresponding to the simulation model to be optimized, and perform model evaluation on the simulation model to be optimized according to the estimated value and the true value to obtain a virtual environment simulation model corresponding to the target intelligent agent.

4. The method according to claim 3, characterized in that: The step of performing a model evaluation on the simulation model to be optimized according to the estimated value and the true value to obtain a virtual environment simulation model corresponding to the target intelligent agent includes: Retrieving a model evaluation function corresponding to the simulation model to be optimized, and determining a model evaluation attribute corresponding to the simulation model to be optimized based on the model evaluation function; When the model evaluation attribute is greater than a preset evaluation attribute, the simulation model to be optimized is used as a virtual environment simulation model corresponding to the target intelligent agent.

5. The method according to claim 1, characterized in that The step of inputting the situation information to be used into a pre-built virtual environment simulation model to obtain the state information to be determined of the target agent at the next data collection moment includes: Determining the action information to be executed by the target agent at the next data collection moment according to the to-be-used situation information and the initial strategy information corresponding to the current data collection moment; The to-be-executed action information is input into the virtual environment simulation model to obtain the to-be-determined state information of the target intelligent agent at the next data collection moment.

6. The method according to claim 1, characterized in that The iterative updating of the model parameters of the initial game model corresponding to the target agent based on the to-be-determined state information corresponding to at least one data collection moment to obtain the target game decision model corresponding to the target agent includes: Determine, based on the reward function pre-deployed in the initial game model, a reward value corresponding to the state information to be determined at at least one data collection moment; The model parameters of the initial game model are iteratively updated according to the reward value corresponding to at least one state information to be determined, so as to obtain a target game decision model corresponding to the target intelligent agent.

7. A device for constructing a target game decision model, characterized in that: include: Applied to the aerial confrontation game scenario, the aircraft is used as an intelligent agent, and the intelligent agent corresponding to one of the aircraft is used as the target intelligent agent. The maneuvers, detections, and strikes performed by the target intelligent agent are the state information of the target intelligent agent. A simulation model construction module, used to obtain an offline game data set corresponding to a target intelligent agent, and to construct a virtual environment simulation model corresponding to the target intelligent agent based on the offline game data set; wherein the offline game data set includes offline game data of the target intelligent agent at at least one data collection moment; A situation information determination module is used to determine the current offline game data corresponding to the current data collection time for the offline game data at each data collection time, and determine the situation information to be used corresponding to the target intelligent agent at the current data collection time; wherein, the current state information corresponding to the target intelligent agent in the situation information to be used; A state information determination module, used to input the situation information to be used into a pre-built virtual environment simulation model to obtain the state information to be determined of the target intelligent agent at the next data collection moment; A target game decision model determination module is used to iteratively update the model parameters of the initial game model corresponding to the target intelligent agent based on the state information to be determined corresponding to at least one data collection moment, so as to obtain a target game decision model corresponding to the target intelligent agent; The simulation model building module includes: The submodule for determining the data set to be used is used to obtain the offline game data set and initial strategy information corresponding to the target agent, and update the offline game data set based on the initial strategy information to obtain the data set to be used; wherein the initial strategy information includes the action information to be executed corresponding to the offline game data at each data collection moment; The function to be optimized determination submodule is used to call the initial environment simulation model, and input the data set to be used into the initial environment simulation model to obtain the function to be optimized corresponding to the initial environment simulation model; wherein the initial environment simulation model is an environment simulation model whose model parameters are randomly generated and not set, and the function to be optimized is a model optimization function deployed in the initial environment simulation model; The simulation environment model determination submodule is used to determine the to-be-determined mean and the to-be-determined variance corresponding to the function to be optimized, and to update the model parameters of the initial environment simulation model according to the to-be-determined mean and the to-be-determined variance to obtain a virtual environment simulation model corresponding to the target intelligent agent.

8. An electronic device, characterized in that: The electronic device comprises: at least one processor; and a memory communicatively connected to the at least one processor; wherein, The memory stores a computer program executable by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can execute the method for constructing a target game decision model described in any one of claims 1-6.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a processor to implement the method for constructing a target game decision model according to any one of claims 1 to 6 when executed.

Images