A decision-making assistance method and system based on AI big model
By using auxiliary decision-making methods based on AI large models in virtual military training, dynamically adjusting the training and resource utilization of agents, the problem of waste of server processing capabilities is solved, and efficient confrontation training and resource optimization of agents is achieved.
Patent Information
- Application Number
- CN202411171786.3
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-26
- Publication Date
- 2024-12-03
- Estimated Expiration
- 2044-08-26
AI Technical Summary
In virtual military training, the data processing demand of the agent increases dramatically during the period of high intensity of confrontation, resulting in waste of server processing capabilities. It is difficult for the existing technology to effectively allocate data processing tasks for game operation and agent training, resulting in waste of resources.
The auxiliary decision-making method based on AI large model is adopted to dynamically adjust the training startup time and experience of the agent through neural networks, and combine the DQN model and target network to optimize the server resource utilization rate and reduce the peak of server task processing.
It effectively reduces the peak of server processing task resource requirements, while ensuring that the agent maintains a certain degree of confrontation intensity during the game, avoiding resource waste.
Smart Images

Figure CN118662913B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of reinforcement learning technology, and more specifically, to an auxiliary decision-making method and system based on an AI big model. Background Art
[0002] In military training in a virtual environment, an intelligent agent trained by reinforcement learning is used to assist the commander in training. Reinforcement learning needs to optimize the intelligent agent by continuously iterating and updating the parameters of the intelligent agent. In order for the intelligent agent to quickly adapt to the commander's strategy changes and continue to innovate, the intelligent agent needs to be learned and adjusted online. During the commander training process, the intelligent agent needs to reason and learn at the same time. The training of the intelligent agent and the operation of the game both rely on the network bandwidth and processing power of the server. The intensity of confrontation in the military training process is different. The intelligent agent will start a round of training after receiving a set number of behavioral rewards. In the time period with higher confrontation intensity, the frequency of environmental feedback to the intelligent agent's behavior is higher, so the frequency of intelligent agent updates is higher. At the same time, the virtual environment of military training requires more data processing, which generates a sudden increase in data processing during this period. If the processing capacity of the server is redundantly configured to ensure the maximum data processing capacity, the processing capacity of the server will be wasted in other time periods. It is necessary to reasonably allocate the game operation data processing tasks that fluctuate with the game process and the data processing tasks of the intelligent agent training to reduce the demand for server processing capacity. Summary of the invention
[0003] The present invention provides an auxiliary decision-making method and system based on an AI large model, which adds a neural network to the online training of the intelligent agent, so that the intelligent agent can dynamically adjust the training start time and the number of selected experiences through the neural network, so that the intelligent agent can more intelligently reduce the task processing peak of the server, while also ensuring a certain confrontation strength of the intelligent agent, and solving the problem that the processing capacity of the server is wasted in other time periods if the processing capacity of the redundantly configured server is used to ensure the maximum data processing capacity.
[0004] The present invention provides an auxiliary decision-making method based on an AI big model, comprising the following steps:
[0005] Step 100, collecting training data and encoding them into observation vectors;
[0006] The training data includes current weather data, combat training map data, enemy information, own information, and server information;
[0007] Weather data: temperature, humidity, visibility;
[0008] Combat training map data: topographic map, combat area division map, position marking map;
[0009] Enemy information: enemy quantity, movement speed, location, and equipment;
[0010] Your own information: your own soldier type, position, number, and equipment;
[0011] Server information: server resource utilization;
[0012] Server resource utilization is the weighted sum of CPU utilization, memory utilization, disk I / O utilization, and network bandwidth utilization.
[0013] Step 200: The observation vector currently acquired Input into the DQN (Deep Q Network) model, output the Q values corresponding to all first actions in the first action space, use the first action selection strategy to select a first action from all first actions, decode and execute;
[0014] In one embodiment of the present invention, a first action in the first action space is expressed as:
[0015] ;
[0016] in, , and They represent the target battlefield areas to which the 1st, 2nd, and nth combat units need to move respectively;
[0017] Combat behavior , and They represent the combat operations that the 1st, 2nd, and nth combat units need to perform respectively;
[0018] In one embodiment of the present invention, the battlefield area is an area divided by a training battlefield, and combat operations include marching, retreating, attacking, and position defense.
[0019] In one embodiment of the present invention, the first action selection strategy includes:
[0020] according to -The greedy strategy chooses the first action:
[0021] set up The initial value is 0.1;
[0022] Generate a random number r∈[0,1], if r< , then randomly select a first action, otherwise, select the first action with the largest Q value.
[0023] Step 300, define time t as the current time, and the first action taken at time t is , execute the first action The observation vector collected at time t+1 is , execute the first action Rewards , combined into a high-dimensional vector exp and stored in the experience replay pool;
[0024] , represents the observation vector collected at time t;
[0025] The calculation formula is:
[0026]
[0027] in , , , Represents the first, second, third, and fourth reward values respectively. The initial value is 0. Execute the first action back The cumulative number of enemy soldiers killed. Add up the number of your own soldiers killed, Add up the number of positions occupied by your side. The number of positions lost by the enemy is accumulated. Represents the total number of enemy soldiers, Indicates the total number of soldiers on one's side. represents the total number of enemy positions at the beginning of training, Indicates the total number of friendly positions at the beginning of training. , , , They represent the first, second, third, and fourth tendency coefficients respectively. The sum of the four tendency coefficients is 1. The default values are 0.2, 0.2, 0.3, and 0.3 respectively.
[0028] Step 400, input the observation vector from time tn to time t into the trained first neural network, output the time when the training starts and the number of experiences collected during the training, and when the time when the training starts is reached, collect the corresponding number of experiences from the experience replay pool to train the DQN model.
[0029] As a further optimization solution of the present invention, the DQN model includes a Q network and Network, Q Network and The network is the same;
[0030] In one embodiment of the present invention, the calculation formula of Q network is as follows:
[0031]
[0032]
[0033] Where Z represents the latent vector, and Both represent trainable weight parameters, and Both represent trainable bias parameters, represents the first output vector, the value of the i-th component of the first output vector represents the Q value of the i-th first action, Represents the Sigmoid function.
[0034] In one embodiment of the present invention, the DQN model adopts a recurrent neural network.
[0035] In one embodiment of the present invention, the training steps of the DQN model include:
[0036] Step 101, randomly obtain an experience from the experience pool;
[0037]
[0038] express The maximum Q value output by the target network;
[0039] according to and Calculate the loss value loss and update the Q network;
[0040]
[0041] Indicates status and the first action The output Q value of the next Q network is, Indicates status and the first action The output Q value of the target network is:
[0042] is the discount factor, which is a value between 0 and 1, used to balance the weights of immediate rewards and future rewards. The default value is 0.6;
[0043] Step 102, updating the target network every fixed number of network updates so that its parameters are the same as those of the current Q network;
[0044] The default value for the fixed network update interval is 10.
[0045] Step 103, until the number of collected experiences equals the number of experiences collected in this training, the step is terminated.
[0046] In one embodiment of the present invention, the calculation formula of the first neural network is as follows:
[0047]
[0048]
[0049]
[0050]
[0051]
[0052]
[0053] in, and They represent the Lth observation vector of the input (the observation vector at time tn is the first observation vector), n≥L≥1, , , and Both represent trainable weight parameters, , , and Both represent trainable bias parameters, represents the dot product, Indicates The first intermediate feature, Indicates The second intermediate feature, Indicates A third intermediate feature, and Respectively represent the L-1th and output features, represents the second output vector, the value of the i-th component of the second output vector represents the Q value of the i-th second action, tanh represents the tanh function, Represents the Sigmoid function.
[0054] The beneficial effects of the present invention are:
[0055] The present invention provides an auxiliary decision-making method based on an AI large model and an online training mechanism for an intelligent agent in the system, which can effectively reduce the peak value of the server's task processing resource requirements while ensuring that the intelligent agent can maintain a certain confrontation strength during the game. BRIEF DESCRIPTION OF THE DRAWINGS
[0056] Figure 1It is a flowchart of the auxiliary decision-making method based on the AI big model and the online training process of the intelligent agent in the system of the present invention; DETAILED DESCRIPTION
[0057] The subject matter described herein will now be discussed with reference to example implementations. It should be understood that the discussion of these implementations is only to enable those skilled in the art to better understand and implement the subject matter described herein, and the functions and arrangements of the elements discussed may be changed without departing from the scope of protection of the present specification. Various examples may omit, replace, or add various processes or components as needed. In addition, the features described in some examples may also be combined in other examples.
[0058] At least one embodiment of the present invention discloses an auxiliary decision-making method based on an AI big model, such as Figure 1 As shown, the following steps are included:
[0059] Step 100, collecting training data and encoding them into observation vectors;
[0060] The training data includes current weather data, combat training map data, enemy information, own information, and server information;
[0061] Weather data: temperature, humidity, visibility;
[0062] Combat training map data: topographic map, combat area division map, position marking map;
[0063] Enemy information: enemy quantity, movement speed, location, and equipment;
[0064] Your own information: your own soldier types (sniper, demolition, assault, machine gun, decryption, medical), location, number, and equipment;
[0065] Server information: server resource utilization;
[0066] Server resource utilization is the weighted sum of CPU utilization, memory utilization, disk I / O utilization, and network bandwidth utilization.
[0067] The formula for calculating server resource utilization is:
[0068]
[0069]
[0070] is the resource utilization of the server at time t, is the CPU utilization at time t, is the memory utilization at time t, is the disk I / O utilization at time t, is the network bandwidth utilization at time t, is the weight of CPU utilization, is the weight of memory utilization, is the weight of disk I / O utilization, is the weight of network bandwidth utilization.
[0071] As an example, , , , , , , , .
[0072] Step 200: The observation vector currently acquired Input into the DQN (Deep Q Network) model, output the Q values corresponding to all first actions in the first action space, use the first action selection strategy to select a first action from all first actions, decode and execute;
[0073] In one embodiment of the present invention, a first action in the first action space is expressed as:
[0074] ;
[0075] in, , and They represent the target battlefield areas to which the 1st, 2nd, and nth combat units need to move respectively;
[0076] Combat behavior , and They represent the combat operations that the 1st, 2nd, and nth combat units need to perform respectively;
[0077] In one embodiment of the present invention, the battlefield area is an area divided by a training battlefield, and combat operations include marching, retreating, attacking, and position defense.
[0078] In one embodiment of the present invention, the first action selection strategy includes:
[0079] according to -The greedy strategy chooses the first action:
[0080] set up The initial value is 0.1;
[0081] Generate a random number r∈[0,1], if r< , then randomly select a first action, otherwise, select the first action with the largest Q value.
[0082] Step 300, define time t as the current time, and the first action taken at time t is , execute the first action The observation vector collected at time t+1 is , execute the first action Rewards , combined into a high-dimensional vector exp and stored in the experience replay pool;
[0083] , represents the observation vector collected at time t;
[0084] The calculation formula is:
[0085]
[0086] in , , , Represents the first, second, third, and fourth reward values respectively. The initial value is 0. Execute the first action back The cumulative number of enemy soldiers killed. Add up the number of your own soldiers killed, The number of positions occupied by your side is accumulated. The number of positions lost by the enemy is accumulated. Represents the total number of enemy soldiers, Indicates the total number of soldiers on one's side. represents the total number of enemy positions at the beginning of training, Indicates the total number of friendly positions at the beginning of training. , , , They represent the first, second, third, and fourth tendency coefficients respectively. The sum of the four tendency coefficients is 1. The default values are 0.2, 0.2, 0.3, and 0.3 respectively.
[0087] Step 400, input the observation vector from time tn to time t into the trained first neural network, output the time when the training starts and the number of experiences collected during the training, and when the time when the training starts is reached, collect the corresponding number of experiences from the experience replay pool to train the DQN model.
[0088] As a further optimization solution of the present invention, the DQN model includes a Q network and a target network, and the Q network is the same as the target network;
[0089] In one embodiment of the present invention, the calculation formula of the DQN model is as follows:
[0090]
[0091]
[0092] Where Z represents the latent vector, and Both represent trainable weight parameters, and Both represent trainable bias parameters, represents the first output vector, the value of the i-th component of the first output vector represents the Q value of the i-th first action, Represents the Sigmoid function.
[0093] In one embodiment of the present invention, the DQN model adopts a recurrent neural network.
[0094] In one embodiment of the present invention, the training steps of the DQN model include:
[0095] Step 101, randomly obtain an experience from the experience pool;
[0096]
[0097] express The maximum Q value of the target network output is the termination, which means the end of the virtual military training;
[0098] according to and Calculate the loss value loss and update the Q network;
[0099]
[0100] Indicates status and the first action The output Q value of the next Q network is, Indicates status and the first action The output Q value of the target network is:
[0101] is the discount factor, which is a value between 0 and 1, used to balance the weights of immediate rewards and future rewards. The default value is 0.6;
[0102] Step 102, updating the target network every fixed number of network updates so that its parameters are the same as those of the current Q network;
[0103] The default value for the fixed network update interval is 10.
[0104] Step 103, until the number of collected experiences equals the number of experiences collected in this training, the step is terminated.
[0105] In one embodiment of the present invention, the calculation formula of the first neural network is as follows:
[0106]
[0107]
[0108]
[0109]
[0110]
[0111]
[0112] in, and They represent the Lth observation vector of the input (the observation vector at time tn is the first observation vector), n≥L≥1, , , and Both represent trainable weight parameters, , , and Both represent trainable bias parameters, represents the dot product, Indicates The first intermediate feature, Indicates The second intermediate feature, Indicates A third intermediate feature, and Respectively represent the L-1th and output features, represents the second output vector, the value of the i-th component of the second output vector represents the Q value of the i-th second action, tanh represents the tanh function, Represents the Sigmoid function.
[0113] In one embodiment of the present invention, the second action belongs to the second action space. The method for generating the second action in the second action space is as follows:
[0114] The range from 0% to 100% is discretized into multiple point values, which are then combined with an optional number of experience points collected for training.
[0115] For example, a second action is expressed as: ;
[0116] in, The ratio of the time difference between the start of training and time t to the time difference between t and t+1 is ; 10 represents the number of experiences collected during training.
[0117] In one embodiment of the present invention, the first neural network is trained using a Q learning method, and the state of Q learning is defined as a set of observation vectors from time tn to time t;
[0118] The reward function for Q learning is:
[0119]
[0120] in, is the reward obtained by taking the second action selected from time t to t+1, is the resource utilization of the server at time t. When the utilization is below 80%, optimization is considered to maximize utilization. When it is above 80%, it is punished for excessive load. When it is above 100%, it is severely punished.
[0121] It should be noted that the training of the first neural network is carried out before the military training, so it will not occupy server resources during the military training.
[0122] The above describes an embodiment of the present invention, but this embodiment is not limited to the above-mentioned specific implementation mode. The above-mentioned specific implementation mode is merely illustrative and not restrictive. Under the guidance of this embodiment, ordinary technicians in this field can also make more forms of equivalent embodiments, all of which are within the protection of this embodiment.
Claims
1. An auxiliary decision-making method based on AI big model, It is characterized in that Online training of the agent consists of the following steps: Step 100, collecting training data and encoding them into observation vectors; Step 200: The observation vector currently acquired Input into the DQN model, output the Q values corresponding to all first actions in the first action space, use the first action selection strategy to select a first action from all first actions to decode and execute; Step 300, define time t as the current time, and the first action taken at time t is , execute the first action The observation vector collected at time t+1 is , execute the first action Rewards , combined into a high-dimensional vector exp and stored in the experience replay pool; , represents the observation vector collected at time t; Step 400, input the observation vector from time tn to time t into the trained first neural network, output the time when the training starts and the number of experience collected during the training, and when the time when the training starts is reached, collect the corresponding number of experience from the experience replay pool to train the DQN model; In step 400, the DQN model includes a Q network, and the calculation formula of the Q network is as follows: ; ; Where Z represents the latent vector, and Both represent trainable weight parameters, and Both represent trainable bias parameters, represents the first output vector, the value of the i-th component of the first output vector represents the Q value of the i-th first action, Represents the Sigmoid function; The training steps of the DQN model in step 400 include: Step 101, randomly obtain an experience from the experience pool; ; in, express The maximum Q value output by the target network; according to and Calculate the loss value loss and update the Q network; ; Indicates status and the first action The output Q value of the next Q network is, Indicates status and the first action The output Q value of the target network is: is the discount factor, which is a value between 0 and 1, used to balance the weights of immediate rewards and future rewards. The default value is 0.6; Step 102, updating the target network every fixed number of network updates so that its parameters are the same as those of the current Q network; Step 103, until the number of collected experiences is equal to the number of experiences collected in this training, the step is terminated; The calculation formula is: ; in , , , Represents the first, second, third, and fourth reward values respectively. The initial value is 0. Execute the first action back The cumulative number of enemy soldiers killed. Add up the number of your own soldiers killed, Add up the number of positions occupied by your side. The number of positions lost by the enemy is accumulated. Represents the total number of enemy soldiers, Indicates the total number of soldiers on one's side. represents the total number of enemy positions at the beginning of training, Indicates the total number of friendly positions at the beginning of training. , , , They represent the first, second, third, and fourth tendency coefficients respectively. The sum of the four tendency coefficients is 1. The default values are 0.2, 0.2, 0.3, and 0.3 respectively. The calculation formula of the first neural network in step 400 is as follows: ; ; ; ; ; ; in, and They represent the Lth observation vector of the input, n≥L≥1, , , and Both represent trainable weight parameters, , , and Both represent trainable bias parameters, represents the dot product, Indicates The first intermediate feature, Indicates The second intermediate feature, Indicates A third intermediate feature, and Respectively represent the L-1th and output features, represents the second output vector, the value of the i-th component of the second output vector represents the Q value of the i-th second action, tanh represents the tanh function, Represents the Sigmoid function; The second action belongs to the second action space. The generation method of the second action in the second action space is as follows: Discretize 0% to 100% into multiple point values, and then combine with the optional experience number collected during training to obtain, wherein the discrete point value represents the ratio of the time difference between the training start time and time t to the time difference between t and time t+1; The first neural network is trained using the Q learning method, and the state of the Q learning is defined as the set of observation vectors from time tn to time t; The reward function for Q learning is: in, is the reward obtained by taking the second action selected from time t to t+1, is the resource utilization of the server at time t.
2. According to the AI big model-based decision-making method of claim 1, It is characterized in that The training data in step 100 includes current weather data, combat training map data, enemy information, own information, and server information; Weather data: temperature, humidity, visibility; Combat training map data: topographic map, combat area division map, position marking map; Enemy information: enemy quantity, movement speed, location, and equipment; Your own information: your own soldier type, position, number, and equipment; Server information: server resource utilization.
3. According to the AI big model-based decision-making method of claim 2, It is characterized in that Server resource utilization is the weighted sum of CPU utilization, memory utilization, disk I / O utilization, and network bandwidth utilization.
4. According to the AI big model-based decision-making method of claim 1, It is characterized in that A first action in the first action space in step 200 is expressed as: ; in, , and They represent the target battlefield areas to which the 1st, 2nd, and nth combat units need to move respectively; Combat behavior , and They respectively represent the combat operations that the 1st, 2nd, and nth combat units need to perform.
5. According to the AI big model-based decision-making method of claim 1, It is characterized in that The first action selection strategy in step 200 includes: according to -The greedy strategy chooses the first action: set up The initial value is 0.1; Generate a random number r∈[0,1], if r< , then randomly select a first action, otherwise, select the first action with the largest Q value.
6. An auxiliary decision-making system based on AI big model, It is characterized in that It is used to execute the auxiliary decision-making method based on the AI big model as described in any one of claims 1-5.
Citation Information
Cited By
Combat agent reinforcement learning training method and system based on self-game mode
CN122114052A
Combat intelligent agent reinforcement learning training method and system based on self-game mode
CN122114052B
