Battlefield simulation method, device and equipment and storage medium
Through normalized data processing and model training of target computer nodes, the problem of increased computing complexity in battlefield simulation software is solved, and efficient resource allocation and simulation quality improvement is achieved.
Patent Information
- Application Number
- CN202411901360.9
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-12-23
- Publication Date
- 2025-04-01
AI Technical Summary
In modern military operations, battlefield simulation software faces the problem of increased computational complexity, which makes it difficult to take into account both simulation quality and computational cost.
By normalizing the node performance and task status data of the target computer node, an observation matrix is constructed, and the action space is defined based on the task action. The initial strategy network model is trained using the preset bidding scheduling method and the soft actor-critician algorithm to obtain the target simulation decision model.
It improves the quality of battlefield simulation, optimizes resource allocation, reduces task completion time, improves the utilization rate of computing resources, reduces data synchronization delay, and controls computing costs and time.
Smart Images
Figure CN119740486A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of deep learning technology, and in particular to a battlefield simulation method, device, equipment and storage medium. Background Art
[0002] In modern military operations, the application of information and digital technology has become a key factor in improving combat effectiveness. As an important tool, battlefield simulation software can simulate the real battlefield and provide a scientific basis for military training, equipment simulation, tactical evaluation and decision support. With the advancement of technology, the functions and complexity of battlefield simulation software are constantly improving. How to achieve efficient resource scheduling and task allocation in dynamic and complex environments has become a hot topic in current research. Battlefield simulation software plays a key role in modern military operations. It provides a virtual but brand-new capability for military, equipment simulation, tactical evaluation and decision support, allowing commanders and troops to conduct these operations without the risk of actual combat. The origin of these software can be traced back to the mid-twentieth century, when military simulations mainly relied on simple mathematical and physical sandboxes to help commanders understand the basic principles of tactics and strategy. In the development of battlefield simulation software, although technology has made significant progress, it still faces a series of sudden challenges in practical applications, which put forward higher requirements on the performance and effect of simulation systems. Computational complexity is a primary issue. The complexity and dynamics of the modern battlefield require the simulation system to handle a large number of combat nodes and tasks, which inevitably increases the computational complexity. Therefore, as the simulation accuracy and scope expand, how to ensure the quality of simultaneous simulation and control computing cost and time becomes a forward-looking problem to be solved. Summary of the invention
[0003] In view of this, the purpose of the present invention is to provide a battlefield simulation method, device, equipment and storage medium, which can improve the quality of battlefield simulation. The specific scheme is as follows:
[0004] In a first aspect, the present application discloses a battlefield simulation method, comprising:
[0005] Performing data normalization processing on the node performance and task status data of the target computer node to obtain an observation vector of the target computer node, and constructing an observation matrix according to the node quantity of the target computer node and the observation vector;
[0006] Defining a corresponding action space based on the task actions corresponding to each node task, and determining a target reward function according to the node task and the target computer node;
[0007] Determine the target executable action corresponding to the target computer node from the action space using a preset bidding scheduling method, and determine a model training data set based on the target reward function, the observation matrix and the target executable action corresponding to the target computer node;
[0008] The initial policy network model is trained using a soft actor-critic algorithm and the model training data set to obtain a target simulation decision model, so as to complete the simulation of the target battlefield based on the target simulation decision model.
[0009] Optionally, performing data normalization processing on the node performance and task status data of the target computer node to obtain the observation vector of the target computer node includes:
[0010] The current load, available computing power, network delay, current task queue length and task priority of the target computer node are obtained, and the current load, the available computing power, the network delay, the current task queue length and the task priority are respectively normalized to obtain an observation vector of the target computer node.
[0011] Optionally, the action space corresponding to the task action definition corresponding to each node task includes:
[0012] Obtaining the node allocation action of each node task, and determining the resource adjustment action of each node task based on the node calculation total resources and the preset action adjustment amount;
[0013] A priority adjustment action is determined based on the priority of each of the node tasks, and a corresponding action space is determined based on the node allocation action, the resource adjustment action and the priority adjustment action.
[0014] Optionally, determining a target reward function according to the node task and the target computer node includes:
[0015] A target reward function is constructed based on the task completion time and task quantity corresponding to the node task, and the resource utilization rate and node quantity of the target computer node.
[0016] Optionally, the determining the target executable action corresponding to the target computer node from the action space by using a preset bidding scheduling method includes:
[0017] Taking each node action in the action space as an auction object, and obtaining an auction bid for the auction object in each target computer node; the auction bid is a bid determined by each target computer node based on its own cost bid and resource utilization bid for the auction object;
[0018] The target computer node with the highest auction bid for the auction object among the target computer nodes is determined as a successful node, and the auction object is determined as a target executable action corresponding to the successful node.
[0019] Optionally, the determining of the model training data set based on the target reward function, the observation matrix and the target executable action corresponding to the target computer node comprises:
[0020] Inputting the target reward function, the observation matrix, and the target executable action corresponding to the target computer node into a preset Markov decision modeling formula, and solving the preset Markov decision modeling formula to obtain a reward function matrix corresponding to the observation matrix and the target executable action;
[0021] A model training data set is determined based on the observation matrix, the target executable action, and the reward function matrix.
[0022] Optionally, the using the soft actor-critic algorithm and the model training data set to perform model training on the initial policy network model to obtain a target simulation decision model includes:
[0023] Initializing the parameters of the initialization policy network and the Q-value network in the initial policy network model to obtain an initialized policy network model;
[0024] The model training data set is stored in a preset experience replay buffer, and a target training data set is selected from all the model training data sets in the preset experience replay buffer;
[0025] Substituting the target training data set into a preset target value calculation formula to obtain a target value, and using the target value to update the parameters of the initialization policy network and the Q value network in the initialized policy network model to obtain an updated policy network model;
[0026] Determining whether the updated policy network model meets the preset model training end condition based on the preset expected reward function;
[0027] If the updated policy network model meets the preset model training end condition, the updated policy network model is determined as the target simulation decision model;
[0028] If the updated policy network model does not meet the preset model training end condition, jump to the step of selecting a target training data set from all model training data sets in the preset experience playback buffer.
[0029] In a second aspect, the present application discloses a battlefield simulation device, comprising:
[0030] An observation matrix construction module is used to perform data normalization processing on the node performance and task status data of the target computer node to obtain the observation vector of the target computer node, and to construct an observation matrix according to the node number of the target computer node and the observation vector;
[0031] An objective function determination module, used to define a corresponding action space based on the task actions corresponding to each node task, and determine a target reward function according to the node task and the target computer node;
[0032] A data group generation module, used to determine the target executable action corresponding to the target computer node from the action space by using a preset bidding scheduling method, and determine a model training data group based on the target reward function, the observation matrix and the target executable action corresponding to the target computer node;
[0033] The model training module is used to perform model training on the initial policy network model using the soft actor-critic algorithm and the model training data group to obtain a target simulation decision model, so as to complete the simulation of the target battlefield based on the target simulation decision model.
[0034] In a third aspect, the present application discloses an electronic device, comprising:
[0035] Memory, used to store computer programs;
[0036] A processor is used to execute the computer program to implement the aforementioned battlefield simulation method.
[0037] In a fourth aspect, the present application discloses a computer-readable storage medium for storing a computer program, which implements the aforementioned battlefield simulation method when executed by a processor.
[0038] It can be seen that in this application, the node performance and task status data of the target computer node are normalized to obtain the observation vector of the target computer node, and the observation matrix is constructed according to the node number of the target computer node and the observation vector; the corresponding action space is defined based on the task action corresponding to each node task, and the target reward function is determined according to the node task and the target computer node; the target executable action corresponding to the target computer node is determined from the action space using a preset bidding scheduling method, and the model training data group is determined based on the target reward function, the observation matrix and the target executable action corresponding to the target computer node; the initial policy network model is trained using the soft actor-critic algorithm and the model training data group to obtain a target simulation decision model, so as to complete the simulation of the target battlefield based on the target simulation decision model. That is, by first obtaining the node performance and task status data of the target computer node, constructing the observation matrix, and then defining the action space, using the bidding scheduling strategy to obtain data and modeling for reinforcement learning model training data, and finally using the soft actor-critic algorithm for pre-training and training. In this way, by optimizing resource allocation, the system can reduce task completion time, improve the utilization of computing resources, and reduce data synchronization delays, thereby improving the quality of battlefield simulation and controlling computing costs and time. BRIEF DESCRIPTION OF THE DRAWINGS
[0039] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying creative work.
[0040] Figure 1 A flowchart of a battlefield simulation method disclosed in this application;
[0041] Figure 2 A specific MDP modeling diagram disclosed in this application;
[0042] Figure 3 A specific strategy network model training flow chart disclosed in this application;
[0043] Figure 4 A flowchart of a specific battlefield simulation method disclosed in this application;
[0044] Figure 5 This is a schematic diagram of the structure of a battlefield simulation device disclosed in this application;
[0045] Figure 6 This is a structural diagram of an electronic device disclosed in this application. DETAILED DESCRIPTION
[0046] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.
[0047] Battlefield simulation software is not only a tool for drivers in modern military operations, but also an important direction for the development of military science and technology. By continuously absorbing the latest scientific and technological achievements, especially artificial intelligence and big data analysis technology, battlefield simulation software will play a more important role in future military training and operations. Therefore, this application will specifically introduce a method for improving the quality of battlefield simulation by training a decision model and improving the efficiency and performance of a distributed battlefield simulation system.
[0048] See also Figure 1 As shown, the embodiment of the present application discloses a battlefield simulation method, including:
[0049] Step S11: performing data normalization processing on the node performance and task status data of the target computer node to obtain the observation vector of the target computer node, and constructing an observation matrix according to the node quantity of the target computer node and the observation vector.
[0050] In this embodiment, the battlefield combat node model is distributed to multiple computer nodes, that is, to each target computer node. The current load, available computing power, network delay, current task queue length and task priority of the target computer node are obtained, and the current load, the available computing power, the network delay, the current task queue length and the task priority are normalized to obtain the observation vector of the target computer node. In other words, the current load, available computing power, network delay, current task queue length and task priority of the target computer node are obtained, and the data is normalized to construct the observation vector of each node, and the observation matrix S is constructed according to the number of nodes. Specifically, take the current load of the computing node o , available computing power A o , network delay o , the current number of simulation processes Q o , and each task priority P o , normalize each value data. The normalization formula is as follows:
[0051] .
[0052] Among them, min means taking the minimum value in vector x, max means taking the maximum value in vector x, and x is the independent variable of the function. Then, according to the L obtained after normalization o 'A o 'N o 'Q o 'P o 'Construct observation vector: O T =[L o 'A o 'N o 'Q o 'P o '], where T in the upper right corner is the transposed symbol of the vector. Finally, the observation matrix S is constructed according to the number of simulation nodes, where the observation matrix S is as follows:
[0053] ;
[0054] Here, n represents the number of nodes, which changes according to the number of tasks.
[0055] Step S12: defining a corresponding action space based on the task actions corresponding to each node task, and determining a target reward function according to the node task and the target computer node.
[0056] In this embodiment, the node allocation action of each node task is obtained, and the resource adjustment action of each node task is determined based on the total node computing resources and the preset action adjustment amount; the priority adjustment action is determined based on the priority of each node task, and the corresponding action space is determined based on the node allocation action, the resource adjustment action and the priority adjustment action. That is, the action space is defined, i.e., the distributed computing online scheduling scheme, including the allocation of specific tasks to a certain node, the adjustment of the node's computing resource allocation, and the adjustment of the task priority. Specifically, the task T is defined. i Assigned to node N j The action is a ij , define task T i The resource adjustment action is r i， Define Task T i The priority adjustment action is p i Combining the above actions, we can define a total action vector A to represent all possible actions in the scheduling scheme: A={a ij , r i , p i|i=1,…,M;j=1,…,N}, where the subscripts i and j represent constants, and M and N represent natural number constants. Next, a target reward function is constructed based on the task completion time and task quantity corresponding to the node task, as well as the resource utilization rate and node quantity of the target computer node. That is, a reward function R is designed based on the information of task completion time, resource utilization rate, data synchronization delay, total number of tasks, and total number of nodes. The reward function R is as follows:
[0057] ;
[0058] In this formula, t i is the completion time of the ith task, is the resource utilization of the ith node, defined as the ratio of actually used resources to available resources; D sync is the data synchronization delay, N is the total number of tasks, M is the total number of nodes, and w 1 , w 2 , w 3 is a weight coefficient used to balance the importance of task completion time, resource utilization, and synchronization delay, and satisfies w 1 +w 2+ w 3 =1.
[0059] Step S13: Determine the target executable action corresponding to the target computer node from the action space using a preset bidding scheduling method, and determine a model training data group based on the target reward function, the observation matrix and the target executable action corresponding to the target computer node.
[0060] In this embodiment, each node action in the action space is taken as an auction object, and the auction bid for the auction object in each target computer node is obtained; the auction bid is a bid determined by each target computer node based on its own cost bid and resource utilization bid for the auction object; the target computer node with the highest auction bid for the auction object among the target computer nodes is determined as a successful node, and the auction object is determined as the target executable action corresponding to the successful node. That is, using auction scheduling (Auction-Based Scheduling), nodes bid according to the supply and demand relationship of resources, and tasks are assigned to the nodes with the highest bids to obtain offline data of the auction scheduling strategy. The role of these data is to strengthen the experience playback method of learning. Specifically, in a distributed scheduling system, the task manager regards each task as an auction object and publishes it to all participating nodes. Each node submits a bid based on the cost and resource utilization of the task. The task manager selects the node with the highest bid to execute the task. First, the task is published, and each task T iis regarded as an auction object, and the task manager publishes the task to all participating nodes. Nodes bid, and each task T i Considered as an auction object, the task manager publishes the task to all participating nodes, with bids of:
[0061] ;
[0062] Where: w cost is the weight based on the cost bid, w utilization is the weight based on the utilization bid and satisfies w cost +w utilization =1. b cost,ij is node N j For task T i Cost-based bidding. utilization,ij is node N j For task T i The resource utilization-based bidding selects the highest bid for each task T i Select the node N with the highest bid j To perform tasks. It should be noted here that the auction object in the algorithm is to perform tasks, and each computer node can perform the task actions corresponding to each task after auctioning the corresponding task. However, since the present application is adjusted for a specific task assigned to a certain node, it specifically includes determining the resource adjustment action of each node task based on the node calculation total resources and the preset action adjustment amount, and determining the priority adjustment action based on the priority of each node task. Therefore, in the present application, each computer node not only needs to perform the assigned specific task, but also the adjusted resource adjustment action and priority adjustment action. Since the node allocation action, the resource adjustment action and the priority adjustment action constitute the action space, the present application has been adjusted for the algorithm, changing the auction object from the task to the node action in the action space to ensure that each computer node can execute all possible actions in the scheduling scheme.
[0063] In this embodiment, the method of determining the model training data set based on the target reward function, the observation matrix, and the target executable action corresponding to the target computer node includes: inputting the target reward function, the observation matrix, and the target executable action corresponding to the target computer node into a preset Markov decision modeling formula, and solving the preset Markov decision modeling formula to obtain the observation matrix and the reward function matrix corresponding to the target executable action; determining the model training data set based on the observation matrix, the target executable action, and the reward function matrix. That is, the target reward function, the observation matrix, and the target executable action corresponding to the target computer node are modeled as a Markov decision process (MDP):
[0064] ;
[0065] in, It is a key concept in reinforcement learning, which represents the expected return starting from state S under a given policy 𝜋. is the symbol of mathematical expectation, S 0 is the first state in the initial state, where express The t power, is the discount factor, t-th power used to reduce the impact of future rewards, R(S t , A t ) is the reward function matrix, which defines the state S at time t t Next take action A t The immediate reward obtained. For each state S t and action A t , the reward can be expressed as R(s, a). Solving this MDP, we can get the optimal strategy, that is, the action A taken in any state can maximize the cumulative reward:
[0066] ;
[0067] Among them, argmax is used to determine the input parameter at which the function achieves the maximum value. It is a key concept in reinforcement learning and represents the optimal strategy. S' is the next state obtained by executing action A in state S. A is the action vector, which represents the set of actions that can be taken in each state S. For each state S, the action set A(s) contains all possible actions a. A is given by the bidding scheduling algorithm based on the state matrix S. Finally, as Figure 2 As shown, according to the reward function, observation matrix, and action vector: .
[0068] Step S14: using the soft actor-critic algorithm and the model training data set to perform model training on the initial policy network model to obtain a target simulation decision model, so as to complete the simulation of the target battlefield based on the target simulation decision model.
[0069] In this embodiment, it should be noted that Soft Actor-Critic (SAC) is a deep reinforcement learning algorithm. It combines policy gradient and Q learning to improve the exploratory and robustness of the strategy through the maximum entropy principle. SAC uses two value networks and a policy network to evaluate the value of state-action pairs and generate actions respectively. Through the experience replay buffer, SAC can reuse experience samples and improve data efficiency. At the same time, the introduction of the target network helps stabilize the update of the value network. In addition, SAC automatically adjusts the temperature parameter to balance the weight between reward and entropy, thereby achieving the best balance between exploration and utilization. Solving the scheduling problem of distributed systems with reinforcement learning relies on a large amount of data. The data obtained by the online system is too slow, and the strategy effect of the model at the beginning of learning is not good. Therefore, filling the experience replay buffer with other strategies plays a big role, reducing the training time while improving the effect. In this embodiment, that is, the parameters of the initialization policy network and the Q-value network in the initial policy network model are initialized to obtain the initialized policy network model; the model training data group is stored in the preset experience replay buffer, and the target training data group is selected from all the model training data groups in the preset experience replay buffer; the target training data group is substituted into the preset target value calculation formula to obtain the target value, and the target value is used to update the parameters of the initialization policy network and the Q-value network in the initialized policy network model to obtain the updated policy network model; based on the preset expected reward function, it is determined whether the updated policy network model meets the preset model training end condition; if the updated policy network model meets the preset model training end condition, the updated policy network model is determined as the target simulation decision model; if the updated policy network model does not meet the preset model training end condition, jump to the step of selecting the target training data group from all the model training data groups in the preset experience replay buffer. Specifically, as Figure 3 As shown in Figure 1, discrete data (training data set) is read from the discrete data buffer (experience playback buffer) to train the initial policy network model. Initialize the parameters θ, Use the experience replay buffer to store training data sets , sample a mini-batch (s, a, r, s') from the experience replay buffer. Where s, r, s', a are mini-batch instances belonging to S, R, S', A respectively. The target value y is calculated as:
[0070] ;
[0071] In the above formula, r is the immediate reward, which is the reward obtained immediately after the current state s takes action a. Used to reduce the impact of future rewards and ensure that the contribution of future rewards gradually decreases, , this is from the strategy Medium sampling action The expected value of Take action Expected cumulative reward. It’s a strategy The entropy term is used to encourage the exploration of the strategy. Randomness of control strategy, higher entropy coefficient Make the strategy more random, lower The preset model training end condition is Q value network parameter θ, Minimize mean square error:
[0072] ;
[0073] Among them, L Q The loss function used to update the Q function (action value function) in reinforcement learning, is the estimated value of the current Q function, which represents the expected cumulative discounted reward for taking action a in state s, and its parameters are . y is the target value to be calculated. This represents the expected value sampled from the experience replay buffer. The parameter θ is regularized to maximize the expected return of entropy:
[0074] .
[0075] It should be noted here that during the entire model training process, an automatic entropy adjustment strategy is used to dynamically adjust the entropy coefficient α to balance exploration and utilization. Automatic entropy adjustment updates the actual entropy of the strategy by minimizing a loss function so that it is as close to the preset entropy target as possible. Specifically, the update of α uses the following loss function:
[0076] ;
[0077] in, is the target entropy, usually a negative value, logπ(a|s) is the logarithmic probability of the strategy. L(α) is the loss function, is the mathematical expectation symbol, and log is the logarithm.
[0078] If the updated policy network model satisfies the conditions of the above formula, the updated policy network model is determined as the target simulation decision model. Finally, the target battlefield simulation software is updated based on the target simulation decision model to complete the simulation of the target battlefield, wherein the process of using the target battlefield simulation software to simulate the target battlefield is as follows: Figure 4 shown.
[0079] It can be seen that in this embodiment, the node performance and task status data of the target computer node are subjected to data normalization processing to obtain the observation vector of the target computer node, and the observation matrix is constructed according to the node number of the target computer node and the observation vector; the corresponding action space is defined based on the task action corresponding to each node task, and the target reward function is determined according to the node task and the target computer node; the target executable action corresponding to the target computer node is determined from the action space using a preset bidding scheduling method, and the model training data group is determined based on the target reward function, the observation matrix and the target executable action corresponding to the target computer node; the initial policy network model is trained using the soft actor-critic algorithm and the model training data group to obtain a target simulation decision model, so as to complete the simulation of the target battlefield based on the target simulation decision model. That is, by first obtaining the node performance and task status data of the target computer node, constructing the observation matrix, and then defining the action space, using the bidding scheduling strategy to obtain offline data and modeling for reinforcement learning model training data, and finally using the soft actor-critic algorithm for pre-training and training. In other words, by building and training a decision-making model that can schedule calculations and learn optimal strategies from them, the efficiency and performance of the distributed battlefield simulation system can be improved; by optimizing resource allocation, the task completion time can be reduced, the utilization of computing resources can be improved, and data synchronization delays can be reduced; the resource allocation algorithm based on deep reinforcement learning provides a solution for distributed battlefield simulation, which can effectively improve the overall performance and efficiency of the system.
[0080] refer to Figure 5 The present application also discloses a battlefield simulation device, including:
[0081] An observation matrix construction module 11 is used to perform data normalization processing on the node performance and task status data of the target computer node to obtain the observation vector of the target computer node, and to construct an observation matrix according to the node number of the target computer node and the observation vector;
[0082] An objective function determination module 12, used to define a corresponding action space based on the task actions corresponding to each node task, and determine a target reward function according to the node task and the target computer node;
[0083] A data group generating module 13 is used to determine the target executable action corresponding to the target computer node from the action space by using a preset bidding scheduling method, and determine a model training data group based on the target reward function, the observation matrix and the target executable action corresponding to the target computer node;
[0084] The model training module 14 is used to perform model training on the initial policy network model using the soft actor-critic algorithm and the model training data set to obtain a target simulation decision model, so as to complete the simulation of the target battlefield based on the target simulation decision model.
[0085] It can be seen that in this embodiment, by first acquiring the node performance and task status data of the target computer node, constructing the observation matrix, and then defining the action space, using the bidding scheduling strategy to acquire data and model the model training data for reinforcement learning, and finally using the soft actor-critic algorithm for pre-training and training. In this way, by optimizing resource allocation, the system can reduce task completion time, improve the utilization of computing resources, and reduce data synchronization delay, thereby improving the quality of battlefield simulation and controlling computing costs and time.
[0086] In some specific embodiments, the observation matrix construction module 11 can be specifically used to obtain the current load, available computing power, network delay, current task queue length and task priority of the target computer node, and normalize the current load, the available computing power, the network delay, the current task queue length and the task priority to obtain the observation vector of the target computer node.
[0087] In some specific embodiments, the objective function determination module 12 may specifically include:
[0088] A resource adjustment action determination unit, used to obtain the node allocation action of each node task, and determine the resource adjustment action of each node task based on the node calculation total resources and the preset action adjustment amount;
[0089] An action space determination unit is used to determine a priority adjustment action based on the priority of each node task, and to determine a corresponding action space based on the node allocation action, the resource adjustment action and the priority adjustment action.
[0090] In some specific embodiments, the objective function determination module 12 may specifically include:
[0091] The reward function construction unit is used to construct a target reward function based on the task completion time and task quantity corresponding to the node task, and the resource utilization rate and node quantity of the target computer node.
[0092] In some specific embodiments, the data group generating module 13 may specifically include:
[0093] A data auction unit, configured to take each node action in the action space as an auction object, and obtain an auction bid for the auction object from each target computer node; the auction bid is a bid determined by each target computer node based on its own cost bid and resource utilization bid for the auction object;
[0094] The action matching unit is used to determine the target computer node with the highest auction bid for the auction object among the target computer nodes as a successful node, and determine the auction object as a target executable action corresponding to the successful node.
[0095] In some specific embodiments, the data group generating module 13 may specifically include:
[0096] a reward function matrix determination unit, configured to input the target reward function, the observation matrix, and the target executable action corresponding to the target computer node into a preset Markov decision modeling formula, and solve the preset Markov decision modeling formula to obtain a reward function matrix corresponding to the observation matrix and the target executable action;
[0097] A training data generating unit is used to determine a model training data group based on the observation matrix, the target executable action and the reward function matrix.
[0098] In some specific embodiments, the model training module 14 may specifically include:
[0099] A model initialization unit, used to initialize the parameters of the initialization policy network and the Q value network in the initial policy network model to obtain an initialized policy network model;
[0100] A training data selection unit, used to store the model training data group into a preset experience replay buffer, and select a target training data group from all model training data groups in the preset experience replay buffer;
[0101] A network updating unit, used to substitute the target training data set into a preset target value calculation formula to obtain a target value, and use the target value to update the parameters of the initialization policy network and the Q value network in the initialized policy network model to obtain an updated policy network model;
[0102] A training result judgment unit, used to judge whether the updated strategy network model meets the preset model training end condition based on the preset expected reward function;
[0103] A model determination unit, configured to determine the updated policy network model as a target simulation decision model if the updated policy network model meets a preset model training end condition;
[0104] A step jump unit is used to jump to the step of selecting a target training data group from all model training data groups in the preset experience playback buffer if the updated policy network model does not meet the preset model training end condition.
[0105] Furthermore, the present application also discloses an electronic device. Figure 6 This is a structural diagram of an electronic device 20 according to an exemplary embodiment. The content in the diagram cannot be regarded as any limitation on the scope of use of the present application.
[0106] Figure 6 A schematic diagram of the structure of an electronic device 20 provided in an embodiment of the present application. The electronic device 20 may specifically include: at least one processor 21, at least one memory 22, a power supply 23, a communication interface 24, an input / output interface 25, and a communication bus 26. The memory 22 is used to store a computer program, which is loaded and executed by the processor 21 to implement the relevant steps in the battlefield simulation method disclosed in any of the aforementioned embodiments. In addition, the electronic device 20 in this embodiment may specifically be an electronic computer.
[0107] In this embodiment, the power supply 23 is used to provide working voltage for each hardware device on the electronic device 20; the communication interface 24 can create a data transmission channel between the electronic device 20 and the external device, and the communication protocol it follows is any communication protocol that can be applied to the technical solution of the present application, and is not specifically limited here; the input and output interface 25 is used to obtain external input data or output data to the outside world, and its specific interface type can be selected according to specific application needs and is not specifically limited here.
[0108] In addition, the memory 22, as a carrier for storing resources, can be a read-only memory, a random access memory, a disk or an optical disk, etc. The resources stored thereon can include an operating system 221, a computer program 222, etc., and the storage method can be temporary storage or permanent storage.
[0109] The operating system 221 is used to manage and control the hardware devices and computer program 222 on the electronic device 20, which can be Windows Server, Netware, Unix, Linux, etc. In addition to the computer program that can be used to complete the battlefield simulation method performed by the electronic device 20 disclosed in any of the aforementioned embodiments, the computer program 222 can further include computer programs that can be used to complete other specific tasks.
[0110] Furthermore, the present application also discloses a computer-readable storage medium for storing a computer program; wherein the computer program, when executed by a processor, implements the battlefield simulation method disclosed above. The specific steps of the method can refer to the corresponding contents disclosed in the above embodiments, and will not be repeated here.
[0111] In this specification, each embodiment is described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the embodiments can be referred to each other. For the device disclosed in the embodiment, since it corresponds to the method disclosed in the embodiment, the description is relatively simple, and the relevant parts can be referred to the method part.
[0112] Professionals may further appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described in the above description according to function. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professionals and technicians may use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.
[0113] The steps of the method or algorithm described in conjunction with the embodiments disclosed herein may be implemented directly using hardware, a software module executed by a processor, or a combination of the two. The software module may be placed in a random access memory (RAM), a memory, a read-only memory (ROM), an electrically programmable ROM, an electrically erasable programmable ROM, a register, a hard disk, a removable disk, a CD-ROM, or any other form of storage medium known in the art.
[0114] Finally, it should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the presence of other identical elements in the process, method, article or device including the elements.
[0115] The technical solution provided by the present application is introduced in detail above. Specific examples are used in this article to illustrate the principles and implementation methods of the present application. The description of the above embodiments is only used to help understand the method of the present application and its core idea. At the same time, for general technicians in this field, according to the idea of the present application, there will be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as a limitation on the present application.
Claims
1. A battlefield simulation method, characterized in that: include: Performing data normalization processing on the node performance and task status data of the target computer node to obtain an observation vector of the target computer node, and constructing an observation matrix according to the node quantity of the target computer node and the observation vector; Defining a corresponding action space based on the task actions corresponding to each node task, and determining a target reward function according to the node task and the target computer node; Determine the target executable action corresponding to the target computer node from the action space using a preset bidding scheduling method, and determine a model training data set based on the target reward function, the observation matrix and the target executable action corresponding to the target computer node; The initial policy network model is trained using a soft actor-critic algorithm and the model training data set to obtain a target simulation decision model, so as to complete the simulation of the target battlefield based on the target simulation decision model.
2. The battlefield simulation method according to claim 1, characterized in that: The step of performing data normalization processing on the node performance and task status data of the target computer node to obtain the observation vector of the target computer node includes: The current load, available computing power, network delay, current task queue length and task priority of the target computer node are obtained, and the current load, the available computing power, the network delay, the current task queue length and the task priority are respectively normalized to obtain an observation vector of the target computer node.
3. The battlefield simulation method according to claim 1, characterized in that: The task action definition corresponding to each node task includes: Obtaining the node allocation action of each node task, and determining the resource adjustment action of each node task based on the node calculation total resources and the preset action adjustment amount; A priority adjustment action is determined based on the priority of each of the node tasks, and a corresponding action space is determined based on the node allocation action, the resource adjustment action and the priority adjustment action.
4. The battlefield simulation method according to claim 1, characterized in that: The determining of the target reward function according to the node task and the target computer node comprises: A target reward function is constructed based on the task completion time and task quantity corresponding to the node task, and the resource utilization rate and node quantity of the target computer node.
5. The battlefield simulation method according to claim 1, characterized in that: The method of using a preset bidding scheduling method to determine the target executable action corresponding to the target computer node from the action space includes: Taking each node action in the action space as an auction object, and obtaining an auction bid for the auction object in each target computer node; the auction bid is a bid determined by each target computer node based on its own cost bid and resource utilization bid for the auction object; The target computer node with the highest auction bid for the auction object among the target computer nodes is determined as a successful node, and the auction object is determined as a target executable action corresponding to the successful node.
6. The battlefield simulation method according to claim 1, characterized in that: The determining of the model training data set based on the target reward function, the observation matrix and the target executable action corresponding to the target computer node comprises: Inputting the target reward function, the observation matrix, and the target executable action corresponding to the target computer node into a preset Markov decision modeling formula, and solving the preset Markov decision modeling formula to obtain a reward function matrix corresponding to the observation matrix and the target executable action; A model training data set is determined based on the observation matrix, the target executable action, and the reward function matrix.
7. The battlefield simulation method according to any one of claims 1 to 6, characterized in that: The method of using the soft actor-critic algorithm and the model training data set to perform model training on the initial policy network model to obtain a target simulation decision model includes: Initializing the parameters of the initialization policy network and the Q-value network in the initial policy network model to obtain an initialized policy network model; The model training data set is stored in a preset experience replay buffer, and a target training data set is selected from all the model training data sets in the preset experience replay buffer; Substituting the target training data set into a preset target value calculation formula to obtain a target value, and using the target value to update the parameters of the initialization policy network and the Q value network in the initialized policy network model to obtain an updated policy network model; Determining whether the updated policy network model meets the preset model training end condition based on the preset expected reward function; If the updated policy network model meets the preset model training end condition, the updated policy network model is determined as the target simulation decision model; If the updated policy network model does not meet the preset model training end condition, jump to the step of selecting a target training data set from all model training data sets in the preset experience playback buffer.
8. A battlefield simulation device, characterized in that: include: An observation matrix construction module is used to perform data normalization processing on the node performance and task status data of the target computer node to obtain the observation vector of the target computer node, and to construct an observation matrix according to the node number of the target computer node and the observation vector; An objective function determination module, used to define a corresponding action space based on the task actions corresponding to each node task, and determine a target reward function according to the node task and the target computer node; A data group generation module, used to determine the target executable action corresponding to the target computer node from the action space by using a preset bidding scheduling method, and determine a model training data group based on the target reward function, the observation matrix and the target executable action corresponding to the target computer node; The model training module is used to perform model training on the initial policy network model using the soft actor-critic algorithm and the model training data group to obtain a target simulation decision model, so as to complete the simulation of the target battlefield based on the target simulation decision model.
9. An electronic device, characterized in that: include: Memory, used to store computer programs; A processor, configured to execute the computer program to implement the battlefield simulation method as described in any one of claims 1 to 7.
10. A computer-readable storage medium, characterized in that: Used to store a computer program, which, when executed by a processor, implements the battlefield simulation method as described in any one of claims 1 to 7.