Battle decision model training method and device, readable storage medium and equipment

Abstract

The invention relates to the technical field of intelligent decision making, and provides a training method of a combat decision model, a training device of the combat decision model, a computer readable storage medium and an electronic device.The training method of the combat decision model comprises the steps that a to-be-trained combat decision model (including a strategy network and a value network, the strategy network is used for outputting a combat strategy, and the value network is used for outputting a first expected value) sequentially inputting the environment state of each time step to obtain combat data for each environment state sequentially output by the to-be-trained combat decision model; the combat data comprises a combat strategy and a first expected value; determining a loss value of a to-be-trained combat decision model according to the combat data of each time step; and performing parameter iteration on a strategy parameter corresponding to the strategy network and a value parameter corresponding to the value network according to the loss value to obtain a trained combat decision model. The model obtained by training in the invention can effectively cope with a changeable battlefield environment.

Description

Background Art

[0002] Under digital conditions, the battlefield of military operations is highly transparent, the entities involved in the combat are complex and exist in multi-dimensional spaces such as sea, land, air and space, and participate in combat operations in a coordinated manner, making the war process more complicated. This puts higher demands on the command and decision-making personnel's global information acquisition capabilities, global observation and operational capabilities, and the control and coordination capabilities of participating entities in multi-dimensional spaces.

[0003] In the related art, generally, pre-set strategies and rules are relied upon to deal with various incoming targets. Obviously, the above scheme cannot automatically update the strategies and rules, and thus cannot effectively deal with the changing battlefield environment.

[0004] In view of this, there is an urgent need in this field to develop a new training method and device for a combat decision-making model.

[0005] It should be noted that the information disclosed in the above background technology section is only used to enhance the understanding of the background of the present disclosure. Summary of the invention

[0006] The purpose of the present disclosure is to provide a training method for a combat decision model, a training device for a combat decision model, a computer-readable storage medium and an electronic device, thereby at least to a certain extent overcoming the technical problem of being unable to effectively cope with a changing battlefield environment due to the limitations of related technologies.

[0007] Other features and advantages of the present disclosure will become apparent from the following detailed description, or may be learned in part by the practice of the present disclosure.

[0008] According to a first aspect of the present disclosure, a training method for a combat decision model is provided, comprising:

[0009] Inputting the environmental state of each time step into the combat decision model to be trained in sequence to obtain combat data for each environmental state outputted in sequence by the combat decision model to be trained; the combat decision model to be trained comprises a strategy network and a value network, the strategy network is used to output the combat strategy for the environmental state of each time step, and the value network is used to output the first expected value for the environmental state; the combat data comprises the combat strategy and the first expected value;

[0010] Determining the loss value of the combat decision model to be trained according to the combat data of each time step;

[0011] According to the loss value, parameter iteration is performed on the strategy parameters corresponding to the strategy network and the value parameters corresponding to the value network to obtain the trained combat decision model.

[0012] In an exemplary embodiment of the present disclosure, determining the loss value of the combat decision model to be trained according to the combat data of each time step includes:

[0013] Determine whether the total number of the multiple time steps currently accumulated reaches a specified value;

[0014] After the total number reaches the specified value, the loss value of the combat decision model to be trained is determined according to the combat data of each time step.

[0015] In an exemplary embodiment of the present disclosure, the combat data for each of the environmental states further includes:

[0016] The reward value of the combat strategy for each time step;

[0017] a new environment state at the next time step, wherein the new environment state is obtained by updating the environment state after executing the combat strategy;

[0018] A second expected value for the new environmental state.

[0019] In an exemplary embodiment of the present disclosure, the reward value of the combat strategy at each time step is determined in the following manner:

[0020] Get multiple pre-configured reward and punishment conditions;

[0021] Matching each of the combat strategies with each of the reward and punishment conditions, and determining reward and punishment parameters corresponding to each of the reward and punishment conditions;

[0022] The reward value is determined according to the weighted result of multiple reward and punishment parameters.

[0023] In an exemplary embodiment of the present disclosure, determining the loss value of the combat decision model to be trained according to the combat data of each time step includes:

[0024] Calculate a value advantage estimate for each time step according to a first expected value for the environmental state and a second expected value for the new environmental state;

[0025] Obtaining the strategy probability ratio for each time step;

[0026] Determining a policy loss of the policy network according to the value advantage estimate for each time step and the policy probability ratio for each time step;

[0027] Calculating the target value for each of the environmental states according to the reward value of the combat strategy for each of the time steps, and determining the value loss of the value network according to the first expected value of each of the environmental states and the target value for each of the environmental states;

[0028] The loss value of the combat decision model to be trained is determined according to the strategy loss and the value loss.

[0029] In an exemplary embodiment of the present disclosure, the step of calculating the value advantage estimation value for each time step according to the first expected value for the environmental state and the second expected value for the new environmental state includes:

[0030] The value advantage estimate for each time step is determined by the following formula:

[0031]

[0032] δ t =r t +γV(s t+1 )-V(s t )

[0033] Wherein, t represents the sequence number of each time step, represents the estimated value of the value advantage corresponding to time step t, T represents the serial number of the maximum time step of each round, l represents the index value that increases by one from 0 to T, γ represents the discount factor, and λ represents the smoothing parameter;

[0034] δ t represents the temporal difference error corresponding to time step t, r t represents the reward value corresponding to time step t, V(s t ) represents the first expected value corresponding to the environmental state at time step t, V(s t+1 ) represents the second expected value corresponding to the environmental state at time step t+1.

[0035] In an exemplary embodiment of the present disclosure, the strategy probability ratio for each time step is determined by the following formula:

[0036]

[0037] Among them, ratio t (θ) represents the probability ratio of the strategy at time step t;

[0038] π θ (a t |s t) represents the time step t, and when the current policy parameter θ is adopted, in the environment state s t Adopt the above mentioned combat strategy a t The first probability of

[0039] Represents the time step t, and takes the old policy parameters θ old When the environment state s t Adopt the above mentioned combat strategy a t The second probability of

[0040] The old policy parameters are the policy parameters before the last round of adjustment, and the current policy parameters are the policy parameters after the last round of adjustment.

[0041] In an exemplary embodiment of the present disclosure, determining the policy loss of the policy network according to the value advantage estimation value for each time step and the policy probability ratio for each time step includes:

[0042] Determining a strategy loss component corresponding to each time step according to the value advantage estimate for each time step and the strategy probability ratio for each time step;

[0043] The policy loss of the policy network is determined according to the expected values ​​of the multiple policy loss components corresponding to the multiple time steps.

[0044] In an exemplary embodiment of the present disclosure, determining the policy loss of the policy network according to the expected values ​​of the plurality of policy loss components corresponding to the plurality of time steps includes:

[0045] The policy loss of the policy network is calculated by the following formula:

[0046]

[0047] in, represents the expectation of multiple strategy loss components with values ​​ranging from 0 to T for time step t;

[0048] Represents the strategy loss component corresponding to time step t;

[0049] The clip function is used to ensure the ratio t The value of (θ) is limited to [1-∈, 1+∈], and the min function represents the value of and The smaller value of the two is used as the policy loss component corresponding to each time step t, and ∈ represents the truncation hyperparameter.

[0050] In an exemplary embodiment of the present disclosure, the calculating of the target value for each of the environmental states according to the reward value of the combat strategy for each of the time steps includes:

[0051] The target value for each of the environmental states is calculated by the following formula:

[0052]

[0053] Among them, V t target represents the target value for time step t, r t+k Represents the reward value corresponding to the combat strategy output at time t+k, and the value of k increases from 0 to Tt.

[0054] In an exemplary embodiment of the present disclosure, determining the value loss of the value network according to the first expected value of each of the environmental states and the target value for each of the environmental states includes:

[0055] The value loss of the value network is calculated based on the following formula:

[0056]

[0057] Among them, L VF (θ) represents the above-mentioned loss of value; represents the expectation of multiple first target values ​​from 0 to T for time step t, where the first target value refers to (V(s t )-V t target ) 2 .

[0058] In an exemplary embodiment of the present disclosure, determining the loss value of the combat decision model to be trained according to the strategy loss and the value loss includes:

[0059] The loss value of the combat decision model to be trained is determined based on the following formula:

[0060]

[0061] Among them, L CLIP (θ) represents the strategy loss, L VF (θ) represents the said loss of value;

[0062] represents the expectation of multiple second target values ​​with values ​​from 0 to T for time step t, each of which refers to the L corresponding to each time step t CLIP (θ)-c1L VF (θ)+c2H(πθ );

[0063] c1 and c2 represent weight hyperparameters;

[0064] H(π θ ) represents the strategy entropy, the combat strategy contains multiple sub-strategies, a represents each sub-strategy in the combat strategy generated corresponding to the environmental state at time step t, π θ (a|s t ) represents the time step t, and when the current policy parameter θ is adopted, in the environment state s t The third probability of taking each sub-strategy a.

[0065] In an exemplary embodiment of the present disclosure, performing parameter iteration on the policy parameters corresponding to the policy network and the value parameters corresponding to the value network according to the loss value to obtain the trained combat decision model includes:

[0066] According to the loss value, the policy parameters corresponding to the policy network and the value parameters corresponding to the value network are updated using a gradient descent algorithm;

[0067] When a preset training termination condition is met, the trained combat decision model is obtained;

[0068] The training termination condition includes that the number of parameter iterations reaches a preset threshold, or the average reward value of multiple rounds tends to converge.

[0069] According to a second aspect of the present disclosure, there is provided a training device for a combat decision model, comprising:

[0070] A combat data generation module, used for sequentially inputting the environmental state of each time step into the combat decision model to be trained, so as to obtain the combat data for each environmental state sequentially output by the combat decision model to be trained; the combat decision model to be trained comprises a strategy network and a value network, the strategy network is used for outputting the combat strategy for the environmental state, and the value network is used for outputting the first expected value for the environmental state; the combat data comprises the combat strategy and the first expected value;

[0071] A loss value determination module, used to determine the loss value of the combat decision model to be trained according to the combat data of each time step;

[0072] A parameter iteration module is used to perform parameter iteration on the strategy parameters corresponding to the strategy network and the value parameters corresponding to the value network according to the loss value to obtain the trained combat decision model.

[0073] According to a third aspect of the present disclosure, there is provided a computer-readable storage medium having a computer program stored thereon, wherein when the computer program is executed by a processor, the training method for the combat decision model described in the first aspect above is implemented.

[0074] According to a fourth aspect of the present disclosure, there is provided an electronic device, comprising: a processor; and a memory for storing executable instructions of the processor; wherein the processor is configured to execute the training method of the combat decision model described in the first aspect above by executing the executable instructions.

[0075] It can be seen from the above technical solutions that the training method of the combat decision model, the training device of the combat decision model, the computer-readable storage medium and the electronic device in the exemplary embodiments of the present disclosure have at least the following advantages and positive effects:

[0076] In the technical solutions provided by some embodiments of the present disclosure, the environmental state of each time step is sequentially input into the combat decision model to be trained (the combat decision model to be trained includes a strategy network and a value network) to obtain the combat data for each environmental state sequentially output by the combat decision model to be trained; the loss value of the combat decision model to be trained is determined according to the combat data of each time step; the policy parameters corresponding to the strategy network and the value parameters corresponding to the value network are iterated according to the loss value to obtain the trained combat decision model. On the one hand, the combat decision model to be trained can be dynamically updated, thereby avoiding the problem of making combat decisions according to unchanging strategies or rules. The problem of being unable to be effectively applied to the changing battlefield environment; further, the joint training of the policy network and the value network, as well as the parameter iteration guided by the loss value, can improve the efficiency and stability of the model training, reduce the fluctuations in the training process, and enable the model to converge to a stable decision-making strategy more quickly; on the other hand, by simultaneously training the policy network and the value network, this method can simultaneously optimize the strategy selection (determine the best action) and value evaluation (predict future returns). The policy network focuses on learning how to make the best decision in a given environment, while the value network helps evaluate the long-term benefits of these decisions. The combination of the two enables the model to make more intelligent and long-term decisions, thereby improving the practicality of the model.

[0077] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0078] The accompanying drawings herein are incorporated into the specification and constitute a part of the specification, illustrate embodiments consistent with the present disclosure, and together with the specification are used to explain the principles of the present disclosure. Obviously, the accompanying drawings described below are only some embodiments of the present disclosure, and for ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without creative work.

[0079] Figure 1 A flowchart showing a method for training a combat decision model in an embodiment of the present disclosure is shown;

[0080] Figure 2 A flowchart showing how to determine the loss value of the combat decision model to be trained according to the combat data at each time step in the embodiment of the present disclosure;

[0081] Figure 3 A flow chart showing how to determine the policy loss of a policy network according to the value advantage estimation value for each time step and the policy probability ratio value for each time step in an embodiment of the present disclosure;

[0082] Figure 4 A flow chart showing how to perform parameter iteration on the policy parameters corresponding to the policy network and the value parameters corresponding to the value network according to the loss value in the embodiment of the present disclosure to obtain a trained combat decision model;

[0083] Figure 5 The overall architecture diagram of the training method of the combat decision model in the embodiment of the present disclosure is shown;

[0084] Figure 6 A schematic diagram showing the structure of a training device for a combat decision model in an exemplary embodiment of the present disclosure;

[0085] Figure 7 A schematic structural diagram of an electronic device in an exemplary embodiment of the present disclosure is shown. DETAILED DESCRIPTION

[0086] Example embodiments will now be described more fully with reference to the accompanying drawings. However, example embodiments can be implemented in a variety of forms and should not be construed as being limited to the examples set forth herein; on the contrary, these embodiments are provided so that the present disclosure will be more comprehensive and complete, and the concepts of the example embodiments are fully conveyed to those skilled in the art. The described features, structures, or characteristics may be combined in one or more embodiments in any suitable manner. In the following description, many specific details are provided to provide a full understanding of the embodiments of the present disclosure. However, those skilled in the art will appreciate that the technical solutions of the present disclosure may be practiced while omitting one or more of the specific details, or other methods, components, devices, steps, etc. may be adopted. In other cases, known technical solutions are not shown or described in detail to avoid obscuring various aspects of the present disclosure.

[0087] The terms "a", "an", "the" and "said" are used in this specification to indicate the presence of one or more elements / components / etc.; the terms "including" and "having" are used to express an open-ended inclusion and mean that additional elements / components / etc. may exist in addition to the listed elements / components / etc.; the terms "first" and "second" etc. are used only as labels and are not intended to limit the quantity of their objects.

[0088] In addition, the accompanying drawings are only schematic illustrations of the present disclosure and are not necessarily drawn to scale. The same reference numerals in the drawings represent the same or similar parts, and their repeated descriptions will be omitted. Some of the block diagrams shown in the accompanying drawings are functional entities and do not necessarily correspond to physically or logically independent entities.

[0089] In the embodiments of the present disclosure, a training method for a combat decision model is first provided, which at least to some extent overcomes the defect of the related technology that it cannot effectively cope with the changing battlefield environment.

[0090] Figure 1 A flow chart of a method for training a combat decision model in an embodiment of the present disclosure is shown. The execution subject of the method for training a combat decision model may be a server for training the combat decision model.

[0091] refer to Figure 1 According to an embodiment of the present disclosure, a training method for a combat decision model includes the following steps:

[0092] Step S110, inputting the environmental state of each time step into the combat decision model to be trained in sequence, so as to obtain the combat data for each environmental state outputted in sequence by the combat decision model to be trained; the combat decision model to be trained comprises a strategy network and a value network, the strategy network is used to output the combat strategy for the environmental state of each time step, and the value network is used to output the first expected value for the environmental state; the combat data comprises the combat strategy and the first expected value;

[0093] Step S120, determining the loss value of the combat decision model to be trained according to the combat data of each time step;

[0094] Step S130, performing parameter iteration on the strategy parameters corresponding to the strategy network and the value parameters corresponding to the value network according to the loss value to obtain a trained combat decision model.

[0095] exist Figure 1 In the technical solution provided by the illustrated embodiment, the environmental state of each time step is sequentially input into the combat decision model to be trained (the combat decision model to be trained includes a strategy network and a value network) to obtain the combat data for each environmental state sequentially output by the combat decision model to be trained; the loss value of the combat decision model to be trained is determined according to the combat data of each time step; the policy parameters corresponding to the strategy network and the value parameters corresponding to the value network are iterated according to the loss value to obtain the trained combat decision model. On the one hand, the combat decision model to be trained can be dynamically updated, thereby avoiding the inability to make combat decisions according to unchanging strategies or rules. The problem of effective application of changing battlefield environments; further, the joint training of policy networks and value networks, as well as parameter iteration guided by loss values, can improve the efficiency and stability of model training, reduce fluctuations during training, and enable the model to converge to a stable decision-making strategy more quickly; on the other hand, by simultaneously training policy networks and value networks, this method can simultaneously optimize policy selection (determine the best action) and value assessment (predict future returns). The policy network focuses on learning how to make the best decisions in a given environment, while the value network helps evaluate the long-term benefits of these decisions. The combination of the two enables the model to make more intelligent and long-term decisions, thereby improving the practicality of the model.

[0096] The following Figure 1 The specific implementation process of each step is described in detail:

[0097] Before step S110, it should be noted that the combat decision model to be trained in the present disclosure may include a strategy network and a value network, the strategy network is used to output the combat strategy for the environmental state at each time step, and the value network is used to output the first expected value for the environmental state.

[0098] After the above-mentioned combat decision model to be trained is configured, the above-mentioned combat decision model to be trained can interact with the environment to output combat data for the environmental state at each time step, so that the experience pool can be updated according to the above-mentioned combat data, and after the total number of multiple time steps accumulated in the experience pool reaches a specified value (for example: 3000, which can be set according to actual conditions, and the present disclosure does not make special limitations on this), the combat data of each time step is used as a training sample to update and iterate the parameters of the combat decision model to be trained.

[0099] Specifically, the above-mentioned environmental state may include characteristic parameters of each incoming target of the combat entity at each time step and overall combat effectiveness parameters of the combat entity at each time step against the incoming target.

[0100] The combat entity in the present disclosure refers to a unit or system that can perform combat missions. For example, it can be a ship, a fighter, a bomber, a tank, an infantry fighting vehicle, etc. It can be set according to the actual situation. The present disclosure does not make any special limitations on this. The following embodiments are described by taking the combat entity as a ship as an example.

[0101] In an exemplary embodiment of the present disclosure, the space surrounding a ship can be divided into multiple defense areas (also referred to as defense sectors), and the multiple defense sectors can completely cover the entire airspace outside the ship, and there is a certain overlapping angle between two adjacent defense sectors.

[0102] The following is a description of how to define each defense sector in the present disclosure:

[0103] First, establish the global NED coordinate system (North East Down, also known as the navigation coordinate system): the center of the ship is the origin O(0,0,0), the N axis is forward along the ship's heading, the E axis is perpendicular to the N axis and points to the right, and the D axis points to the center of the earth. Let the length and width of the ship be L and W respectively.

[0104] Second, establish a local polar coordinate system for each corner:

[0105] Assume that the four corners of the ship are set to A, B, C and D respectively, then the coordinates of point A are (-L / 2, -W / 2), the coordinates of point B are (L / 2, -W / 2), the coordinates of point C are (L / 2, W / 2), and the coordinates of point D are (-L / 2, W / 2).

[0106] Third, establish a local polar coordinate system for each corner:

[0107] A local polar coordinate system is established at each corner of A, B, C, and D, with each corner as the origin. A point in each local polar coordinate system is represented by (r, φ), where r is the distance to the origin (a fixed point at the center of the coordinate system, i.e., any of ABCD above), and φ is the polar angle (measured from the polar axis to the line from the pole to the point).

[0108] Fourth, define the defense sector:

[0109] The defense sectors at each corner of A, B, C, and D have a central angle θ (that is, the angle of each sector's defense), and there is an overlap angle Δθ between adjacent sectors. Assume that the defense sector at point A has an angle range of: The minimum radius of each defense sector is denoted as r min , the maximum radius is represented by r max .

[0110] After defining the above multiple defense sectors, assume that the position of an incoming target is expressed as (N t ,E t ), then the defense sector in which the incoming target is located can be determined based on the following steps:

[0111] First, transform the global NED coordinates into the local rectangular coordinate system:

[0112] For the target position (N t ,E t ), it is necessary to convert it from the global NED coordinate system to the local rectangular coordinate system of each corner. Take point A as an example, the local rectangular coordinates of point A are:

[0113] Second, calculate the coordinates in the local rectangular coordinate system:

[0114] Assume that the coordinates of the target position in the local rectangular coordinate system of point A are (x A ,y A ), the conversion formula is as follows:

[0115] Third, calculate the polar angle φ t :

[0116] Calculate the polar angle φ in the local rectangular coordinate system t :

[0117] Fourth, determine whether the target location is within a certain defense sector:

[0118] Check the polar angle φ of the target position t Is it within the angle range of the defense sector? Also check if the target location is within the radius of the defense sector: If r min ≤r t ≤r max , the target position is within the defense sector.

[0119] It should be noted that the present invention can deploy combat weapons in each defense sector. Exemplarily, the combat weapons can include four types. Exemplarily, the types of combat weapons can include missiles, cannons, decoys, and electronic jamming (ew), among which missiles and cannons are hard weapons, and decoys and electronic jamming are soft weapons.

[0120] More specifically, the missile can include two attack modes (also called two guidance modes), which can be radar mode and infrared mode; the machine gun can include two attack modes (also called two auxiliary aiming modes), which can be radar mode and infrared mode; the decoy bomb can include three attack modes (also called three interference modes), which can be active infrared decoy bomb, outboard active decoy and passive infrared jammer; the electronic jamming can include two attack modes (also called two interference modes), which can be passive chaff interference and active noise interference.

[0121] Exemplarily, the above four types of combat weapons can be deployed in each defense sector. Given that each type of combat weapon can include multiple attack modes, when determining the specific number of combat weapons deployed on the ship, it is necessary to jointly determine the number of types of combat weapons, the number of combat weapons, and the number of attack modes associated with each combat weapon. For example, assuming that two types of combat weapons (missiles and machine guns) are deployed in a certain defense sector, one of each type of combat weapon is deployed, and the missile includes two attack modes, and the machine gun also includes two attack modes, then the number of combat weapons deployed in the defense sector can be 1*2+1*2=4. After calculating the number of combat weapons deployed in each defense sector one by one, the numbers of combat weapons corresponding to the above multiple defense sectors can be accumulated to determine the specific number N of combat weapons deployed on the ship.

[0122] In view that combat weapons may be located in different defense sectors and contain different attack modes, different combat weapons can be distinguished according to the identification of the defense sector and the identification of the attack mode. Specifically, the weapon configuration on the ship can refer to Table 1:

[0123] Table 1

[0124]

[0125] The characteristic parameters of each incoming target can be composed of a set of attributes describing its current state (i.e., characteristic variables of multiple dimensions). These attributes are combined into a vector, which includes the distance, azimuth, pitch, speed, speed (moving direction relative to a certain reference direction), x, y, z (x, y, z correspond to the N-axis coordinates of the NED coordinate system, the E-axis coordinates of the NED coordinate system, and the D-axis coordinates of the NED coordinate system), Vx, Vy, Vz (Vx, Vy, Vz correspond to the N-axis speed of the NED coordinate system, the E-axis speed of the NED coordinate system, and the D-axis speed of the NED coordinate system), absolute heading, absolute speed, etc., as well as electronic reconnaissance parameters such as threat level, credibility, etc. These dimensions include sufficient information about the incoming target itself to support comprehensive evaluation and decision-making of the incoming target.

[0126] Exemplarily, the dimension of the above characteristic parameter may be G. Referring to Table 2, Table 2 shows the specific contents of the dimension included in the above characteristic parameter:

[0127] Table 2

[0128]

[0129]

[0130] The overall combat effectiveness parameters of the combat entity against the incoming target at each time step can be called the overall combat effectiveness parameters. The overall combat effectiveness parameters are composed of the combat effectiveness parameters of each combat weapon against each incoming target at each time step, and the combat effectiveness parameters of each combat weapon against each incoming target at each time step can include parameters from two aspects, namely, the expected combat effect characterization parameters of each combat weapon against each incoming target and the performance parameters of each combat weapon at each time step.

[0131] Specifically, referring to Table 3, the combat effectiveness parameters of each combat weapon against each incoming target at each time step may include the following L (L is an integer greater than 1) dimensions:

[0132] Table 3

[0133]

[0134]

[0135]

[0136] It should be noted that in the above Table 3, Corresponding to the performance parameters of each of the above-mentioned combat weapons at each time step, the rest correspond to the characterization parameters of the expected combat effect of each of the above-mentioned combat weapons against each incoming target.

[0137] Given that the combat effectiveness parameters of each combat weapon against each incoming target at each time step correspond to L dimensions, while the total number of combat weapons is N, and the total number of incoming targets is M, the above-mentioned overall combat effectiveness parameters correspond to N*M*L dimensions, that is, the above-mentioned overall combat effectiveness parameters are composed of M N*L first matrices.

[0138] In each process of interacting with the environment, the environmental state at each time step can be input into the combat decision model to be trained. After that, the combat decision model to be trained (including the policy network and the value network) can perform the following processing:

[0139] The policy network includes a first encoding layer, a first decision layer, and a policy mask layer;

[0140] Among them, the first encoding layer can encode the overall combat effectiveness parameters of each time step to obtain a coding matrix. Specifically, the first encoding layer may include a preprocessing layer, a feature extraction layer, a first multi-head self-attention layer and a first feedforward network; the preprocessing layer can preprocess the overall combat effectiveness parameters to obtain an initial matrix; the feature extraction layer can perform feature extraction on the initial matrix to obtain a word embedding matrix; the word embedding matrix consists of M row vectors; the first multi-head self-attention layer can perform multi-head attention allocation on each row vector to obtain each attention output matrix; the first feedforward network can optimize the M attention output matrices to obtain a coding matrix; the coding matrix consists of M first matrices corresponding to the M attention output matrices.

[0141] The first decision layer can decide the initial combat strategy for the M incoming targets at each time step based on the encoding matrix and the characteristic parameters of the incoming targets. Specifically, the decision layer may include a feature transformation layer, a second multi-head self-attention layer, a multi-head attention layer, and a second feedforward network; the feature transformation layer may perform feature transformation processing on the feature parameters to obtain a transformation matrix; the second multi-head self-attention layer may perform multi-head self-attention allocation on the transformation matrix to obtain a second attention output matrix; the second multi-head self-attention layer includes M attention heads, and the second attention output matrix includes M second matrices corresponding to the M attention heads; each attention head in the multi-head attention layer (the number of attention heads in the multi-head attention layer is equal to N) may perform multi-head attention allocation on the encoding matrix and the second attention output matrix to obtain each third attention output matrix; the multi-head attention layer can perform multi-head attention allocation on the encoding matrix and the second attention output matrix to obtain a third attention output matrix; the number of attention heads in the multi-head attention layer is equal to N; the second feedforward network can optimize the N third attention output matrices to obtain an initial probability distribution matrix to obtain an initial combat strategy for M incoming targets; the number of rows of the above initial probability distribution matrix is ​​equal to the total number N of combat weapons deployed on the combat entity, and the number of columns of the initial probability distribution matrix is ​​equal to the total number M of at least one incoming target; the initial probability distribution matrix is ​​used to describe the attack probability of each combat weapon against each incoming target.

[0142] The above policy mask layer can do the following:

[0143] First, a first conflict resolution matrix is ​​generated (the first conflict resolution matrix has the same dimension as the initial probability distribution matrix, and each element in the first conflict resolution matrix is ​​equal to a first preset value 1). Then, the first conflict resolution matrix can be adjusted based on the first state information of each combat weapon (for example, whether the combat weapon is currently in an available state) and the second state information of each incoming target (for example, whether there is a designated combat weapon at the incoming target) to obtain a second conflict resolution matrix.

[0144] Afterwards, a replica matrix of the above initial probability distribution matrix can be generated, and the corresponding elements of the replica matrix and the second conflict resolution matrix can be multiplied to obtain a first product matrix; the first product matrix contains a second column vector of 1*N corresponding to each of the M incoming targets; and Y groups of joint attack strategies are determined according to the first product matrix. Among them, the joint attack strategy is used to describe the corresponding relationship between each combat weapon and its corresponding attack target. For example, taking M as 2 and N as 3 as an example, assuming that the preferred combat weapon for incoming target 1 is combat weapon 1, and the preferred combat weapons for incoming target 2 are combat weapons 2 and combat weapons 3, the first group of joint attack strategies obtained can be: combat weapon 1-incoming target 1, combat weapon 2-incoming target 2; another group of joint attack strategies can be: combat weapon 1-incoming target 1, combat weapon 3-incoming target 2);

[0145] Afterwards, a final joint attack strategy (without any weapon conflict) can be generated based on the Y group of joint attack strategies; the second conflict resolution matrix can be updated according to the final joint attack strategy. Specifically, the target position of the characteristic element corresponding to the preferred combat weapon for each incoming target in the final joint attack strategy can be determined (the incoming target can determine the column number of the characteristic element, and the preferred combat weapon can determine the row number of the characteristic element, thereby locating a target position). Afterwards, the element corresponding to the target position in the second conflict resolution matrix can be updated to the first preset value 1, and the other elements except the target position can be updated to The second preset value is 0, and the third conflict resolution matrix is ​​obtained; finally, the third conflict resolution matrix can be used to perform conflict resolution processing on the initial probability distribution matrix (for example: multiplying the third conflict resolution matrix by the above initial probability distribution matrix by corresponding elements) to obtain a target probability distribution matrix that represents the optimized combat strategy (the target probability distribution matrix is ​​also an N*M matrix, the number of rows of the matrix is ​​equal to the total number N of combat weapons deployed on the combat entity, and the number of columns is equal to the total number M of at least one incoming target for the combat entity, and each element in the target probability distribution matrix is ​​used to describe the final attack probability of each combat weapon against each incoming target).

[0146] The value network includes a second encoding layer, a second decision layer and a fully connected layer; the second encoding layer can encode the overall combat effectiveness parameters of each time step to obtain a coding matrix; the second decision layer can decide on a value matrix indicating the status of each weapon with respect to each real-time incoming target based on the coding matrix and the characteristic parameters of the incoming target; then, the above-mentioned fully connected layer can map the above-mentioned value matrix into a state value scalar, that is, obtain the first expected value.

[0147] The following is a description of the process of updating and iterating the strategy parameters and value parameters of the combat decision model to be trained:

[0148] In step S110, the environmental state of each time step is sequentially input into the combat decision model to be trained to obtain combat data for each environmental state sequentially output by the combat decision model to be trained.

[0149] In this step, as described above, after the environmental state input value of each time step is input to the combat decision model to be trained, the combat decision model to be trained can output the combat data for the environmental state of each time step in sequence.

[0150] It should be noted that the combat data may include the combat strategies and first expected values ​​corresponding to the above-mentioned time steps. In addition, the combat data may also include the reward value of the combat strategy for each time step; the new environmental state at the next time step (that is, the environmental state after the change after the combat strategy output at the current time is executed); and the second expected value for the above-mentioned new environmental state.

[0151] Among them, the reward value of the above combat strategy for each time step is determined by the following method:

[0152] Multiple pre-configured reward and punishment conditions can be obtained; each combat strategy is matched with each reward and punishment condition to determine the reward and punishment parameters corresponding to each reward and punishment condition; and the reward value is determined according to the weighted result of multiple reward and punishment parameters. Exemplarily, a reward function can be established, and the reward function can include two aspects of reward and punishment conditions. The first aspect is the reward and punishment conditions established based on the combat strategy itself, and the second aspect is the reward and punishment conditions established based on the combat effect of the combat strategy.

[0153] First, the above-mentioned reward and punishment conditions (including 3 items) are explained in detail:

[0154] ① Rewards for coordinated strikes of weapons (i.e. the above-mentioned combat weapons): According to the rules of tactics and methods in prior knowledge, there are specific types of weapon coordinated strikes with effect gains / effect reductions (including hard-hard weapon coordination and hard-soft weapon coordination). Rewards / penalties are given to the weapon combinations generated by the decision based on the following formula 1:

[0155]

[0156] Here, according to experience, a1=10, b=5 can be set.

[0157] ② Weapon allocation overflow: In order to avoid wasting resources, it is necessary to set an upper limit n on the number of combat weapons allocated to a single incoming target. maxWhen the allocated amount exceeds the upper limit, a penalty will be imposed. The specific calculation method is:

[0158] Traverse all specific weapons in the ship defense weapon system and calculate the allocation overflow penalty value r for each specific weapon based on the following formula 2 @ :

[0159]

[0160] Here, based on experience, c=3 can be set;

[0161] After calculating the allocation overflow penalty value for each weapon, the allocation overflow penalties for all specific weapons can be summed based on the following formula 3:

[0162]

[0163] Where N is the number of specific weapons.

[0164] ③Weapon resource consumption: After each missile is launched by a weapon unit, a resource consumption penalty will be incurred based on the following formula 4:

[0165] R3=-d Formula 4

[0166] Here, d=1 is set based on experience.

[0167] The following is a detailed explanation of the above-mentioned second aspect of reward and punishment conditions (including 3 items):

[0168] ① Whether the enemy's incoming target is successfully intercepted: At a certain time step, when the enemy's incoming target in the environment is shot down, the following formula 5 is used to determine which weapon(s) in our weapon system produced the killing effect:

[0169] (1) For a missile launched by our side, when the state parameters of the missile and the state parameters of the enemy's incoming target fail at the same time in the simulation environment, it is determined that the missile has produced an interception effect;

[0170] (2) For close-in defense guns fired by our side, determine which close-in defense gun of our side produced the interception effect based on the time range of the killing distance limit;

[0171]

[0172] Here we set e=20 and f=15 based on experience.

[0173] ② Whether the enemy's incoming target is successfully interfered: At a certain moment, when the enemy's incoming target in the environment is interfered to deviate from our fleet, it means that the enemy's incoming target is successfully interfered by the soft weapon launched by us. The soft weapon interference reward can be determined based on the following formula 6:

[0174] R5=g Formula 6

[0175] Here, g=10 is assumed based on experience.

[0176] ③ The enemy's incoming target is not affected: At each moment when the enemy's incoming target is not affected, determine whether our weapons are assigned to the incoming target based on the time range of the kill distance limit: If yes, it means that our weapon has not caused an interception effect on the enemy's incoming target, and a certain penalty is given based on the following formula 7:

[0177] R6=-h Formula 7

[0178] Here, h=5 is set based on experience.

[0179] After calculating the reward and punishment parameters for each of the above reward and punishment factors, the weighted sum of each of the above rewards can be used to obtain the total reward value corresponding to each time step based on the following formula 8:

[0180]

[0181] It should be noted that when any reward condition is not matched in each time step, the reward R i = 0. i It is the weight or scaling factor of each reward, which can adjust the importance of each reward in the learning and training process. Here, according to experience, q1=1, q2=1.5, q3=1, q4=0.5, q5=1, q6=1 can be set. The present disclosure does not make any special limitation on this.

[0182] In step S120, the loss value of the combat decision model to be trained is determined based on the combat data of each time step.

[0183] In this step, it can be determined whether the total number of the current accumulated multiple time steps reaches the specified value; after the total number reaches the specified value, the loss value of the combat decision model to be trained can be determined based on the combat data of each time step.

[0184] refer to Figure 2 , Figure 2 A flow chart showing how to determine the loss value of the combat decision model to be trained according to the combat data at each time step in the embodiment of the present disclosure includes steps S201 to S203:

[0185] In step S201 , a value advantage estimation value for each time step is calculated based on a first expected value for an environmental state and a second expected value for a new environmental state.

[0186] In this step, the estimated value advantage for each time step can be determined by the following formula 9 and formula 10:

[0187]

[0188] δ t =r t +γV(s t+1 )-V(s t ) Formula 10

[0189] Among them, t represents the serial number of each time step (the value of t will be reinitialized in each round), represents the estimated value of the value advantage corresponding to time step t, T represents the serial number of the maximum time step of each round, l represents the index value that increases by one from 0 to T, γ represents the discount factor, and λ represents the smoothing parameter;

[0190] δ t represents the temporal difference error corresponding to time step t, r t represents the reward value corresponding to time step t, V(s t ) represents the first expected value corresponding to the environmental state at time step t, V(s t+1 ) represents the second expected value corresponding to the environmental state at time step t+1.

[0191] In step S202, the strategy probability ratio for each time step is obtained.

[0192] In this step, the strategy probability ratio for each time step can be determined by the following formula 11:

[0193]

[0194] Among them, ratio t (θ) represents the strategy probability ratio at time step t;

[0195] π θ (a t |s t ) represents the time step t, and when the current policy parameter θ is adopted, in the environment state s t Adopt a combat strategy t The first probability of

[0196] Represents the time step t, and takes the old policy parameters θ old When the environment state s t Adopt a combat strategy t The second probability of

[0197] The old strategy parameters are the strategy parameters before the last round of adjustment, and the current strategy parameters are the strategy parameters after the last round of adjustment.

[0198] In step S203, the policy loss of the policy network is determined based on the value advantage estimate for each time step and the policy probability ratio for each time step.

[0199] In this step, you can refer to Figure 3 , Figure 3 A flow chart showing how to determine the policy loss of a policy network according to the value advantage estimation value for each time step and the policy probability ratio for each time step in an embodiment of the present disclosure includes steps S301-S302:

[0200] In step S301, the strategy loss component corresponding to each time step is determined based on the value advantage estimate for each time step and the strategy probability ratio for each time step.

[0201] In this step, we can To determine the strategy loss component corresponding to each time step t, the clip function is used to ensure the ratio t The value of (θ) is limited to [1-∈, 1+∈] to prevent the update amplitude from being too large. The min function represents the value of and The smaller value of the two is used as the policy loss component corresponding to each time step t, and ∈ represents the truncation hyperparameter, which is used to limit the update amplitude.

[0202] In step S302, the policy loss of the policy network is determined according to the expected values ​​of multiple policy loss components corresponding to multiple time steps.

[0203] In this step, the policy loss of the policy network can be calculated by the following formula 12:

[0204]

[0205] in, represents the expectation of multiple strategy loss components with values ​​ranging from 0 to T for time step t; represents the strategy loss component corresponding to time step t; the clip function is used to ensure ratio t The value of (θ) is limited to [1-∈, 1+∈], and the min function represents the value of The smaller value of the two is used as the policy loss component corresponding to each time step t.

[0206] Next reference Figure 2 In step S204, the target value for each environmental state is calculated according to the reward value of the combat strategy for each time step, and the value loss of the value network is determined according to the first expected value of each environmental state and the target value for each environmental state.

[0207] In this step, the target value for each environmental state can be calculated by the following formula 13:

[0208]

[0209] Among them, V t target represents the target value for time step t, r t+k Represents the reward value corresponding to the combat strategy output at time t+k, and the value of k increases from 0 to Tt.

[0210] Afterwards, the value loss of the value network can be calculated based on the following formula 14:

[0211]

[0212] Among them, L VF (θ) represents the above-mentioned loss of value; represents the expectation of multiple first target values ​​from 0 to T for time step t, where the first target value refers to (V(s t )-V t t89:；t ) @ .

[0213] After calculating the policy loss of the policy network through step S202 and calculating the value loss of the value network through step S204, step S205 can be entered to determine the loss value of the combat decision model to be trained based on the policy loss and the value loss.

[0214] In this step, the loss value of the combat decision model to be trained can be determined based on the following formulas 15 and 16:

[0215]

[0216] H(π θ )=-∑8π θ (a|s t )log π θ (a|s t ) Formula 16

[0217] Among them, L CLIP (θ) represents the policy loss, L VF (θ) represents the value loss;

[0218] It represents the expectation of multiple second target values ​​with values ​​from 0 to T for time step t. Each second target value refers to the L corresponding to each time step t. CLIP (θ)-c1L VF (θ)+c@ H(π θ );

[0219] c1 and c @ Represents the weight hyperparameter, which is used to balance the contribution of each part. c1 can be 0.9, c @ It can be set to 0.9, and can be set according to actual conditions, and the present disclosure does not make any special restrictions on this;

[0220] H(π θ ) represents the strategy entropy. The combat strategy contains multiple sub-strategies (assuming that the combat strategy corresponds to an N*M matrix, the matrix contains N*M elements, corresponding to N*M sub-strategies). a represents each sub-strategy in the combat strategy generated corresponding to the environmental state at time step t. π θ (a|s t ) represents the time step t, and when the current policy parameter θ is adopted, in the environment state s t The third probability of taking each sub-strategy a.

[0221] After calculating the loss value of the combat decision model to be trained, you can then refer to Figure 1 In step S130, the policy parameters corresponding to the policy network and the value parameters corresponding to the value network are iterated according to the loss value to obtain a trained combat decision model.

[0222] In this step, refer to Figure 4 , Figure 4 A flow chart showing how to perform parameter iteration on the policy parameters corresponding to the policy network and the value parameters corresponding to the value network according to the loss value to obtain a trained combat decision model in the embodiment of the present disclosure includes steps S401-S402:

[0223] In step S401, according to the loss value, the gradient descent algorithm is used to update the policy parameters corresponding to the policy network and the value parameters corresponding to the value network.

[0224] In this step, the policy parameters corresponding to the policy network and the value parameters corresponding to the value network can be updated using the gradient descent algorithm according to the loss value L(θ).

[0225] In step S402, when the preset training termination condition is met, a trained combat decision model is obtained.

[0226] In this step, when the number of parameter iterations reaches a preset threshold, for example, 5000 times (which can be set according to actual conditions), or when the average reward value of multiple rounds (a training round refers to a process starting from an initial state and going through a series of actions and environmental feedback until a certain terminal state is reached, for example, when playing an electronic game, a round is the entire process from the beginning of the game to the end of the game, such as reaching the end point or exhausting the life value), the update and iteration process of the above-mentioned strategy parameters and value parameters can be stopped to obtain the above-mentioned trained combat decision model.

[0227] After obtaining the above-mentioned trained combat decision model, the above-mentioned trained combat decision model can be deployed to the actual application scenario, so that the trained combat decision model can decide the combat strategy for the current time step according to the real-time environmental status (including weapon status, status of incoming targets, etc.), and generate a mask matrix based on the strategy mask layer to filter out infeasible or conflicting decisions, and finally output a decision matrix that is adapted to the actual situation. The decision matrix can maximize the combined defense capability of the weapon. After executing the decision, the environmental status can be updated to make the next decision.

[0228] refer to Figure 5 , Figure 5 The overall architecture diagram of the training method of the combat decision model in the embodiment of the present disclosure is shown as follows: Figure 5 As shown:

[0229] The combat decision model to be trained may include a strategy network and a value network; the encoding layer of the strategy network may output a coding matrix, the decision layer may output an initial combat strategy, and the strategy mask layer may resolve conflicts on the initial combat strategy to obtain a final combat strategy;

[0230] After outputting the combat strategy for each time step, the relevant combat data can be cached in the experience pool, and the reward value for each combat strategy decision can be calculated;

[0231] When the number of interactions between the combat decision model to be trained and the environment reaches the specified number, the data in the experience pool can be used as training samples, and the loss value of the combat decision model to be trained can be calculated based on PPO-Clip, and the policy parameters of the above-mentioned policy network and the value parameters of the above-mentioned value network can be updated using the loss value;

[0232] After each round of training, initialize the environment and the above experience pool;

[0233] Until the preset training termination condition is met, the policy parameters of the policy network and the value parameters of the value network are stopped from being updated to obtain a trained combat decision model;

[0234] Among them, PPO-Clip, the full name of Proximal Policy Optimization with Clipping, is a variant of the Proximal Policy Optimization (PPO) algorithm. PPO is a reinforcement learning algorithm improved on the basis of policy gradient, which aims to solve common problems in policy gradient methods, such as unstable training and slow convergence. PPO improves the stability and efficiency of the algorithm by introducing a technology called "approximate policy optimization" and some special techniques such as clipping and multi-step objectives.

[0235] The present disclosure also provides a training device for a combat decision model. Figure 6 A schematic diagram showing the structure of a training device for a combat decision model in an exemplary embodiment of the present disclosure; Figure 6 As shown, the training device 600 for the combat decision model may include a combat data generation module 610, a loss value determination module 620 and a parameter iteration module 630. Among them:

[0236] The combat data generation module 610 is used to sequentially input the environmental state of each time step into the combat decision model to be trained, so as to obtain the combat data for each environmental state sequentially output by the combat decision model to be trained; the combat decision model to be trained comprises a strategy network and a value network, the strategy network is used to output the combat strategy for the environmental state, and the value network is used to output the first expected value for the environmental state; the combat data comprises the combat strategy and the first expected value;

[0237] A loss value determination module 620, used to determine the loss value of the combat decision model to be trained according to the combat data of each time step;

[0238] The parameter iteration module 630 is used to perform parameter iteration on the strategy parameters corresponding to the strategy network and the value parameters corresponding to the value network according to the loss value to obtain the trained combat decision model.

[0239] In an exemplary embodiment of the present disclosure, the loss value determination module 620 determines the loss value of the combat decision model to be trained according to the combat data of each time step, including:

[0240] Determine whether the total number of the multiple time steps currently accumulated reaches a specified value;

[0241] After the total number reaches the specified value, the loss value of the combat decision model to be trained is determined according to the combat data of each time step.

[0242] In an exemplary embodiment of the present disclosure, the combat data for each of the environmental states further includes:

[0243] The reward value of the combat strategy for each time step;

[0244] a new environment state at the next time step, wherein the new environment state is obtained by updating the environment state after executing the combat strategy;

[0245] A second expected value for the new environmental state.

[0246] In an exemplary embodiment of the present disclosure, the reward value of the combat strategy at each time step is determined in the following manner:

[0247] Get multiple pre-configured reward and punishment conditions;

[0248] Matching each of the combat strategies with each of the reward and punishment conditions, and determining reward and punishment parameters corresponding to each of the reward and punishment conditions;

[0249] The reward value is determined according to the weighted result of multiple reward and punishment parameters.

[0250] In an exemplary embodiment of the present disclosure, the loss value determination module 620 determines the loss value of the combat decision model to be trained according to the combat data of each time step, including:

[0251] Calculate a value advantage estimate for each time step according to a first expected value for the environmental state and a second expected value for the new environmental state;

[0252] Obtaining the strategy probability ratio for each time step;

[0253] Determining a policy loss of the policy network according to the value advantage estimate for each time step and the policy probability ratio for each time step;

[0254] Calculating the target value for each of the environmental states according to the reward value of the combat strategy for each of the time steps, and determining the value loss of the value network according to the first expected value of each of the environmental states and the target value for each of the environmental states;

[0255] The loss value of the combat decision model to be trained is determined according to the strategy loss and the value loss.

[0256] In an exemplary embodiment of the present disclosure, the loss value determination module 620 calculates the value advantage estimation value for each time step according to the first expected value for the environmental state and the second expected value for the new environmental state, including:

[0257] The value advantage estimate for each time step is determined by the following formula:

[0258]

[0259] δ t =r t +γV(s t+1 )-V(s t )

[0260] Wherein, t represents the sequence number of each time step, represents the estimated value of the value advantage corresponding to time step t, T represents the serial number of the maximum time step of each round, l represents the index value that increases by one from 0 to T, γ represents the discount factor, and λ represents the smoothing parameter;

[0261] δ t represents the temporal difference error corresponding to time step t, r t represents the reward value corresponding to time step t, V(s t ) represents the first expected value corresponding to the environmental state at time step t, V(s t+1 ) represents the second expected value corresponding to the environmental state at time step t+1.

[0262] In an exemplary embodiment of the present disclosure, the strategy probability ratio for each time step is determined by the following formula:

[0263]

[0264] Among them, ratio t (θ) represents the probability ratio of the strategy at time step t;

[0265] π θ (a t |s t ) represents the time step t, and when the current policy parameter θ is adopted, in the environment state s t Adopt the above mentioned combat strategy a t The first probability of

[0266] Represents the time step t, and takes the old policy parameters θ old When the environment state s t Adopt the above mentioned combat strategy at The second probability of

[0267] The old policy parameters are the policy parameters before the last round of adjustment, and the current policy parameters are the policy parameters after the last round of adjustment.

[0268] In an exemplary embodiment of the present disclosure, the loss value determination module 620 determines the policy loss of the policy network according to the value advantage estimation value for each time step and the policy probability ratio for each time step, including:

[0269] Determining a strategy loss component corresponding to each time step according to the value advantage estimate for each time step and the strategy probability ratio for each time step;

[0270] The policy loss of the policy network is determined according to the expected values ​​of the multiple policy loss components corresponding to the multiple time steps.

[0271] In an exemplary embodiment of the present disclosure, the loss value determination module 620 determines the policy loss of the policy network according to the expected values ​​of the multiple policy loss components corresponding to the multiple time steps, including:

[0272] The policy loss of the policy network is calculated by the following formula:

[0273]

[0274] in, represents the expectation of multiple strategy loss components with values ​​ranging from 0 to T for time step t;

[0275] Represents the strategy loss component corresponding to time step t;

[0276] The clip function is used to ensure the ratio t The value of (θ) is limited to [1-∈, 1+∈], and the min function represents the value of and The smaller value of the two is used as the policy loss component corresponding to each time step t, and ∈ represents the truncation hyperparameter.

[0277] In an exemplary embodiment of the present disclosure, the loss value determination module 620 calculates the target value for each of the environmental states according to the reward value of the combat strategy for each of the time steps, including:

[0278] The target value for each of the environmental states is calculated by the following formula:

[0279]

[0280] Among them, V t t89:；t represents the target value for time step t, r t+< Represents the reward value corresponding to the combat strategy output at time t+k, and the value of k increases from 0 to Tt.

[0281] In an exemplary embodiment of the present disclosure, the loss value determination module 620 determines the value loss of the value network according to the first expected value of each of the environmental states and the target value for each of the environmental states, including:

[0282] The value loss of the value network is calculated based on the following formula:

[0283]

[0284] Among them, L VF (θ) represents the above-mentioned loss of value; represents the expectation of multiple first target values ​​from 0 to T for time step t, where the first target value refers to (V(s t )-V t t89:；t ) @ .

[0285] In an exemplary embodiment of the present disclosure, the loss value determination module 620 determines the loss value of the combat decision model to be trained according to the strategy loss and the value loss, including:

[0286] The loss value of the combat decision model to be trained is determined based on the following formula:

[0287]

[0288] Among them, L CLIP (θ) represents the strategy loss, L VF (θ) represents the said loss of value;

[0289] represents the expectation of multiple second target values ​​with values ​​from 0 to T for time step t, each of which refers to the L corresponding to each time step t CLIP (θ)-c1L VF (θ)+c2H(π θ );

[0290] c1 and c @ represents the weight hyperparameter;

[0291] H(π θ) represents the strategy entropy, the combat strategy contains multiple sub-strategies, a represents each sub-strategy in the combat strategy generated corresponding to the environmental state at time step t, π θ (a|s t ) represents the time step t, and when the current policy parameter θ is adopted, in the environment state s t The third probability of taking each sub-strategy a.

[0292] In an exemplary embodiment of the present disclosure, the parameter iteration module 630 performs parameter iteration on the policy parameters corresponding to the policy network and the value parameters corresponding to the value network according to the loss value to obtain the trained combat decision model, including:

[0293] According to the loss value, the policy parameters corresponding to the policy network and the value parameters corresponding to the value network are updated using a gradient descent algorithm;

[0294] When a preset training termination condition is met, the trained combat decision model is obtained;

[0295] The training termination condition includes that the number of parameter iterations reaches a preset threshold, or the average reward value of multiple rounds tends to converge.

[0296] The specific details of each module in the training device of the above-mentioned combat decision model have been described in detail in the corresponding training method of the combat decision model, so they will not be repeated here.

[0297] It should be noted that, although several modules or units of the device for action execution are mentioned in the above detailed description, this division is not mandatory. In fact, according to the embodiments of the present disclosure, the features and functions of two or more modules or units described above can be embodied in one module or unit. On the contrary, the features and functions of one module or unit described above can be further divided into multiple modules or units to be embodied.

[0298] In addition, although the steps of the method in the present disclosure are described in a specific order in the drawings, this does not require or imply that the steps must be performed in this specific order, or that all the steps shown must be performed to achieve the desired results. Additionally or alternatively, some steps may be omitted, multiple steps may be combined into one step, and / or one step may be decomposed into multiple steps, etc.

[0299] Through the description of the above implementation, it is easy for those skilled in the art to understand that the example implementation described here can be implemented by software, or by software combined with necessary hardware. Therefore, the technical solution according to the implementation of the present disclosure can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (which can be a CD-ROM, a USB flash drive, a mobile hard disk, etc.) or on a network, including several instructions to enable a computing device (which can be a personal computer, a server, a mobile terminal, or a network device, etc.) to execute the method according to the implementation of the present disclosure.

[0300] The present disclosure also provides a computer-readable storage medium, which may be included in the electronic device described in the above embodiments; or may exist independently without being assembled into the electronic device.

[0301] The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus or device, or any combination thereof. More specific examples of computer-readable storage media may include, but are not limited to, an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof. In the present disclosure, a computer-readable storage medium may be any tangible medium containing or storing a program that may be used by or in conjunction with an instruction execution system, apparatus, or device.

[0302] Computer-readable storage media can send, propagate or transmit programs for use by or in conjunction with an instruction execution system, apparatus or device. The program code contained on the computer-readable storage medium can be transmitted using any appropriate medium, including but not limited to: wireless, wire, optical cable, RF, etc., or any suitable combination of the above.

[0303] The computer-readable storage medium carries one or more programs. When the one or more programs are executed by an electronic device, the electronic device implements the method described in the above embodiments.

[0304] In addition, an electronic device capable of implementing the above method is also provided in an embodiment of the present disclosure.

[0305] Those skilled in the art will appreciate that various aspects of the present disclosure may be implemented as systems, methods or program products. Therefore, various aspects of the present disclosure may be specifically implemented in the following forms, namely: complete hardware implementation, complete software implementation (including firmware, microcode, etc.), or a combination of hardware and software, which may be collectively referred to herein as "circuits", "modules" or "systems".

[0306] Refer to the following Figure 7 An electronic device 700 according to this embodiment of the present disclosure is described. Figure 7 The electronic device 700 shown is merely an example and should not bring any limitation to the functions and scope of use of the embodiments of the present disclosure.

[0307] like Figure 7 As shown, the electronic device 700 is in the form of a general computing device. The components of the electronic device 700 may include, but are not limited to: at least one processor 710, at least one memory 720, a bus 730 connecting different system components (including the memory 720 and the processor 710), and a display 740.

[0308] The memory stores program codes, which can be executed by the processor 710, so that the processor 710 executes the steps according to various exemplary embodiments of the present disclosure described in the above “Exemplary Method” section of this specification. For example, the processor 710 can execute the following steps: Figure 1 As shown in: Step S110, the environmental state of each time step is sequentially input into the combat decision model to be trained to obtain the combat data for each environmental state sequentially output by the combat decision model to be trained; the combat decision model to be trained includes a strategy network and a value network, the strategy network is used to output the combat strategy for the environmental state at each time step, and the value network is used to output the first expected value for the environmental state; the combat data includes the combat strategy and the first expected value; Step S120, according to the combat data of each time step, the loss value of the combat decision model to be trained is determined; Step S130, according to the loss value, the strategy parameters corresponding to the strategy network and the value parameters corresponding to the value network are iterated to obtain the trained combat decision model.

[0309] The memory 720 may include a readable medium in the form of a volatile storage, such as a random access memory (RAM) 7201 and / or a cache memory 7202 , and may further include a read-only memory (ROM) 7203 .

[0310] The memory 720 may also include a program / utility 7204 having a set (at least one) of program modules 7205, such program modules 7205 including but not limited to: an operating system, one or more application programs, other program modules, and program data, each of which or some combination may include an implementation of a network environment.

[0311] Bus 730 may be a bus representing one or more of several types of bus structures, including a memory bus or memory controller, a peripheral bus, an accelerated graphics port, a processor, or a local bus using any of a variety of bus architectures.

[0312] The electronic device 700 may also communicate with one or more external devices 800 (e.g., keyboards, pointing devices, Bluetooth devices, etc.), may also communicate with one or more devices that enable a user to interact with the electronic device 700, and / or communicate with any device that enables the electronic device 700 to communicate with one or more other computing devices (e.g., routers, modems, etc.). Such communication may be performed via an input / output (I / O) interface 750. Furthermore, the electronic device 700 may also communicate with one or more networks (e.g., local area networks (LANs), wide area networks (WANs), and / or public networks, such as the Internet) via a network adapter 760. As shown, the network adapter 760 communicates with other modules of the electronic device 700 via a bus 730. It should be understood that, although not shown in the figure, other hardware and / or software modules may be used in conjunction with the electronic device 700, including but not limited to: microcode, device drivers, redundant processors, external disk drive arrays, RAID systems, tape drives, and data backup storage systems, etc.

[0313] Those skilled in the art will readily appreciate other embodiments of the present disclosure after considering the specification and practicing the invention disclosed herein. The present disclosure is intended to cover any variations, uses, or adaptations of the present disclosure that follow the general principles of the present disclosure and include common knowledge or customary technical means in the art that are not disclosed in the present disclosure. The description and examples are to be considered exemplary only, and the true scope and spirit of the present disclosure are indicated by the claims.

Claims

1. A training method for a combat decision model, characterized in that: include: Inputting the environmental state of each time step into the combat decision model to be trained in sequence to obtain combat data for each environmental state outputted in sequence by the combat decision model to be trained; the combat decision model to be trained comprises a strategy network and a value network, the strategy network is used to output the combat strategy for the environmental state of each time step, and the value network is used to output the first expected value for the environmental state; the combat data comprises the combat strategy and the first expected value; Determining the loss value of the combat decision model to be trained according to the combat data of each time step; According to the loss value, parameter iteration is performed on the strategy parameters corresponding to the strategy network and the value parameters corresponding to the value network to obtain the trained combat decision model.

2. The method according to claim 1, characterized in that Determining the loss value of the combat decision model to be trained according to the combat data of each time step includes: Determine whether the total number of the multiple time steps currently accumulated reaches a specified value; After the total number reaches the specified value, the loss value of the combat decision model to be trained is determined according to the combat data of each time step.

3. The method according to claim 1 or 2, characterized in that: The combat data for each of the environmental states also includes: The reward value of the combat strategy for each time step; a new environment state at the next time step, wherein the new environment state is obtained by updating the environment state after executing the combat strategy; A second expected value for the new environmental state.

4. The method according to claim 3, characterized in that The reward value of the combat strategy at each time step is determined in the following way: Get multiple pre-configured reward and punishment conditions; Matching each of the combat strategies with each of the reward and punishment conditions, and determining reward and punishment parameters corresponding to each of the reward and punishment conditions; The reward value is determined according to the weighted result of multiple reward and punishment parameters.

5. The method according to claim 3, characterized in that: Determining the loss value of the combat decision model to be trained according to the combat data of each time step includes: Calculate a value advantage estimate for each time step according to a first expected value for the environmental state and a second expected value for the new environmental state; Obtaining the strategy probability ratio for each time step; Determining a policy loss of the policy network according to the value advantage estimate for each time step and the policy probability ratio for each time step; Calculating the target value for each of the environmental states according to the reward value of the combat strategy for each of the time steps, and determining the value loss of the value network according to the first expected value of each of the environmental states and the target value for each of the environmental states; The loss value of the combat decision model to be trained is determined according to the strategy loss and the value loss.

6. The method according to claim 5, characterized in that The calculating the value advantage estimation value for each time step according to the first expected value for the environmental state and the second expected value for the new environmental state comprises: The value advantage estimate for each time step is determined by the following formula: δ t =r t +γV(s t+1 )-V(s t ) Wherein, t represents the sequence number of each time step, represents the estimated value advantage corresponding to time step t, T represents the serial number of the maximum time step of each round, l represents the index value that increases by one from 0 to T, γ represents the discount factor, and λ represents the smoothing parameter; δ t represents the temporal difference error corresponding to time step t, r t represents the reward value corresponding to time step t, V(s t ) represents the first expected value corresponding to the environmental state at time step t, V(s t+1 ) represents the second expected value corresponding to the environmental state at time step t+1.

7. The method according to claim 5, characterized in that The strategy probability ratio for each time step is determined by the following formula: Among them, ratio t (θ) represents the probability ratio of the strategy at time step t; π θ (a t | t ) represents the time step t, and when the current policy parameter θ is adopted, in the environment state s t Adopt the above mentioned combat strategy a t The first probability of Represents the time step t, and takes the old policy parameters θ ％l& When the environment state s t Adopt the above mentioned combat strategy a t The second probability of The old policy parameters are the policy parameters before the last round of adjustment, and the current policy parameters are the policy parameters after the last round of adjustment.

8. A training device for a combat decision model, characterized in that: include: A combat data generation module, used to sequentially input the environmental state of each time step into the combat decision model to be trained, so as to obtain combat data for each environmental state sequentially output by the combat decision model to be trained; the combat decision model to be trained comprises a strategy network and a value network, the strategy network is used to output the combat strategy for the environmental state, and the value network is used to output the first expected value for the environmental state; the combat data comprises the combat strategy and the first expected value; A loss value determination module, used to determine the loss value of the combat decision model to be trained according to the combat data of each time step; A parameter iteration module is used to perform parameter iteration on the strategy parameters corresponding to the strategy network and the value parameters corresponding to the value network according to the loss value to obtain the trained combat decision model.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the training method of the combat decision model described in any one of claims 1 to 7 is implemented.

10. An electronic device, characterized in that: include: processor; as well as A memory, configured to store executable instructions of the processor; Wherein, the processor is configured to execute the combat decision model training method described in any one of claims 1 to 7 by executing the executable instructions.

Images