A method for generating target search strategies for unmanned combat vehicles

Abstract

The present disclosure relates to a method for generating a target search strategy for an unmanned combat vehicle (UCV), comprising: constructing an UCV target search strategy generation model, the UCV target search strategy generation model comprising a set of states of multiple UCVs at any given time, a set of actions of the multiple UCVs at any given time, and environmental feedback received by the multiple UCVs; training the UCV target search strategy generation model using a meta-reinforcement learning algorithm; incorporating a weighted loss function into the training process to obtain target search strategies for the multiple UCVs; and deploying the obtained target search strategies for the multiple UCVs in a real mission environment. This method can address the biased estimation problem of the strategy model in the meta-reinforcement learning algorithm and improve the efficiency and quality of generating target search action strategies for UCVs.

Description

Technical Field

[0001] The present disclosure relates to the field of military information technology, and in particular to a method for generating a target search strategy for an unmanned combat vehicle. Background Art

[0002] Unmanned combat vehicles (UCVs) are a crucial force in future intelligent warfare and will play a key role on future battlefields. However, they are not yet truly unmanned. When operating outdoors, soldiers still need to carry remote controls or use satellite vehicles for remote control.

[0003] With the development of artificial intelligence technology, reinforcement learning-based strategy generation methods have become increasingly popular in recent years. This approach directly learns strategies through continuous "interactive error" in adversarial task environments, demonstrating strong robustness and versatility. Meta-reinforcement learning algorithms, as a typical deep reinforcement learning method for solving continuous state-action space tasks, can effectively address the generation of action strategies for unmanned combat vehicles (UCVs) in high-dimensional, highly dynamic battlefield environments. They are more consistent with the characteristics and patterns of action strategy generation for UCVs. However, a significant problem with meta-reinforcement learning algorithms is the biased estimation of the policy model, which directly prevents UCVs from learning effective strategies in complex task environments. Therefore, addressing the biased estimation of the policy model in meta-reinforcement learning algorithms and improving the efficiency and quality of target search action strategy generation for UCVs under limited computing resources has become a pressing issue. Therefore, it is necessary to address one or more of the issues mentioned above in order to address the biased estimation of the policy model.

[0004] It should be noted that the information disclosed in the above background technology section is only used to enhance the understanding of the background of the present disclosure, and therefore may include information that does not constitute prior art known to ordinary technicians in the field. Summary of the Invention

[0005] The purpose of the embodiments of the present disclosure is to provide a method for generating a target search strategy for an unmanned combat vehicle, which can solve the problem of biased estimation of the strategy model and improve the efficiency and quality of generating the target search action strategy for the unmanned combat vehicle.

[0006] The present disclosure provides a method for generating a target search strategy for an unmanned combat vehicle, the method comprising the following steps:

[0007] Constructing an unmanned combat vehicle target search strategy generation model, wherein the unmanned combat vehicle target search strategy generation model includes a state set of multiple unmanned combat vehicles at any time, a set of actions of multiple unmanned combat vehicles at any time, and environmental feedback obtained by multiple unmanned combat vehicles;

[0008] The target search strategy generation model of the unmanned combat vehicle is trained using a meta-reinforcement learning algorithm; a weighted loss function is incorporated into the training process to obtain target search strategies for multiple unmanned combat vehicles;

[0009] The target search strategies obtained for multiple unmanned combat vehicles are deployed in a real mission environment.

[0010] In an exemplary embodiment of the present disclosure, the step of constructing a target search strategy generation model for an unmanned combat vehicle includes: a set of environmental states s of a plurality of said unmanned combat vehicles at any time t t ; The plurality of unmanned combat vehicles are set according to the environmental state s t The action set a is composed of the corresponding actions in the corresponding states t ; and the environmental feedback r obtained by the plurality of said unmanned combat vehicles t .

[0011] In an exemplary embodiment of the present disclosure, the environment state set s t The formulas include:

[0012] s t ={(x1,y1),(x2,y2),(x3,y3),...,(x n ,y n )}

[0013] Among them, x and y represent the horizontal and vertical coordinates of the state of each unmanned combat vehicle, x∈[0,5000], y∈[0,5000]; n represents the number of the unmanned combat vehicle.

[0014] In an exemplary embodiment of the present disclosure, the action set a t The formulas include:

[0015] a t ={move1,move2,move3,move4,move5,move6,move7,move8}

[0016] Among them, move1 indicates that the unmanned combat vehicle maneuvers to the east; move2 indicates that the unmanned combat vehicle maneuvers to the southeast; move3 indicates that the unmanned combat vehicle maneuvers to the south; move4 indicates that the unmanned combat vehicle maneuvers to the southwest; move5 indicates that the unmanned combat vehicle maneuvers to the west; move6 indicates that the unmanned combat vehicle maneuvers to the northwest; move7 indicates that the unmanned combat vehicle maneuvers to the north; move8 indicates that the unmanned combat vehicle maneuvers to the northeast; the value range of the maneuvering action includes move∈[0,50], 0 indicates that the unmanned combat vehicle has not made any action, and 1-100 indicates the number of steps of the unmanned combat vehicle.

[0017] In an exemplary embodiment of the present disclosure, the environmental feedback r t It includes rewards for unmanned combat vehicles reaching any target location, penalties for unmanned combat vehicles touching obstacles, penalties for unmanned combat vehicles touching boundaries, and penalties for unmanned combat vehicles colliding with each other.

[0018] In an exemplary embodiment of the present disclosure, a meta-reinforcement learning algorithm is used to train the unmanned combat vehicle target search strategy generation model; in the step of incorporating a weighted loss function into the training process to obtain target search strategies for multiple unmanned combat vehicles, the training process of the unmanned combat vehicle target search strategy generation model includes multiple parameters, and the multiple parameters include:

[0019] The strategy generation model is π θ , the target search task distribution is D(T), the basic learning rate is α, the meta-learning rate is β, the outer loop time is M, and the number of samples is N.

[0020] In an exemplary embodiment of the present disclosure, a meta-reinforcement learning algorithm is used to train the target search strategy generation model of the unmanned combat vehicle. In the step of incorporating a weighted loss function into the training process to obtain the target search strategies of multiple unmanned combat vehicles, the calculation formula of the weighted loss function includes:

[0021]

[0022] in, To find the derivative symbol, represents the kth trajectory The weight of represents the kth trajectory The loss function of .

[0023] In an exemplary embodiment of the present disclosure, the process of obtaining target search strategies for multiple unmanned combat vehicles includes:

[0024] Initializing a plurality of said parameters;

[0025] Initialization time t=1;

[0026] Collect N task samples from the target search task distribution D(T);

[0027] For each task sample, generate model π according to the strategy θ , sample K trajectories

[0028] According to the trajectory and loss function Calculating gradients

[0029] Using gradient descent Get the optimized parameter θ'; get the new strategy generation model π θ' , sample C trajectories

[0030] Calculate the weight of each of the C trajectories in the loss function;

[0031] Using C tracks and weights To do a meta update:

[0032] Among them, ← means the assignment operation;

[0033] If the time t is equal to the outer loop time M, a target search strategy for multiple unmanned combat vehicles is obtained.

[0034] In an exemplary embodiment of the present disclosure, the process of obtaining target search strategies for multiple unmanned combat vehicles includes:

[0035] Initializing a plurality of said parameters;

[0036] Initialization time t=1;

[0037] Collect N task samples from the target search task distribution D(T);

[0038] For each task sample, generate model π according to the strategy θ , sample K trajectories

[0039] According to the trajectory and loss function Calculating gradients

[0040] Using gradient descent Get the optimized parameter θ'; get the new strategy generation model π θ' , sample C trajectories

[0041] Calculate the weight of each of the C trajectories in the loss function;

[0042] Using C tracks and weights To do a meta update:

[0043] Among them, ← means the assignment operation;

[0044] If the time t is not equal to the outer loop time M, it is necessary to return to step t+1 and generate a model π for each task sample according to the strategy. θ , sample K trajectories Continue with the training process.

[0045] In an exemplary embodiment of the present disclosure, the calculation formula for calculating the weight of each of the C trajectories in the loss function includes:

[0046]

[0047] in, w i represents the information entropy of the i-th trajectory; ∑ k w k represents the sum of information entropy of all sampled trajectories; l i represents the loss value of the i-th trajectory; l represents the average loss of all sampled trajectories.

[0048] The technical solutions provided by the present disclosure may have the following beneficial effects:

[0049] In an embodiment of the present disclosure, a method for generating a target search strategy for an unmanned combat vehicle is proposed. By introducing a weighted loss function in the process of training the target search strategy generation model of the unmanned combat vehicle using a meta-reinforcement learning algorithm, the biased estimation problem of the strategy model of the meta-reinforcement learning algorithm is solved to a certain extent, and the efficiency and quality of the target search action strategy generation of the unmanned combat vehicle are improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] The accompanying drawings are incorporated into and constitute a part of the specification, illustrate embodiments consistent with the present disclosure, and together with the specification, serve to explain the principles of the present disclosure. Obviously, the drawings described below are only some embodiments of the present disclosure, and those skilled in the art can derive other drawings based on these drawings without inventive effort.

[0051] Figure 1A schematic diagram showing the steps of a method for generating a target search strategy for an unmanned combat vehicle in an exemplary embodiment of the present disclosure;

[0052] Figure 2 A flow chart showing a method for generating a target search strategy for an unmanned combat vehicle in an exemplary embodiment of the present disclosure is provided;

[0053] Figure 3 A schematic diagram illustrating an exemplary embodiment of the present disclosure introducing a weighted objective function during the training process of a meta-reinforcement learning algorithm. DETAILED DESCRIPTION

[0054] Example embodiments will now be described more fully with reference to the accompanying drawings. However, example embodiments can be embodied in many forms and should not be construed as limited to the examples set forth herein; rather, these embodiments are provided so that this disclosure will be thorough and complete and will fully convey the concepts of the example embodiments to those skilled in the art. The described features, structures, or characteristics may be combined in any suitable manner in one or more embodiments.

[0055] In addition, the accompanying drawings are merely schematic illustrations of the present disclosure and are not necessarily drawn to scale. Identical reference numerals in the figures denote identical or similar parts, and thus repetitive descriptions thereof will be omitted. Some of the block diagrams shown in the accompanying drawings are functional entities that do not necessarily correspond to physically or logically separate entities. These functional entities may be implemented in software, in one or more hardware modules or integrated circuits, or in different networks and / or processor devices and / or microcontroller devices.

[0056] This example embodiment provides a method for generating a target search strategy for an unmanned combat vehicle. Figure 1 As shown, the method may include the following steps:

[0057] Step S101: constructing an unmanned combat vehicle target search strategy generation model, which includes a state set of multiple unmanned combat vehicles at any time, a set of actions of multiple unmanned combat vehicles at any time, and environmental feedback obtained by multiple unmanned combat vehicles;

[0058] Step S102: Using a meta-reinforcement learning algorithm to train the unmanned combat vehicle target search strategy generation model; introducing a weighted loss function into the training process to obtain target search strategies for multiple unmanned combat vehicles;

[0059] Step S103: Deploy the obtained target search strategies of multiple unmanned combat vehicles in a real mission environment.

[0060] In this embodiment, a method for generating a target search strategy for an unmanned combat vehicle is proposed. By introducing a weighted loss function in the process of training the unmanned combat vehicle target search strategy generation model using a meta-reinforcement learning algorithm, the biased estimation problem of the strategy model of the meta-reinforcement learning algorithm is solved to a certain extent, and the efficiency and quality of the generation of the target search action strategy of the unmanned combat vehicle are improved.

[0061] The following describes each step of the calculation method in this exemplary embodiment in more detail.

[0062] The Model Agnostic Meta Learning (MAML) algorithm, when solving the generation and optimization problems of unmanned combat vehicle target search action strategies, can effectively solve the problem of unmanned combat vehicles generating action strategies in high-latitude and high-dynamic battlefield environments. It has strong robustness and versatility, and is more in line with the characteristics and laws of unmanned combat vehicle search action strategy generation. However, in the process of training unmanned combat vehicles, the algorithm has a problem of biased estimation of the strategy model, which reduces the learning efficiency. The present disclosure introduces a weighted loss function in the process of training the unmanned combat vehicle target search strategy generation model using the meta-reinforcement learning algorithm, thereby solving the problem of biased estimation of the strategy model of the meta-reinforcement learning algorithm to a certain extent and improving the efficiency and quality of the generation of unmanned combat vehicle target search action strategies. The details are as follows:

[0063] like Figure 2 As shown, in step S101, an unmanned combat vehicle target search strategy generation model is constructed.

[0064] The goal of this exemplary embodiment is to use multiple unmanned combat vehicles to find multiple predetermined targets in the shortest possible time in a complex battlefield environment.

[0065] During the war, unmanned combat vehicles (UCVs) were tasked with searching for firepower points at the front lines. According to the combat plan, the UAVs assembled from a pre-set area. In the complex mountainous environment, which presents numerous obstacles, the UAVs needed to independently and collaboratively search for enemy firepower points and complete the search for all predetermined targets. It is estimated that after 10,000 steps, the UAVs would have located five firepower points, A, B, C, D, and E, completing the task of collecting target location information and ensuring the entry of subsequent deep attack forces into battle.

[0066] In this example implementation, the unmanned combat vehicles are divided into five groups, each representing an action unit. All unmanned combat vehicles are pre-deployed in the southeast corner of the designated area;

[0067] The conditions for completing the target search mission are: after the unmanned combat vehicle has passed the preset combat time, that is, 10,000 time steps, all five fire point targets have been reached and detected by at least one unmanned combat vehicle.

[0068] So the set of environmental states s at any time t t for:

[0069] s t ={(x1, y1),(x2, y2),(x3, y3),(x4, y4),(x5, y5)}, where x and y represent the horizontal and vertical coordinates of the position of each group of unmanned combat vehicles in the environment, respectively. Although it is only used to represent the position of the unmanned combat vehicles, the environmental state set s t This can include a variety of environmental factors, such as the terrain, altitude, weather visibility, and wind speed of the vehicle's location. The subscripts 1, 2, and 3 represent the numbers of the three groups of unmanned combat vehicles, respectively. The value range of the state variables in each dimension is: x∈[0,5000], y∈[0,5000].

[0070] Multiple unmanned ships are combined according to the environmental state s t The action set a is composed of the corresponding actions made by the corresponding states in t For a t ={move1,move2,move3,move4,mov5e,mov6e,mov7e,mov8e}, where move1 indicates that the unmanned combat vehicle maneuvers toward the east; move2 indicates that the unmanned combat vehicle maneuvers toward the southeast; move3 indicates that the unmanned combat vehicle maneuvers toward the south; move4 indicates that the unmanned combat vehicle maneuvers toward the southwest; move5 indicates that the unmanned combat vehicle maneuvers toward the west; move6 indicates that the unmanned combat vehicle maneuvers toward the northwest; move7 indicates that the unmanned combat vehicle maneuvers toward the north; move8 indicates that the unmanned combat vehicle maneuvers toward the northeast; the value range of the maneuvering action includes move∈[0,50]. The entire specified area is regarded as a coordinate system, 0 indicates no action, and 50 indicates that it is divided into 50 action zones.

[0071] The environmental feedback r t include:

[0072] Reward for the unmanned combat vehicle reaching any of the firepower targets A, B, C, D and E: r t =+1000;

[0073] Penalty for unmanned combat vehicles touching obstacles: r t =-400;

[0074] Penalty for unmanned combat vehicles touching the boundary: rt =-2000;

[0075] Penalty for collision between unmanned combat vehicles: r t =-200.

[0076] like Figure 3 As shown, in step S102, the unmanned combat vehicle target search strategy generation model is trained using the meta-reinforcement learning algorithm; when the weighted loss function is incorporated into the training process, the target search strategies of multiple unmanned combat vehicles are obtained.

[0077] In this step, the training process of the unmanned combat vehicle target search strategy generation model includes multiple parameters:

[0078] The strategy generation model is π θ , a fully connected neural network with 3 hidden layers (128 neurons in total) is used.

[0079] The target search task distribution is D(T).

[0080] The basic learning rate is α, α = 0.005.

[0081] The meta-learning rate is β, β = 0.01.

[0082] The number of outer loops is M, M=10000.

[0083] The process of obtaining target search strategies for multiple unmanned combat vehicles includes:

[0084] Initialize multiple parameters;

[0085] Initialization time t=1;

[0086] Collect N task samples from the target search task distribution D(T);

[0087] For each task sample, generate model π according to the strategy θ , sample K trajectories

[0088] According to the trajectory and loss function Calculating gradients

[0089] Using gradient descent Get the optimized parameter θ'; get the new strategy generation model π θ' , sample C trajectories

[0090] Calculate the weight of each of the C trajectories in the loss function;

[0091] Using C tracks and weights To do a meta update:

[0092] Among them, ← means the assignment operation; here the assignment operation means assigning the size of the value on the right side of ← to the left side of ←.

[0093] If the time t is equal to the outer loop time M, the target search strategy of multiple unmanned combat vehicles is obtained;

[0094] If the size at time t is not equal to the outer loop time M, it is necessary to return to step t+1 and generate a model π for each task sample according to the strategy. θ , sample K trajectories Continue with the training process.

[0095] It should be noted that the calculation formula for calculating the weight of each of the C trajectories in the loss function includes:

[0096]

[0097] in, w i represents the information entropy of the i-th trajectory; ∑ k w k represents the sum of information entropy of all sampled trajectories; l i represents the loss value of the i-th trajectory; l represents the average loss of all sampled trajectories.

[0098] Finally, step S103 is performed to deploy the obtained target search strategy of multiple unmanned combat vehicles in a real mission environment.

[0099] It should be noted that although the steps of the method disclosed herein are depicted in a particular order in the accompanying drawings, this does not require or imply that the steps must be performed in that particular order, or that all steps must be performed to achieve the desired result. Additionally or alternatively, certain steps may be omitted, multiple steps may be combined into a single step, and / or a single step may be decomposed into multiple steps. Furthermore, it is readily understood that these steps may be executed synchronously or asynchronously, for example, in multiple modules / processes / threads.

[0100] It should be noted that although several units of the system for action execution are mentioned in the above detailed description, this division is not mandatory. In fact, according to the embodiment of the present disclosure, the features and functions of two or more units described above can be concretized in one unit. Conversely, the features and functions of a unit described above can be further divided into multiple units for concretization. Some or all of the units can be selected according to actual needs to achieve the purpose of the disclosed solution. Those of ordinary skill in the art can understand and implement it without paying creative work.

[0101] Those skilled in the art will readily appreciate other embodiments of the present disclosure after considering the specification and practicing the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of the present disclosure that follow the general principles of the present disclosure and include common knowledge or customary techniques in the art not disclosed herein. The description and examples are to be considered as exemplary only, with the true scope and spirit of the present disclosure being indicated by the appended claims.

Claims

1. A method for generating a target search strategy for an unmanned combat vehicle, characterized in that: The following steps are involved: Constructing an unmanned combat vehicle target search strategy generation model, wherein the unmanned combat vehicle target search strategy generation model includes a state set of multiple unmanned combat vehicles at any time, a set of actions of multiple unmanned combat vehicles at any time, and environmental feedback obtained by multiple unmanned combat vehicles; The target search strategy generation model of the unmanned combat vehicle is trained using a meta-reinforcement learning algorithm; a weighted loss function is introduced into the training process to obtain target search strategies of multiple unmanned combat vehicles; The training process of the target search strategy generation model of the unmanned combat vehicle includes multiple parameters, and the multiple parameters include: the strategy generation model is , the target search task distribution is , the basic learning rate is , the meta-learning rate is , the outer loop time is ; The calculation formula of the weighted loss function includes: ； in, To find the derivative symbol, Indicates the Tracks The weight of Indicates the Tracks The loss function of The process of obtaining target search strategies for multiple unmanned combat vehicles includes: Initializing a plurality of said parameters; Initialization time ; From the target search task distribution Collect N task samples; For each task sample, generate a model according to the strategy ,sampling Tracks ; According to the trajectory and loss function , calculate the gradient ; Using gradient descent , and get the optimized parameters ; Get a new strategy generation model , sample C trajectories ; Calculate the weight of each of the C trajectories in the loss function. The calculation formula includes: ； in, ; represents the information entropy of the i-th trajectory; represents the sum of information entropy of all sampled trajectories; l i Represents the loss value of the i-th trajectory; represents the average of the losses of all sampled trajectories; Using C tracks and weights , do a meta-update: ,in, Indicates the meaning of assignment operation; If the time t is equal to the outer loop time M, a target search strategy for multiple unmanned combat vehicles is obtained; If the time t is not equal to the outer loop time M, it is necessary to return to step t+1 and generate a model for each task sample according to the strategy. ,sampling Tracks ;Continue the training process; The target search strategies obtained for multiple unmanned combat vehicles are deployed in a real mission environment.

2. The method for generating a target search strategy for an unmanned combat vehicle according to claim 1, characterized in that: The steps of constructing the target search strategy generation model of the unmanned combat vehicle include: a plurality of the unmanned combat vehicles at any time The set of environmental states ; A plurality of said unmanned combat vehicles are assembled according to said environmental state The action set consisting of the corresponding actions in the corresponding states ; and environmental feedback obtained by multiple unmanned combat vehicles .

3. The method for generating a target search strategy for an unmanned combat vehicle according to claim 2, characterized in that: The environment state set The formulas include: in, and The horizontal and vertical coordinates represent the status of each unmanned combat vehicle, , ; n Indicates the number of the unmanned combat vehicle.

4. The method for generating a target search strategy for an unmanned combat vehicle according to claim 3, characterized in that: The action set The formulas include: in, This indicates that the UCAV is maneuvering in the due east direction; It indicates that the UCAV is maneuvering in a southeasterly direction; It indicates that the UCAV is maneuvering in the direction of due south; It indicates that the UCAV is maneuvering in a southwesterly direction; It indicates that the UCAV’s maneuver is heading due west; It indicates that the UCAV is maneuvering in a northwest direction; It indicates that the UCAV’s maneuver is heading due north; The maneuver of the unmanned combat vehicle moves toward the northeast; the range of maneuvering action includes , 0 means the unmanned combat vehicle has not taken any action, and 1-100 means the number of steps the unmanned combat vehicle has taken.

5. The method for generating a target search strategy for an unmanned combat vehicle according to claim 4, characterized in that: The environmental feedback It includes rewards for unmanned combat vehicles reaching any target location, penalties for unmanned combat vehicles touching obstacles, penalties for unmanned combat vehicles touching boundaries, and penalties for unmanned combat vehicles colliding with each other.

Images