A reinforcement learning model training method for unmanned aerial vehicle air combat decision
By introducing a large language model into the UAV air combat decision-making process to design a reward function, and combining it with an asynchronous update strategy, the problems of reward sparsity and insufficient exploration are solved, thereby improving the training efficiency and decision-making ability of the UAV air combat decision-making model.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- UNIV OF CHINESE ACAD OF SCI
- Filing Date
- 2023-12-14
- Publication Date
- 2026-07-07
AI Technical Summary
In the field of UAV air combat decision-making, reinforcement learning models suffer from low training efficiency due to problems such as sparse rewards, overexploration, or underexploration. Furthermore, the difficulty in constructing reward functions can easily lead to suboptimal or erroneous behaviors.
By employing a large language model in the design of the reward function, and by calculating the superposition of the basic reward and the predicted gain reward, combined with an asynchronous update strategy, the training process of the reinforcement learning model is optimized.
It improved training efficiency, enhanced the prediction and decision-making capabilities of the UAV air combat decision model, reduced the exploration space, overcame the feedback time limitation, and achieved efficient training results.
Smart Images
Figure CN117787384B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of unmanned aerial vehicle (UAV) air combat decision-making technology, and in particular to a reinforcement learning model training method for UAV air combat decision-making. Background Technology
[0002] In recent years, deep reinforcement learning has become one of the important methods for achieving optimal decision-making by intelligent agents. Its core idea is to learn how an agent interacts with its environment and make decisions through trial and error and the accumulation of experience. In this process, deep reinforcement learning mainly focuses on how to take actions to maximize numerical rewards.
[0003] However, the inventors of this application discovered in their research that, in the field of UAV air combat decision-making, reinforcement learning environments are characterized by large decision spaces and long task planning periods. Reinforcement learning models that rely on reward and punishment mechanisms for learning can suffer from problems such as reward sparsity, overexploration, or underexploration, ultimately affecting the efficiency of reinforcement learning training. Therefore, in the field of UAV air combat decision-making, there is a problem of difficulty in constructing reward functions, which can easily lead to the agent (UAV) learning suboptimal or erroneous behaviors. Summary of the Invention
[0004] To address the aforementioned problems, the purpose of this invention is to provide a reinforcement learning model training method for UAV air combat decision-making. During the model training process, a large language model is used to participate in the design of the reward function to solve the reward sparsity problem, reduce the exploration space of the early behavior of the reinforcement learning model, and improve training efficiency, thereby obtaining a reinforcement learning model that can optimize air combat decision-making.
[0005] To achieve the above objectives, the present invention adopts the following technical solution:
[0006] Firstly, this application provides a reinforcement learning model training method for UAV air combat decision-making, the training method comprising several training rounds, each training round including:
[0007] (1) Set the network architecture and initial network parameters of the reinforcement learning model in the current training round;
[0008] (2) Obtain input data and input it into the reinforcement learning model of the current training round to obtain output data, wherein the input data includes battlefield environment data and air combat type data, and the output data is the decision data output by the reinforcement learning model;
[0009] (3) Calculate the reward function for the current training round based on the decision data output by the reinforcement learning model. The reward function for the current training round is obtained by superimposing the basic reward and the prediction gain reward. The basic reward is related to the air combat type data and the battlefield environment data. The prediction gain reward is determined by the decision difference, which is the difference between the decision data output by the reinforcement learning model and the pre-determined large language model for the input data.
[0010] (4) Adjust the network parameters of the reinforcement learning model according to the reward function of the current training round to obtain the initial network parameters for the next training round;
[0011] (5) Return to (1) to execute the next training round until the training reaches the preset stopping condition.
[0012] In one implementation of this application, the calculation formula for the reward function is as follows:
[0013] R Total = (1-α(t))*R baseline +α(t)*R LLM
[0014] Where t is a time series; R Total R is the reward function; baseline As the basic reward; R LLM The predicted gain is α(t), which is a preset time function relationship.
[0015] In one implementation of this application, the prediction gain reward is inversely proportional to the decision difference;
[0016] The decision difference is the vector difference between the decision data output by the reinforcement learning model and the pre-trained large language model for the input data.
[0017] In one implementation of this application, the method further includes the step of pre-determining the large language model;
[0018] The pre-determined large language model includes:
[0019] Choose a large language model that sets the network structure and initial network parameters;
[0020] Data sets are obtained from historical data, including question-and-answer datasets and decision datasets. The question-and-answer datasets include text data pairs of battlefield environment data and decision data under a set air combat type data. The decision datasets are time-series data of battlefield environment data and decision data under a set air combat type data in an expert simulation environment.
[0021] Based on the question-and-answer dataset, the network parameters of the large language model are fine-tuned so that the network structure of the large language model has the predictive ability for air combat decision-making.
[0022] Based on the decision dataset, a structured knowledge base is established for use by large language models for invocation and querying.
[0023] In one implementation of this application, after the determined large language model obtains the input data, it first uses a structured knowledge base to query the corresponding decision data based on the vector representation of the battlefield environment data, and outputs the vector representation of the decision data.
[0024] Once the determined large language model has no corresponding vector representation of battlefield environment data in the structured knowledge base, it converts the vector representation of battlefield environment data into text data, then predicts the text of the corresponding decision data based on its own network structure, and finally converts the text into a vector representation of decision data for output.
[0025] In one implementation of this application, α(t) has a function that decays over time.
[0026] In one implementation of this application, α(t) is a negative linear time decay, inverse proportional time decay, or negative exponential decay function.
[0027] In one implementation of this application, the battlefield environment data includes the kinematic parameters of the UAV and the enemy aircraft observed by the UAV.
[0028] The decision data includes instructions from the drone's own action state space.
[0029] In one implementation of this application, the kinematic parameters include position coordinates, motion angle, and motion velocity;
[0030] The motion state space includes wing control, elevator control, rudder control, and throttle control.
[0031] Secondly, this application provides a computer-readable storage medium storing a computer program, wherein the computer program, when running, controls the device where the computer-readable storage medium is located to execute the reinforcement learning model training method for UAV air combat decision-making as described in the first aspect.
[0032] The present invention has the following advantages due to the adoption of the above technical solutions: (1) Training a large language model as a decision expert for air combat missions and constructing a knowledge base based on the air combat mission decision dataset can achieve accurate prediction in air combat missions and achieve efficient training results with relatively low economic and time costs. (2) Combining the prediction reward of the large language model with the reward of deep reinforcement learning, and using the large language model to train and assist UAV decision-making can not only speed up the training process, but also improve the prediction and decision-making capabilities of the model. (3) Overcoming the problem of feedback time limitation of large language model, and using an asynchronous update strategy to solve the problem of feedback time difference between language model and deep reinforcement learning model. Attached Figure Description
[0033] Figure 1 This is an experimental result diagram of the reinforcement learning model training method according to an embodiment of the present invention. Detailed Implementation
[0034] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. All other embodiments obtained by those skilled in the art based on the described embodiments of the present invention are within the scope of protection of the present invention.
[0035] In one embodiment of this application, a reinforcement learning model training method for UAV air combat decision-making is provided.
[0036] The training method in this application includes several training rounds, and each training round includes:
[0037] (1) Set the network architecture and initial network parameters of the reinforcement learning model in the current training round.
[0038] Specifically, the network architecture of a reinforcement learning model can be an existing network architecture or a new architecture obtained by adaptively modifying an existing network architecture. Examples of such network architectures include MAPPO and MADDPG. Once selected, the network architecture remains unchanged throughout the training process.
[0039] A network architecture includes network units such as convolutional layers and connection layers. These network units are connected with different weight values, which are called network parameters. The network parameters are ultimately determined through the training process. They need to be initialized before training and updated after each training epoch.
[0040] (2) Obtain input data and input it into the reinforcement learning model of the current training round to obtain output data, wherein the input data includes battlefield environment data and air combat type data, and the output data is the decision data output by the reinforcement learning model;
[0041] Specifically, the air combat type data is a pre-set air combat type, which may include 1-on-1 air combat or many-on-many air combat training. In many-on-many scenarios, 2-on-2 is the most typical. For example, in an air combat environment, two drones usually form a host and a wingman to perform cooperative missions.
[0042] Battlefield environmental data includes kinematic parameters of the UAV and enemy aircraft as observed by the UAV, such as position coordinates, motion angles, and motion speeds.
[0043] Decision data includes instructions for the UAV's own action state space. The action state space can include wing control, elevator control, rudder control, and throttle control, among others.
[0044] Reinforcement learning models, after setting the network architecture and initializing the network parameters, can calculate the output based on the input and then feed the output back to adjust the network parameters.
[0045] (3) Calculate the reward function for the current training round based on the decision data output by the reinforcement learning model. The reward function for the current training round is obtained by superimposing the basic reward and the prediction gain reward. The basic reward is related to the air combat type data and the battlefield environment data. The prediction gain reward is determined by the decision difference, which is the difference between the decision data output by the reinforcement learning model and the pre-determined large language model for the input data.
[0046] Specifically, the formula for calculating the reward function is as follows:
[0047] R Total = (1-α(t))*R baseline +α(t)*R LLM
[0048] Where t is a time series; R Total R is the reward function; baseline As the basic reward; R LLM The predicted gain is α(t), which is a preset time function relationship.
[0049] More specifically, prior to this step, a large language model needs to be determined in advance for the determination of the prediction gain reward.
[0050] The large language model is predetermined, including:
[0051] A. Select and set the large language model with network structure and initial network parameters;
[0052] B. Obtain datasets from historical data, including question-and-answer datasets and decision datasets. The question-and-answer datasets include text data pairs of battlefield environment data and decision data under a set air combat type data. The decision datasets are time-series data of battlefield environment data and decision data under a set air combat type data in an expert simulation environment.
[0053] C. Based on the question-and-answer dataset, fine-tune the network parameters of the large language model so that the network structure of the large language model has the predictive ability for air combat decision-making;
[0054] D. Based on the aforementioned decision dataset, establish a structured knowledge base for use by large language models for invocation and querying.
[0055] In the process of determining the prediction gain reward in the large language model, after obtaining the input data, the system first uses a structured knowledge base to query the corresponding decision data based on the vector representation of the battlefield environment data and outputs the vector representation of the decision data. If there is no corresponding vector representation of the battlefield environment data in the structured knowledge base, the vector representation of the battlefield environment data is converted into text data. Then, based on its own network structure, the system predicts the text of the corresponding decision data and converts the text into the vector representation of the decision data for output.
[0056] The vector difference between the vector representation of the decision data output by the large language model and the vector representation of the decision data output by the reinforcement learning model with current network parameters is used to determine R. LLM The basis for this can generally be an inverse proportional relationship.
[0057] In the embodiments of this application, α(t) has a function relationship of decaying with time, such as a negative linear time decay, an inverse proportional time decay, or a negative exponential decay function relationship.
[0058] (4) Adjust the network parameters of the reinforcement learning model according to the reward function of the current training round to obtain the initial network parameters for the next training round;
[0059] (5) Return to (1) to execute the next training round until the training reaches the preset stopping condition.
[0060] The following experimental setup demonstrates how the above method can improve the convergence speed of training.
[0061] This experiment includes the following steps:
[0062] 1) Define the types of drone air combat.
[0063] The defined air combat types can be applied to simulations or actual combat.
[0064] In this implementation case, an air combat mission was constructed based on an unmanned aerial vehicle (UAV) air combat simulation environment, consisting of 1vs.1 tracking and confrontation and 2vs.2 cooperative confrontation.
[0065] 2) Obtain datasets from historical data.
[0066] The datasets include question-and-answer datasets and decision datasets.
[0067] The question-and-answer dataset includes textual data on battlefield environment and decision-making under defined air combat types. Specifically, based on the air combat type, information related to air combat, position prediction, and air combat strategies is extracted from publicly available open-domain datasets to generate a standard question-and-answer format dataset. Training samples are in question and answer formats, used for fine-tuning the large language model. This allows the fine-tuned large language model to learn knowledge about air combat decision-making, enabling the model to be trained as an expert in air combat mission decision-making.
[0068] The decision dataset is a time-series dataset of battlefield environment data and decision data under a set air combat type in an expert simulation environment. Based on the adversarial scenario in the expert simulation environment, decision datasets for different air combat types are constructed. Specifically, game data can be obtained from the simulation environment using open-source decision rule libraries, expert decision-making, or manual decision-making methods. Each piece of game data is organized chronologically, and each data point is assigned corresponding transformation logic and calculation methods according to their sequential relationships. The two types of data are then integrated to obtain time-series datasets of battlefield environment data and decision data under the set air combat type. After structured processing, the dataset retains its corresponding structure and logical relationships even after vectorization. In this implementation case, expert decision-making and manual decision-making methods were used to acquire game data in a simulation environment. The transformation logic and calculation methods are as follows: A three-dimensional coordinate system was established with the center position of the air combat space as the origin. The latitude and longitude in the simulation environment were converted into coordinate data, and the speed unit was converted into a unified standard. Based on the changes in the speed and heading angle of the UAV in a short period of time and the corresponding position information, a formula was obtained to calculate the area coordinates of the UAV's expected position in the next moment. Then, the two types of data were organized and structured, and the data samples were saved in dictionary form to reflect the structure and logical relationship between the data.
[0069] 3) Fine-tune the large language model.
[0070] First, a pre-trained large language model needs to be selected. This model should be lightweight, and the choice should be based on the complexity of the air combat mission and the hardware available. It should not only be able to load for training and inference evaluation but also ensure that the model's inference speed meets the requirements of multi-threaded calls in a reinforcement learning environment. Based on the specific parameters and network structure of the large language model, relevant fine-tuning parameters are set. Then, the model is fine-tuned using a question-answering dataset. The fine-tuned model should be able to accurately understand and generate state descriptions of the air combat simulation environment. After fine-tuning, the model learns the knowledge of the air combat mission, and the large language model can act as a decision expert for air combat missions. In this implementation case, for example, the ChatGLM2-6B model can be selected for fine-tuning. This model has 6 billion parameters and requires 8GB of GPU memory. After fine-tuning, it can accurately understand and generate state descriptions of the air combat simulation environment, making correct decisions for tracking and combat missions, and its inference speed meets the requirements of air combat mission training.
[0071] 4) Build a knowledge base.
[0072] The knowledge base is used for calling and querying large language models to improve the efficiency of large language models in outputting decision data.
[0073] Based on the text type and length of the air combat mission decision dataset, an appropriate embedding model is selected. The chosen embedding model needs to be adapted to the language and text length of the dataset. The air combat mission decision dataset is then vectorized using the embedding model to obtain a knowledge base. The game data and computational logic in the air combat mission decision dataset enhance the predictive and decision-making capabilities of the large language model. Vectorization allows for rapid data querying and inference. By calling the knowledge base, not only are the large language model's position predictions and decision evaluations more accurate, but its inference speed is also accelerated, ensuring rapid results during training without affecting the training process. This implementation case selects OpenAI's embedding model `text-embedding-ada-002` to vectorize the air combat mission decision dataset. This model can vectorize Chinese data and long documents. After adding the knowledge base, the inference speed and accuracy of the large language model are further improved.
[0074] 5) Construct a basic reward system based on air combat type and battlefield environment. baseline .
[0075] The basic rewards are set according to the rules and objectives of the air combat mission. These rewards are mainly used to evaluate the decisions of the UAV, including basic rewards such as the UAV's attitude, altitude, survivability, and distance from the enemy aircraft.
[0076] 6) Set the structure of the reward function.
[0077] The formula for calculating the reward function is:
[0078] R Total = (1-α(t))*R baseline +α(t)*R LLM
[0079] Where t is a time series; R Total R is the reward function; baseline As the basic reward; R LLM For predicted gain reward; R LLM This measure assesses the difference between the decisions made by the UAV's reinforcement learning model and those made by the large language model. α(t) is a time-decreasing weight, the specific value of which can be dynamically set based on the air combat mission objective and training status. This parameter is used to adjust the weight of the large language model's prediction reward. In this implementation, the initial value is set to 0.4, and it decays linearly with each step until it reaches 0.1. This ensures that in the early stages of the mission, the UAV relies more on the prediction reward provided by the large language model for decision-making, while in the later stages, it gradually shifts to relying on the agent's own exploration and experience. This reward mechanism design utilizes the predictive power of the large language model, reducing the exploration space in the early stages of combat, while also ensuring the fundamental role of the original reward, providing effective guidance for the UAV's decision-making in complex air combat scenarios.
[0080] 7) Train the reinforcement learning model according to the set reward function.
[0081] First, the parameters of the UAV model and the relevant parameters of the training task are set according to the air combat mission and the reinforcement learning algorithm. Then, the large language model prediction parameters are set. After the large language model is introduced, an asynchronous update strategy is needed to solve the feedback time difference problem between the large language model and the deep reinforcement learning model. The large language model prediction parameter is the frequency at which the sampling thread calls the large language model during training. It needs to be determined according to the number of threads in the training task, hardware conditions and training stage. The role of this parameter is to adjust the degree of intervention of the large language model. In this implementation case, 64 sampling threads are set for the training task, and the frequency of each thread calling the large language model is set to once every 50 steps (10 seconds).
[0082] The training effect of this application is as follows: Figure 1 As shown, the baseline is a technical approach that does not involve a large language model in the reward function design. Compared to the baseline, this application (LLM.RS) significantly improves the training performance.
[0083] In summary, this application has the following advantages: (1) Training a large language model as a decision expert for air combat missions and constructing a knowledge base based on the air combat mission decision dataset can achieve accurate predictions in air combat missions, achieving efficient training results with relatively low economic and time costs. (2) Combining the prediction reward of the large language model with the reward of deep reinforcement learning, and using the large language model to train and assist UAV decision-making can not only accelerate the training process, but also improve the model's prediction and decision-making capabilities. (3) Overcoming the problem of feedback time limitation of the large language model, and using an asynchronous update strategy to solve the problem of feedback time difference between the language model and the deep reinforcement learning model.
[0084] In this application embodiment, a computer-readable storage medium is also provided, which stores a computer program that, when executed by a computer device, implements the method described in this application embodiment.
[0085] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.
[0086] In the embodiments provided by this invention, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of the units described above is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be an indirect coupling or communication connection between devices or units through some interfaces, and may be electrical, mechanical, or other forms.
[0087] The integrated units implemented as software functional units described above can be stored in a computer-readable storage medium. These software functional units, stored in a storage medium, include several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) or processor to execute some steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.
[0088] The above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.
Claims
1. A reinforcement learning model training method for UAV air combat decision-making, characterized in that, The training method includes several training rounds, and each training round includes: (1) Set the network architecture and initial network parameters of the reinforcement learning model in the current training round; (2) Obtain input data and input it into the reinforcement learning model of the current training round to obtain output data, wherein the input data includes battlefield environment data and air combat type data, and the output data is the decision data output by the reinforcement learning model; the battlefield environment data includes the kinematic parameters of the UAV and the enemy aircraft observed by the UAV; the decision data includes the instructions of the UAV's own action state space determined by the UAV. (3) Calculate the reward function for the current training round based on the decision data output by the reinforcement learning model. The reward function for the current training round is obtained by superimposing the basic reward and the prediction gain reward. The basic reward is related to the air combat type data and the battlefield environment data. The prediction gain reward is determined by the decision difference, which is the difference between the decision data output by the reinforcement learning model and the pre-determined large language model for the input data. (4) Adjust the network parameters of the reinforcement learning model according to the reward function of the current training round to obtain the initial network parameters for the next training round; (5) Return to (1) to execute the next training round until the training reaches the preset stopping condition; The formula for calculating the reward function is as follows: = (1 - α ( t )) * +α ( t ) * in, t It is a time series; For the reward function; As a basic reward; For predicted gain rewards; α ( t ) represents a preset time function relationship, the α ( t It exhibits a function that decays over time; The prediction gain reward is inversely proportional to the decision difference; the decision difference is the vector difference between the decision data output by the reinforcement learning model and the pre-trained large language model for the input data.
2. The reinforcement learning model training method for UAV air combat decision-making according to claim 1, characterized in that, The method further includes the step of pre-determining the large language model; The pre-determined large language model includes: Choose a large language model that sets the network structure and initial network parameters; Data sets are obtained from historical data, including question-and-answer datasets and decision datasets. The question-and-answer datasets include text data pairs of battlefield environment data and decision data under a set air combat type data. The decision datasets are time-series data of battlefield environment data and decision data under a set air combat type data in an expert simulation environment. Based on the question-and-answer dataset, the network parameters of the large language model are fine-tuned so that the network structure of the large language model has the predictive ability for air combat decision-making. Based on the decision dataset, a structured knowledge base is established for use by large language models for invocation and querying.
3. The reinforcement learning model training method for UAV air combat decision-making according to claim 2, characterized in that, Once the large language model is determined, after obtaining the input data, it prioritizes the structured knowledge base, queries the corresponding decision data based on the vector representation of the battlefield environment data, and outputs the vector representation of the decision data. Once the determined large language model has no corresponding vector representation of battlefield environment data in the structured knowledge base, it converts the vector representation of battlefield environment data into text data, then predicts the text of the corresponding decision data based on its own network structure, and finally converts the text into a vector representation of decision data for output.
4. The reinforcement learning model training method for UAV air combat decision-making according to claim 1, characterized in that, The α ( t ) represents a negative linear time decay, inverse proportional time decay, or negative exponential decay function relationship.
5. The reinforcement learning model training method for UAV air combat decision-making according to claim 1, characterized in that, The kinematic parameters include position coordinates, motion angle, and motion velocity; The motion state space includes wing control, elevator control, rudder control, and throttle control.
6. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program, which, when executed, controls the device containing the computer-readable storage medium to perform the reinforcement learning model training method for UAV air combat decision-making as described in any one of claims 1 to 5.