Mission planning method, device, medium and program product for simulation deduction
Through the simulation-oriented task planning method, the Markov decision-making process and multi-layer reinforcement learning algorithm are used to automate task planning, which solves the problem that traditional methods are difficult to meet the real-time and accuracy of modern battles, and achieves fast and efficient combat tasks 规划和资源配置,提升了作战任务的智能化和适应性。
Patent Information
- Application Number
- CN202411164018.5
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-23
- Publication Date
- 2024-11-26
- Estimated Expiration
- 2044-08-23
AI Technical Summary
Traditional manual task planning methods are difficult to meet the real-time and accuracy requirements of combat mission planning in modern battles, especially in complex and dynamic battlefield environments, which are difficult to quickly generate efficient task planning solutions.
Using a simulation-oriented deduction task planning method, the UAV team combat mission model is designed as the Markov decision-making process, and the tasks are decomposed through the multi-layer reinforcement learning algorithm, a hierarchical decision-making framework is built, neural network models are trained, and task planning is automated to improve decision-making. Efficiency and adaptability.
It significantly improves the intelligence level and decision-making flexibility of combat mission planning, quickly generates combat mission planning plans, ensures the optimal allocation of combat resources, enhances the real-time and adaptability of mission planning, and reduces the work burden of command personnel.
Smart Images

Figure CN118691049B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of mission planning, and in particular to a simulation-based mission planning method, device, medium and program product. Background Art
[0002] In the military command system, mission planning is a basic and complex task that requires accurate answers to a series of questions about the time, location, method, goal and results of mission execution. With the continuous evolution of science and technology and combat concepts, the modern combat environment presents the characteristics of fast-paced and high-intensity confrontation, the battlefield situation is unpredictable, and the fire interception and electronic interference suffered by weapon systems are becoming increasingly complex. The traditional manual mission planning method is difficult to meet the real-time and accuracy requirements for combat mission planning in modern campaigns. Summary of the invention
[0003] The embodiments of the present invention provide a simulation-based mission planning method, device, medium, and program product, which are intended to improve the accuracy and planning efficiency of combat mission planning.
[0004] In order to achieve the above object, on the one hand, a simulation-based task planning method is provided, the method comprising:
[0005] S1, designing a combat mission model, defining combat mission end conditions, action sets, state sets, reward functions and objective functions, wherein the combat mission model is a UAV swarm combat mission model, and the UAV swarm combat mission model is a mathematical model based on a Markov decision process MDP;
[0006] The combat mission end condition is that the combat mission end time is reached or the UAV group completes the combat mission within the specified time;
[0007] The action set is the basic mission of the drone, including taking cover, waiting in place, searching, heading to the bombing point and / or bombing;
[0008] The state set is a system environment state and a combat unit state, wherein the system environment state includes an enemy base intact, an enemy base severely destroyed, or an enemy base slightly destroyed, and the combat unit state is a drone state, and the drone state includes: drone position, drone fuel level, drone damage status, and / or drone ammunition level;
[0009] The reward function represents the immediate reward obtained by the drone after completing the subtask, and the immediate reward includes the reward obtained by the drone from the system environment after completing the task and the cost of the drone taking action;
[0010] The objective function is the behavior criterion of the UAV, and the objective function is defined as:
[0011]
[0012] Execute for said drone Rewards after actions, is the current state of the drone, is the discount factor, Execute for said drone The next state after the action, is the maximum action value in the next state of the drone, is the duration of the drone action;
[0013] S2, establish a simulation environment for combat mission planning;
[0014] S3, decomposing the combat mission into a series of subtasks executed by parallel combat units, wherein each subtask represents an option, the option is an action sequence, and the subtasks include a task of bombing an enemy base A, a task of bombing an enemy base B, and / or a task of bombing an enemy base C;
[0015] S4, constructing a hierarchical decision framework, wherein the hierarchical decision framework is divided into a high-level decision framework and a low-level decision framework, the high-level decision framework is used to select an action sequence to be executed, automatically generate an action sequence, and adjust the strategy according to the feedback of the low-level decision framework, and the low-level decision framework is used to process the specific action selection in the action sequence and feedback the processing result;
[0016] S5, train the combat mission planning neural network model and build a combat mission planning system.
[0017] Preferably, in the simulation-based mission planning method, the simulation-based mission planning environment in step S2 is a simulation platform in which a pre-selected UAV performs actions and returns a report.
[0018] Preferably, in the simulation-based task planning method, the action sequence is based on a Q-learning algorithm, and the Q-learning algorithm includes: a state set, an action set, an immediate reward, a Q value table, and an action selection, wherein:
[0019] The state set is a subset of the system environment state in step S1;
[0020] The action set is a subset of the action set in step S1;
[0021] The instant reward includes state action benefit and action cost. The state action benefit is based on the principle that each action will have an instant reward. The state action benefit is expressed as , is a predefined benefit value, and the action cost is expressed as , is a predefined cost value, is the duration of the action, is the number of executing drones;
[0022] The Q value table is an action-time pair, representing the benefit of taking a under s;
[0023] The action selection specifies the criteria and basis for the action, wherein the action selection is selected by substituting the following formula into the Q value table:
[0024]
[0025] is the probability of selecting action a in state s, is an action-time pair, e is a constant greater than 0, represents the i-th action in A(s), and A(s) represents the set of all possible actions that can be taken in state s.
[0026] Preferably, in the simulation-based task planning method, the high-level decision framework in step S4 automatically generates an action sequence, including: dynamically generating an action sequence through system environment and drone status evaluation, and when an action sequence ends or an external event occurs, performing an evaluation to determine the next action sequence, wherein the external event includes: damage to a friendly drone and / or changes in the status of an enemy base.
[0027] Preferably, in the simulation-based task planning method, the optional multi-layer reinforcement learning algorithm includes: loading a Q value table, initializing a state, selecting an action sequence at the upper layer, executing an action sequence, updating action sequence information, time monitoring, and state judgment, wherein the updating of the action sequence information is implemented by the following formula:
[0028]
[0029] is the state-action value function after the action sequence is updated, is the learning rate, is the discount factor, Is the action sequence to perform the action After the reward, Is the action sequence to perform the action Then transfer to the next state, is the action sequence that maximizes The action value of the value is the optimal action value.
[0030] On the other hand, a device for task planning for simulation and deduction is provided, which includes a memory and a processor, wherein the memory stores at least one program, and the at least one program is executed by the processor to implement the steps of any of the above-described methods for task planning for simulation and deduction.
[0031] On the other hand, a computer-readable storage medium is provided, wherein at least one program is stored in the storage medium, and the at least one program is executed by a processor to implement the steps of any of the simulation-oriented task planning methods described above.
[0032] On the other hand, a computer program product includes a computer program, wherein when the computer program is executed by a processor, the steps of any of the simulation-oriented task planning methods described above are implemented.
[0033] The above technical solution has one or more of the following technical effects:
[0034] (1) By introducing the optional multi-layer reinforcement learning algorithm, combat tasks are layered and quickly decomposed, and long-term and high-level decision-making problems are automatically processed. Combat mission planning schemes can be quickly generated in complex and dynamic battlefield environments, which improves the intelligence level of combat mission planning and the flexibility of decision-making, speeds up the decision-making process, and significantly improves the efficiency of combat mission planning. It is especially suitable for large-scale, high-intensity campaign-level battlefield simulation scenarios.
[0035] (2) By defining the end conditions, action sets, and state sets of combat missions, we ensure that combat resources are optimally configured according to the status and number of combat units, thereby maximizing combat effectiveness.
[0036] (3) The simulation environment can simulate the real battlefield environment, allowing the combat mission planning system to dynamically adjust the planning scheme based on the latest situation information, thereby enhancing the real-time and adaptability of mission planning.
[0037] (4) Automated mission planning reduces the workload of commanders, allowing them to focus on higher-level strategic decisions, thereby improving command effectiveness and operational response speed. BRIEF DESCRIPTION OF THE DRAWINGS
[0038] Figure 1 A flowchart of a simulation-based task planning method according to an embodiment of the present invention;
[0039] Figure 2 A schematic diagram of decomposition of UAV swarm combat tasks in a simulation-based task planning method according to an embodiment of the present invention;
[0040] Figure 3 A schematic diagram of the steps of a multi-layer reinforcement learning algorithm for a simulation-based task planning method according to an embodiment of the present invention;
[0041] Figure 4 A schematic diagram of the workflow of each module in a task planning system for implementing a simulation-based task planning method according to an embodiment of the present invention;
[0042] Figure 5 It is a structural schematic diagram of a task planning device for simulation and deduction according to another embodiment of the present invention. DETAILED DESCRIPTION
[0043] To further illustrate the various embodiments, the present invention provides drawings. These drawings are part of the disclosure of the present invention, which are mainly used to illustrate the embodiments and can be used in conjunction with the relevant descriptions in the specification to explain the operating principles of the embodiments. With reference to these contents, a person of ordinary skill in the art should be able to understand other possible implementations and advantages of the present invention. The components in the figures are not drawn to scale, and similar component symbols are generally used to represent similar components.
[0044] The present invention will now be further described with reference to the accompanying drawings and specific implementation methods.
[0045] Embodiment 1:
[0046] In order to improve the real-time and accuracy of combat mission planning, the present invention provides a simulation-based mission planning method. Figure 1 The flowchart of a simulation-based task planning method according to an embodiment of the present invention is shown in FIG. The simulation-based task planning method according to an embodiment of the present invention implements task planning based on a multi-layer reinforcement learning algorithm of options, such as Figure 1 , the method of this embodiment comprises the following steps:
[0047] S1, design a combat mission model, define the combat mission end conditions, action set, state set, reward function and objective function, where the combat mission model is a UAV swarm combat mission model, and the UAV swarm combat mission model is a mathematical model based on the Markov decision process MDP;
[0048] The combat mission end condition is that the pre-set combat mission end time is reached or the drone group completes the combat mission within the specified time;
[0049] The action set is the basic mission of the drone, including taking cover, waiting in place, searching, heading to the bombing point and / or bombing;
[0050] The state set is the system environment state and the combat unit state, wherein the system environment state includes the enemy base intact, the enemy base severely destroyed or the enemy base slightly destroyed, and the combat unit state is the drone state, and the drone state includes: drone position, drone fuel level, drone damage and / or drone ammunition level;
[0051] The reward function represents the immediate reward obtained by the drone after completing the subtask. The immediate reward includes the reward obtained from the system environment after the drone completes the task and the cost of the drone taking action;
[0052] The objective function is the behavior criteria of the drone, and the objective function is defined as:
[0053]
[0054] Execution for drones Rewards after actions, is the current state of the drone, is the discount factor, Execution for drones The next state after the action, is the maximum action value in the next state of the drone, is the duration of the drone action. In a specific implementation of the embodiment of the present application, the designed combat mission model is a mathematical model of an MDP; the MDP model consists of five basic elements, namely, a five-tuple .in The system state set includes the state of the system environment, such as regulations and restrictions, battlefield environment and weather, and the state of the combat unit, such as position, number of weapons and lethality, etc. It is the system action set, i.e. the basic tasks of the combat unit, such as maneuvering, attacking, etc. is the state transition probability; It is the system reward, which means the reward that the combat unit can get after completing the task; It is the objective function of the system, that is, the code of conduct for combat units and the basis for decision-making. Its result reflects the execution result of the task.
[0055] The UAV swarm combat model is a multi-task combat process, and the combat environment itself has a strong dynamic change, so the UAV swarm combat is a multi-task dynamic programming problem. First, the UAV swarm combat task can be decomposed into Figure 2 shown. Figure 2The figure is a schematic diagram of the task decomposition process of the drone swarm agent. In this model, the combat mission is first decomposed into sub-tasks in different areas according to the different bombing targets; then the sub-tasks in each area can be divided into many specific tasks according to the bombing stage and time. Finally, the sub-tasks can be decomposed into the action level. After decomposing to the action level, the MDP data model can be established, and then the Q-learning algorithm can be used to solve it. The system state set, system action set, system reward function and system objective function of the MDP model will be explained in detail below.
[0056] In a specific implementation, the above-mentioned task end conditions include the lower limit of the task time has been reached and the task is completed. The lower limit of the task time is set to be reached because in the process of executing the drone group combat mission, it is often necessary to limit the task execution time to ensure that the overall task is executed in a time sequence. Once this period is exceeded, the task execution effect is often greatly reduced or counterproductive, and the system reward will return a negative number. Task completion refers to the combat unit completing the task within a specified time, such as the task of destroying the enemy base.
[0057] In a specific implementation, the action set in step S1 includes the basic tasks of the drone; the basic tasks of the drone include: cover, waiting on the spot, searching, going to the bombing point and / or bombing, etc.; the state set in step S1 includes: system environment state and drone state, the system environment state includes: enemy base state, including the enemy base intact, the enemy base severely destroyed, and the enemy base slightly destroyed. The drone state includes: drone position, drone fuel level, drone damage and / or drone ammunition level.
[0058] In a specific implementation, the reward function in step S1 is specified according to the specific action task. Specifically, when the drone swarm agent completes a subtask, it will receive an immediate reward from the system environment. This immediate reward mainly comes from two aspects, one is the reward from the system environment after successfully attacking the enemy base, and the other is the cost of taking these actions. Depending on the distance between the drone and the enemy base to be attacked and the difficulty of the task, the reward obtained is also different. The reward function of the system can be simply described as: .
[0059] In a specific implementation, the objective function in step S1 is both the basis for the agent's action selection and an indicator for measuring the quality of the bombing. Given that the bombing mission is a task closely related to time, the rewards brought by the actions taken by the agent will be discounted as time goes by, but the drone swarm combat mission model is a finite-stage model. In summary, the objective function can be defined as:
[0060] (1)
[0061] Furthermore, the agent selects an action strategy through formula (1). The meanings of the variables shown in formula (1) are as follows: For execution Rewards after actions, is the current state, is the discount factor, For execution The next state after the action, is the maximum action value in the next state, The duration of the action.
[0062] S2, establish a simulation environment for combat mission planning.
[0063] In an embodiment of the present application, the mission planning simulation deduction environment established in step S2 includes a deduction platform in which combat units perform actions and return feedback during the learning process. In a specific implementation, the deduction platform includes common simulation platforms on the market such as AFSIM and Command, among which AFSIM is a simulation software developed by the US Air Force, which provides a comprehensive simulation environment for the Air Force and other military departments to evaluate the effects of different tactics and techniques. This embodiment uses the AFSIM platform to construct a drone swarm bombing scenario, deploy the forces of both sides, and build a simulation environment that can be used for reinforcement learning training.
[0064] S3, decompose the combat mission into a series of subtasks executed by parallel combat units, where each subtask represents an option, the option is an action sequence, and the subtasks include bombing enemy base A mission, bombing enemy base B mission and / or bombing enemy base C mission.
[0065] In a specific implementation of an embodiment of the present application, the action sequence in step S3 represents an action sequence under a combat mission. In the process of constructing a combat mission, the entire task is decomposed into execution subtasks for each combat unit, and these subtasks are not subtasks executed sequentially, but are performed concurrently. The meaning of the subtask is that each task is evaluated by the command center according to the current status, and the evaluation results are divided into levels. The task obtains the current level n, and by assigning a combat unit to execute it, the current task level is n-1, and this process is a subtask one by one.
[0066] The combat mission is a very complex task model with high timeliness requirements. Using action sequences can quickly stratify tasks, quickly decompose tasks, and greatly reduce the state space of each layer, improving task allocation and decision-making efficiency. Adding action sequences to the drone swarm combat model will make the model solution more effective and make full use of process resources. An action sequence represents the encapsulation of a subtask, which can form relatively independent modules. Through continuous learning in the early stage, a strategy template suitable for action sequences will be formed, which greatly saves learning time.
[0067] In a specific implementation, there is a Q-learning algorithm inside the action sequence, which includes the following basic elements: state set, action set, immediate reward, Q value table, and action selection. The state set is a subset of the environment state set, which means that only the task state needs to be considered in Option. The action set is also a subset of the combat unit action set, and there is no need to consider the combat unit state transfer problem. The immediate reward also does not need to consider the transfer cost, which includes state action benefit and action cost. The state action benefit takes each action, and there must be an immediate return. The state action benefit can be expressed as , is the defined benefit value; the action cost is mainly related to time and the number of combat units, and the action cost can be expressed as , is the defined cost value, is the duration of the action, is the number of combat units. The Q value table is an action-time pair, that is, , represents the benefit of the combat unit taking a under s. The length of the Q table depends on the product of the size of the state space and the size of the action set. Action selection stipulates the criteria and basis for the action. The action not only affects the reward, but also affects the state of the environment at the next moment and the final reinforcement value, so a corresponding selection mechanism is needed. Action selection is performed according to the following formula: ,in The probability of choosing action a when in state s, is an action-time pair, e is a constant greater than 0, represents the i-th action in A(s), A(s) represents the set of all possible actions in state s. The optimal action can be selected by substituting the Q value table into the above formula.
[0068] S4, build a hierarchical decision framework, where the hierarchical decision framework is divided into a high-level decision framework and a low-level decision framework. The high-level decision framework is used to select the action sequence to be executed, automatically generate the action sequence, and adjust the strategy according to the feedback of the low-level decision framework. The low-level decision framework is used to process the specific action selection within the action sequence and feedback the processing results.
[0069] In a specific implementation of the embodiment of the present application, the biggest problem in the optional multi-layer reinforcement learning is how to discover the hierarchical structure. In the drone swarm scenario, the automatic generation of action sequences can be divided into two situations: during the execution of a task, after an action sequence or an action has ended, the environment and the state of the drone swarm agent should be evaluated, and the next action sequence should be determined based on the evaluation results; when an action sequence has not ended, but a message is received that a friendly drone is damaged or the enemy base has been destroyed, it is also necessary to evaluate the state of the environment and the agent, and determine the next action sequence decision based on the evaluation results.
[0070] Furthermore, the UAV swarm combat mission model based on action sequences has been constructed, and the internal algorithm of the action sequence and the automatic generation of the action sequence have also been determined. Figure 3 As shown, the multi-layer reinforcement learning algorithm includes the following steps:
[0071] S41, loading Q value table information;
[0072] S42, obtaining a startup status set;
[0073] S43, enter the upper layer learning and select the action sequence;
[0074] S44, execute action sequence;
[0075] S45, update the action sequence information, update the state according to formula (2);
[0076] S46, determining whether the upper limit time has expired, if so, proceeding to S48;
[0077] S47, determine whether the action sequence state is zero, if so, proceed to S48, otherwise, proceed to S41;
[0078] S48, end.
[0079] (2)
[0080] Furthermore, the meanings of the variables in formula (2) are as follows: is the state-action value function after the action sequence is updated, is the learning rate, is the discount factor, Is an action sequence that performs an action After the reward, Is an action sequence that performs an action Then transfer to the next state, is the action sequence that maximizes The action value of the value is the optimal action value.
[0081] S5, train the combat mission planning neural network model and build a combat mission planning system.
[0082] In a specific implementation of an embodiment of the present application, the combat mission planning system constructed by step S5 is applied to automatically generate a mission planning plan, wherein the mission planning plan is a sequence of actions executed by multiple combat units, including a set of actions such as cover, waiting on the spot, searching, going to the bombing point, bombing, etc. in a drone swarm combat scenario.
[0083] According to the embodiments of this application, a realistic UAV combat scenario is constructed with the help of the AFSIM simulation platform, and an option layer reinforcement learning algorithm is introduced to train a neural network model for UAV swarm combat mission planning. This model is integrated into the mission planning system, which can intelligently generate efficient and adaptable combat mission plans based on the real-time combat situation in the simulation environment.
[0084] Figure 4 The following is a schematic diagram of the workflow of each module in the task planning system. Figure 4 As shown in the figure, the system includes a combat mission planning module, a combat mission planning agent, and a simulation environment. The combat mission planning module gives the combat scenario / scope to the combat mission planning agent based on the combat mission planning neural network model; the agent makes decisions based on state characteristics and immediate rewards, and selects the optimal execution option, i.e., option; the execution option selected by the agent is tested and verified in the simulation environment, and the agent adjusts or optimizes the previously generated combat plan based on the environmental feedback results. The whole process is a cyclic iterative process until the optimal combat plan is obtained.
[0085] Embodiment 2:
[0086] The present invention also provides a device for task planning for simulation deduction, such as Figure 5 As shown, the device includes a processor 501, a memory 502, a bus 503, and a computer program stored in the memory 502 and executable on the processor 501. The processor 501 includes one or more processing cores. The memory 502 is connected to the processor 501 via the bus 503. The memory 502 is used to store program instructions. When the processor executes the computer program, the steps in the simulation-oriented task planning method of the first embodiment of the present invention are implemented.
[0087] Further, as an executable solution, the device for simulation-oriented task planning can be a computer unit, which can be a computing device such as a desktop computer, a notebook, a PDA, and a cloud server. The computer unit may include, but is not limited to, a processor and a memory. Those skilled in the art will appreciate that the composition structure of the above-mentioned computer unit is only an example of a computer unit and does not constitute a limitation on the computer unit, and may include more or less components than the above, or a combination of certain components, or different components. For example, the computer unit may also include input and output devices, network access devices, buses, etc., which are not limited in the embodiments of the present invention.
[0088] Further, as an executable solution, the processor may be a central processing unit (CPU), other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor, etc. The processor is the control center of the computer unit, and various interfaces and lines are used to connect various parts of the entire computer unit.
[0089] The memory can be used to store the computer program and / or module, and the processor realizes various functions of the computer unit by running or executing the computer program and / or module stored in the memory, and calling the data stored in the memory. The memory can mainly include a program storage area and a data storage area, wherein the program storage area can store an operating system and at least one application required for a function; the data storage area can store data created according to the use of the mobile phone, etc. In addition, the memory can include a high-speed random access memory, and can also include a non-volatile memory, such as a hard disk, a memory, a plug-in hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (SecureDigital, SD) card, a flash card (Flash Card), at least one disk storage device, a flash memory device, or other volatile solid-state storage devices.
[0090] Embodiment three:
[0091] The present invention also provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, the steps of the simulation-oriented task planning method described in Example 1 of the present invention are implemented.
[0092] If the module / unit integrated in the computer unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the present invention implements all or part of the processes in the above-mentioned embodiment method, and can also be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a computer-readable storage medium. When the computer program is executed by the processor, the steps of the above-mentioned various method embodiments can be implemented. Among them, the computer program includes computer program code, and the computer program code can be in source code form, object code form, executable file or some intermediate form. The computer-readable medium may include: any entity or device that can carry the computer program code, recording medium, U disk, mobile hard disk, disk, optical disk, computer memory, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory) and software distribution medium. It should be noted that the content contained in the computer-readable medium can be appropriately increased or decreased according to the requirements of legislation and patent practice in the jurisdiction.
[0093] Embodiment 4:
[0094] The present invention also provides a computer program product, including a computer program, which, when executed by a processor, implements the steps of the simulation-oriented task planning method described in Embodiment 1 of the present invention.
[0095] Although the present invention has been specifically shown and described in conjunction with the preferred embodiments, it should be understood by those skilled in the art that various changes may be made to the present invention in form and details without departing from the spirit and scope of the present invention as defined by the appended claims, all of which are within the scope of protection of the present invention.
Claims
1. A task planning method for simulation deduction, characterized in that: The option-based multi-layer reinforcement learning algorithm implements task planning, which includes the following steps: S1, designing a combat mission model, defining combat mission end conditions, action sets, state sets, reward functions and objective functions, wherein the combat mission model is a UAV swarm combat mission model, and the UAV swarm combat mission model is a mathematical model based on a Markov decision process MDP; The combat mission end condition is that the combat mission end time is reached or the UAV group completes the combat mission within the specified time; The action set is the basic mission of the drone, including taking cover, waiting in place, searching, heading to the bombing point and / or bombing; The state set is a system environment state and a combat unit state, wherein the system environment state includes an enemy base intact, an enemy base severely destroyed, or an enemy base slightly destroyed, and the combat unit state is a drone state, and the drone state includes: drone position, drone fuel level, drone damage status, and / or drone ammunition level; The reward function represents the immediate reward obtained by the drone after completing the subtask, and the immediate reward includes the reward obtained by the drone from the system environment after completing the task and the cost of the drone taking action; The objective function is the behavior criterion of the UAV, and the objective function is defined as: Execute for said drone Rewards after actions, is the current state of the drone, is the discount factor, Execute for said drone The next state after the action, is the maximum action value in the next state of the drone, is the duration of the drone action; S2, establish a simulation environment for combat mission planning; S3, decomposing the combat mission into a series of subtasks executed by parallel combat units, wherein the combat mission includes a combat mission level, and when each subtask is decomposed from the combat mission, the combat mission level changes according to a predetermined rule, wherein the meaning of each subtask is evaluated by the command center according to the current state; each subtask represents an option, and the option is an action sequence, and the subtask includes a task of bombing an enemy base A, a task of bombing an enemy base B, and / or a task of bombing an enemy base C; S4, constructing a hierarchical decision framework, wherein the hierarchical decision framework is divided into a high-level decision framework and a low-level decision framework, the high-level decision framework is used to select an action sequence to be executed, automatically generate an action sequence, and adjust the strategy according to the feedback of the low-level decision framework, and the low-level decision framework is used to process the specific action selection in the action sequence and feedback the processing result; S5, training the combat mission planning neural network model and building the combat mission planning system; The action sequence is based on the Q-learning algorithm, which includes: a state set, an action set, an immediate reward, a Q value table, and an action selection, wherein: The state set is a subset of the system environment state in step S1; The action set is a subset of the action set in step S1; The instant reward includes state action benefit and action cost. The state action benefit is based on the principle that each action will have an instant reward. The state action benefit is expressed as , is a predefined benefit value, and the action cost is expressed as , is a predefined cost value, is the duration of the action, is the number of executing drones; The Q value table is an action-time pair, representing the benefit of taking a under s; The action selection specifies the criteria and basis for the action, wherein the action selection is selected by substituting the following formula into the Q value table: is the probability of selecting action a in state s, is an action-time pair, e is a constant greater than 0, represents the i-th action in A(s), and A(s) represents the set of actions taken in state s; The automatic generation of action sequences by the high-level decision framework in step S4 includes: dynamically generating action sequences through system environment and drone status evaluation; when an action sequence ends or an external event occurs, an evaluation is performed to determine the next action sequence, wherein the external event includes: damage to a friendly drone and / or changes in the status of an enemy base.
2. The simulation-based task planning method according to claim 1, wherein: The simulation deduction environment for the combat mission planning in step S2 is a deduction platform in which a pre-selected UAV performs actions and returns a report.
3. The simulation-based task planning method according to claim 1, wherein: The optional multi-layer reinforcement learning algorithm includes: loading Q value table, state initialization, upper layer selection action sequence, execution action sequence, updating action sequence information, time monitoring, state judgment, wherein the updating action sequence information is realized by the following formula: is the state-action value function after the action sequence is updated, is the learning rate, is the discount factor, Is the action sequence to perform the action After the reward, Is the action sequence to perform the action Then transfer to the next state, is the action sequence that maximizes The action value of the value is the optimal action value.
4. A device for task planning for simulation deduction, characterized in that: It includes a memory and a processor, the memory stores at least one program, and the at least one program is executed by the processor to implement the steps of the simulation-oriented task planning method as described in any one of claims 1 to 3.
5. A computer-readable storage medium, characterized in that: The storage medium stores at least one program, and the at least one program is executed by a processor to implement the steps of the simulation-oriented task planning method as described in any one of claims 1 to 3.
6. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the simulation-oriented task planning method as described in any one of claims 1 to 3 are implemented.
