Heterogeneous multi-unmanned-system task decision-making and path planning method based on multi-modal multi-task reinforcement learning

Abstract

The invention discloses a heterogeneous multi-unmanned-system task decision-making and path planning method based on multi-modal multi-task reinforcement learning, and belongs to the field of artificial intelligence. According to the invention, the problem that a single-class agent cannot complete complex tasks is solved. And uniformly coding the multi-modal information, extracting position information and obstacle information of the unmanned surface vehicles of the two parties from a coding result, inputting the position information and the obstacle information into a reinforcement learning algorithm based on PPO for task decision making, and sending the multi-task information to the unmanned surface vehicles. The red unmanned ship preprocesses the received multi-task text information, takes the obtained position of the red unmanned ship, the target position and the obstacle position information as a state space of path planning, and executes a specified task after arriving at the target position according to a planned path until a task switching moment is reached; and updating network parameters in the reinforcement learning algorithm according to the updated total reward of each unmanned aerial vehicle, and re-deciding a task based on the updated parameters. The method can be applied to task decision making and path planning of the heterogeneous multi-unmanned system.

Description

Technical Field

[0001] The present invention belongs to the field of artificial intelligence, and specifically relates to a heterogeneous multi-unmanned system task decision-making and path planning method based on multi-modal multi-task reinforcement learning. Background Art

[0002] In recent years, with the continuous advancement of science and technology, the field of artificial intelligence has developed rapidly. As an important branch of artificial intelligence, multi-agent reinforcement learning has made breakthrough progress in the fields of industry, military, transportation, etc. It aims to study how multiple agents can complete tasks together through interaction and collaboration in a common environment. Thanks to the communication and cooperation between agents, multi-intelligence systems can have stronger adaptability and higher efficiency to collaborate and complete complex tasks in complex environments. With the continuous advancement of data acquisition technology and data fusion technology, agents can obtain rich information through various channels, such as image information, situation data and other multimodal information. This enables dozens or even hundreds of agents to fully understand the changes in their environment under the input of multimodal information, and make autonomous task decisions based on this through reinforcement learning.

[0003] However, in real life, many complex tasks cannot be completed by a single type of intelligent agent. They require multiple intelligent agents with different functions, characteristics and capabilities to work together. These systems with different types of intelligent agents are called heterogeneous multi-unmanned systems. The automation and intelligence of complex tasks are achieved through the collaborative work of multiple intelligent agents. Therefore, in the scenario of collaborative autonomous confrontation, in order to solve the problem that a single type of intelligent agent cannot complete complex tasks, it is necessary to organize and control a heterogeneous multi-unmanned system composed of multiple types of intelligent agents to complete specific complex tasks. When a heterogeneous multi-unmanned system completes a complex task, it needs to go through two stages: task decision-making and path planning. First, the complex task is decomposed into several types of subtasks. Each intelligent agent considers the task requirements and its own capabilities to autonomously decide to execute the subtask, thereby improving the overall efficiency. Secondly, in the process of executing the task, the intelligent agent plans the path according to the surrounding environment, so that it can complete its own task efficiently and accurately. Summary of the invention

[0004] The purpose of the present invention is to solve the problem that a single category of intelligent agents cannot complete complex tasks, and proposes a heterogeneous multi-unmanned system task decision-making and path planning method based on multimodal multi-task reinforcement learning.

[0005] The technical solution adopted by the present invention to solve the above technical problems is: a heterogeneous multi-unmanned system task decision and path planning method based on multi-modal multi-task reinforcement learning, the method comprising a multi-unmanned aerial vehicle task decision layer and a multi-unmanned boat path planning layer;

[0006] The working process at the multi-UAV mission decision layer is:

[0007] Step 1: The vth Red UAV obtains modal data u containing the status information of both UAVs, the location information of both UAVs, and the obstacle information in the environment, u=(u1,u 2,…， u U ), where u1,u 2,…, u U Respectively represent the first type of modal data, the second type of modal data, …, the Uth type of modal data;

[0008] Then the acquired modal data is encoded to obtain a modal data encoding result;

[0009] Step 2: The modal data encoding result is used as the input of the task decision algorithm embedded in the vth red drone, and then the task decision algorithm outputs the action, the position information of the unmanned boats of both sides and the position information of obstacles, and sends the position information of the unmanned boats of both sides, the position information of obstacles and the task information represented by the action to the red surface multi-unmanned boats;

[0010] The working process in the multi-unmanned boat path planning layer is as follows:

[0011] Step 3: After the Red Team's multiple unmanned boats receive the position information of both unmanned boats, obstacle position information, and mission information sent by the vth Red Team UAV, each Red Team's unmanned boat uses its own current position information, target position information, and obstacle position information to form its own embedded state space of the path planning algorithm;

[0012] The path planning algorithm then outputs the actions performed at each step in the process of the Red Army's unmanned boat moving towards the target location. After the Red Army's unmanned boat reaches the target location, it starts to perform the mission until the mission switching moment is reached or the confrontation ends.

[0013] If the confrontation ends, the algorithm ends;

[0014] If the task switching time is reached, return to step 1.

[0015] The beneficial effects of the present invention are:

[0016] The present invention designs a heterogeneous multi-unmanned system task decision and planning integrated method based on multimodal multi-task reinforcement learning, uniformly encodes multimodal information, extracts the position information and obstacle information of both unmanned boats from the encoding results, inputs the extracted information into the reinforcement learning algorithm based on PPO, and the unmanned boat uses the reinforcement learning algorithm based on PPO to make task decisions, and then sends the position information, obstacle information and multi-task information of both unmanned boats to the unmanned boat. The red party unmanned boat pre-processes the received multi-task text information to obtain its own position information, target position information and obstacle position information, uses the target position information, obstacle position information and the current position information of the red party unmanned boat as the state space of path planning, uses the LSTM-TD3 algorithm based on heuristic knowledge to plan the path to reach the target position to perform the specified task, updates the total reward of each unmanned boat after reaching the switching time of the task, and updates the parameters in the reinforcement learning algorithm network based on PPO according to the total reward to optimize the algorithm, and then the unmanned boat re-decides the task for the unmanned boat to execute. The method of the present invention enables multiple intelligent agents to make task decisions and path planning, complete complex tasks in an interactive and collaborative manner, and organize these heterogeneous intelligent agents into an efficient and collaborative group, thereby improving overall performance. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Figure 1 This is a framework diagram of a heterogeneous multi-unmanned system task decision-making and path planning method based on multi-modal multi-task reinforcement learning;

[0018] Figure 2 It is a flowchart of the task decision process of heterogeneous multi-unmanned system based on PPO algorithm;

[0019] Figure 3 It is a flow chart for encoding multimodal data;

[0020] Figure 4 This is a framework diagram of the multi-unmanned boat path planning process based on the LSTM-TD3 algorithm based on heuristic knowledge;

[0021] Figure 5 This is a flow chart of the task decision-making and path planning method for heterogeneous multi-unmanned systems based on multimodal multi-task reinforcement learning. DETAILED DESCRIPTION

[0022] Specific implementation method 1: This implementation method describes a heterogeneous multi-unmanned system task decision and path planning method based on multi-modal multi-task reinforcement learning, the method comprising a multi-unmanned aerial vehicle task decision layer and a multi-unmanned boat path planning layer;

[0023] The working process at the multi-UAV mission decision layer is:

[0024] Step 1: The vth Red UAV obtains modal data u containing the status information of both UAVs, the location information of both UAVs, and the obstacle information in the environment, u=(u1,u 2,…， u U ), where u1,u 2,…, u U Respectively represent the first type of modal data, the second type of modal data, …, the Uth type of modal data;

[0025] The two parties in the present invention are the red party and the blue party, and the red party and the blue party are in confrontation;

[0026] Then the acquired modal data is encoded to obtain a modal data encoding result;

[0027] Step 2: The modal data encoding result is used as the input of the task decision algorithm embedded in the vth red drone, and then the task decision algorithm outputs the action, the position information of the unmanned boats of both sides and the position information of obstacles, and sends the position information of the unmanned boats of both sides, the position information of obstacles and the task information represented by the action to the red surface multi-unmanned boats;

[0028] The working process in the multi-unmanned boat path planning layer is as follows:

[0029] Step 3: After the Red Team's multiple unmanned boats receive the position information of both unmanned boats, obstacle position information and mission information sent by the vth Red Team UAV (only the Red Team's multiple unmanned boats in the mission area of ​​the vth Red Team UAV receive the information sent by the vth Red Team UAV), each Red Team's unmanned boat (i.e., each Red Team's unmanned boat that receives the information of the vth Red Team UAV) uses its own current position information, target (i.e., all Blue Team's unmanned boats in the mission area of ​​the vth Red Team UAV) position information and obstacle position information to form its own embedded state space of the path planning algorithm;

[0030] Then, the path planning algorithm outputs the corresponding actions of the Red Army's unmanned boat in each step of its journey to the target location. After the Red Army's unmanned boat reaches the target location, it starts to perform the mission until the mission switching moment is reached or the confrontation ends.

[0031] If the confrontation ends, the algorithm ends;

[0032] If the task switching time is reached, use the updated reward of the vth Red Team UAV to return to step 1.

[0033] Each drone of the Red Team executes the process of this embodiment. The method of the present invention is composed of a multi-drone task decision layer and a multi-unmanned boat path planning layer. Each drone of the task decision layer is responsible for its own task area, that is, when making a task decision, each drone obtains multimodal data in its own task area, encodes the obtained multimodal data to obtain the encoding result, and then inputs the encoding result into its own embedded task decision algorithm, and sends task information to the multi-unmanned boats controlled by itself through the task decision algorithm (it should be noted that the tasks decided by a drone to each unmanned boat controlled by itself can be different). Taking the vth Red Team drone as an example, the vth Red Team drone sends the decided task to each unmanned boat controlled by itself, and each unmanned boat that receives the task performs path planning to reach the target position and executes the corresponding task. The cumulative total reward of the vth Red Team drone is updated according to the task execution situation, and then the network parameters of the task decision algorithm corresponding to the vth Red Team drone are updated according to the cumulative total reward, and the updated network parameters are used for the next task decision.

[0034] Specific implementation method 2: Combination Figure 3 The difference between this embodiment and the first embodiment is that when the modal data u=(u1,u2) acquired by the vth Red Team UAV, where u1 represents image data and u2 represents situation data, the encoding method of the modal data u is:

[0035] The dimension of the image data u1 is recorded as H×W×3, where H is the height of the image data u1 and W is the width of the image data u1;

[0036] The image data u1 is preprocessed to generate RGB image data of dimension N×N×3, where N is the width and height of the RGB image data. The generated RGB image is then passed through the convolution layer, pooling layer, and fully connected layer in sequence, and the fully connected layer outputs a d1×1-dimensional image data feature vector;

[0037] The situation data u2 containing the status information of both unmanned boats, the position information of both unmanned boats and the obstacle information in the environment is processed to generate structured situation data, and a plane map is generated based on the structured situation data. The generated plane map is then passed through a fully connected layer to generate a d2×1-dimensional situation data feature vector;

[0038] Then, the image data feature vector and the situation data feature vector are concatenated row by row to obtain a (d1+d2)×1-dimensional feature vector, and the concatenated feature vector is used as the modal data encoding result.

[0039] The other steps and parameters are the same as those in the first embodiment.

[0040] Specific implementation method three: This implementation method is different from specific implementation method one in that when the modal data u acquired by the vth Red Team UAV only includes situation data, the situation data is first preprocessed to generate structured situation data, and then a plane graph is generated based on the structured situation data. The generated plane graph is passed through a fully connected layer to generate a d2×1-dimensional situation data feature vector, and the generated situation data feature vector is used as the modal data encoding result.

[0041] The other steps and parameters are the same as those in the first embodiment.

[0042] Specific implementation method 4: This implementation method is different from specific implementation method 1 in that the task decision algorithm is a reinforcement learning algorithm based on PPO.

[0043] The other steps and parameters are the same as those in the first embodiment.

[0044] Specific implementation method 5: This implementation method is different from the specific implementation method 4 in that the action space output by the task decision algorithm is m, where m = (m1, m2, ..., m M ), m1,m2,...,m M They represent the first type of tasks, the second type of tasks, …, the Mth type of tasks respectively.

[0045] The other steps and parameters are the same as those in the fourth embodiment.

[0046] Specific implementation method 6: This implementation method is different from specific implementation method 5 in that the value of M is 3, that is, the action space m = (m1, m2, m3), m1 is the inspection task, m2 is the strike task, and m3 is the reinforcement task;

[0047] And design the reward function according to the action space. The specific process is:

[0048] If all the Red Army UAVs that received the mission information of the vth Red Army UAV are destroyed while performing the current round mission, execute step 1;

[0049] If all the Red UAVs that have received the mission information of the vth Red UAV are performing the mission of the current round, and all the Blue UAVs in the mission area of ​​the vth Red UAV are destroyed, then execute step 2;

[0050] Otherwise, go to step 3;

[0051] Step 1: After all the Red Team UAVs that have received the mission information of the vth Red Team UAV have completed the current round of missions, the cumulative total reward of the vth Red Team UAV is:

[0052] R v＝ R' v +R lose

[0053] Among them, R v is the cumulative total reward of the vth Red UAV after the current round of mission decision (including the total reward of the current round and all previous rounds), R' v is the cumulative total reward of the vth Red UAV after the last round of mission decision (i.e., the sum of the rewards obtained from the last round and the mission decision before the last round), R lose It is the penalty that the vth Red UAV receives when all the Red UAVs that received the mission information of the vth Red UAV are destroyed;

[0054] Step 2: After all the Red Team UAVs that have received the mission information of the vth Red Team UAV have completed the current round of missions, the cumulative total reward of the vth Red Team UAV is:

[0055] R v＝ R' v +R win

[0056] Among them, R win It is the reward that the vth Red UAV receives when all the Red UAVs that have received the mission information of the vth Red UAV have performed their missions and all the Blue UAVs in the mission area of ​​the vth Red UAV have been destroyed;

[0057] Step 3: For the i-th Red Army UAV that receives the mission information of the v-th Red Army UAV:

[0058] When the i-th Red Army unmanned boat performs a patrol mission, give the v-th Red Army UAV K1*R p K1 is the total number of blue unmanned boats found by the i-th red unmanned boat during the patrol mission of the current round, R p It is the reward for discovering a blue team's unmanned boat.

[0059] When the i-th Red Army unmanned boat performs a strike mission, if the i-th Red Army unmanned boat is not destroyed, give the v-th Red Army drone K2*R e The reward is , where K2 is the total number of blue unmanned boats destroyed by the i-th red unmanned boat when performing the strike mission in the current round, R e is the reward for destroying a blue unmanned boat during the strike mission. If the i-th red unmanned boat is destroyed, the v-th red drone will receive R d punishment;

[0060] When the i-th Red Army unmanned boat is performing a reinforcement mission, if the i-th Red Army unmanned boat arrives at the location of the friendly party seeking help, it will give the v-th Red Army drone R hIf the i-th Red UAV fails to reach the location of the friendly party seeking help, then the v-th Red UAV will neither be rewarded nor punished;

[0061] Similarly, according to the task execution status of each Red Army unmanned boat that received the task information of the vth Red Army UAV, the total reward R obtained by the vth Red Army UAV for deciding the current round of tasks is calculated, and then the cumulative total reward R of the vth Red Army UAV after the current round of task decision is obtained. v＝ R' v +R.

[0062] The other steps and parameters are the same as those in the fifth embodiment.

[0063] For a drone, after each mission decision, the multi-drone mission decision layer obtains the red drone reward based on the red drone’s mission performance, updates the network parameters based on the PPO reinforcement learning algorithm based on the reward, and outputs a new decision task based on the updated network parameters for the red drone to execute.

[0064] Specific implementation method seven: This implementation method is different from specific implementation method one in that the path planning algorithm is an LSTM-TD3 algorithm based on heuristic knowledge.

[0065] The other steps and parameters are the same as those in the first embodiment.

[0066] Specific implementation method 8: This implementation method is different from specific implementation method 7 in that the action space a of the path planning algorithm is a=(a1,a 2,…， a A ), a1,a 2,…， a A They are the first action, the second action,…, the Ath action in the action space a respectively.

[0067] The other steps and parameters are the same as those in the seventh embodiment.

[0068] Specific implementation method 9: This implementation method is different from specific implementation method 8 in that the value of A is 3, that is, a=(a1, a2, a3), a1 represents the left row, a2 represents the straight row, and a3 represents the right row.

[0069] The other steps and parameters are the same as those in the eighth embodiment.

[0070] Specific implementation method ten: This implementation method is different from specific implementation method nine in that the reward function of the path planning algorithm is:

[0071] R k =R tar +R d +R num+R a +R int , the unmanned boat reaches the target location

[0072] R k =R col +R d +R num +R a +R int , unmanned boat collides with obstacles

[0073] Among them, R k is the cumulative total reward obtained by the kth Red Team’s unmanned boat after completing the current step when sailing towards the target position, R d is the cumulative distance reward, R num is the cumulative step penalty, R a is the cumulative position reward, R int is the cumulative heuristic knowledge reward, R tar is the target reward obtained after the unmanned boat reaches the target position, R col It is the penalty that the Red Team’s unmanned boat receives when it collides with an obstacle;

[0074] R d The calculation method is:

[0075] R d =R' d +d*(2 / 3) dcur / dstart

[0076] Among them, R' d is the cumulative distance reward of the kth Red Team’s unmanned boat after executing the previous step, R d is the cumulative distance reward of the kth red unmanned boat after executing the current step, d cur is the distance between the position reached by the kth unmanned boat after executing the current step and the target position, d start is the distance between the initial position and the target position of the kth unmanned boat, and d is the distance reward parameter;

[0077] R num The calculation method is:

[0078] R num =R' num +β

[0079] Among them, R' num is the cumulative step penalty after the kth Red Team’s unmanned boat executes the previous step, R num is the cumulative step penalty after the kth Red UAV executes the current step, and β is the step penalty parameter;

[0080] R a The calculation method is:

[0081] R a =R' a +α*cosθ

[0082] Among them, R' a is the cumulative position reward of the kth Red Team’s unmanned boat after executing the previous step, R a is the cumulative azimuth reward of the kth red unmanned boat after executing the current step, α is the azimuth reward parameter, and θ is the angle between the line between the position reached by the kth unmanned boat after executing the current step and the target position and the heading of the kth unmanned boat;

[0083] R int The calculation method is:

[0084] R int =R' int +R i

[0085] Among them, R' int is the cumulative heuristic knowledge reward of the kth Red Team’s unmanned boat after executing the previous step, R i is the heuristic knowledge reward brought by executing the current step, R int It is the cumulative heuristic knowledge reward of the kth red unmanned boat after executing the current step.

[0086] The other steps and parameters are the same as those in the ninth embodiment.

[0087] In the path planning algorithm, after each action is completed, an unmanned boat will feedback rewards to the path planning algorithm embedded in itself to guide the next action of the path planning output.

[0088] Example

[0089] This embodiment proposes a heterogeneous multi-unmanned system task decision and path planning method based on multi-modal multi-task reinforcement learning, such as Figure 1 As shown, the method of this embodiment includes a multi-UAV task decision layer and a multi-UAV path planning layer;

[0090] 1. Multi-UAV mission decision design based on multi-modal multi-task reinforcement learning

[0091] The process of the multi-UAV task decision algorithm based on multi-modal multi-task reinforcement learning in this embodiment is as follows: Figure 2As shown, first, the multimodal data is uniformly encoded, and useful information is preferentially extracted from the image information as the input of the PPO algorithm. If there is no image information input, a plane map is formed by structured situation data, and the state information and position information of the multi-agent are extracted from the plane map as the input of the PPO algorithm. According to the above input, the multi-UAVs make decisions based on the PPO reinforcement learning training algorithm and send the tasks to the surface unmanned boat for execution. If the switching moment of the periodic task has not been reached, the unmanned boat continues to perform the original task. On the contrary, if the switching moment of the periodic task is reached, the PPO algorithm is optimized by updating the reward and then updating the parameters in the algorithm network, and a new decision task is output for the unmanned boat to execute. This cycle is repeated until the confrontation round ends. The present invention designs the unified encoding of the multimodal data of the unmanned aerial vehicle, the reward function of the reinforcement learning, and the multi-task output action space, so that the red side's multi-UAVs can make decisions on multiple tasks for the red side's surface unmanned boat to execute under the input of multimodal information.

[0092] (1) Unified coding design for multi-UAV multi-modal data input

[0093] The input of multiple drones is defined as multimodal data u=(u1,u 2,…， u U ), including U types of modal data. The present invention takes the modal type number U=2 of the input data as an example, that is, u=(u1,u2), where u1 is the image data and u2 is the situation data. The multimodal data processing flow is as follows Figure 3 As shown in the figure, the H×W×3-dimensional image data is input, and the N×N×3-dimensional RGB image data is generated through the image preprocessing process. Then, the d1×1-dimensional image feature vector is extracted through the convolution layer, pooling layer and fully connected layer to complete the encoding process of the image data. The input includes situation data such as "status information of both unmanned boats", "position information of both unmanned boats" and "obstacle information in the environment". After the data preprocessing process, the structured situation data is generated, and then the d2×1-dimensional situation data feature vector is generated by the fully connected layer to complete the situation data encoding process. Finally, the two types of feature vectors are spliced ​​into (d1+d2)×1-dimensional feature vectors by row to achieve unified encoding, and the feature vectors obtained by unified encoding are used as the input of the PPO reinforcement learning training algorithm.

[0094] (2) Design of reward function for multi-UAV reinforcement learning with multi-modal input and multi-task output, specifically:

[0095] If all the Red Army UAVs that received the mission information of the vth Red Army UAV are destroyed while performing the current round mission, execute step 1;

[0096] If all the Red UAVs that have received the mission information of the vth Red UAV are performing the mission of the current round, and all the Blue UAVs in the mission area of ​​the vth Red UAV are destroyed, then execute step 2;

[0097] Otherwise, go to step 3;

[0098] Step 1: After all the Red Team UAVs that have received the mission information of the vth Red Team UAV have completed the current round of missions, the cumulative total reward of the vth Red Team UAV is:

[0099] R v＝ R' v +R lose

[0100] Among them, R v is the cumulative total reward of the vth Red UAV after the current round of mission decision, R' v is the cumulative total reward of the vth Red UAV after the last round of mission decision, R lose It is the penalty that the vth Red UAV receives when all the Red UAVs that received the mission information of the vth Red UAV are destroyed;

[0101] Step 2: After all the Red Team UAVs that have received the mission information of the vth Red Team UAV have completed the current round of missions, the cumulative total reward of the vth Red Team UAV is:

[0102] R v＝ R' v +R win

[0103] Among them, R win It is the reward that the vth Red UAV receives when all the Red UAVs that have received the mission information of the vth Red UAV have performed their missions and all the Blue UAVs in the mission area of ​​the vth Red UAV have been destroyed;

[0104] Step 3: For the i-th Red Army UAV that receives the mission information of the v-th Red Army UAV:

[0105] When the i-th Red Army unmanned boat performs a patrol mission, give the v-th Red Army UAV K1*R p K1 is the total number of blue unmanned boats found by the i-th red unmanned boat during the patrol mission of the current round, R p It is the reward for discovering a blue team's unmanned boat.

[0106] When the i-th Red Army unmanned boat performs a strike mission, if the i-th Red Army unmanned boat is not destroyed, give the v-th Red Army drone K2*R e The reward is , where K2 is the total number of blue unmanned boats destroyed by the i-th red unmanned boat when performing the strike mission in the current round, Re is the reward for destroying a blue unmanned boat during the strike mission. If the i-th red unmanned boat is destroyed, the v-th red drone will receive R d punishment;

[0107] When the i-th Red Army unmanned boat is performing a reinforcement mission, if the i-th Red Army unmanned boat arrives at the location of the friendly party seeking help, it will give the v-th Red Army drone R h If the i-th Red UAV fails to reach the location of the friendly party seeking help, then the v-th Red UAV will neither be rewarded nor punished;

[0108] Similarly, according to the task execution status of each Red Army unmanned boat that received the task information of the vth Red Army UAV, the total reward R obtained by the vth Red Army UAV for deciding the current round of tasks is calculated, and then the cumulative total reward R of the vth Red Army UAV after the current round of task decision is obtained. v＝ R' v +R.

[0109] Since the confrontation environment is relatively complex, various factors in various mission situations will affect the success rate of the red drone's decision-making. Therefore, the present invention designs a reward function according to the algorithm flow in Table 1, and designs the reward function of the multi-drone reinforcement learning training algorithm for multi-modal input and multi-task output to consist of the following rewards:

[0110] ①Reward for finding the target

[0111] The red drone makes a decision to output the patrol mission and sends it to the red unmanned boat. When the red unmanned boat receives the mission, each time it finds a blue unmanned boat during the patrol mission, the drone that makes the decision will receive a reward R. p , the reference value of this reward is 50.

[0112] ②Rewards for successful attack

[0113] The Red UAV makes a decision to output the strike mission and sends it to the Red UAV. When the Red UAV that receives the mission performs the strike mission, for every Blue UAV destroyed, the UAV that makes the decision will receive a reward R e , the reference value of this reward is 200.

[0114] ③Penalty for failure to strike

[0115] The Red UAV makes a decision to output a strike mission and sends it to the Red UAV. If the Red UAV that receives the mission is destroyed while executing the strike mission, the decision-making UAV will be penalized R. d , the reference value of this penalty is -200.

[0116] ④Rewards for successful reinforcement

[0117] The Red UAV makes a decision to output a reinforcement mission and sends it to the Red UAV. When the Red UAV receives the mission and reaches the location of the friendly party seeking assistance while performing the reinforcement mission, the UAV that made the decision will receive a reward R. h , the reference value of this reward is 100.

[0118] ⑤ Overall victory and defeat rewards

[0119] When all the red or blue unmanned boats in a drone's mission area are eliminated, or when the maximum number of rounds Pmax is reached, all red drones will be given an overall victory reward. When all blue unmanned boats are eliminated, all red drones that make decisions will receive a reward Rwin, with a reference value of 500. When all red unmanned boats are eliminated, all red drones that make decisions will be penalized Rlose, with a reference value of -500. If the maximum number of rounds is reached, no rewards or penalties will be given to red drones.

[0120] Table 1

[0121]

[0122]

[0123] (3) Action space design for multi-UAV multi-task output

[0124] The action value output by the algorithm network is defined as the multi-UAV decision task m = (m1, m2, ..., m M ), m includes M kinds of tasks. The present invention takes 3 as an example, that is, m = (m1, m2, m3), where m1 is the inspection task, m2 is the strike task, and m3 is the reinforcement task. For a multi-task agent, after the task switching cycle, the agent autonomously decides the task to be executed in the next cycle. The definitions of the three tasks selected for decision-making in the present invention are introduced as follows:

[0125] ① Inspection tasks

[0126] The Red side's drone flies at a low speed, taking pictures of the sea surface. When it finds the Blue side's unmanned boat, it sends the Blue side's unmanned boat's location information and information about obstacles such as sea reefs to the nearby Red side's unmanned boat so that the Red side's unmanned boat can plan a path to complete the attack on the Blue side's unmanned boat.

[0127] ②Strike mission

[0128] The red drone observes the blue drone entering the field of view and attacks the blue drone. And when it finds that the number of blue drones in the field of view is greater than the number of red drones, it sends its own location information to the nearby red drones to request to increase.

[0129] ③Reinforcement mission

[0130] When receiving a request for help from friendly forces, the Red Army drones that are not carrying out strike missions will fly to the location of the friendly forces requesting help to provide reinforcements.

[0131] After deciding on the mission, multiple UAVs send mission information to the Red side's multiple UAVs in the text format of <Red side K UAV, mission name, target location, obstacle location> for the UAVs to execute, such as <Red side 1 UAV, patrol mission, 0010 location, 0001 location>.

[0132] 2. Path planning design of surface unmanned boat based on LSTM-TD3 algorithm based on heuristic knowledge

[0133] The framework of multi-unmanned boat path planning method based on deep reinforcement learning is as follows Figure 4 As shown, after receiving the multi-task information sent by the multi-drone, the multi-unmanned boat performs text preprocessing on it, and obtains the target position information, obstacle position information and current position information as the state space of path planning. By using the LSTM-TD3 algorithm based on heuristic knowledge as the path planning algorithm of the multi-unmanned boat to output the next action, after multiple steps, the target position is reached to complete the corresponding task. The present invention respectively designs the preprocessing of the multi-task information input of the unmanned boat planning layer, the state space of the multi-unmanned boat path planning, the reward function for the path planning algorithm and the action space output by the multi-unmanned boat path planning, so that the red side's multi-unmanned boat can reach the target position and complete the corresponding task under the input of multi-task information.

[0134] (1) Preprocessing of multi-task information input at the unmanned boat planning layer

[0135] After receiving the multi-task information in the text format of <Red Party K Unmanned Boat, Mission Name, Target Location, Obstacle Location>, the Red Party’s multi-unmanned boat preprocesses it and extracts the target location information and obstacle location information, providing guidance for the subsequent Red Party K Unmanned Boat to implement path planning based on the LSTM-TD3 algorithm of heuristic knowledge to reach the target location to perform the task.

[0136] (2) State space design for multi-UAV path planning

[0137] When the Red Team’s unmanned boat is planning its path, the state space of the unmanned boat is designed as s = (s1, s 2,…， s S ). Since the unmanned boat needs to reach the target position to complete the specified task while avoiding collision with obstacles in the environment, the present invention selects the number of unmanned boat states to be 3, that is, s = (s1, s2, s3), where s1 is the position of obstacles such as reefs, s2 is the target position that the red party unmanned boat needs to reach, and s3 is the current position of the red party unmanned boat.

[0138] (3) Reward function design for multi-UAV path planning algorithm

[0139] Since the path planning process of the Red Team’s unmanned boat takes a long time, a continuous reward function is used to reshape the training process and provide reward feedback in real time. The design method of the reward function for the multi-unmanned boat path planning algorithm is shown in Algorithm 2 in Table 2:

[0140] Table 2

[0141]

[0142]

[0143] The present invention takes into account various influencing factors in the path planning process of the unmanned boat, so the reward function of the path planning of the unmanned boat is designed to consist of the following rewards:

[0144] ① Rewards for reaching the target

[0145] After the Red Team's unmanned boat reaches the target location, it will receive the target reward R tar , the reference value of this reward is 500.

[0146] ②Breakage Penalty

[0147] When the Red Team's unmanned boat collides with an obstacle such as a reef, it will be penalized with a collision obstacle penalty R col , the reference value of this reward is -200.

[0148] ③Distance Reward

[0149] In order to guide the red team's unmanned boat to reach the target point efficiently and accurately, a distance reward is given to the unmanned boat according to the distance between the unmanned boat and the target point. The present invention selects the distance reward parameter as d, and the reference value of this parameter is 8.

[0150] ④ Step Penalty

[0151] In order to enable the red party's unmanned boat to find an optimal path in the shortest possible time and enable the intelligent agent's unmanned boat to reach the target point in the least possible number of steps, a step penalty is introduced. The present invention selects a step penalty parameter of β, and the reference value of this parameter is -10.

[0152] ⑤ Position Rewards

[0153] Since the angle between the direction of the red team's unmanned boat and the target point during its movement has a certain influence on the planning, the smaller the angle, the better the planning effect, so the orientation reward is introduced. The present invention selects the orientation reward parameter as α, and the reference value of this parameter is 6.

[0154] ⑥Inspirational knowledge rewards

[0155] In order to prevent the Red Team’s unmanned boat from blindly exploring the path during the planning process, the present invention introduces the heuristic knowledge reward R, taking into account the influence of factors such as the distance change between the unmanned boat and the target point and the distance between the unmanned boat and the obstacle. i The method for selecting the reference value of this reward is shown in Table 3.

[0156] Table 3 Selection method of reference value of heuristic knowledge reward

[0157]

[0158] Specifically, when the unmanned boat approaches an obstacle, it should be penalized to ensure that the unmanned boat maintains a safe distance from the obstacle during driving. When the unmanned boat successfully avoids obstacles and moves away from them, it should be rewarded accordingly. And when the unmanned boat gets closer and closer to the target location, it should be rewarded, and when it moves away from the target location, it should be penalized.

[0159] (4) Action space design for multi-UAV path planning output

[0160] When the Red Agent Unmanned Boat performs path planning, the continuous actions of the unmanned boat are discretized, so its action space is designed to be a = (a1, a 2,…， a A ). Considering the kinematic characteristics of the unmanned boat, the present invention selects the number of unmanned boat actions as 3, that is, a = (a1, a2, a3). Among them, a1 is left movement, a2 is straight movement, and a3 is right movement. The linear speeds of the three actions are all set to 15m, the angular velocity of the left movement is set to -0.8rad / s, the angular velocity of the right movement is set to 0.8rad / s, and the angular velocity of the straight movement is set to 0.

[0161] 3. Integrated process of task decision-making and planning for heterogeneous multi-unmanned systems based on multi-modal multi-task reinforcement learning

[0162] The process of the integrated method for task decision-making and planning of heterogeneous multi-unmanned systems based on multi-modal multi-task reinforcement learning designed by the present invention is as follows: Figure 5 As shown in the figure, first, the multimodal information is uniformly encoded, and the state information and position information of both intelligent agents are extracted from the encoding results. The extracted information is input into the PPO algorithm, and the UAV makes task decisions based on the algorithm, and then sends the multi-task information to the unmanned boat. The red unmanned boat uses the LSTM-TD3 algorithm based on heuristic knowledge to plan the path to the target location to perform the specified task. After reaching the switching time of the task, the total reward of each UAV is updated, and the parameters in the PPO algorithm network are updated according to the total reward to optimize the algorithm. Then the UAV re-decides the task for the unmanned boat to execute. On the contrary, if the switching time of the periodic task has not been reached, the original task will continue to be executed. This cycle continues until the confrontation round ends.

[0163] The above calculation examples of the present invention are only used to explain the calculation model and calculation process of the present invention in detail, and are not intended to limit the implementation methods of the present invention. For ordinary technicians in the relevant field, other different forms of changes or modifications can be made based on the above description. It is impossible to list all the implementation methods here. All obvious changes or modifications derived from the technical solution of the present invention are still within the scope of protection of the present invention.

Claims

1. A heterogeneous multi-unmanned system task decision-making and path planning method based on multi-modal multi-task reinforcement learning, characterized in that: The method includes a multi-UAV mission decision layer and a multi-UAV path planning layer; The working process at the multi-UAV mission decision layer is: Step 1: The vth Red UAV obtains modal data u containing the status information of both UAVs, the location information of both UAVs, and the obstacle information in the environment, u=(u1,u 2,…， u U ), where u1,u 2,…, u U Respectively represent the first type of modal data, the second type of modal data, …, the Uth type of modal data; Then the acquired modal data is encoded to obtain a modal data encoding result; Step 2: The modal data encoding result is used as the input of the task decision algorithm embedded in the vth red drone, and then the task decision algorithm outputs the action, the position information of the unmanned boats of both sides and the position information of obstacles, and sends the position information of the unmanned boats of both sides, the position information of obstacles and the task information represented by the action to the red surface multi-unmanned boats; The working process in the multi-unmanned boat path planning layer is as follows: Step 3: After the Red Team's multiple unmanned boats receive the position information of both unmanned boats, obstacle position information, and mission information sent by the vth Red Team UAV, each Red Team's unmanned boat uses its own current position information, target position information, and obstacle position information to form its own embedded state space of the path planning algorithm; The path planning algorithm then outputs the actions performed at each step in the process of the Red Army's unmanned boat moving towards the target location. After the Red Army's unmanned boat reaches the target location, it starts to perform the mission until the mission switching moment is reached or the confrontation ends. If the confrontation ends, the algorithm ends; If the task switching time is reached, return to step 1.

2. The method for task decision and path planning of heterogeneous multi-unmanned systems based on multi-modal multi-task reinforcement learning according to claim 1 is characterized in that: When the modal data u=(u1,u2) acquired by the vth Red Team UAV, where u1 represents image data and u2 represents situation data, the encoding method of the modal data u is: The dimension of the image data u1 is recorded as H×W×3, where H is the height of the image data u1 and W is the width of the image data u1; The image data u1 is preprocessed to generate RGB image data of dimension N×N×3, where N is the width and height of the RGB image data. The generated RGB image is then passed through the convolution layer, pooling layer, and fully connected layer in sequence, and the fully connected layer outputs a d1×1-dimensional image data feature vector; The situation data u2 is preprocessed to generate structured situation data, and a plane graph is generated based on the structured situation data. The generated plane graph is then passed through a fully connected layer to generate a d2×1-dimensional situation data feature vector. Then, the image data feature vector and the situation data feature vector are concatenated row by row to obtain a (d1+d2)×1-dimensional feature vector, and the concatenated feature vector is used as the modal data encoding result.

3. The method for task decision and path planning of heterogeneous multi-unmanned systems based on multi-modal multi-task reinforcement learning according to claim 1 is characterized in that: When the modal data u acquired by the vth red team drone only includes situation data, the situation data is first preprocessed to generate structured situation data, and then a plane map is generated based on the structured situation data. The generated plane map is passed through a fully connected layer to generate a d2×1-dimensional situation data feature vector, and the generated situation data feature vector is used as the modal data encoding result.

4. The method for task decision and path planning of heterogeneous multi-unmanned systems based on multi-modal multi-task reinforcement learning according to claim 1 is characterized in that: The task decision algorithm is a reinforcement learning algorithm based on PPO.

5. The method for task decision and path planning of heterogeneous multi-unmanned systems based on multi-modal multi-task reinforcement learning according to claim 4 is characterized in that: The action space output by the task decision algorithm is m, where m=(m1,m2,...,m M ), m1,m2,...,m M They represent the first type of tasks, the second type of tasks, …, the Mth type of tasks respectively.

6. The method for task decision and path planning of heterogeneous multi-unmanned systems based on multi-modal multi-task reinforcement learning according to claim 5 is characterized in that: The value of M is 3, that is, the action space m = (m1, m2, m3), m1 is the inspection task, m2 is the strike task, and m3 is the reinforcement task; And design the reward function according to the action space. The specific process is: If all the Red Army UAVs that received the mission information of the vth Red Army UAV are destroyed while performing the current round mission, execute step 1; If all the Red UAVs that have received the mission information of the vth Red UAV are performing the mission of the current round, and all the Blue UAVs in the mission area of ​​the vth Red UAV are destroyed, then execute step 2; Otherwise, go to step 3; Step 1: After all the Red Team UAVs that have received the mission information of the vth Red Team UAV have completed the current round of missions, the cumulative total reward of the vth Red Team UAV is: R v＝ R' v +R lose Among them, R v is the cumulative total reward of the vth Red UAV after the current round of mission decision, R' v is the cumulative total reward of the vth Red UAV after the last round of mission decision, R lose It is the penalty that the vth Red UAV receives when all the Red UAVs that received the mission information of the vth Red UAV are destroyed; Step 2: After all the Red Team UAVs that have received the mission information of the vth Red Team UAV have completed the current round of missions, the cumulative total reward of the vth Red Team UAV is: R v＝ R' v +R win Among them, R win It is the reward that the vth Red UAV receives when all the Red UAVs that have received the mission information of the vth Red UAV have performed their missions and all the Blue UAVs in the mission area of ​​the vth Red UAV have been destroyed; Step 3: For the i-th Red Army UAV that receives the mission information of the v-th Red Army UAV: When the i-th Red Army unmanned boat performs a patrol mission, give the v-th Red Army UAV K1*R p K1 is the total number of blue unmanned boats found by the i-th red unmanned boat during the patrol mission of the current round, R p It is the reward for discovering a blue team's unmanned boat. When the i-th Red Army unmanned boat performs a strike mission, if the i-th Red Army unmanned boat is not destroyed, give the v-th Red Army drone K2*R e The reward is , where K2 is the total number of blue unmanned boats destroyed by the i-th red unmanned boat when performing the strike mission in the current round, R e is the reward for destroying a blue unmanned boat during the strike mission. If the i-th red unmanned boat is destroyed, the v-th red drone will receive R d punishment; When the i-th Red Army unmanned boat is performing a reinforcement mission, if the i-th Red Army unmanned boat arrives at the location of the friendly party seeking help, it will give the v-th Red Army drone R h If the i-th Red UAV fails to reach the location of the friendly party seeking help, then the v-th Red UAV will neither be rewarded nor punished; Similarly, according to the task execution status of each Red Army unmanned boat that received the task information of the vth Red Army UAV, the total reward R obtained by the vth Red Army UAV for deciding the current round of tasks is calculated, and then the cumulative total reward R of the vth Red Army UAV after the current round of task decision is obtained. v＝ R' v +R.

7. The method for task decision and path planning of heterogeneous multi-unmanned systems based on multi-modal multi-task reinforcement learning according to claim 1 is characterized in that: The path planning algorithm is an LSTM-TD3 algorithm based on heuristic knowledge.

8. The method for task decision and path planning of heterogeneous multi-unmanned systems based on multi-modal multi-task reinforcement learning according to claim 7 is characterized in that: The action space a of the path planning algorithm is a=(a1,a 2,…， a A ), a1,a 2,…， a A They are the first action, the second action,…, the Ath action in the action space a respectively.

9. The method for task decision and path planning of heterogeneous multi-unmanned systems based on multi-modal multi-task reinforcement learning according to claim 8 is characterized in that: The value of A is 3, that is, a=(a1, a2, a3), a1 represents the left row, a2 represents the straight row, and a3 represents the right row.

10. The method for task decision and path planning of heterogeneous multi-unmanned systems based on multi-modal multi-task reinforcement learning according to claim 9 is characterized in that: The reward function of the path planning algorithm is: R k =R tar +R d +R num +R a +R int , the unmanned boat reaches the target location R k =R col +R d +R num +R a +R int , unmanned boat collides with obstacles Among them, R k is the cumulative total reward obtained by the kth Red Team’s unmanned boat after completing the current step when sailing towards the target position, R d is the cumulative distance reward, R num is the cumulative step penalty, R a is the cumulative position reward, R int is the cumulative heuristic knowledge reward, R tar is the target reward obtained after the unmanned boat reaches the target position, R col It is the penalty that the Red Team’s unmanned boat receives when it collides with an obstacle; R d The calculation method is: R d =R' d +d*(2 / 3) dcur / dstart Among them, R' d is the cumulative distance reward of the kth Red Team’s unmanned boat after executing the previous step, R d is the cumulative distance reward of the kth red unmanned boat after executing the current step, d cur is the distance between the position reached by the kth unmanned boat after executing the current step and the target position, d start is the distance between the initial position and the target position of the kth unmanned boat, and d is the distance reward parameter; R num The calculation method is: R num =R' num +b Among them, R' num is the cumulative step penalty after the kth Red Team’s unmanned boat executes the previous step, R num is the cumulative step penalty after the kth Red UAV executes the current step, and β is the step penalty parameter; R a The calculation method is: R a =R' a +α*cosθ Among them, R' a is the cumulative position reward of the kth Red Team’s unmanned boat after executing the previous step, R a is the cumulative azimuth reward of the kth red unmanned boat after executing the current step, α is the azimuth reward parameter, and θ is the angle between the line between the position reached by the kth unmanned boat after executing the current step and the target position and the heading of the kth unmanned boat; R int The calculation method is: R int =R' int +R i Among them, R' int is the cumulative heuristic knowledge reward of the kth Red Team’s unmanned boat after executing the previous step, R i is the heuristic knowledge reward brought by executing the current step, R int It is the cumulative heuristic knowledge reward of the kth red unmanned boat after executing the current step.

Images