Unmanned aerial vehicle-vehicle cooperation system dual-objective optimization method based on evolutionary reinforcement learning

Abstract

The invention relates to an evolutionary reinforcement learning-based dual-objective optimization method for an unmanned aerial vehicle-vehicle cooperation system. The method comprises the following steps of S1, initializing a strategy population; s2, training all team strategies in the population by using a multi-agent reinforcement learning algorithm; s3, evaluating fitness values of all the teams; s4, iterating a team strategy in the population by using an inter-team coevolution method to generate a new population; s5, judging whether the population strategy converges or not; s6, the unmanned aerial vehicle-vehicle system executes sensing and data unloading tasks according to the strategy network collaborative decision; and S7, deducing data of an unperceived area according to the collected data. According to the method, the advantages of a co-evolution algorithm and multi-agent reinforcement learning are combined, a multi-agent reinforcement learning model learning strategy is utilized, the global optimization direction is controlled through co-evolution among teams, and dual-target optimization of bidirectional strategy collaboration and complex cooperative tasks of heterogeneous unmanned aerial vehicles and vehicles is achieved.

Description

Technical Field

[0001] The present invention relates to the technical field of unmanned aerial vehicle (UAV)-vehicle cooperative system, and in particular to a dual-objective optimization method for an unmanned aerial vehicle (UAV)-vehicle cooperative system based on evolutionary reinforcement learning. Background Art

[0002] Sparse crowd sensing is a low-cost and high-efficiency sensing mode that uses high-quality data from a small area to infer data from the remaining area. Existing work ignores the timeliness of the information of the sensing data and is difficult to meet the sparse crowd sensing scenarios that require high data freshness. Summary of the invention

[0003] In view of this, the purpose of the present invention is to provide a dual-objective optimization method for a UAV-vehicle collaborative system based on evolutionary reinforcement learning, which can effectively schedule the collaborative system composed of UAVs and vehicles to cooperate in completing perception and data offloading tasks, and achieve dual-objective collaborative optimization of data quality and information timeliness.

[0004] To achieve the above object, the present invention adopts the following technical solution: a dual-objective optimization method for a UAV-vehicle cooperative system based on evolutionary reinforcement learning, comprising the following steps:

[0005] Step S1: Initialize the strategy population;

[0006] Step S2: Use a multi-agent reinforcement learning algorithm to train all team strategies in the population;

[0007] Step S3: Evaluate the fitness values ​​of all teams and perform non-dominated sorting and crowding calculations;

[0008] Step S4: Use the inter-team co-evolution method to iterate the team strategies in the population and generate a new population;

[0009] Step S5: Determine whether the population strategy has converged, if yes, proceed to step S6, otherwise proceed to step S2;

[0010] Step S6: The UAV-vehicle system performs the perception and data offloading tasks according to the collaborative decision of the policy network;

[0011] Step S7: Infer the data of the unperceived area based on the collected data.

[0012] In a preferred embodiment, the operation process of initializing the strategy population in step S1 is:

[0013] Step S101: Create w team strategies, each team consists of a group of heterogeneous multi-agents of the same number, and each team maintains its own strategy π independently; the strategy population π composed of multiple teams pop ={π 1 ,π2 ,…,π w}; π 1 represents the first team strategy, π 2 represents the second team strategy, π w represents the w-th team strategy;

[0014] Step S102: Initialize all team strategies π in the population and create an evaluation network and a target network corresponding to each agent;

[0015] Step S103: Initialize the shared experience pool of the entire population.

[0016] In a preferred embodiment: the operation process of using the multi-agent reinforcement learning algorithm to train all team strategies in the population in step S2 is:

[0017] Step S201: taking out team strategies from the population in sequence, and resetting the virtual environment to an initial state;

[0018] Step S202: Obtain the current state s of the team strategy t And local observations of all drones and vehicles represents the local observation of a certain agent in period t, where jk = j / k, which is used to distinguish between the two heterogeneous agents of drones and cars; Contains reachable area locations Historical Trajectory And its remaining power Three characteristics; the state of period t is represented by s t Indicates the position of all cars and power and the current location of all drones and power Power The remaining percentage indicates the battery life of the vehicle. The status characteristic is always 1, indicating the battery level of the drone. Gradually decreases as you move.

[0019] Step S203: Current location of the drone and all possible target regions a i Distance The moving distance D within the decision interval u Limit, where 1≤i≤m, m is the total number of target areas. and reach a i Power consumption Remaining power The influence of , get the available action set of the drone at the current moment

[0020] Step S204: Vehicle current position and all possible target regions a i Distance The moving distance D within the decision interval v Limit, get the set of available actions for the vehicle at the current moment

[0021]

[0022] Using the shortest path algorithm Select arrive A set of areas with the least required cycles; for vehicle v k Expert guidance actions come from With all drones The set union of the guidance areas between and then the intersection of the actions with movement restrictions constitute the final available actions for the vehicle:

[0023]

[0024] Step S205: Obtain the action set of all drones and vehicles from the available actions according to the action selection method ∈-greedy algorithm trained by reinforcement learning

[0025] Step S206: The drone and the vehicle perform actions in the virtual environment and arrive at the target area to perform the task; the drone collects data in the area and unloads the data to the vehicle in the same area; the vehicle provides charging services for the drone; and updates the drone related information and vehicle related information The amount of electricity The update formula is:

[0026]

[0027] Step S207: Calculate uniformity coefficient ω t ∈(0,1] and cycle perception number count t ∈[1,p] is used for reward calculation of non-final states, where B t (a i ) represents the amount of data collected in region i up to period t:

[0028]

[0029] The reward function formula is as follows: the reward obtained in the non-final state is the balance factor and the uniformity coefficient ωt And cycle perception count t The product of the final state reward we directly use the real data E and the inferred data Error As a reward t :

[0030]

[0031] Step S208: Set the current state s t , the next moment state s t+1 , the agent's current local observation The agent's local observation at the next moment Agent work Reward t and the final state identifier terminal are stored in the shared experience pool;

[0032] Step S209: extract small batches of data from the experience pool to train the target network parameters of the team strategy, and assign the parameter values ​​of the target network to the evaluation network after a certain number of iterations:

[0033]

[0034] L(θ)=(r t +γ(terminal-1)Q tatget -Q eval ) 2

[0035] Step S210: Repeat steps S202 to S209 for the team strategy until a final state is reached;

[0036] Step S211: Repeat steps S201 to S210 for all team strategies in the population until a preset upper limit of iteration rounds is reached.

[0037] In a preferred embodiment, the operation process of evaluating the fitness values ​​of all teams and performing non-dominated sorting and congestion calculation in step S3 is as follows:

[0038] Step S301: taking out team strategies from the population in sequence and resetting the virtual environment to an initial state;

[0039] Step S302: Obtain local observations of all drones and vehicles of the team strategy at the current moment At the same time, all drones and cars obtain their own actions based on local observations and team strategies. Drones and vehicles move to the target area specified by the action to perform the mission;

[0040] Step S303: Calculate the drone u j Data collection time Time until intersection with vehicle Difference DT Co :

[0041]

[0042] Step S304: Repeat steps S302 to S303 for the team strategy until the final state is reached, and calculate the data inference error And data freshness index AoI:

[0043] AoI=∑ j,t,s DT co / N

[0044] Step S305: Repeat steps S301 to S304 for all team strategies in the population to obtain and AoI;

[0045] Step S306: Based on all team strategies and AoI, search for all teams that are not dominated by other team strategies and store them in the set R i The index i represents the iteration round and also represents the different levels in the non-dominated sorting.

[0046] Step S307: Ignore from the population the data that have been stored in the set R i The team strategy of the embodiment of the present invention is repeated to execute step S306 until all team strategies complete the non-dominated sorting;

[0047] Step S308: According to the non-dominated sorting result, the crowding value of each team strategy in the solution space with other adjacent strategies is calculated in each level, and each team strategy is further sorted in the same level.

[0048] In a preferred embodiment, the operation process of using the inter-team co-evolution method to iterate the team strategies in the population and generate a new population in step S4 is as follows:

[0049] Step S401: From population π pop Select the top eNUM elite teams from the list to form part of the team strategy for the new population;

[0050] Step S402: Starting from π in turn through a tournament competition pop Select team strategies from the non-elite teams as the parent generation for subsequent evolutionary operations;

[0051] Step S403: The champion team π of the tournamentc Will work with a randomly selected elite team π e Crossover is performed, and the offspring produced have a certain probability of mutation, and the offspring produced are added to the new population:

[0052]

[0053] Mutate(π)=(π-π jk )∪G(π jk )

[0054] Step S404: Repeat steps S402 to S403 until the number of the new population is equal to the number of the original population.

[0055] In a preferred embodiment: in step S5, it is determined whether the population strategy has converged. If it is determined to be convergent, the process proceeds to step S6. Otherwise, the process continues with step S2. The operation process is:

[0056] Step S501: after a number of iterations, if the optimal strategy in the population does not change, or if the preset number of iterations is reached, proceed to step S6;

[0057] Step S502: The optimal strategy in the population is still changing in the last few rounds, and the process returns to step S2.

[0058] In a preferred embodiment, the operation process of the drone-vehicle system in step S6 performing the perception and data offloading tasks according to the policy network collaborative decision is as follows:

[0059] Step S601: Obtain the highest-ranked team strategy in the population, and use the strategy as a decision-making strategy for controlling drones and vehicles in a real mission environment;

[0060] Step S602: Obtain local observations of all drones and vehicles at the current moment At the same time, all drones and cars obtain their own actions based on local observations and team strategies. Drones and vehicles move to the target area specified by the action to perform the mission;

[0061] Step S603: The drone and the vehicle move to the target area specified by the action to perform the task; the drone is responsible for collecting data values ​​in the area; the vehicle is responsible for receiving data collected by the drone in the same area and providing power support for the drone;

[0062] Step S604: Repeat steps S602 to S603 until all tasks are completed or the task time is reached.

[0063] In a preferred embodiment, the operation process of inferring the data of the unperceived area based on the collected data in step S7 is as follows:

[0064] Step S701: All drones upload the unuploaded data to the vehicle to complete the data aggregation;

[0065] Step S702: During the mission, the vehicle uploads the real-time acquired data to the cloud platform, and the cloud platform infers the data of the remaining unperceived areas based on the collected data.

[0066] Compared with the prior art, the present invention has the following beneficial effects: The method of the present invention mainly solves the scheduling problem of cooperative systems composed of drones and vehicles to complete perception and data unloading tasks. In order to achieve the collaborative scheduling of two heterogeneous intelligent agents, drones and vehicles, and balance the two coupled goals of data collection and data unloading, the method fully combines the complementary advantages of co-evolutionary algorithms and multi-agent reinforcement learning to complete strategy training. The method uses a multi-agent reinforcement learning model to learn strategies, and controls the global optimization direction through co-evolution between teams, thereby achieving two-way strategy coordination of heterogeneous drones and vehicles and dual-objective optimization of complex cooperative tasks. BRIEF DESCRIPTION OF THE DRAWINGS

[0067] Figure 1 The figure is a schematic diagram of a method flow of a preferred embodiment of the present invention. DETAILED DESCRIPTION

[0068] The present invention will be further described below in conjunction with the accompanying drawings and embodiments.

[0069] It should be noted that the following detailed descriptions are illustrative and are intended to provide further explanation of the present application. Unless otherwise specified, all technical and scientific terms used herein have the same meanings as those commonly understood by those skilled in the art to which the present application belongs.

[0070] It should be noted that the terms used herein are only for describing specific embodiments and are not intended to limit the exemplary embodiments according to the present application; as used herein, unless the context clearly indicates otherwise, the singular form is also intended to include the plural form. In addition, it should be understood that when the terms "comprise" and / or "include" are used in this specification, they indicate the presence of features, steps, operations, devices, components and / or their combinations.

[0071] This embodiment provides a dual-objective optimization method for a UAV-vehicle cooperative system based on evolutionary reinforcement learning, such as Figure 1 As shown, the following steps are included:

[0072] Step S1: Initialize the strategy population;

[0073] Step S2: Use a multi-agent reinforcement learning algorithm to train all team strategies in the population;

[0074] Step S3: Evaluate the fitness values ​​of all teams and perform non-dominated sorting and crowding calculations;

[0075] Step S4: Use the inter-team co-evolution method to iterate the team strategies in the population and generate a new population;

[0076] Step S5: Determine whether the population strategy has converged, if yes, proceed to S6, otherwise continue to S2;

[0077] Step S6: The UAV-vehicle system performs the perception and data offloading tasks according to the collaborative decision of the policy network;

[0078] Step S7: Infer the data of the unperceived area based on the collected data.

[0079] In this embodiment, the operation process of initializing the strategy population is:

[0080] Step S101: Create w team strategies, each team consists of a group of heterogeneous multi-agents of the same number, and they independently maintain their own strategy π. The population π composed of multiple teams pop ={π 1 ,π 2 ,…,π w};

[0081] Step S102: Initialize all team strategies π in the population and create an evaluation network and a target network corresponding to each agent;

[0082] Step S103: Initialize the shared experience pool of the entire population.

[0083] In this embodiment, the operation process of using the multi-agent reinforcement learning algorithm to train all team strategies in the population is as follows:

[0084] Step S201: Take out team strategies from the population in sequence and reset the virtual environment to an initial state.

[0085] Step S202: Obtain the current state s of the team strategy t And local observations of all drones and vehicles Represents the local observation of a certain agent in period t, where jk = j / k, which is used to distinguish between the two heterogeneous agents of drones and cars. Contains reachable area locations Historical Trajectory And its remaining power Three characteristics. The state of period t is represented by s t Indicates the position of all cars and and the locations of all drones and power

[0086] Step S203: Current location of the drone and all possible target regions a i Distance The moving distance D within the decision interval u Limit, while current location and reach a i Power consumption Remaining power The influence of , get the available action set (reachable area set) of the drone at the current moment

[0087]

[0088] Step S204: Vehicle current position and all possible target regions a i Distance The moving distance D within the decision interval v Restrictions, get the set of available actions (reachable area set) of the vehicle at the current moment

[0089]

[0090] Furthermore, using the shortest path algorithm Select arrive The set of regions with the least required cycles. k Expert guidance actions come from With all drones The set union of the guidance areas between , and then the intersection with the movement restriction actions, constitutes the final available vehicle actions:

[0091]

[0092] Step S205: Obtain the action set of all drones and vehicles from the available actions according to the action selection method ∈-greedy algorithm trained by reinforcement learning

[0093] Step S206: The drone and vehicle perform actions in the virtual environment and arrive at the target area to perform the task. The drone collects data in the area and unloads the data to the vehicle in the same area. The vehicle provides charging services for the drone. Update drone related information and vehicle related information The amount of electricity The update formula is:

[0094]

[0095] Step S207: Calculate uniformity coefficient ω t ∈(0,1] and cycle perception number count t ∈[1,p] is used for reward calculation of non-final states, where B t (a i ) represents the amount of data collected in region i up to period t:

[0096]

[0097] The reward function formula is as follows: the reward obtained in the non-final state is the balance factor and the uniformity coefficient ω t And cycle perception count t The reward of the final state is directly the error between the inferred data and the real data. As a bonus:

[0098]

[0099] Step S208: Set the current state s t , the next moment state s t+1 , the agent's current local observation The agent's local observation at the next moment Agent work Reward t And the final state identifier terminal is stored in the shared experience pool.

[0100] Step S209: extract small batches of data from the experience pool to train the target network parameters of the team strategy, and assign the parameter values ​​of the target network to the evaluation network after a certain number of iterations:

[0101]

[0102] L(θ)=(r t +γ(terminal-1)Q tatget -Q eval ) 2

[0103] Step S210: Repeat S202 to S209 for the team strategy until the final state (preset task time) is reached.

[0104] Step S211: Repeat S201 to S210 for all team strategies in the population until a preset upper limit of iteration rounds is reached.

[0105] In this embodiment, the operation process of evaluating the fitness values ​​of all teams and performing non-dominated sorting and congestion calculation is as follows:

[0106] Step S301: Take out team strategies from the population in sequence and reset the virtual environment to an initial state.

[0107] Step S302: Obtain local observations of all drones and vehicles of the team strategy at the current moment At the same time, all drones and cars obtain their own actions based on local observations and team strategies. Drones and vehicles move to the target area specified by the action to perform the mission.

[0108] Step S303: Calculate the drone u j Data collection time Time until intersection with vehicle Difference DT Co :

[0109]

[0110] Step S304: Repeat S302 to S303 for the team strategy until the final state (preset task time) is reached, and calculate the data inference error And data freshness index AoI:

[0111] AoI=∑ j,t,s DT co / N

[0112] Step S305: Repeat S301 to S304 for all team strategies in the population to obtain and AoI.

[0113] Step S306: According to all team strategies and AoI, search for all teams that are not dominated by other team strategies and store them in the set R i The index i represents the iteration round and also represents the different levels in the non-dominated sort.

[0114] Step S307: Ignore from the population the data that have been stored in the set R i The team strategy of the embodiment of the present invention is repeated to execute step S306 until all team strategies complete the non-dominated sorting.

[0115] Step S308: According to the non-dominated sorting result, the crowding value of each team strategy in the solution space with other adjacent strategies is calculated in each level, and each team strategy is further sorted in the same level.

[0116] In this embodiment, the operation process of using the inter-team co-evolution method to iterate the team strategies in the population and generate a new population is as follows:

[0117] Step S401: From population π pop Select the top eNUM elite teams from the group to form part of the team strategy for the new population.

[0118] Step S402: Starting from π in turn through a tournament competition pop Select team strategies from the non-elite teams as the parents for subsequent evolutionary operations.

[0119] Step S403: The champion team π of the tournament c Will work with a randomly selected elite team π e Crossover is performed, and the offspring produced have a certain probability of mutation, and the offspring produced are added to the new population:

[0120]

[0121] Mutate(π)=(π-π jk )∪G(π jk )

[0122] Step S404: Repeat steps S402 to S403 until the number of the new population is equal to the number of the original population.

[0123] In this embodiment, it is determined whether the population strategy has converged. If it is determined to be convergent, the process proceeds to S6. Otherwise, the process continues with S2 as follows:

[0124] Step S501: After several iterations, if the optimal strategy in the population does not change, or the preset iteration number is reached, proceed to step S6

[0125] Step S502: The optimal strategy in the population is still changing in the last few rounds, and the process returns to step S2.

[0126] In this embodiment, the operation process of the drone-vehicle system performing the perception and data offloading tasks according to the policy network collaborative decision is as follows:

[0127] Step S601: Obtain the highest-ranked team strategy in the population, and use the strategy as the decision-making strategy for controlling drones and vehicles in a real mission environment.

[0128] Step S602: Obtain local observations of all drones and vehicles at the current moment At the same time, all drones and cars obtain their own actions based on local observations and team strategies. Drones and vehicles move to the target area specified by the action to perform the mission.

[0129] Step S603: The drone and the vehicle move to the target area specified by the action to perform the task. The drone is responsible for collecting data values ​​in the area. The vehicle is responsible for receiving data collected by the drone in the same area and providing power support for the drone.

[0130] Step S604: Repeat steps S602 to S603 until all tasks are completed or the task time is reached.

[0131] In this embodiment, the operation process of inferring the data of the unperceived area based on the collected data is as follows:

[0132] Step S701: All drones upload the unuploaded data to the vehicle to complete the data aggregation.

[0133] Step S702: During the mission, the vehicle uploads the real-time acquired data to the cloud platform, and the cloud platform infers the data of the remaining unperceived areas based on the collected data.

[0134] The above description is only a preferred embodiment of the present invention. All equivalent changes and modifications made according to the scope of the patent application of the present invention should fall within the scope of the present invention.

Claims

1. A dual-objective optimization method for UAV-vehicle cooperative system based on evolutionary reinforcement learning, characterized by: The following steps are involved: Step S1: Initialize the strategy population; Step S2: Use a multi-agent reinforcement learning algorithm to train all team strategies in the population; Step S3: Evaluate the fitness values ​​of all teams and perform non-dominated sorting and crowding calculations; Step S4: Use the inter-team co-evolution method to iterate the team strategies in the population and generate a new population; Step S5: Determine whether the population strategy has converged, if yes, proceed to step S6, otherwise proceed to step S2; Step S6: The UAV-vehicle system performs the perception and data offloading tasks according to the collaborative decision of the policy network; Step S7: Infer the data of the unperceived area based on the collected data.

2. The dual-objective optimization method for a UAV-vehicle cooperative system based on evolutionary reinforcement learning according to claim 1 is characterized in that: The operation process of initializing the strategy population in step S1 is: Step S101: Create w team strategies, each team consists of a group of heterogeneous multi-agents of the same number, and each team maintains its own strategy π independently; the strategy population π composed of multiple teams pop ={π 1 ,π 2 ,…,π w }; π 1 represents the first team strategy, π 2 represents the second team strategy, π w represents the w-th team strategy; Step S102: Initialize all team strategies π in the population and create an evaluation network and a target network corresponding to each agent; Step S103: Initialize the shared experience pool of the entire population.

3. The dual-objective optimization method for a UAV-vehicle cooperative system based on evolutionary reinforcement learning according to claim 1, characterized in that: The operation process of using the multi-agent reinforcement learning algorithm to train all team strategies in the population in step S2 is: Step S201: taking out team strategies from the population in sequence, and resetting the virtual environment to an initial state; Step S202: Obtain the current state s of the team strategy t And local observations of all drones and vehicles represents the local observation of a certain agent in period t, where jk = j / k, which is used to distinguish between the two heterogeneous agents of drones and cars; Contains reachable area locations Historical Trajectory And its remaining power Three characteristics; the state of period t is represented by s t Indicates the position of all cars and power and the current location of all drones and power Power The remaining percentage indicates the battery life of the vehicle. The status characteristic is always 1, the drone's battery level gradually decreases with movement; Step S203: Current location of the drone and all possible target regions a i Distance The moving distance D within the decision interval u Limit, where 1≤i≤m, m is the total number of target areas; at the same time, the current position and reach a i Power consumption Remaining power The influence of , get the available action set of the drone at the current moment Step S204: Vehicle current position and all possible target regions a i Distance The moving distance D within the decision interval v Limit, get the set of available actions for the vehicle at the current moment Using the shortest path algorithm Select arrive A set of areas with the least required cycles; for vehicle v k Expert guidance actions come from With all drones The set union of the guidance areas between and then the intersection of the actions with movement restrictions constitute the final available actions for the vehicle: Step S205: Obtain the action set of all drones and vehicles from the available actions according to the action selection method ∈-greedy algorithm trained by reinforcement learning Step S206: The drone and the vehicle perform actions in the virtual environment and arrive at the target area to perform the task; the drone collects data in the area and unloads the data to the vehicle in the same area; the vehicle provides charging services for the drone; and updates the drone related information and vehicle related information The amount of electricity The update formula is: Step S207: Calculate uniformity coefficient ω t ∈(0,1] and cycle perception number count t ∈[1,p] is used for reward calculation of non-final states, where B t (a i ) represents the amount of data collected in region i up to period t: The reward function formula is as follows: the reward obtained in the non-final state is the balance factor and the uniformity coefficient ω t And cycle perception count t The product of the final state reward we directly use the real data E and the inferred data Error As a reward t : Step S208: Set the current state s t , the next moment state s t+1 , the agent's current local observation The agent's local observation at the next moment Agent work Reward t and the final state identifier terminal are stored in the shared experience pool; Step S209: extract small batches of data from the experience pool to train the target network parameters of the team strategy, and assign the parameter values ​​of the target network to the evaluation network after a certain number of iterations: L(θ)=(r t +γ(terminal-1)Q tatget -Q eval ) 2 Step S210: Repeat steps S202 to S209 for the team strategy until a final state is reached; Step S211: Repeat steps S201 to S210 for all team strategies in the population until a preset upper limit of iteration rounds is reached.

4. The dual-objective optimization method for a UAV-vehicle cooperative system based on evolutionary reinforcement learning according to claim 1, characterized in that: The operation process of evaluating the fitness values ​​of all teams and performing non-dominated sorting and congestion calculation in step S3 is as follows: Step S301: taking out team strategies from the population in sequence and resetting the virtual environment to an initial state; Step S302: Obtain local observations of all drones and vehicles of the team strategy at the current moment At the same time, all drones and cars obtain their own actions based on local observations and team strategies. Drones and vehicles move to the target area specified by the action to perform the mission; Step S303: Calculate the drone u j Data collection time Time until intersection with vehicle Difference DT Co : Step S304: Repeat steps S302 to S303 for the team strategy until the final state is reached, and calculate the data inference error And data freshness index AoI: AoI=∑ j,t,s DT co / N Step S305: Repeat steps S301 to S304 for all team strategies in the population to obtain and AoI; Step S306: According to all team strategies and AoI, search for all teams that are not dominated by other team strategies and store them in the set R i The index i represents the iteration round and also represents the different levels in the non-dominated sorting. Step S307: Ignore from the population the data that have been stored in the set R i The team strategy of the embodiment of the present invention is repeated to execute step S306 until all team strategies complete the non-dominated sorting; Step S308: According to the non-dominated sorting result, the crowding value of each team strategy in the solution space with other adjacent strategies is calculated in each level, and each team strategy is further sorted in the same level.

5. The dual-objective optimization method for a UAV-vehicle cooperative system based on evolutionary reinforcement learning according to claim 1, characterized in that: In step S4, the operation process of using the inter-team co-evolution method to iterate the team strategies in the population and generate a new population is as follows: Step S401: From population π pop Select the top eNUM elite teams from the list to form part of the team strategy for the new population; Step S402: Starting from π in turn through a tournament competition pop Select team strategies from the non-elite teams as the parent generation for subsequent evolutionary operations; Step S403: The champion team π of the tournament c Will work with a randomly selected elite team π e Crossover is performed, and the offspring produced have a certain probability of mutation, and the offspring produced are added to the new population: Change(π)=(π-π jk )∪G(π jk ) Step S404: Repeat steps S402 to S403 until the number of the new population is equal to the number of the original population.

6. The dual-objective optimization method for UAV-Vehicle cooperative system based on evolutionary reinforcement learning according to claim 1, characterized in that: In step S5, it is determined whether the population strategy has converged. If it is, the process proceeds to step S6. Otherwise, the process continues with step S2. Step S501: after a number of iterations, if the optimal strategy in the population does not change, or if the preset number of iterations is reached, proceed to step S6; Step S502: The optimal strategy in the population is still changing in the last few rounds, and the process returns to step S2.

7. The dual-objective optimization method for UAV-Vehicle cooperative system based on evolutionary reinforcement learning according to claim 1 is characterized by: The operation process of the drone-vehicle system performing the perception and data offloading tasks according to the collaborative decision of the policy network in step S6 is as follows: Step S601: Obtain the highest-ranked team strategy in the population, and use the strategy as a decision-making strategy for controlling drones and vehicles in a real mission environment; Step S602: Obtain local observations of all drones and vehicles at the current moment At the same time, all drones and cars obtain their own actions based on local observations and team strategies. Drones and vehicles move to the target area specified by the action to perform the mission; Step S603: The drone and the vehicle move to the target area specified by the action to perform the task; the drone is responsible for collecting data values ​​in the area; The vehicle is responsible for receiving data collected by drones in the same area and providing power support for the drones; Step S604: Repeat steps S602 to S603 until all tasks are completed or the task time is reached.

8. The dual-objective optimization method for UAV-vehicle cooperative system based on evolutionary reinforcement learning according to claim 1, characterized in that: The operation process of inferring the data of the unperceived area based on the collected data in step S7 is as follows: Step S701: All drones upload the unuploaded data to the vehicle to complete the data aggregation; Step S702: During the mission, the vehicle uploads the real-time acquired data to the cloud platform, and the cloud platform infers the data of the remaining unperceived areas based on the collected data.

Images