Multi-task group perception method for empirical weight updating and time delay sensitive reward

Abstract

The invention relates to a multi-task group perception method for empirical weight updating and time delay sensitive reward. The method comprises the following steps that 1, a simulation platform carries out algorithm initialization; 2, the command center initializes the environment; step 3, the simulation platform performs task allocation according to the currently stored task allocator strategy pi; and 4, updating the emergency task interest points in each individual experience buffer area according to the currently stored time sequence predictor parameter psi, and the like. The method has the advantages that the processing efficiency and the energy utilization rate of multi-task mobile group perception are effectively improved through a hierarchical multi-agent reinforcement learning algorithm in combination with an experience buffer area and a time predictor; the mobile group can process the monitoring task by using the remaining time after completing the emergency task, so that efficient task scheduling and trajectory planning are realized, the overall response speed and the processing efficiency of the system are remarkably improved, and meanwhile, the energy consumption is minimized.

Description

Technical Field

[0001] The present invention belongs to the field of mobile group perception, and in particular relates to a multi-task group perception method with experience weight update and delay-sensitive reward. Background Art

[0002] Traffic monitoring is essential for managing urban congestion and promoting efficient transportation systems. Traditional data collection methods mainly rely on individual sensing devices and sensor networks, which have limitations such as small coverage, low data quality, and shallow perception levels. Although individual sensing devices such as smartphones and fixed sensors have alleviated the pressure of traffic monitoring to a certain extent, their coverage is limited and it is difficult to meet the large-scale and comprehensive traffic data needs. Sensor networks are costly to build, difficult to maintain, and difficult to cope with rapidly changing traffic environments, which makes it difficult for traditional methods to achieve maximum effectiveness in practical applications.

[0003] To solve these problems, Mobile Crowd Sensing (MCS) technology came into being. MCS uses groups of people or unmanned equipment (such as drones and self-driving cars) to collect and share environmental data. It has the advantages of low cost, wide coverage, and strong real-time performance. In the MCS scenario, the mobile group has become an ideal sensing device with its flexibility and efficient data collection capabilities. The mobile group can quickly cover a large area and can collect data in places that are difficult for traditional equipment to reach, thereby significantly improving the efficiency and accuracy of traffic monitoring.

[0004] However, traditional MCS methods mainly focus on the optimization of a single task and have difficulty in processing multiple types of perception tasks at the same time. In actual traffic monitoring scenarios, it is often necessary to process multiple different types of tasks at the same time, such as conventional traffic flow monitoring and emergency accident scene processing. These tasks have different delay threshold requirements. How to efficiently allocate and execute these tasks with limited resources has become a major challenge. Specifically, these challenges include the following aspects:

[0005] First, due to the limited perception capabilities of mobile swarms, they must be distributed in different geographical areas and collaboratively plan their trajectories to meet various mission requirements while minimizing total energy consumption. Second, the time and location of emergency tasks are unpredictable. Therefore, mobile swarms need to react quickly to handle multiple emergency situations while maintaining a good balance between ongoing tasks. Finally, the difference in requirements between routine monitoring tasks and emergency tasks further increases the complexity. Routine monitoring tasks may require long-term stable coverage, while emergency tasks require immediate response, which requires a delicate balance in task allocation and resource scheduling. How mobile swarms can effectively balance routine monitoring tasks while handling emergency tasks while ensuring the efficient operation of the entire system has become a difficult problem.

[0006] In order to meet these challenges, traditional optimization algorithms (such as convex optimization) often have difficulty finding suitable solutions because they show limitations when facing dynamic changes and multi-objective optimization problems. Therefore, more flexible and intelligent algorithms are needed, such as the multi-agent deep reinforcement learning (MADRL) algorithm. The MADRL algorithm can gradually optimize the decision-making process in a complex dynamic environment by simulating the interaction and learning of multiple agents, thereby more effectively handling the concurrent execution of multiple tasks. Summary of the invention

[0007] In view of the defects and gaps in the prior art, the present invention proposes a multi-task group perception method with experience weight update and delay-sensitive reward (called DRL-MTUCS), which includes a high-level task allocator and a mobile group that executes a group of low-level tasks. Each individual in the mobile group corresponds to an experience buffer and a delay-sensitive reward for multi-task execution. The task allocator is a neural network, which determines whether to insert the task into the buffer corresponding to the individual by observing the global state information of the environment, the local characteristics of the individual, and the characteristics of the current urgent interest point. The intrinsic reward is a reward generated in combination with each individual's own motivation and is unrelated to the reward provided by the environment.

[0008] The method of the present invention comprises the following steps:

[0009] Step 1: The simulation platform initializes the algorithm;

[0010] Step 2: Initialize the command center environment;

[0011] Step 3: The simulation platform allocates tasks according to the currently saved task allocator strategy π; if there is an emergency point of interest, strategy π observes the global state and outputs the allocation result, inserting the emergency point of interest into the experience buffer of the corresponding mobile group; if there is no emergency task point of interest, go to step 4;

[0012] Step 4: Update the urgent task interest points in each individual experience buffer according to the currently saved timing predictor parameter ψ;

[0013] Step 5: The simulation platform saves the strategy π of each individual. k , in the time slice [t, t+1), each individual u processes two types of interest points. If it is a monitoring interest point, go to step 6; if it is an emergency interest point, go to step 7;

[0014] Step 6: When processing the monitoring task, the individual spends τ move Time at speed Direction Movement; individual costs τ-τ move The data is collected from each point of interest via a ground-to-air uplink channel based on orthogonal frequency division multiple access (OFDMA) technology at a time; the ground-to-air channel includes line-of-sight (LoS) and non-line-of-sight (NLoS) losses, with a unit bandwidth of B and a noise power spectral density of W. n ;

[0015] Step 7: When handling an emergency task, the individual takes a photo to scan the emergency task area. The individual is still moving when taking photos, so the photos taken will be displaced and blurred. In order to effectively handle the emergency task, the photos taken need to meet the maximum blur requirement δ max , that is, the speed of the individual must satisfy

[0016] Step 8: Individual u obtains rewards from the environment based on current observations and current goals and the local observation of the next time slice The extrinsic reward refers to the reward for completing the multi-task group perception task in HiMDP. At the same time, the individual sends the local observation of the current time slice, the local observation of the next time slice, the extrinsic reward of the current time slice, and the interaction experience to the strategy trajectory buffer of the simulation platform. The strategy trajectory buffer stores the interaction experience data. When the strategy trajectory buffer is full of interaction experience data, the earliest stored interaction experience data is overwritten.

[0017] Step 9: The simulation platform calculates the final reward for each individual

[0018] Step 10: At the end of the time slice [t, t+1), the delay counters of all points of interest are updated according to the processing situation; t = t+1, the simulation platform repeatedly executes steps 4 to 9 until t = T, and the round ends;

[0019] Step 11, the simulation platform updates the strategy π and value function V of the task allocator;

[0020] Step 12: training a time series predictor parameterized by ψ;

[0021] Step 13: According to step 11, for each individual strategy π u and the value function V u Calculate the loss function and perform gradient descent on the total loss function, as shown in formula (8):

[0022]

[0023] Step 14: When the simulation platform tests the current mobile group strategy {π u |u=1,2,…,U} The effective task processing index obtained is higher than the results of all previous tests, and the current mobile group strategy parameters are saved;

[0024] Step 15: The simulation platform repeatedly executes steps 3 to 14 until the number of interaction rounds reaches an upper limit;

[0025] Step 16: The simulation platform retrieves the saved optimal strategy parameters of the mobile group of the DRL-MTUCS algorithm, that is, obtains the optimal strategy for the mobile group that can maximize the effective task processing index, obtains the best action for each step of the mobile group based on the optimal strategy, sends control instructions to the mobile group and the task distributor to collect data on points of interest in the task area and sends the data back to the simulation platform.

[0026] Furthermore, the simulation platform in step 1 initializes the algorithm, including:

[0027] Step 101, initializing the parameters of the timing predictor ψ;

[0028] Step 102: Initialize the strategy π of each individual u u Parameters and value function V u Parameters;

[0029] Step 103, initializing the strategy π of the upper allocator and the parameters of the value function V;

[0030] Step 104: Initialize the experience buffer of each individual.

[0031] Furthermore, the command center initializes the environment in step 2; including:

[0032] There are two types of interest points in the environment: monitoring task interest points and emergency task interest points. The total number of these two types of interest points is P. The monitoring task interest points are data sensors whose locations are given by specific real data sets. The emergency task interest points are data sensors with a side length of l. emerThe entire perception task is divided into T equal time slices, each time slice has the same duration τ. When processing the monitoring task interest point, a time slice is divided into a mobile group moving time τ move and data collection time τ coll There are two parts. When processing emergency mission points of interest, the mobile group covers the entire emergency area in a zigzag shape at a specific speed. A delay counter is set at each point of interest. As long as the point of interest is not processed in the time slice [t, t+1), the counter is incremented. The command center deploys a task distributor and U full-energy E max The mobile group acts as the executor of perception data collection. The command center sets t=0 and is ready to schedule the mobile group to interact with the environment for one round. Here and below, a round is referred to as the trajectory planning problem of the mobile group in the considered scenario, which is modeled as a hierarchical Markov decision process (HiMDP).

[0033] Furthermore, in step 4, updating the urgent task interest points in each individual experience buffer according to the currently saved timing predictor parameter ψ includes:

[0034] Step 401: The simulation platform sends the current local observation of each individual u and the experience buffer of the individual to the time series predictor;

[0035] Step 402: For each emergency task interest point in the experience buffer, the time series predictor constructs the target observation state when the individual u arrives at the emergency task interest point;

[0036] Step 403: The timing predictor reads the current local observation and target observation Predict the expected number of steps N for the individual to reach the point of interest for the emergency task π , as shown in formula (1):

[0037]

[0038] in, Represents the consumption function from the current local observation to the target observation, E π represents expectations;

[0039] Step 404, sending the expected number of steps back to the experience buffer;

[0040] Step 405: Each individual's experience buffer sets the highest weighted emergency task interest point as the target of the corresponding individual u And sent to individual u.

[0041] Furthermore, in step 9, the simulation platform calculates the final reward for each individual include:

[0042] Step 901: Individual u calculates the normalized distance from his current emergency task point of interest g to the point of interest And the total delay calculated from the receipt of the task is sent to the simulation platform;

[0043] Step 902: The simulation platform obtains the neighbor information of individual u, selects another individual u' closest to individual u as the anti-target, and calculates the normalized distance between the two individuals. intrinsic rewards as anti-goals;

[0044] Step 903: The simulation platform delays and normalized distance Multiply, as the intrinsic reward of the goal;

[0045] Step 904: The simulation platform calculates the intrinsic reward according to the intrinsic reward weight coefficient ω as shown in formula (2):

[0046]

[0047] Furthermore, in step 11, the simulation platform updates the strategy π and value function V of the task allocator, including

[0048] Step 1101: Calculate the clipping loss function for updating the policy function As shown in formula (3):

[0049]

[0050] Among them, r t (θ) is the importance sampling ratio of the PPO algorithm, is the estimated advantage function, clip is the clipping function; ∈ is a hyperparameter used to control the magnitude of clipping;

[0051] Step 1102: Calculate the estimated value function in the time slice [t, t+1) As shown in formula (4):

[0052]

[0053] Among them, V(s t ; φ) is s t The true value function under φ represents the estimated parameters of the neural network;

[0054] Step 1103: Calculate the mean square error loss function for updating the value function V, as shown in formula (5):

[0055]

[0056] Among them, ∈' is a hyperparameter used to control the magnitude of value function clipping;

[0057] Step 1104: Calculate the total loss function, as shown in formula (6):

[0058]

[0059] Among them, c critic ,c entropy is a hyperparameter, is the entropy of policy π.

[0060] Furthermore, the step 12 of training the time series predictor parameterized by ψ includes:

[0061] Step 1201, the collection strategy π collects trajectories and generates labels between the state pairs visited by π;

[0062] Step 1202, select a segment starting from the segment where the emergency task point of interest appears in the trajectory (starting when an individual is assigned an emergency point of interest and ending when an individual successfully handles the emergency point of interest), and create a state pair between the initial state and each state in the subsequent time slice;

[0063] Step 1203: Generate the actual time slice label of strategy π from the initial state to any future state in the segment; train the network of the time series predictor by minimizing formula (7):

[0064]

[0065] Compared with the prior art in this technical field, the present invention has the following beneficial effects:

[0066] 1. The method described in the present invention effectively improves the processing efficiency and energy utilization of multi-task mobile group perception through a hierarchical multi-agent reinforcement learning algorithm combined with an experience buffer and a time predictor; after completing an emergency task, the mobile group can use the remaining time to process monitoring tasks, achieve efficient task scheduling and trajectory planning, significantly improve the overall response speed and processing efficiency of the system, and minimize energy consumption.

[0067] 2. In terms of improving data timeliness, the method of the present invention designs a new metric "effective task processing index" to measure the overall performance of the algorithm in processing various types of tasks. By combining the delay requirements and weight settings of different tasks, an effective balance between monitoring tasks and emergency tasks is achieved, ensuring the real-time and timeliness of data. The algorithm keeps the delay of each point of interest below the specified threshold, significantly reducing threshold violations, and can better support traffic management and decision-making.

[0068] 3. In order to enhance the robustness of the system, the method described in the present invention adopts a delay-sensitive reward mechanism, which improves the robustness of the system by guiding the mobile group to maintain a balance between emergency tasks and monitoring tasks; when faced with a variety of unpredictable task scenarios, the mobile group can flexibly adjust strategies to ensure the efficient completion of tasks, which significantly improves the adaptability and stability of the system in practical applications.

[0069] 4. The method described in the present invention is highly rigorous in system modeling, and has carried out detailed modeling of the processing methods of monitoring tasks and emergency tasks, such as the communication model of monitoring tasks and the photo processing method of emergency tasks, and the maximum blur requirement of photos; corresponding processing strategies are formulated for different task types to ensure that various tasks can be completed efficiently and accurately; through precise model and algorithm design, the actual application effect of the system in complex environments is improved.

[0070] 5. The method described in the present invention is based on the WarpDrive development training framework, which extends the advantages of GPU parallelism to multi-agent reinforcement learning. It not only moves the neural network training to the graphics card, but also parallelizes the environmental interaction process, realizes efficient environmental simulation and training, and can run hundreds of environmental instances on the graphics card at the same time, significantly improving the training efficiency and adaptability of the algorithm, and supporting larger-scale multi-task execution scenarios. BRIEF DESCRIPTION OF THE DRAWINGS

[0071] Figure 1 is an overall flow chart of the method of the present invention;

[0072] Figure 2 This is a schematic diagram of the effect of the number of mobile groups on the effective task processing index in the method of the present invention. DETAILED DESCRIPTION

[0073] In order to better understand the technical solution of the method of the present invention, the specific implementation mode of the present invention is further described in detail below in conjunction with the accompanying drawings of the specification.

[0074] Example

[0075] like Figure 1 As shown, the method of the present invention comprises the following steps:

[0076] Step 1: The simulation platform initializes the algorithm;

[0077] Step 2: Initialize the command center environment;

[0078] Step 3: The simulation platform allocates tasks according to the currently saved task allocator strategy π; if there is an emergency point of interest, strategy π observes the global state and outputs the allocation result, inserting the emergency point of interest into the experience buffer of the corresponding mobile group; if there is no emergency task point of interest, go to step 4;

[0079] Step 4: Update the urgent task interest points in each individual experience buffer according to the currently saved timing predictor parameter ψ;

[0080] Step 5: The simulation platform saves the strategy π of each individual. k , in the time slice [t, t+1), each individual u processes two types of interest points. If it is a monitoring interest point, go to step 6; if it is an emergency interest point, go to step 7;

[0081] Step 6: When processing the monitoring task, the individual spends τ move Time at speed Direction Movement; individual costs τ-τ move The data is collected from each point of interest via a ground-to-air uplink channel based on orthogonal frequency division multiple access (OFDMA) technology at a time; the ground-to-air channel includes line-of-sight (LoS) and non-line-of-sight (NLoS) losses, with a unit bandwidth of B and a noise power spectral density of W. n ;

[0082] Step 7: When handling an emergency task, the individual takes a photo to scan the emergency task area. The individual is still moving when taking photos, so the photos taken will be displaced and blurred. In order to effectively handle the emergency task, the photos taken need to meet the maximum blur requirement δ max , that is, the speed of the individual must satisfy

[0083] Step 8: Individual u obtains rewards from the environment based on current observations and current goals and the local observation of the next time slice The extrinsic reward refers to the reward for completing the multi-task group perception task in HiMDP. At the same time, the individual sends the local observation of the current time slice, the local observation of the next time slice, the extrinsic reward of the current time slice, and the interaction experience to the strategy trajectory buffer of the simulation platform. The strategy trajectory buffer stores the interaction experience data. When the strategy trajectory buffer is full of interaction experience data, the earliest stored interaction experience data is overwritten.

[0084] Step 9: The simulation platform calculates the final reward for each individual

[0085] Step 10: At the end of the time slice [t, t+1), the delay counters of all points of interest are updated according to the processing situation; t = t+1, the simulation platform repeatedly executes steps 4 to 9 until t = T, and the round ends;

[0086] Step 11, the simulation platform updates the strategy π and value function V of the task allocator;

[0087] Step 12: training a time series predictor parameterized by ψ;

[0088] Step 13: According to step 11, for each individual strategy π u and the value function V u Calculate the loss function and perform gradient descent on the total loss function, as shown in formula (8):

[0089]

[0090] Step 14: When the simulation platform tests the current mobile group strategy {π u |u=1,2,…,U} The effective task processing index obtained is higher than the results of all previous tests, and the current mobile group strategy parameters are saved;

[0091] Step 15: The simulation platform repeatedly executes steps 3 to 14 until the number of interaction rounds reaches an upper limit;

[0092] Step 16: The simulation platform retrieves the saved optimal strategy parameters of the mobile group of the DRL-MTUCS algorithm, that is, obtains the optimal strategy for the mobile group that can maximize the effective task processing index, obtains the best action for each step of the mobile group based on the optimal strategy, sends control instructions to the mobile group and the task distributor to collect data on points of interest in the task area and sends the data back to the simulation platform.

[0093] As a preferred specific embodiment, the simulation platform in step 1 performs algorithm initialization, including:

[0094] Step 101, initializing the parameters of the timing predictor ψ;

[0095] Step 102: Initialize the strategy π of each individual u u Parameters and value function V u Parameters;

[0096] Step 103, initializing the strategy π of the upper allocator and the parameters of the value function V;

[0097] Step 104: Initialize the experience buffer of each individual.

[0098] As a preferred specific embodiment, the command center initializes the environment in step 2; including:

[0099] There are two types of interest points in the environment: monitoring task interest points and emergency task interest points. The total number of these two types of interest points is P. The monitoring task interest points are data sensors whose locations are given by specific real data sets. The emergency task interest points are data sensors with a side length of l. emerThe entire perception task is divided into T equal time slices, each time slice has the same duration τ. When processing the monitoring task interest point, a time slice is divided into a mobile group moving time τ move and data collection time τ coll There are two parts. When processing emergency mission points of interest, the mobile group covers the entire emergency area in a zigzag shape at a specific speed. A delay counter is set at each point of interest. As long as the point of interest is not processed in the time slice [t, t+1), the counter is incremented. The command center deploys a task distributor and U full-energy E max The mobile group acts as the executor of perception data collection. The command center sets t=0 and is ready to schedule the mobile group to interact with the environment for one round. Here and below, a round is referred to as the trajectory planning problem of the mobile group in the considered scenario, which is modeled as a hierarchical Markov decision process (HiMDP).

[0100] As a preferred specific embodiment, the updating of the emergency task interest points in each individual experience buffer according to the currently saved timing predictor parameter ψ in step 4 includes:

[0101] Step 401: The simulation platform sends the current local observation of each individual u and the experience buffer of the individual to the time series predictor;

[0102] Step 402: For each emergency task interest point in the experience buffer, the time series predictor constructs the target observation state when the individual u arrives at the emergency task interest point;

[0103] Step 403: The timing predictor reads the current local observation and target observation Predict the expected number of steps N for the individual to reach the point of interest for the emergency task π , as shown in formula (1):

[0104]

[0105] in, Represents the consumption function from the current local observation to the target observation, E π represents expectations;

[0106] Step 404, sending the expected number of steps back to the experience buffer;

[0107] Step 405: Each individual's experience buffer sets the highest weighted emergency task interest point as the target of the corresponding individual u And sent to individual u.

[0108] As a preferred specific embodiment, the simulation platform in step 9 calculates the final reward of each individual include:

[0109] Step 901: Individual u calculates the normalized distance from his current emergency task point of interest g to the point of interest And the total delay calculated from the receipt of the task is sent to the simulation platform;

[0110] Step 902: The simulation platform obtains the neighbor information of individual u, selects another individual u' closest to individual u as the anti-target, and calculates the normalized distance between the two individuals. intrinsic rewards as anti-goals;

[0111] Step 903: The simulation platform delays and normalized distance Multiply, as the intrinsic reward of the goal;

[0112] Step 904: The simulation platform calculates the intrinsic reward according to the intrinsic reward weight coefficient ω as shown in formula (2):

[0113]

[0114] As a preferred specific embodiment, the simulation platform in step 11 updates the strategy π and the value function V of the task allocator, including:

[0115] Step 1101: Calculate the clipping loss function for updating the policy function As shown in formula (3):

[0116]

[0117] Among them, r t (θ) is the importance sampling ratio of the PPO algorithm, is the estimated advantage function, clip is the clipping function; ∈ is a hyperparameter used to control the magnitude of clipping;

[0118] Step 1102: Calculate the estimated value function in the time slice [t, t+1) As shown in formula (4):

[0119]

[0120] Among them, V(s t ; φ) is s t The true value function under φ represents the estimated parameters of the neural network;

[0121] Step 1103: Calculate the mean square error loss function for updating the value function V, as shown in formula (5):

[0122]

[0123] Among them, ∈' is a hyperparameter used to control the magnitude of value function clipping;

[0124] Step 1104: Calculate the total loss function, as shown in formula (6):

[0125]

[0126] Among them, c critic ,c entropy is a hyperparameter, is the entropy of policy π.

[0127] As a preferred specific embodiment, the training of the time series predictor parameterized by ψ in step 12 includes:

[0128] Step 1201, the collection strategy π collects trajectories and generates labels between the state pairs visited by π;

[0129] Step 1202, select a segment starting from the segment where the emergency task point of interest appears in the trajectory (starting when an individual is assigned an emergency point of interest and ending when an individual successfully handles the emergency point of interest), and create a state pair between the initial state and each state in the subsequent time slice;

[0130] Step 1203: Generate the actual time slice label of strategy π from the initial state to any future state in the segment; train the network of the time series predictor by minimizing formula (7):

[0131]

[0132] Experimental example

[0133] In order to better reflect the actual situation of traffic monitoring, the experimental examples of the present invention use data sets from the real-world cities of San Francisco and Chengdu. Both data sets contain taxi trajectories. The verification example selects the top 300 points most frequently visited by taxis as the monitored points of interest. In order to simulate the unpredictability of emergency points of interest, the locations of these emergency points of interest are randomly selected and generated at fixed intervals throughout the round. In the verification example, drones are used as collection devices for unmanned groups, which will fly at low altitudes in urban environments and collect data related to traffic information.

[0134] In the experimental example, the technical parameters in the environment setting come from the research work of Zhao et al., while the parameters related to drones come from the technical specifications of industrial drones, such as DJI Matrice 600. Four drones (U=4) were used in the experiment, and the delay threshold of the monitoring task Set to 35 time slices, the delay threshold for urgent tasks Set to 20 time slices, the maximum allowed blur value of the image (δ max ) is 5.

[0135] In the experiment, the present invention uses the following performance indicators to evaluate the effect of the algorithm:

[0136] 1. Effective processing ratio of task m I m The effective processing ratio measures whether the algorithm can effectively control the delay of a certain type of task m below the threshold and avoid violating the threshold. It is defined as follows:

[0137]

[0138] in, represents the set of points of interest for task m, is the delay of the interest point p in time slice t, is the maximum allowable delay of task m. The scenarios considered by the invention include the effective processing ratio of emergency tasks and the effective processing ratio of monitoring tasks.

[0139] 2. The energy consumption ratio η measures the energy usage efficiency of the algorithm during task execution and is defined as the ratio of total energy consumption to the maximum allowed energy.

[0140] 3. The effective task processing index combines all the above indicators to evaluate the comprehensive performance of the algorithm in multiple task categories as follows:

[0141]

[0142] In the scenario considered by the present invention, there are two types of tasks, surveillance tasks and emergency tasks, so the effective task processing index is actually

[0143] In order to achieve the best effect of the method proposed in the present invention, the intrinsic reward weight and buffer length are adjusted to adapt to different data sets. Usually, a shorter buffer length (such as l buf =3) can effectively extract the most relevant task information while avoiding overly complex task planning and achieving the best task processing efficiency. The adjustment of the intrinsic reward weight ω needs to balance the processing priority of emergency tasks and monitoring tasks. Smaller ω (such as 0.1 and 0.3) tends to prioritize emergency tasks, while larger ω (such as 0.7 and 0.9) is conducive to the continuous processing of monitoring tasks. Medium ω (such as 0.5) can usually achieve the best balance between the two types of tasks. Reasonable parameter settings can significantly improve the overall performance of the algorithm in a multi-task environment.

[0144] During the training process, other algorithm hyperparameters are configured as follows: KL coefficient is 0.2, single batch round number is 10, KL coefficient lower limit is 1e-20, sampling segment length is 120, SGD iteration number is 5, optimizer is Adam, learning rate is 0.0005, discount factor is 0.99, total training steps are 10000000, truncated round batch mode is used, and the number of environments is 500. The neural network structure is 3 fully connected layers, hidden state size is 128; 1 ​​convolution layer, convolution kernel size is 3, convolution step size is 1, convolution padding is 2, and pooling size is 3.

[0145] In order to evaluate the performance gap between the present invention and other standard methods, the following benchmark methods were compared to verify the effectiveness of DRL-MTUCS. The comparison methods include:

[0146] HAPPO (Heterogeneous Proximal Policy Optimization), a policy iteration method with a central value function in multi-agent scenarios, is an enhanced version of MAPPO;

[0147] AIM (Adversarial Intrinsic Motivation), a hierarchical reinforcement learning algorithm that introduces an intrinsic reward that minimizes the Wasserstein distance between the agent’s state visit distribution and the target distribution;

[0148] DRL-EMS (DRL Emergency Multi-UAV Scheduling), which solves the problem of drone trajectory planning in disaster response data collection and uses a single-agent deep reinforcement learning method based on a multi-head attention mechanism for drone scheduling;

[0149] mTSP (Multiple Traveling Salesman Problem), this data collection task is modeled as a traditional Multiple Traveling Salesman Problem (mTSP), where the emergency mission points of interest that need to be visited are regarded as traveling salesman cities, and the cost function is the distance between different points of interest;

[0150] Random, each drone randomly samples actions from the action space.

[0151] like Figure 2 As shown, the number of drones has a significant impact on the effectiveness of the present invention: while keeping the monitoring task threshold at 35 time slices and the fuzzy requirement at 5, the number of drones (U) is changed from 2 to 10 for testing. The results show that the increase in the number of drones significantly improves the effective task processing index and the emergency task processing ratio. When the number of drones is small, it is difficult for the system to process multiple emergency tasks and monitoring tasks at the same time, resulting in low task processing efficiency. As the number of drones increases, the system is able to better allocate and process tasks, thereby improving the overall task processing efficiency and energy utilization.

[0152] Specifically, when the number of drones increases to 5, the effective task processing index of the present invention reaches a peak value of 1.462. The system's ability to balance processing between emergency tasks and monitoring tasks is significantly improved. In the comparative experiment, the performance of the baseline method is significantly inferior to that of the present invention. For example, the effective task processing index of HAPPO is only 1.203 when the number of drones is 5. The main reason is that it cannot effectively handle complex priority scheduling in a multi-task environment. The effective task processing index of the AIM algorithm under the same conditions is 1.178, showing its limitations in processing target condition tasks. The effective task processing index of the DRL-EMS algorithm is 1.092 when the number of drones is 5, mainly because its single-agent method performs poorly in a multi-agent task environment. The effective task processing index of the mTSP method is 1.024, and the effective task processing index of the random method is the lowest, only 0.905.

[0153] The excellent performance of the present invention is mainly due to two important contributions: the experience buffer and the delay-sensitive reward: the experience buffer can effectively manage and prioritize multiple emergency tasks and monitoring tasks, so that the UAV can efficiently schedule tasks in complex environments; the delay-sensitive reward mechanism automatically adjusts the reward weight between emergency tasks and monitoring tasks to achieve the best balance in task processing. These innovative designs make the present invention far superior to traditional methods in multi-task mobile group perception tasks.

[0154] In summary, the experimental results verify the effectiveness of the present invention in multi-task mobile group perception tasks, especially when the number of drones is properly configured, it can significantly improve the overall performance and task processing efficiency of the system. The above results further show that the present invention has good adaptability and practical value in complex practical application scenarios.

[0155] The present invention is not limited to the above-mentioned embodiments. Without departing from the essential content of the present invention, any deformation, improvement and substitution that can be thought of by those skilled in the art shall fall into the protection scope of the present invention.

Claims

1. A multi-task group perception method with experience weight update and delay-sensitive reward, the method comprising the following steps: Step 1: The simulation platform initializes the algorithm; Step 2: Initialize the command center environment; Step 3: The simulation platform allocates tasks according to the currently saved task allocator strategy π; if there is an emergency point of interest, strategy π observes the global state and outputs the allocation result, inserting the emergency point of interest into the experience buffer of the corresponding mobile group; if there is no emergency task point of interest, go to step 4; Step 4: Update the urgent task interest points in each individual experience buffer according to the currently saved timing predictor parameter ψ; Step 5: The simulation platform saves the strategy π of each individual. k , in the time slice [t, t+1), each individual u processes two types of interest points. If it is a monitoring interest point, go to step 6; if it is an emergency interest point, go to step 7; Step 6: When processing the monitoring task, the individual spends τ move Time at speed Direction Movement; individual costs τ-τ move The data is collected from various points of interest via the ground-to-air uplink channel in a period of time, based on the orthogonal frequency division multiple access technology; the ground-to-air channel includes line-of-sight and non-line-of-sight losses, the unit bandwidth is B, and the power spectral density of the noise is W n ; Step 7: When handling an emergency task, the individual takes a photo to scan the emergency task area. The individual is still moving when taking photos, so the photos taken will be displaced and blurred. In order to effectively handle the emergency task, the photos taken need to meet the maximum blur requirement δ max , that is, the speed of the individual must satisfy Step 8: Individual u obtains rewards from the environment based on current observations and current goals and the local observation of the next time slice The extrinsic reward refers to the reward for completing the multi-task group perception task in HiMDP. At the same time, the individual sends the local observation of the current time slice, the local observation of the next time slice, the extrinsic reward of the current time slice, and the interaction experience to the strategy trajectory buffer of the simulation platform. The strategy trajectory buffer stores the interaction experience data. When the strategy trajectory buffer is full of interaction experience data, the earliest stored interaction experience data is overwritten. Step 9: The simulation platform calculates the final reward for each individual Step 10: At the end of the time slice [t, t+1), the delay counters of all points of interest are updated according to the processing situation; t = t+1, the simulation platform repeatedly executes steps 4 to 9 until t = T, and the round ends; Step 11, the simulation platform updates the strategy π and value function V of the task allocator; Step 12: training a time series predictor parameterized by ψ; Step 13: According to step 11, for each individual strategy π u and the value function V u Calculate the loss function and perform gradient descent on the total loss function, as shown in formula (8): Step 14: When the simulation platform tests the current mobile group strategy {π u |u=1,2,…,U} The effective task processing index obtained is higher than the results of all previous tests, and the current mobile group strategy parameters are saved; Step 15: The simulation platform repeatedly executes steps 3 to 14 until the number of interaction rounds reaches an upper limit; Step 16: The simulation platform retrieves the saved optimal strategy parameters of the mobile group of the DRL-MTUCS algorithm, that is, obtains the optimal strategy for the mobile group that can maximize the effective task processing index, obtains the best action for each step of the mobile group based on the optimal strategy, sends control instructions to the mobile group and the task distributor to collect data on points of interest in the task area and sends the data back to the simulation platform. As a preferred specific embodiment, the simulation platform in step 1 performs algorithm initialization, including: Step 101, initializing the parameters of the timing predictor ψ; Step 102: Initialize the strategy π of each individual u u Parameters and value function V u Parameters; Step 103, initializing the strategy π of the upper allocator and the parameters of the value function V; Step 104: Initialize the experience buffer of each individual. As a preferred specific embodiment, the command center initializes the environment in step 2; including: There are two types of interest points in the environment: monitoring task interest points and emergency task interest points. The total number of these two types of interest points is P. The monitoring task interest points are data sensors whose locations are given by specific real data sets. The emergency task interest points are data sensors with a side length of l. emer The entire perception task is divided into T equal time slices, each time slice has the same duration τ. When processing the monitoring task interest point, a time slice is divided into a mobile group moving time τ move and data collection time τ coll There are two parts. When processing emergency mission points of interest, the mobile group covers the entire emergency area in a zigzag shape at a specific speed. A delay counter is set at each point of interest. As long as the point of interest is not processed in the time slice [t, t+1), the counter is incremented. The command center deploys a task distributor and U full-energy E max The mobile group acts as the executor of perception data collection. The command center sets t=0 and is ready to schedule the mobile group to interact with the environment for one round. Here and below, a round is referred to as the trajectory planning problem of the mobile group in the considered scenario, which is modeled as a hierarchical Markov decision process. Furthermore, in step 4, updating the urgent task interest points in each individual experience buffer according to the currently saved timing predictor parameter ψ includes: Step 401: The simulation platform sends the current local observation of each individual u and the experience buffer of the individual to the time series predictor; Step 402: For each emergency task interest point in the experience buffer, the time series predictor constructs the target observation state when the individual u arrives at the emergency task interest point; Step 403: The timing predictor reads the current local observation and target observation Predict the expected number of steps N for the individual to reach the point of interest for the emergency task π , as shown in formula (1): in, Represents the consumption function from the current local observation to the target observation, E π represents expectations; Step 404, sending the expected number of steps back to the experience buffer; Step 405: Each individual's experience buffer sets the highest weighted emergency task interest point as the target of the corresponding individual u And sent to individual u. As a preferred specific embodiment, the simulation platform in step 9 calculates the final reward of each individual include: Step 901: Individual u calculates the normalized distance from his current emergency task point of interest g to the point of interest And the total delay calculated from the receipt of the task is sent to the simulation platform; Step 902: The simulation platform obtains the neighbor information of individual u, selects another individual u' closest to individual u as the anti-target, and calculates the normalized distance between the two individuals. intrinsic rewards as anti-goals; Step 903: The simulation platform delays and normalized distance Multiply, as the intrinsic reward of the goal; Step 904: The simulation platform calculates the intrinsic reward according to the intrinsic reward weight coefficient ω as shown in formula (2): As a preferred specific embodiment, the simulation platform in step 11 updates the strategy π and the value function V of the task allocator, including: Step 1101: Calculate the clipping loss function for updating the policy function As shown in formula (3): Among them, r t (θ) is the importance sampling ratio of the PPO algorithm, is the estimated advantage function, clip is the clipping function; ∈ is a hyperparameter used to control the magnitude of clipping; Step 1102: Calculate the estimated value function in the time slice [t, t+1) As shown in formula (4): Among them, V(s t ; φ) is s t The true value function under φ represents the estimated parameters of the neural network; Step 1103: Calculate the mean square error loss function for updating the value function V, as shown in formula (5): Among them, ∈' is a hyperparameter used to control the magnitude of value function clipping; Step 1104: Calculate the total loss function, as shown in formula (6): Among them, c critic ,c entropy is a hyperparameter, is the entropy of policy π. As a preferred specific embodiment, the training of the time series predictor parameterized by ψ in step 12 includes: Step 1201, the collection strategy π collects trajectories and generates labels between the state pairs visited by π; Step 1202, selecting a segment starting from the segment where the emergency mission point of interest appears in the trajectory, and creating a state pair between the initial state and each state in the subsequent time slice; Step 1203: Generate the actual time slice label of strategy π from the initial state to any future state in the segment; train the network of the time series predictor by minimizing formula (7):

Images