Group perception method based on vehicle-mounted unmanned aerial vehicle observation enhancement and asynchronous decision

By introducing observation encoder and action generator into vehicle-mounted drone group perception, combined with the asynchronous decision strategy update framework, the problems of space-time interleaving and asynchronous decision-making are solved, and fast perception and low-latency vehicle-mounted drone group perception tasks are realized. Improves the performance and scalability of the method.

CN119087991APending Publication Date: 2024-12-06BEIJING INST OF TECH
View PDF 0 Cites 2 Cited by

Patent Information

Application Number
CN202410873298.0
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-07-01
Publication Date
2024-12-06

AI Technical Summary

Technical Problem

The existing technology is difficult to effectively solve the problems of space-time interleaving and asynchronous decision-making in vehicle-mounted drone cluster perception, which makes it difficult to minimize data delays, and traditional methods have low performance and are difficult to generalize.

Method used

Adopting a group perception method based on vehicle-mounted drone observation enhancement and asynchronous decision-making, through the simulation platform initialization and policy update framework, using observation encoder and action generator based on historical action autoregression, three types of reward functions are introduced, and asynchronous decision-making strategies are designed. Update the framework, considering the interdependence of space-time interweaving and decision-making phases.

Benefits of technology

It realizes rapid inference in the multi-agent deep reinforcement learning framework, reduces data perception delay, avoids high GPU resource requirements, improves the scalability and deployment ease of the method, and solves the traditional method's "slow solution," The problem of difficulty in generalization.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119087991A_ABST
    Figure CN119087991A_ABST
Patent Text Reader

Abstract

The invention relates to a group perception method based on vehicle-mounted unmanned aerial vehicle observation enhancement and asynchronous decision. The method comprises the steps that 1, a simulation platform carries out algorithm initialization; step 2, initializing a problem example by the simulation platform; and step 3, inputting the environmental information into an observation encoder for two-stage vehicle-mounted unmanned aerial vehicle perception to obtain code embedding of each observed entity and global aggregation embedding of problem instances, and the like. The method has the advantages that a new vehicle-mounted unmanned aerial vehicle group sensing problem is proposed, sensing of information points is rapidly achieved through two-stage vehicle-mounted unmanned aerial vehicle group scheduling, and minimization of information time delay is achieved; the space-time information observed by the unmanned aerial vehicle / unmanned vehicle can be enhanced and aggregated, the space-time interleaving information is fully considered during action decision making, and the problem that the space-time interleaving information cannot be efficiently and comprehensively extracted for decision making in a traditional method is solved.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention belongs to the technical field of Internet of Things, and in particular relates to a group perception method based on vehicle-mounted unmanned aerial vehicle observation enhancement and asynchronous decision-making. Background Art

[0002] Crowd sensing has become an efficient computing model for obtaining data for smart city applications. Thanks to the rapid development of mobile networks and unmanned devices, the use of mobile drones / unmanned vehicles equipped with high-speed network sensors to collect and disseminate perception data in urban environments is becoming a reality. Compared with human participants, mobile unmanned devices represented by drones / unmanned vehicles have higher flexibility and lower information transmission delays, and can better cope with crowd sensing tasks in complex urban environments.

[0003] In existing applications, unmanned vehicles have large loading capacity and long-distance transportation capabilities, but are limited to specific roads. Drones have the ability to move flexibly in the environment, but their range of activity is limited by battery capacity. If the advantages of both are combined to form a vehicle-mounted drone swarm, unmanned vehicles can load drones for long-distance transportation, and release drones at unmanned vehicle stops to collect data on the local environment. This model fully utilizes the respective advantages of drones / unmanned vehicles and significantly improves the collection efficiency and latency control of perception data.

[0004] The present invention constructs the above-mentioned data collection task as a two-stage vehicle-mounted drone swarm perception problem with minimized data delay. The perception scenario of this problem includes a vehicle-mounted drone swarm (each vehicle-mounted drone consists of an unmanned vehicle and multiple drones loaded thereon), several unmanned vehicle stops and several points of interest where sensor data to be collected is stored (such as closed-circuit television cameras and alarm sensors). The first-stage perception problem plans the moving path of the unmanned vehicle from the unmanned vehicle starting point to the stop, and the second-stage perception problem plans the moving paths of multiple drones from the stop to the points of interest. The goal is to minimize the perception data delay while ensuring that the sensor data of all points of interest are perceived.

[0005] However, the two-stage vehicle-mounted drone swarm perception problem with data latency minimization faces the challenges of spatiotemporal interweaving of observation information and interdependence of the two-stage optimization problems. First, from the perspective of spatiotemporal interweaving of observation information, the observations of unmanned equipment contain spatial information (such as the geographical distribution of unmanned vehicle stops and points of interest) and temporal information (such as the time consumed by the task and the requirements of each point of interest for data latency). How to efficiently and comprehensively extract spatiotemporal information for decision-making becomes a challenge; second, from the perspective of interdependence of the two-stage optimization problems, the two-stage decision-making is asynchronous, and the initial state of the second-stage perception task depends on the result of the first-stage perception task, which makes it difficult to obtain the global optimal solution of the system through independent optimization of the two stages, posing a challenge to the design of an efficient asynchronous decision-making optimization framework.

[0006] The two-stage vehicle-mounted drone swarm perception problem with data delay minimization is NP-hard, that is, it is usually impossible to give an optimal solution in polynomial time. The academic community has proposed solvers based on heuristic methods or approximate algorithms (such as OR-Tools, Gurobi, etc.), but they face the problem of "slow solution and difficult generalization", that is, when applied to problem instances with new geographical distribution, it still takes tens of seconds or even minutes. Methods based on deep learning also face challenges in the following two aspects. In terms of vehicle-mounted drone swarm observation extraction, the example methods of pointer network (Pointer Network), recurrent neural network (RNN), and transformer neural network use the representation ability of deep networks to extract the location information of entities such as unmanned vehicle stops and points of interest in the perception scene. However, these methods only process spatial information and are difficult to deal with complex observations intertwined in time and space. When these methods are directly applied to vehicle-mounted drone data collection tasks, the performance is still not high and cannot meet the needs of reducing data perception latency. In terms of vehicle-mounted drone swarm decision-making, the task is defined as a time-slot system, which can be considered as a Markov decision process (MDP). Among the many solutions for MDP, Multi-agent Deep Reinforcement Learning (MADRL) has gradually become a popular technology for dealing with continuous decision-making problems involving multiple controllable agents. However, existing methods such as typical IPPO (Independent Proximal Policy Optimization) assume that agents make decisions synchronously and that there is no mutual dependence between agents, making it difficult to better solve the decision asynchrony in the two-stage vehicle-mounted drone swarm perception problem with minimized data delay. Summary of the invention

[0007] In view of the defects in the prior art, the present invention proposes a group perception method based on vehicle-mounted UAV observation enhancement and asynchronous decision-making.

[0008] The group perception method based on vehicle-mounted drone observation enhancement and asynchronous decision-making of the present invention comprises the following steps:

[0009] Step 1: The simulation platform initializes the algorithm;

[0010] Step 2: Example of simulation platform initialization problem;

[0011] Step 3: Input the environmental information into the observation encoder for two-stage vehicle-mounted drone perception (hereinafter referred to as: observation encoder) to obtain the coded embedding for each observed entity and the global aggregate embedding of the problem instance. The observation encoder can enhance and aggregate the spatiotemporal information observed by the drone / unmanned vehicle, and fully consider the spatiotemporal interleaved information when making action decisions, solving the problem that traditional methods cannot efficiently and comprehensively extract spatiotemporal interleaved information for decision-making;

[0012] Step 4: Input the enhanced embeddings for the UAV agent and the UGV agent into the UAV / UGV action generator based on historical action autoregression, and obtain the decisions of the UAV agent and the UGV agent at their respective time steps;

[0013] Step 5: After the round ends, the simulation platform calculates the reward function after asynchronous decision-making, and introduces three types of reward functions that can accurately and meticulously describe the asynchronous decision-making in the perception process of vehicle-mounted drone swarms. A policy update framework for asynchronous decision-making of vehicle-mounted drone swarms is designed, taking into account the difficulties of each decision-making stage and the mutual dependence of the two decision-making stages;

[0014] Step 6: Use the policy update framework for asynchronous decision-making of vehicle-mounted drone groups to update the policy parameters θ of the drone / unmanned vehicle agent u and θ c Make updates;

[0015] Step 7: Update the baseline policy.

[0016] Step 8: Repeat steps 2 to 7 until the number of model training cycles reaches the upper limit;

[0017] Step 9. The simulation platform retrieves the saved optimal strategy parameters of the group perception method based on vehicle-mounted drone observation enhancement and asynchronous decision-making, that is, obtains the optimal scheduling strategy for the vehicle-mounted drone group that can minimize information delay, obtains the optimal path planning result for each drone / unmanned vehicle based on the optimal scheduling strategy, sends control instructions to the vehicle-mounted drone group to complete the group perception task and send data back to the simulation platform.

[0018] Furthermore, in step 1, the simulation platform initializes the algorithm, and the specific steps are:

[0019] Step 101: UAVs and unmanned vehicles are aggregated into a UAV group super-agent (hereinafter referred to as "UAV agent") and an unmanned vehicle group super-agent (hereinafter referred to as "unmanned vehicle agent"), respectively, and the super-agent performs the allocation and trajectory planning of different UAV / unmanned vehicle tasks;

[0020] Step 102: Initialize the parameters θ of the UAV agent and the unmanned vehicle agent strategy network respectively. u and θc ;

[0021] Step 103: Initialize the baseline policy parameters of the UAV agent and the unmanned vehicle agent respectively and The definition of the baseline policy is the policy network parameters with the best performance so far in the training process. The objective function of the training policy network is obtained by comparing the relative advantages and disadvantages of the current policy and the baseline policy to reduce the variance of the objective function and improve the training stability. The initialization satisfies

[0022] Step 104: Initialize data buffers for the UAV agent and the unmanned vehicle agent.

[0023] Furthermore, in step 2, the simulation platform initializes the problem instance, and the specific steps are:

[0024] Step 201: Initialize the environment information. There are 1 starting point D and N in the environment. B Unmanned vehicle stops B, N P Each point of interest P is called an entity in the environment. The location of the entity in the environment is randomly generated on the map by the simulation platform at each initialization, and the task is divided into several time steps;

[0025] Step 202: define the observation space and action space of the UAV agent. In each time step, the UAV agent will observe the location of each unmanned vehicle stop, the location of each point of interest, and the current decision sequence of the UAV agent. The UAV agent is required to make a decision in each time step: 1) select a point of interest to be visited by a certain UAV in the next time step; 2) choose whether to end the path planning of the current UAV and proceed with the path planning of the next UAV.

[0026] Step 203, define the observation space and action space of the unmanned vehicle agent. After the drone agent completes the task, the unmanned vehicle agent starts the task. In each time step, the unmanned vehicle agent will observe the starting point position, the unmanned vehicle stop position, the decision sequence of the drone agent and the decision sequence of the current unmanned vehicle agent. The unmanned vehicle agent is required to make decisions in each time step: 1) select the stop point to go to in the next time step for each unmanned vehicle or whether to return to the starting point, 2) select whether to end the path planning of the current unmanned vehicle and proceed with the path planning of the next unmanned vehicle;

[0027] Step 204: The simulation platform assigns an unmanned vehicle parking station to each unmanned vehicle and initializes each unmanned vehicle at the unmanned vehicle starting point;

[0028] Step 205, the simulation platform sets the time step to 0 and starts to prepare to schedule the UAV agent and the unmanned vehicle agent to interact with the environment. The entire asynchronous decision-making process in which the UAV agent first completes the first stage decision and then the unmanned vehicle agent completes the second stage decision based on the decision result of the UAV agent is defined as a round (hereinafter referred to as: round).

[0029] Furthermore, in step 3, the environmental information is input into the observation encoder for two-stage vehicle-mounted drone perception (hereinafter referred to as: observation encoder) to obtain the encoding embedding for each observed entity and the global aggregate embedding of the problem instance. The observation encoder can enhance and aggregate the spatiotemporal information observed by the drone / unmanned vehicle, and fully consider the spatiotemporal interleaved information when making action decisions, solving the problem that traditional methods cannot efficiently and comprehensively extract spatiotemporal interleaved information for decision-making. The specific steps are as follows:

[0030] Step 301: Given d x The input features x of each observed entity of dimension i , the initial d is calculated as follows (1) h dimensional observation entity encoding embedding In the following formula (1), d h =128:

[0031]

[0032] Where: W X and b X is the linear transformation obtained through learning;

[0033] Step 302: Update the observed entity encoding embedding obtained above through N attention layers. Each attention layer consists of two sub-layers, a multi-head attention (MHA) layer for performing information transfer between nodes, and a node-level fully connected feed-forward (FF) layer. Each sub-layer adds residual connection and batch normalization (BN). The overall calculation process is as follows (2) and (3):

[0034]

[0035] In the above formula, the layer index l indicates that the parameters of each layer are not shared. The MHA sublayer uses M = 8 heads, and the dimension of each head is In addition, the FF sublayer contains a hidden sublayer with 512 dimensions and ReLU activation function;

[0036] Step 303: embed the final codes of all observed entities obtained by the above calculation into Take the average to get the global aggregate embedding of the problem instance As shown in formula (4):

[0037]

[0038] The encoding of each observed entity is embedded and the global aggregate embedding of the problem instance It is passed as input to the drone / unmanned vehicle action generator based on historical action autoregression and is uniformly defined as the enhanced embedding of the agent for the observed information (hereinafter referred to as: enhanced embedding).

[0039] Furthermore, in step 4, the enhanced embeddings for the UAV agent and the unmanned vehicle agent are successively input into the UAV / unmanned vehicle action generator based on historical action autoregression to obtain the decisions of the UAV agent and the unmanned vehicle agent at their respective time steps. The specific steps are as follows:

[0040] Step 401: Strengthen the enhanced embedding of the drone agent. When calculating the code embedding of the observed entity, the heterogeneity of different entities observed by the drone agent is taken into account. The initial code embedding of the drone agent is enhanced as shown in the following formula (5):

[0041]

[0042] in: are all linear transformations obtained by the UAV agent through learning. idx is the number of the UAV currently making decisions. is the distance vector from the point of interest to each unmanned vehicle stop;

[0043] Step 402: Similarly, the enhanced embedding of the unmanned vehicle agent is enhanced. The initial encoding embedding of the unmanned vehicle agent is enhanced as shown in the following formula (6):

[0044]

[0045] in: are all linear transformations obtained by the unmanned vehicle agent through learning. It represents the time required for the UAV to reach the corresponding point of interest according to the decision sequence of the UAV agent from the UAV parking station. is the information vector of the time information, including the minimum value, average value and maximum value;

[0046] Step 403: Based on the input enhanced embedding, a context embedding based on historical actions and historical spatiotemporal information is calculated as follows:

[0047]

[0048] Where: N Kt represents the number of drones / unmanned vehicles for which the super-agent has not started making decisions at time step t, n' t Indicates the number of entities that the drone / unmanned vehicle currently making a decision has visited. Indicates the distance that the current decision-making UAV / unmanned vehicle has traveled. represents the distance traveled by the UAV / unmanned vehicle with the longest travel distance at present, d(p t ,k t ) represents the distance between the drone / unmanned vehicle that is making a decision and its starting point (the drone corresponds to the vehicle stop point, and the unmanned vehicle corresponds to the starting point). At time step t = 1, v 1 ,v f represents the d learned by the agent h Dimension placeholder;

[0049] Step 404: The drone / unmanned vehicle action generator based on historical action autoregression uses a strategy acquisition method based on a multi-head attention mechanism. The attention weight is determined by calculating the dot product of the query (q) representing the historical action and historical spatiotemporal information and the key (k) representing the optional action of the drone / unmanned vehicle at the current time step. These weights are then applied to the value (v) representing the correlation between the query and the key, so as to be used for the final action output. The query, key and value are calculated using the following formula (8):

[0050]

[0051] Where: W Q ,W K ,W V All are learned linear transformations;

[0052] Step 405: Calculate query q (c) The compatibility with all actions (i.e., the selection of points in this time step) is limited to a certain range using the tanh function, as calculated in the following formula (9):

[0053]

[0054] Step 406: interpret the above compatibility as unnormalized logarithmic probability, and use the softmax function to calculate the final action output probability vector p as follows:

[0055]

[0056] Furthermore, in step 5, after the round ends, the simulation platform calculates the reward function after asynchronous decision-making, and introduces three types of reward functions that can accurately and meticulously describe the asynchronous decision-making situation in the perception process of vehicle-mounted drone swarms. A policy update framework for asynchronous decision-making of vehicle-mounted drone swarms is designed, taking into account the difficulties of each decision-making stage and the mutual dependence of the two decision-making stages. The specific steps are as follows:

[0057] Step 501: Calculate the reward function R1 related only to the decision sequence of the UAV agent according to formula (11):

[0058]

[0059] in: represents the time required for the UAV to reach the corresponding point of interest according to the decision sequence from the UAV stop, ω i Indicates the weight corresponding to each point of interest. The higher the weight, the more priority the point of interest should be perceived.

[0060] Step 502: Calculate the reward function R2 related only to the decision sequence of the unmanned vehicle agent according to formula (12):

[0061]

[0062] in: It represents the time required for the unmanned vehicle to reach the corresponding unmanned vehicle stop according to the decision sequence from the starting point;

[0063] Step 503: Calculate the reward function R of the environment as a whole according to formula (13): total :

[0064]

[0065] Where: time i It represents the data delay from the departure of the unmanned vehicle to the completion of the perception of the corresponding point of interest by the UAV.

[0066] Furthermore, in step 6, the policy parameter θ of the UAV / unmanned vehicle agent is updated using the policy update framework for asynchronous decision-making of vehicle-mounted UAV swarms. u and θ c To update, the specific steps are as follows:

[0067] Step 601: Repeat steps 4 to 7 for a total of T times, using reward functions R1 and R2 each time, and calculate the gradient estimate based on the reinforcement algorithm, as shown in the following formula (14):

[0068]

[0069] Where: b u(s0) and b c (s'0) represents the reward function obtained by the baseline strategy corresponding to the UAV / Unmanned Vehicle agent, and the above gradient estimation is used to simultaneously update the strategy network of the UAV agent and the Unmanned Vehicle agent, so that the UAV / Unmanned Vehicle agent can quickly learn from the pure random strategy to the local optimal strategy of each of the two stages;

[0070] Step 602: Repeat steps 4 to 6 for a total of T times, using the reward function R each time. total , the gradient estimate is calculated based on the reinforcement algorithm, as shown in the following formula (15):

[0071]

[0072] As above, policy gradient updates are performed to enable the UAV / unmanned vehicle agent to learn the global optimal strategy for the vehicle-mounted UAV swarm perception task.

[0073] Furthermore, in step 7, the baseline policy is updated:

[0074] The definition of the baseline policy is the policy network parameters with the best performance so far in the training process. At the end of each model training cycle, the performance of the current policy of the drone / unmanned vehicle agent is compared with its corresponding baseline policy. Only when the performance improvement of the current policy compared with the baseline policy is significant in the paired t-test (α = 5%), the parameters θ of the baseline policy corresponding to the corresponding agent are updated. BL , tested using 10,000 problem instances.

[0075] The beneficial effects of the method of the present invention are:

[0076] 1. The method described in the present invention proposes a new perception problem of vehicle-mounted drone swarms, and explores how to quickly realize the perception of information points and minimize information delay through two-stage vehicle-mounted drone swarm scheduling; the research on this problem by the method described in the present invention further promotes the application of vehicle-mounted drone swarms in real-life rapid perception scenarios.

[0077] 2. The method described in the present invention designs an observation encoder for two-stage vehicle-mounted drone perception and a drone / unmanned vehicle action generator based on historical action autoregression for drone / unmanned vehicle decision-making. It can enhance and aggregate the spatiotemporal information observed by the drone / unmanned vehicle, and fully consider the spatiotemporal interleaved information when making action decisions, solving the problem that traditional methods cannot efficiently and comprehensively extract spatiotemporal interleaved information for decision-making.

[0078] 3. The method described in the present invention accurately models the vehicle-mounted drone swarm perception task. The three types of reward functions introduced can accurately and meticulously describe the asynchronous decision-making situation in the vehicle-mounted drone swarm perception process, and designs a strategy update framework for asynchronous decision-making of vehicle-mounted drone swarms. It fully considers the difficulties of each decision-making stage and the interdependence of the two decision-making stages, and solves the challenge of interdependence in the two-stage optimization problem.

[0079] 4. The strategy update framework and baseline update strategy introduced by the method of the present invention are only used in training and no longer need to be used in reasoning. Therefore, fast reasoning can be achieved using only the CPU, avoiding the problem of using expensive GPUs with large physical sizes, and solving the problem of "slow solution and difficult generalization" of traditional optimization methods. At the same time, the method does not increase time and space costs and is easier to deploy in real drones / unmanned vehicles.

[0080] 5. The method described in the present invention is a highly scalable multi-agent deep reinforcement learning framework for vehicle-mounted drone swarm perception tasks, which can be built on the basis of any multi-agent actor-critic algorithm. BRIEF DESCRIPTION OF THE DRAWINGS

[0081] Figure 1 It is a schematic diagram of the overall process of the method of the present invention;

[0082] FIG2 is a solution result of the method of the present invention on an example of a vehicle-mounted drone perception problem, where: Figure 2a Schematic diagram of the UAV task allocation and scheduling trajectory; Figure 2b Schematic diagram of the unmanned vehicle scheduling trajectory. Specific implementation plan

[0083] The present invention is further described in detail below in conjunction with the accompanying drawings. Figure 1 shown.

[0084] In a specific embodiment of the present invention, the simulation platform generates entities in a two-stage vehicle-mounted drone swarm perception mission scenario with minimized data delay by a random generation method, and the random generation method is as follows:

[0085] The entities in the scene are divided into starting points, driverless car stops, and points of interest. The locations of the entities are randomly distributed to simulate the possible situations in different scenes. i Coordinates Uniform sampling is performed from a 1×1 square area, and the x and y coordinates are uniformly distributed in the range of (0,1). After the random generation of the above entities is completed, an unmanned vehicle docking station is assigned to each drone, and each unmanned vehicle is initialized at the unmanned vehicle starting point.

[0086] For the parameters of drone / unmanned vehicle agent strategy training, the discount factor for calculating cumulative returns is set to 0.95, the batch training size is 512, the learning rate of all learnable parameters is 0.0001, and the weight decay during network parameter update is set to 0.00001. A total of 30 training cycles are designed, and 1,280,000 rounds are set for training in each training cycle, and the model that minimizes the perception data delay is selected.

[0087] In order to demonstrate the performance of the method of the present invention in minimizing data perception latency in vehicle-mounted drone swarm perception, a detailed and complete system test was conducted, and problem instances of different entity scales (2 unmanned vehicle stops and 20 points of interest, 5 unmanned vehicle stops and 50 points of interest, 10 unmanned vehicle stops and 100 points of interest) were designed. The specific evaluation form is the following three indicators when the entire vehicle-mounted drone perception task is completed:

[0088] 1. Average data perceived delay (R time ): Calculate as follows (16):

[0089]

[0090] Where: time i It represents the data delay from the departure of the unmanned vehicle to the completion of the perception of the corresponding point of interest by the drone;

[0091] 2. Data perception delay satisfaction (R satisfaction ): At a given threshold τ th If the perceived delay of the point of interest is less than the threshold, it is considered to have reached a satisfactory level. If it exceeds this threshold, it is considered to have not reached a satisfactory level, as calculated by the following formula (17):

[0092]

[0093] 3. Total driving distance of vehicle-mounted drone swarm (R dis ): The sum of the distances traveled by the vehicle-mounted drone swarm (including drones and unmanned vehicles) after completing the perception task.

[0094] like Figure 2a-2bAs shown in the figure, taking an example of a perception task of a vehicle-mounted drone swarm with many entities and relatively complex decisions, the method of the present invention is used to demonstrate the visualization results of scheduling a single vehicle-mounted drone in a vehicle-mounted drone swarm on a simulation platform. It can be seen that the task division of the vehicle-mounted drone swarm (dividing each point of interest into perceptions by different drones) and strategy scheduling have achieved good results, and when scheduling vehicle-mounted drones, the intelligent agent can fully consider spatial and temporal information. When scheduling drones / unmanned vehicles, while considering spatial information, it can first perform actions with higher perception information delay costs (such as unmanned vehicles first visiting and assigning unmanned vehicle stops where drones with more perception tasks are located), which fully proves the effectiveness of the method of the present invention when spatiotemporal information is intertwined and the two-stage optimization problem is interdependent.

[0095] The present invention is not limited to the above-mentioned embodiments. Without departing from the essential content of the present invention, any deformation, improvement and substitution that can be thought of by those skilled in the art shall fall into the protection scope of the present invention.

Claims

1. A group sensing method based on vehicle-mounted drone observation enhancement and asynchronous decision-making, comprising the following steps: Step 1: The simulation platform initializes the algorithm; Step 2: Example of simulation platform initialization problem; Step 3: Input the environmental information into the observation encoder for two-stage vehicle-mounted drone perception to obtain the coded embedding for each observed entity and the global aggregate embedding of the problem instance. The observation encoder can enhance and aggregate the spatiotemporal information observed by the drone / unmanned vehicle, and fully consider the spatiotemporal interleaved information when making action decisions, solving the problem that traditional methods cannot efficiently and comprehensively extract spatiotemporal interleaved information for decision-making; Step 4: Input the enhanced embeddings for the UAV agent and the UGV agent into the UAV / UGV action generator based on historical action autoregression, and obtain the decisions of the UAV agent and the UGV agent at their respective time steps; Step 5. After the round ends, the simulation platform calculates the reward function after asynchronous decision-making, and introduces three types of reward functions to accurately and meticulously describe the asynchronous decision-making of vehicle-mounted drone swarms during perception. It designs a policy update framework for asynchronous decision-making of vehicle-mounted drone swarms, and considers the difficulty of each decision-making stage and the mutual dependence of the two decision-making stages. Step 6: Use the policy update framework for asynchronous decision-making of vehicle-mounted drone groups to update the policy parameters θ of the drone / unmanned vehicle agent u and θ c Make updates; Step 7: Update the baseline strategy. Step 8: Repeat steps 2 to 7 until the number of model training cycles reaches the upper limit; Step 9. The simulation platform retrieves the saved optimal strategy parameters of the group perception method based on vehicle-mounted drone observation enhancement and asynchronous decision-making, that is, obtains the optimal scheduling strategy for the vehicle-mounted drone group that can minimize information delay, obtains the optimal path planning result for each drone / unmanned vehicle based on the optimal scheduling strategy, sends control instructions to the vehicle-mounted drone group to complete the group perception task and send data back to the simulation platform.

2. According to the group perception method based on vehicle-mounted drone observation enhancement and asynchronous decision-making as described in claim 1, in step 1, the simulation platform initializes the algorithm, and the specific steps are: Step 101: UAVs and unmanned vehicles are aggregated into a UAV group super-agent (hereinafter referred to as "UAV agent") and an unmanned vehicle group super-agent (hereinafter referred to as "unmanned vehicle agent"), respectively, and the super-agent performs the allocation and trajectory planning of different UAV / unmanned vehicle tasks; Step 102: Initialize the parameters θ of the UAV agent and the unmanned vehicle agent strategy network respectively. u and θ c ; Step 103: Initialize the baseline policy parameters of the UAV agent and the unmanned vehicle agent respectively and The definition of the baseline policy is the policy network parameters with the best performance so far in the training process. The objective function of the training policy network is obtained by comparing the relative advantages and disadvantages of the current policy and the baseline policy to reduce the variance of the objective function and improve the training stability. The initialization satisfies Step 104: Initialize data buffers for the UAV agent and the unmanned vehicle agent.

3. According to the group perception method based on vehicle-mounted drone observation enhancement and asynchronous decision-making as described in claim 1, in step 2, the simulation platform initializes the problem instance, and the specific steps are: Step 201: Initialize the environment information. There are 1 starting point D and N in the environment. B Unmanned vehicle stops B, N P Each point of interest P is called an entity in the environment. The location of the entity in the environment is randomly generated on the map by the simulation platform at each initialization, and the task is divided into several time steps; Step 202: define the observation space and action space of the drone agent. In each time step, the drone agent will observe the location of each unmanned vehicle stop, the location of each point of interest, and the current decision sequence of the drone agent. The drone agent is required to make a decision in each time step: select a point of interest to go to in the next time step for a certain drone, choose whether to end the path planning for the current drone, and proceed with the path planning for the next drone. Step 203, define the observation space and action space of the unmanned vehicle agent. After the drone agent completes the task, the unmanned vehicle agent starts the task. In each time step, the unmanned vehicle agent will observe the starting point position, the unmanned vehicle stop position, the decision sequence of the drone agent and the current decision sequence of the unmanned vehicle agent. The unmanned vehicle agent is required to make a decision in each time step: select the stop point to go to in the next time step for each unmanned vehicle or whether to return to the starting point, choose whether to end the path planning of the current unmanned vehicle, and proceed with the path planning of the next unmanned vehicle; Step 204: The simulation platform assigns an unmanned vehicle parking station to each unmanned vehicle and initializes each unmanned vehicle at the unmanned vehicle starting point; Step 205, the simulation platform sets the time step to 0 and starts to prepare to schedule the UAV agent and the unmanned vehicle agent to interact with the environment. The entire asynchronous decision-making process in which the UAV agent first completes the first stage decision and the unmanned vehicle agent completes the second stage decision based on the decision result of the UAV agent is defined as a round.

4. According to the group perception method based on vehicle-mounted drone observation enhancement and asynchronous decision-making as described in claim 1, in step 3, the environmental information is input into the observation encoder for two-stage vehicle-mounted drone perception to obtain the encoding embedding for each observed entity and the global aggregate embedding of the problem instance. The observation encoder can enhance and aggregate the spatiotemporal information observed by the drone / unmanned vehicle, and fully consider the spatiotemporal interleaved information when making action decisions, which solves the problem that traditional methods cannot efficiently and comprehensively extract spatiotemporal interleaved information for decision-making. The specific steps are as follows: Step 301: Given d x The input features x of each observed entity of dimension i , the initial d is calculated as follows (1) h dimensional observation entity encoding embedding In the following formula (1), d h =128: in: W X and b X is the linear transformation obtained through learning; Step 302: Update the observed entity encoding embedding obtained above through N attention layers. Each attention layer consists of two sub-layers, a multi-head attention layer for performing information transfer between nodes, and a node-level fully connected feed-forward layer. Residual connections and batch normalization are added to each sub-layer. The overall calculation process is as follows: In the above formula, the layer index l indicates that the parameters of each layer are not shared. The MHA sublayer uses M = 8 heads, and the dimension of each head is In addition, the FF sublayer contains a hidden sublayer with 512 dimensions and ReLU activation function; Step 303: embed the final codes of all observed entities obtained by the above calculation into Take the average to get the global aggregate embedding of the problem instance As shown in formula (4): The encoding of each observed entity is embedded and the global aggregate embedding of the problem instance It is passed as input to the drone / unmanned vehicle action generator based on historical action autoregression and is uniformly defined as the enhanced embedding of the agent for the observed information.

5. According to the group perception method based on vehicle-mounted drone observation enhancement and asynchronous decision-making as claimed in claim 1, in step 4, the enhanced embeddings for the drone agent and the unmanned vehicle agent are successively input into the drone / unmanned vehicle action generator based on historical action autoregression, and the decisions of the drone agent and the unmanned vehicle agent at their respective time steps are obtained respectively. The specific steps are as follows: Step 401: Strengthen the enhanced embedding of the drone agent. When calculating the code embedding of the observed entity, the heterogeneity of different entities observed by the drone agent is taken into account. The initial code embedding of the drone agent is enhanced as shown in the following formula (5): in: are all linear transformations obtained by the UAV agent through learning. idx is the number of the UAV currently making decisions. is the distance vector from the point of interest to each unmanned vehicle stop; Step 402: Similarly, the enhanced embedding of the unmanned vehicle agent is enhanced. The initial encoding embedding of the unmanned vehicle agent is enhanced as shown in the following formula (6): in: are all linear transformations obtained by the unmanned vehicle agent through learning. It represents the time required for the UAV to reach the corresponding point of interest according to the decision sequence of the UAV agent from the UAV parking station. is the information vector of the time information, including the minimum value, average value and maximum value; Step 403: Based on the input enhanced embedding, a context embedding based on historical actions and historical spatiotemporal information is calculated as follows: Where: N Kt represents the number of drones / unmanned vehicles for which the super-agent has not started making decisions at time step t, n' t Indicates the number of entities that the drone / unmanned vehicle currently making a decision has visited. Indicates the distance that the current decision-making drone / unmanned vehicle has traveled. represents the distance traveled by the UAV / unmanned vehicle with the longest travel distance at present, d(p t ,k t ) represents the distance between the drone / unmanned vehicle that is making the decision and its starting point, that is, the distance between the drone and the vehicle stop and the unmanned vehicle and the starting point. At time step t = 1, v 1 ,v f represents the d learned by the agent h Dimension placeholder; Step 404: The drone / unmanned vehicle action generator based on historical action autoregression uses a strategy acquisition method based on a multi-head attention mechanism. The attention weight is determined by calculating the dot product of the query (q) representing the historical action and historical spatiotemporal information and the key (k) representing the optional action of the drone / unmanned vehicle at the current time step. These weights are then applied to the value (v) representing the correlation between the query and the key, so as to be used for the final action output. The query, key and value are calculated using the following formula (8): q (c) =W Q h (c) k i =W K h i ,v i =W V h i ……(8), Where: W Q ,W K ,W V All are learned linear transformations; Step 405: Calculate query q (c) The compatibility with all actions is limited to a certain range using the tanh function, as calculated by the following formula (9): Step 406: interpret the above compatibility as unnormalized logarithmic probability, and use the softmax function to calculate the final action output probability vector p as follows:

6. According to the group perception method based on vehicle-mounted drone observation enhancement and asynchronous decision-making as described in claim 1, in step 5, after the round ends, the simulation platform calculates the reward function after asynchronous decision-making, and introduces three types of reward functions that can accurately and meticulously describe the situation of asynchronous decision-making in the process of vehicle-mounted drone group perception. A strategy update framework for asynchronous decision-making of vehicle-mounted drone groups is designed, taking into account the difficulties of each decision-making stage and the mutual dependence of the two decision-making stages. The specific steps are as follows: Step 501: Calculate the reward function R1 related only to the decision sequence of the UAV agent according to formula (11): in: represents the time required for the UAV to reach the corresponding point of interest according to the decision sequence from the UAV stop, ω i Indicates the weight corresponding to each point of interest. The higher the weight, the more priority the point of interest should be perceived. Step 502: Calculate the reward function R2 related only to the decision sequence of the unmanned vehicle agent according to formula (12): in: It represents the time required for the unmanned vehicle to reach the corresponding unmanned vehicle stop according to the decision sequence from the starting point; Step 503: Calculate the reward function R of the environment as a whole according to formula (13): total : Where: time i It represents the data delay from the departure of the unmanned vehicle to the completion of the perception of the corresponding point of interest by the UAV.

7. According to the group perception method based on vehicle-mounted drone observation enhancement and asynchronous decision-making as claimed in claim 1, in step 6, the policy parameter θ of the drone / unmanned vehicle agent is updated using a policy update framework for asynchronous decision-making of vehicle-mounted drone groups. u and θ c To update, the specific steps are as follows: Step 601: Repeat steps 4 to 7 for a total of T times, using reward functions R1 and R2 each time, and calculate the gradient estimate based on the reinforcement algorithm, as shown in the following formula (14): in: b u (s0) and b c (s'0) represents the reward function obtained by the baseline strategy corresponding to the UAV / Unmanned Vehicle agent, and the above gradient estimation is used to simultaneously update the strategy network of the UAV agent and the Unmanned Vehicle agent, so that the UAV / Unmanned Vehicle agent can quickly learn from the pure random strategy to the local optimal strategy of each of the two stages; Step 602: Repeat steps 4 to 6 for a total of T times, using the reward function R each time. total , the gradient estimate is calculated based on the reinforcement algorithm, as shown in the following formula (15): As above, policy gradient updates are performed to enable the drone / unmanned vehicle agent to learn the global optimal strategy for the vehicle-mounted drone swarm perception task.

8. According to the group perception method based on vehicle-mounted drone observation enhancement and asynchronous decision-making as claimed in claim 1, in step 7, the baseline strategy is updated: The definition of the baseline policy is the policy network parameters with the best performance so far in the training process. At the end of each model training cycle, the performance of the current policy of the drone / unmanned vehicle agent is compared with its corresponding baseline policy. Only when the performance improvement of the current policy compared to the baseline policy is significant in the paired t-test α=5%, the parameters θ of the baseline policy corresponding to the corresponding agent are updated. BL , tested using 10,000 problem instances.

Citation Information

Cited By

  • Internet of vehicles cooperative sensing parameter high-efficiency domain self-adaption method and device and medium

    CN121711658A

  • An efficient domain adaptation method and device for cooperative perception parameters of internet of vehicles and a medium

    CN121711658B