Methods, devices, equipment and media for multi-UAV collaborative traffic condition monitoring

Abstract

This application discloses a method, apparatus, device, and medium for multi-UAV collaborative traffic condition monitoring, relating to the field of multi-UAV collaborative control technology. The method includes: multiple UAVs conducting inspections within a preset inspection area; identifying traffic anomalies using a trained deep learning model; selecting cooperative UAVs and assigning them to the abnormal traffic area when anomalies are detected, and collaboratively collecting traffic condition data; outputting fused traffic condition data after each cooperative UAV merges the traffic condition data from other cooperative UAVs; and obtaining real-time traffic condition monitoring results based on the fused traffic condition data from the cooperative UAVs. This application utilizes multiple UAVs to perceive the same scene from different angles, enabling the fusion of data from multiple perspectives, improving the accuracy of accident detection, and providing data support for relevant departments in handling abnormal traffic events.

Description

Technical Field

[0001] This application relates to the field of multi-UAV collaborative control technology, and in particular to a method, device, equipment and medium for multi-UAV collaborative traffic condition monitoring. Background Technology

[0002] With my country's economic development and social progress, the number of vehicles has increased rapidly, and traffic congestion and violations have gradually become hot issues in social development. Currently, monitoring equipment installed on roads is commonly used to monitor vehicles, and the monitoring information can be used to calculate traffic flow, measure vehicle speed, and identify traffic conditions. However, roadside monitoring equipment has the following drawbacks:

[0003] Limited field of view means the monitoring area can only cover a few nodes and has blind spots, making it impossible to obtain macroscopic traffic data for a specific traffic node in a long-distance, multi-segment accident scenario. Response is delayed, requiring manual analysis and making it difficult to take action based on limited information in a short time, resulting in low accident handling efficiency. Deployment and construction costs are high, requiring significant upfront investment and ongoing maintenance. Data transmission and communication bottlenecks exist; the large volume of data generated by the monitoring system is limited by bandwidth in remote road environments, failing to meet real-time transmission and response requirements, and hindering the immediate monitoring and handling of traffic conditions.

[0004] To address the shortcomings of traditional traffic monitoring methods, drones can be used for traffic monitoring. Compared to traditional methods, drones offer advantages such as flexibility, a wide field of view, and rapid deployment. However, the use of drones in this context does not fundamentally solve the aforementioned technical problems.

[0005] While related technologies utilize drones to replace traditional monitoring equipment, personnel are still needed to determine traffic conditions and whether accidents have occurred. For accidents with a wide impact, it is difficult to provide evacuation plans in a short time to alleviate widespread congestion. Therefore, the use of drones in these technologies has not changed the problem of manpower and material resources being consumed in traffic monitoring. Furthermore, the coverage and field of view of a single drone are limited. Multiple drones can collect data comprehensively from multiple angles, but multi-drone coordination is difficult. Traditional coordination methods typically require each drone to transmit all sensing data. While this preserves complete scene information, it also leads to low bandwidth utilization and excessive communication load.

[0006] Therefore, there is currently a lack of a method that can utilize drones to achieve real-time and efficient traffic condition monitoring. Summary of the Invention

[0007] This application provides a method, apparatus, device, and medium for multi-UAV collaborative traffic condition monitoring to address the shortcomings of the aforementioned related technologies. The technical solution is as follows:

[0008] In a first aspect, embodiments of this application provide a traffic condition monitoring method involving multiple unmanned aerial vehicles (UAVs) in collaboration, comprising:

[0009] Multiple drones are used to conduct inspections within a pre-defined inspection area; each drone inspects a sub-area within the pre-defined inspection area based on a pre-defined inspection route.

[0010] The trained deep learning model on each drone identifies whether there are any anomalies in the traffic conditions within the corresponding sub-region.

[0011] When an abnormal traffic condition is detected, a cooperating drone is selected based on the real-time status of each drone, and each cooperating drone is assigned to the area with abnormal traffic conditions, so as to collect traffic condition data collaboratively through multiple cooperating drones.

[0012] Each of the cooperative drones merges traffic status data from other cooperative drones and outputs merged traffic status data.

[0013] Real-time traffic condition monitoring results are obtained based on the fused traffic condition data of each of the collaborative drones.

[0014] In one alternative embodiment of the first aspect, each UAV identifies whether there are anomalies in the traffic conditions within its corresponding sub-region using a trained deep learning model, including:

[0015] Image data of the corresponding sub-region is collected using an image acquisition device mounted on a drone;

[0016] The image data is input into the trained deep learning model, and the trained deep learning model extracts features from the image data to identify whether there are abnormal traffic conditions.

[0017] When an anomaly in traffic conditions is detected, point cloud data is collected in the corresponding sub-region using a lidar mounted on a drone. Based on the point cloud data, the location range of the traffic anomaly is determined, thus identifying the area of ​​traffic anomaly.

[0018] In one alternative embodiment of the first aspect, the step of selecting cooperative drones based on the real-time status of each drone and assigning each cooperative drone to an area with abnormal traffic conditions includes:

[0019] Determine the distance from the real-time location coordinates of each drone to the area of ​​abnormal traffic conditions;

[0020] Based on the distance, estimate the remaining battery power of each drone after it reaches the traffic anomaly area. If the remaining battery power is greater than a first battery power threshold, then the corresponding drone is selected as a cooperative drone.

[0021] Update the inspection routes of the collaborative drones to the areas with abnormal traffic conditions, and assign each collaborative drone to the areas with abnormal traffic conditions.

[0022] In one alternative embodiment of the first aspect, the method further includes:

[0023] If the remaining battery power of any drone is less than or equal to the second battery threshold, update the return route of the low-battery drone to the drone nest, and generate a return command to the low-battery drone so that the low-battery drone executes the return command and returns to the drone nest according to the return route;

[0024] When generating a return command to the low-power drone, the real-time location and inspected sub-area of ​​the low-power drone are obtained, a flight path from the backup drone to the real-time location of the low-power drone is generated, and a control command is generated to the backup drone so that the backup drone executes the control command and flies to the real-time location of the low-power drone according to the flight path, and continues to inspect the sub-area originally inspected by the low-power drone.

[0025] In one alternative of the first aspect, each collaborative UAV generates a first feature distribution matrix based on the acquired feature map, and generates a second feature distribution map that is negatively correlated with the first feature distribution matrix;

[0026] Each of the cooperative drones, after fusing traffic state data from other cooperative drones, outputs fused traffic state data, including:

[0027] The first collaborative drone receives the second feature distribution map from the second collaborative drone and constructs a feature selection matrix based on the first feature distribution matrix of the first collaborative drone and the second feature distribution map of the second collaborative drone.

[0028] Based on the feature selection matrix, the selection weight of each feature in the first feature distribution matrix of the first cooperative UAV is determined, the corresponding feature is selected based on the value of the selection weight, a sparse priority feature map is generated, and the sparse priority feature map is sent to the second cooperative UAV.

[0029] The feature map of the second cooperative UAV is fused with the sparse priority feature map, and the fused feature map is output as the fused traffic state data.

[0030] In this matrix, each element represents the confidence level of the traffic state data at each location.

[0031] In one alternative of the first aspect, obtaining real-time traffic condition monitoring results based on the fused traffic condition data of each of the cooperative drones includes:

[0032] Modeling is performed based on the fused traffic state data of each of the collaborative drones to determine the vehicle distribution and average vehicle speed on each road, and output real-time traffic state monitoring results.

[0033] In one alternative embodiment of the first aspect, after obtaining the real-time traffic condition monitoring results based on the fused traffic condition data of each of the cooperative drones, the method further includes:

[0034] Based on the real-time traffic condition monitoring results, congested road sections and traffic flow on congested road sections are determined, and traffic anomaly alerts are sent to other drones within a preset range of the traffic anomaly area through the cooperative drone.

[0035] After receiving the traffic anomaly alert, the other drones determine the traffic status data of the connecting road segments of the congested road segment;

[0036] A driving route that bypasses the congested road section is generated based on the traffic status data of the connected road sections.

[0037] Secondly, embodiments of this application also provide a traffic condition monitoring device for multi-UAV collaborative operation, comprising:

[0038] The drone scheduling module is used to conduct inspections within a preset inspection area using multiple drones; wherein each drone inspects a sub-area within the inspection area based on a preset inspection route;

[0039] The information recognition module is used to identify whether there are any abnormalities in the traffic conditions within the corresponding sub-region using the trained deep learning model carried by each drone.

[0040] When the information identification module detects an abnormal traffic condition, the drone scheduling module is also used to select a cooperating drone based on the real-time status of each drone and assign each cooperating drone to the area with abnormal traffic conditions, so as to collect traffic condition data collaboratively through multiple cooperating drones.

[0041] The information fusion module is used to enable each of the cooperative drones to merge traffic status data from other cooperative drones and then output the merged traffic status data.

[0042] The information processing module is used to obtain real-time traffic condition monitoring results based on the fused traffic condition data of each of the cooperative drones.

[0043] Thirdly, embodiments of this application also provide an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the method provided by the first aspect or any implementation thereof of the embodiments of this application.

[0044] Fourthly, this application also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the method provided by the first aspect of the embodiments of this application or any implementation thereof.

[0045] The beneficial effects of the technical solutions provided in some embodiments of this application include at least the following:

[0046] This application provides a multi-UAV collaborative traffic condition monitoring method, apparatus, equipment, and medium. This application first uses individual UAVs to collect information within corresponding sub-regions, thereby achieving large-scale monitoring and quickly identifying approximate areas with abnormal traffic conditions. More collaborative UAVs can then be allocated to these abnormal areas, saving UAV computing resources. The UAVs can move freely and cover a wide area, overcoming the limitations of fixed monitoring equipment. Furthermore, a multi-UAV collaborative sensing approach is used in abnormal traffic areas. Multiple UAVs perceive the same scene from different angles, allowing the system to fuse data from multiple perspectives and improve the accuracy of accident detection. After detecting an accident, this application can generate traffic condition monitoring results in real time and transmit them to a server via a communication network. This provides users with efficient traffic anomaly warning information and offers real-time, efficient data support for relevant departments handling traffic anomalies, thus shortening response time. Attached Figure Description

[0047] To more clearly illustrate the technical solutions in this application or related technologies, the drawings used in the description of the embodiments or related technologies will be briefly introduced below. Obviously, the drawings described below are some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0048] Figure 1 This is a flowchart illustrating a multi-UAV collaborative traffic condition monitoring method provided in an embodiment of this application;

[0049] Figure 2This is a schematic diagram of the structure of a multi-UAV collaborative traffic condition monitoring device provided in an embodiment of this application;

[0050] Figure 3 This is a schematic diagram of the structure of the electronic device provided in the embodiments of this application. Detailed Implementation

[0051] To make the objectives, technical solutions, and advantages of this application clearer, the technical solutions of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0052] The terms "comprising" and "having," and any variations thereof, in the specification, claims, and accompanying drawings of this application are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or apparatus that includes a series of steps or modules is not limited to the steps or modules listed, but may optionally include steps or modules not listed, or may optionally include other steps or modules inherent to such process, method, product, or apparatus.

[0053] It should be noted that the terms "first" and "second" used in this application are merely to distinguish similar objects and do not represent a specific ordering of the objects. It is understood that "first" and "second" can be interchanged in a specific order or sequence where permitted. It should be understood that the objects distinguished by "first" and "second" can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in an order other than those described or illustrated herein.

[0054] The present application will now be described in detail with reference to specific embodiments.

[0055] Next, combine Figure 1 This application introduces a multi-UAV collaborative traffic condition monitoring method provided by its embodiments. For details, please refer to... Figure 1 , Figure 1 This illustration shows a flowchart of a multi-UAV collaborative traffic condition monitoring method provided in an embodiment of this application. Figure 1 As shown, the method includes the following steps:

[0056] S101 uses multiple drones to conduct inspections within a pre-defined inspection area;

[0057] S102, using the trained deep learning model carried by each drone to identify whether there are any abnormalities in the traffic status within the corresponding sub-region;

[0058] If an anomaly in the traffic condition is detected, execute S103:

[0059] S103, Select cooperative drones based on the real-time status of each drone, and assign each cooperative drone to an area with abnormal traffic conditions, so as to collect traffic condition data collaboratively through multiple cooperative drones;

[0060] S104, Each of the cooperative drones merges the traffic status data of other cooperative drones and outputs the merged traffic status data;

[0061] S105, real-time traffic condition monitoring results are obtained based on the fused traffic condition data of each of the cooperative drones.

[0062] Specifically, in S101, each drone inspects a sub-region within the inspection area based on a preset inspection route. There may be overlapping areas between sub-regions, and each sub-region includes at least one drone; however, this embodiment does not limit this.

[0063] Understandably, based on historical data, areas with high traffic volume and accident-prone areas can be identified. It is advisable to increase the deployment of drones in these areas. Multiple drones can increase the inspection frequency in areas with high traffic volume or accident-prone areas, and can be prioritized for monitoring high-risk or critical road sections through a dynamic task allocation mechanism.

[0064] Understandably, for areas with high traffic volume, accident-prone areas, and locations with numerous obstacles that create blind spots for drone photography, smaller sub-areas can be set up, and / or the number of drones in the corresponding sub-areas can be increased.

[0065] The inspection area can be an urban area where monitoring tasks need to be performed, and one or more blocks can be designated for monitoring. This application embodiment does not limit this.

[0066] Specifically, in S102, each drone is equipped with a pre-trained deep learning model. The deep learning model can extract features from the information collected by the drone to quickly identify whether there are abnormal traffic conditions. For example, the drone can collect image data in the corresponding sub-area through its image acquisition device, input the image data into the pre-trained deep learning model, and extract features from the image data to identify whether there are abnormal traffic conditions. Abnormal traffic conditions can be understood as a situation where the traffic conditions on the road are significantly different from those during off-peak hours. For example, whether there is a situation where the traffic flow in a lane is greater than the traffic flow during off-peak hours, and / or, detecting accident features such as rear-end collisions, rollovers, and impacts.

[0067] Furthermore, if the aforementioned abnormal traffic conditions are detected, point cloud data in the corresponding sub-region can be collected by the lidar carried by the UAV. Based on the point cloud data, the location range where the traffic condition is abnormal can be determined, thus identifying the abnormal traffic condition area.

[0068] In some embodiments, a multimodal feature fusion module can combine 2D imagery with 3D point cloud data, and a neural network model can be used to spatially align and semantically fuse the information captured by the camera and LiDAR. To ensure the consistency of multimodal data, the system can project the 3D point cloud data and 2D imagery onto a unified bird's-eye view using a geometric transformation method before fusion. Specifically, by using coordinate mapping and feature point matching, data from different sensors are spatially overlapped, thereby generating a unified view, reducing dimensional differences between data, and improving the accuracy of subsequent feature extraction and information fusion.

[0069] In some embodiments, the specific method for generating a bird's-eye view includes: firstly, extracting features from a 2D image and a 3D point cloud, and then projecting the 3D information onto the bird's-eye view. Using camera intrinsic and extrinsic parameter matrices, as well as perspective or orthophoto transformation, the coordinate system of the LiDAR's 3D point cloud is ensured to be consistent with that of the 2D image. The resulting bird's-eye view can simultaneously possess both the global spatial layout and local details of the road environment, which is helpful for accident detection and vehicle monitoring tasks.

[0070] It should be noted that the abnormal traffic conditions identified by a single drone are only preliminary identifications. The monitoring perspective and data collection of a single drone are limited, making it difficult to ensure a comprehensive understanding of the accident scene.

[0071] Therefore, it is necessary to introduce more collaborative drones to monitor areas with abnormal traffic conditions from multiple angles.

[0072] In some embodiments, if an abnormal traffic condition is detected in S102, step S103 is executed, which involves selecting a cooperative drone and assigning it to the area with the abnormal traffic condition. Specifically, this includes:

[0073] Determine the distance from the real-time location coordinates of each drone to the area of ​​abnormal traffic conditions;

[0074] Based on the distance, estimate the remaining battery power of each drone after it reaches the traffic anomaly area. If the remaining battery power is greater than a first battery power threshold, then the corresponding drone is selected as a cooperative drone.

[0075] Update the inspection routes of the collaborative drones to the areas with abnormal traffic conditions, and assign each collaborative drone to the areas with abnormal traffic conditions.

[0076] Specifically, the distance and real-time battery level can be combined for weighted calculation. The distance is assigned a weight from high to low in ascending order, and the real-time battery level is assigned a weight from high to low in descending order. This determines the order of weights from high to low, and a list of drones that can be selected as cooperative drones is determined. From this list, a preset number of drones are selected as cooperative drones.

[0077] Understandably, the first power threshold is the minimum power required for the drone to perform inspection tasks after flying into an area with abnormal traffic conditions.

[0078] Specifically, after some drones are selected as cooperative drones, the original inspection sub-area may lack inspection drones. When it is determined that there is an abnormal traffic condition, a backup drone can be selected to perform the inspection task of the corresponding sub-area. Alternatively, a backup drone can be directly selected as a cooperative drone. This application embodiment does not limit this.

[0079] In some embodiments, considering the power consumption of the drone, it is necessary to allocate the drone's inspection tasks based on the drone's real-time power level, specifically including:

[0080] If the remaining battery power of any drone is less than or equal to the second battery threshold, update the return route of the low-battery drone to the drone nest, and generate a return command to the low-battery drone so that the low-battery drone executes the return command and returns to the drone nest according to the return route;

[0081] When generating a return command to the low-power drone, the real-time location and inspected sub-area of ​​the low-power drone are obtained, a flight path from the backup drone to the real-time location of the low-power drone is generated, and a control command is generated to the backup drone so that the backup drone executes the control command and flies to the real-time location of the low-power drone according to the flight path, and continues to inspect the sub-area originally inspected by the low-power drone.

[0082] Understandably, the second power threshold is the minimum power threshold at which the drone can return to the drone nest. The second power threshold can be dynamically updated in real time. The value of the second power threshold is estimated based on the real-time location of the drone and the path to the nearest drone nest. This application does not limit this.

[0083] Furthermore, in multi-drone collaborative monitoring, drones need to communicate frequently to share data, resulting in huge bandwidth consumption. Traditional collaborative methods usually require each drone to transmit all perception data. While this preserves complete scene information, it also leads to low bandwidth utilization, excessive communication load, and multiple drones may collect the same or similar traffic information from different perspectives. Therefore, it is necessary to effectively integrate the perception results of multiple drones to avoid data redundancy and communication congestion.

[0084] Therefore, in S104, it is necessary to integrate traffic status data collected by multiple collaborative drones, specifically including:

[0085] Each collaborative drone generates a first feature distribution matrix based on the collected feature map, and generates a second feature distribution map that is negatively correlated with the first feature distribution matrix;

[0086] Each of the cooperative drones, after fusing traffic state data from other cooperative drones, outputs fused traffic state data, including:

[0087] The first collaborative drone receives the second feature distribution map from the second collaborative drone and constructs a feature selection matrix based on the first feature distribution matrix of the first collaborative drone and the second feature distribution map of the second collaborative drone.

[0088] Based on the feature selection matrix, the selection weight of each feature in the first feature distribution matrix of the first cooperative UAV is determined, the corresponding feature is selected based on the value of the selection weight, a sparse priority feature map is generated, and the sparse priority feature map is sent to the second cooperative UAV.

[0089] The feature map of the second cooperative UAV is fused with the sparse priority feature map, and the fused feature map is output as the fused traffic state data.

[0090] In this matrix, each element represents the confidence level of the traffic state data at each location.

[0091] Understandably, all cooperative drones can be either first cooperative drones or second cooperative drones. Here, the first and second cooperative drones are only used to distinguish between drones that send the second feature distribution map and drones that receive the second feature distribution map.

[0092] Understandably, the first collaborative UAV constructs a first feature distribution matrix based on the collected feature map, i.e., traffic state data. This first feature distribution matrix characterizes the confidence level of features at each location in the feature map collected by the first collaborative UAV. It represents the distribution of confidence levels for features at each location and can be used to characterize the perception importance of different spatial regions, identifying key areas for the perception task. For example, areas containing target objects have greater perception value than background areas. The first feature distribution matrix guides the system to share only the most important perception information under bandwidth constraints, saving communication costs and improving perception efficiency. In traditional collaborative systems, information from all spatial regions is often processed and transmitted in equal amounts.

[0093] Understandably, the second feature distribution matrix is ​​negatively correlated with the first feature distribution matrix, indicating the spatial areas where the UAV needs to obtain more information from other collaborating UAVs. It reflects areas with weaker perception, typically where information may be missing due to factors such as obstruction and distance. The second feature distribution matrix is ​​generated based on the first feature distribution matrix and is used to indicate the uncertainty of the UAV's perception data in certain areas, thereby requesting supplementary data from other collaborating UAVs.

[0094] Therefore, the first cooperative drone can receive the second feature distribution matrix of the second cooperative drone, thereby determining the data missing by the second cooperative drone, and can further determine whether to transmit data to the second cooperative drone in response to the second feature distribution matrix of the second cooperative drone.

[0095] Furthermore, the first cooperative UAV can combine its first feature distribution matrix and the second feature distribution matrix received from the second cooperative UAV to select the spatial region of perception criticality from the feature map, thus constructing a sparse-priority feature map. Specifically, a feature selection matrix is ​​constructed using the first feature distribution matrix of the first cooperative UAV and the second feature distribution map of the second cooperative UAV, and then the value of the feature selection matrix is ​​used to determine whether each feature needs to be transmitted to the second cooperative UAV.

[0096] In some embodiments, the weights in the feature selection matrix can be binarized to 0 or 1. If the weight is 1, it indicates that the corresponding feature needs to be transmitted; otherwise, it does not need to be transmitted.

[0097] In some embodiments, the feature selection matrix can be used to characterize the information overlap between the first feature distribution matrix of the first cooperative UAV and the second feature distribution map of the second cooperative UAV. A higher value in the feature selection matrix indicates a higher information overlap between the first and second feature distribution matrices of the first cooperative UAV, meaning the first cooperative UAV possesses more information required by the second cooperative UAV. In this case, it is determined that data can be transmitted to the second cooperative UAV. Conversely, if the feature selection matrix value is low, a selection threshold can be set. If the feature selection matrix value is below the selection threshold, it indicates that the first feature distribution map of the first cooperative UAV contains less data required by the second feature distribution map. In this case, data transmission to the second cooperative UAV is unnecessary, as the communication overhead outweighs the benefits of communication, thus avoiding unnecessary bandwidth consumption.

[0098] Based on this processing, in some embodiments, if at least one feature needs to be transmitted between every two cooperative drones according to the feature selection matrix, then only the second feature distribution matrix and the sparse priority feature map need to be transmitted to the other cooperative drone. Transmitting only this sparse information can greatly reduce the amount of information and reduce bandwidth usage, thereby ensuring the efficient sharing of key perception information.

[0099] Furthermore, after each collaborating drone receives sparse-priority feature maps from other drones, the received information needs to be fused, specifically including:

[0100] Each drone integrates its own feature map with sparse-priority features from other drones to further improve perception accuracy, specifically through a multi-head attention mechanism. This fusion process, by combining the first feature distribution matrix of all drones, accurately identifies which regions deserve more attention. The multi-head attention mechanism assigns greater weight to these high-priority regions during fusion, ensuring that the most critical perception information is effectively transmitted and shared among different drones. This results in the output of fused traffic state data.

[0101] Furthermore, in S105, obtaining real-time traffic condition monitoring results based on the fused traffic condition data of each of the cooperative drones includes:

[0102] Modeling is performed based on the fused traffic state data of each of the collaborative drones to determine the vehicle distribution and average vehicle speed on each road, and output real-time traffic state monitoring results.

[0103] Specifically, the fused traffic state data of each UAV can be fed back to the constructed traffic state model, thereby reflecting the traffic flow and vehicle speed on each lane, determining the specific location of traffic accidents and the starting and ending points of congestion caused by traffic accidents, and providing early warning information based on lane direction when traffic flow is high in some lanes. This application embodiment does not limit this.

[0104] In some embodiments, after S105, the method further includes:

[0105] Based on the real-time traffic condition monitoring results, congested road sections and traffic flow on congested road sections are determined, and traffic anomaly alerts are sent to other drones within a preset range of the traffic anomaly area through the cooperative drone.

[0106] After receiving the traffic anomaly alert, the other drones determine the traffic status data of the connecting road segments of the congested road segment;

[0107] A driving route that bypasses the congested road section is generated based on the traffic status data of the connected road sections.

[0108] Specifically, the system where the drone is located can be linked with the systems of relevant departments and / or third-party map providers. The information collected by the drone in real time can be input into the systems of relevant departments and / or third-party map providers to determine the traffic flow of congested road sections and surrounding road sections. The destinations of various users around the congested road sections can be captured by combining the system of third-party map providers to calculate detour routes. This application embodiment does not limit this.

[0109] The following are apparatus embodiments of this application, which can be used to execute the method embodiments of this application. For details not disclosed in the apparatus embodiments of this application, please refer to the method embodiments of this application.

[0110] Please see below. Figure 2 This is a schematic diagram of a multi-UAV collaborative traffic condition monitoring device provided as an exemplary embodiment of this application. This device can be implemented as all or part of a terminal through software, hardware, or a combination of both, or it can be integrated as an independent module on a server. The multi-UAV collaborative traffic condition monitoring device in this embodiment can be applied to a terminal or the cloud. The device 20 includes a UAV scheduling module 201, an information identification module 202, an information fusion module 203, and an information processing module 204, wherein:

[0111] The drone scheduling module 201 is used to conduct inspections within a preset inspection area using multiple drones; wherein each drone inspects a sub-area within the inspection area based on a preset inspection route;

[0112] The information recognition module 202 is used to identify whether there are any abnormalities in the traffic status within the corresponding sub-region through the trained deep learning model carried by each UAV;

[0113] When the information identification module 202 identifies an abnormal traffic condition, the drone scheduling module 201 is further configured to select a cooperating drone based on the real-time status of each drone and assign each cooperating drone to the area with abnormal traffic conditions, so as to collect traffic condition data collaboratively through multiple cooperating drones.

[0114] The information fusion module 203 is used to enable each of the cooperative drones to merge the traffic status data of other cooperative drones and output the merged traffic status data.

[0115] The information processing module 204 is used to obtain real-time traffic condition monitoring results based on the fused traffic condition data of each of the cooperative drones.

[0116] It should be noted that the above-described embodiment of the device 20, when executing the multi-UAV collaborative traffic condition monitoring method, is only illustrated by the division of the above functional modules. In practical applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above. Furthermore, the device provided in the above embodiment and the multi-UAV collaborative traffic condition monitoring method embodiment belong to the same concept, and their implementation process is detailed in the method embodiment, which will not be repeated here.

[0117] This application also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the steps of any of the methods described above.

[0118] Please see Figure 3 This is a structural block diagram of an electronic device provided in an embodiment of this application.

[0119] like Figure 3 As shown, the electronic device 300 includes a processor 301 and a memory 302.

[0120] In this embodiment, the processor 301 is the control center of the computer system, and can be a processor of a physical machine or a processor of a virtual machine. The processor 301 may include one or more processing cores, such as a 4-core processor or an 8-core processor. The processor 301 can be implemented using at least one hardware form selected from DSP (Digital Signal Processing), FPGA (Field-Programmable Gate Array), and PLA (Programmable Logic Array).

[0121] Processor 301 may also include a main processor and a coprocessor. The main processor is a processor used to process data in the wake-up state, also known as a CPU (Central Processing Unit). The coprocessor is a low-power processor used to process data in the standby state.

[0122] Memory 302 may include one or more computer-readable storage media, which may be non-transitory. Memory 302 may also include high-speed random access memory and non-volatile memory, such as one or more disk storage devices or flash memory devices. In some embodiments of this application, the non-transitory computer-readable storage media in memory 302 is used to store at least one instruction, which is executed by processor 301 to implement the method in the embodiments of this application.

[0123] In some embodiments, the electronic device 300 further includes a peripheral device interface 303 and at least one peripheral device 304. The processor 301, memory 302, and peripheral device interface 303 can be connected via a bus or signal line. Each peripheral device 304 can be connected to the peripheral device interface 303 via a bus, signal line, or circuit board. Specifically, the peripheral device 304 includes: a display screen, a camera, and audio circuitry. The peripheral device interface 303 can be used to connect at least one I / O (Input / Output) related peripheral device to the processor 301 and memory 302.

[0124] In some embodiments of this application, the processor 301, memory 302, and peripheral device interface 303 are integrated on the same chip or circuit board; in other embodiments of this application, any one or two of the processor 301, memory 302, and peripheral device interface 303 can be implemented on separate chips or circuit boards. This application does not specifically limit the implementation in this regard.

[0125] The electronic device structural block diagram shown in the embodiments of this application does not constitute a limitation on the electronic device 300. The electronic device 300 may include more or fewer components than shown, or combine certain components, or use different component arrangements.

[0126] This application also provides a computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the steps of the methods in any of the foregoing embodiments. The computer-readable storage medium may include, but is not limited to, any type of disk, including floppy disks, optical disks, DVDs, CD-ROMs, microdrives, as well as magneto-optical disks, ROMs, RAMs, EPROMs, EEPROMs, DRAMs, VRAMs, flash memory devices, magnetic cards or optical cards, nanosystems (including molecular memory ICs), or any type of medium or device suitable for storing instructions and / or data.

[0127] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the parts that contribute to the related technology, can be embodied in the form of software products. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0128] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application.

Claims

1. A method for traffic condition monitoring using multi-UAV collaborative methods, characterized in that, include: Multiple drones are used to conduct inspections within a pre-defined inspection area; each drone inspects a sub-area within the pre-defined inspection area based on a pre-defined inspection route. The trained deep learning model on each drone identifies whether there are any anomalies in the traffic conditions within the corresponding sub-region. When an abnormal traffic condition is detected, a cooperating drone is selected based on the real-time status of each drone, and each cooperating drone is assigned to the area with abnormal traffic conditions, so as to collect traffic condition data collaboratively through multiple cooperating drones. Each of the cooperative drones merges traffic state data from other cooperative drones and outputs merged traffic state data. Real-time traffic condition monitoring results are obtained based on the fused traffic condition data of each of the cooperative drones; Each collaborative drone generates a first feature distribution matrix based on the collected feature map, and generates a second feature distribution map that is negatively correlated with the first feature distribution matrix; Each of the cooperative drones, after fusing traffic state data from other cooperative drones, outputs fused traffic state data, specifically including: The first collaborative drone receives the second feature distribution map from the second collaborative drone and constructs a feature selection matrix based on the first feature distribution matrix of the first collaborative drone and the second feature distribution map of the second collaborative drone. Based on the feature selection matrix, the selection weight of each feature in the first feature distribution matrix of the first cooperative UAV is determined, the corresponding feature is selected based on the value of the selection weight, a sparse priority feature map is generated, and the sparse priority feature map is sent to the second cooperative UAV. The feature map of the second cooperative UAV is fused with the sparse priority feature map, and the fused feature map is output as the fused traffic state data. In this matrix, each element represents the confidence level of the traffic state data at each location.

2. The traffic condition monitoring method using multi-UAV collaborative operation according to claim 1, characterized in that, Each drone uses a pre-trained deep learning model to identify whether there are any anomalies in the traffic conditions within its corresponding sub-region, including: Image data of the corresponding sub-region is collected using an image acquisition device mounted on a drone; The image data is input into the trained deep learning model, and the trained deep learning model extracts features from the image data to identify whether there are abnormal traffic conditions. When an anomaly in traffic conditions is detected, point cloud data is collected in the corresponding sub-region using a lidar mounted on a drone. Based on the point cloud data, the location range of the traffic anomaly is determined, thus identifying the area of ​​traffic anomaly.

3. The traffic condition monitoring method using multi-UAV collaborative operation according to claim 2, characterized in that, The step of selecting cooperative drones based on the real-time status of each drone and assigning each cooperative drone to an area with abnormal traffic conditions includes: Determine the distance from the real-time location coordinates of each drone to the area of ​​abnormal traffic conditions; Based on the distance, estimate the remaining battery power of each drone after it reaches the traffic anomaly area. If the remaining battery power is greater than a first battery power threshold, then the corresponding drone is selected as a cooperative drone. Update the inspection routes of the collaborative drones to the areas with abnormal traffic conditions, and assign each collaborative drone to the areas with abnormal traffic conditions.

4. The traffic condition monitoring method using multi-UAV collaborative operation according to claim 1, characterized in that, The method further includes: If the remaining battery power of any drone is less than or equal to the second battery threshold, update the return route of the low-battery drone to the drone nest, and generate a return command to the low-battery drone so that the low-battery drone executes the return command and returns to the drone nest according to the return route; When generating a return command to the low-power drone, the real-time location and inspected sub-area of ​​the low-power drone are obtained, a flight path from the backup drone to the real-time location of the low-power drone is generated, and a control command is generated to the backup drone so that the backup drone executes the control command and flies to the real-time location of the low-power drone according to the flight path, and continues to inspect the sub-area originally inspected by the low-power drone.

5. The traffic condition monitoring method using multi-UAV collaborative operation according to claim 1, characterized in that, The real-time traffic condition monitoring results obtained based on the fused traffic condition data of each of the cooperative drones include: Modeling is performed based on the fused traffic state data of each of the collaborative drones to determine the vehicle distribution and average vehicle speed on each road, and output real-time traffic state monitoring results.

6. The traffic condition monitoring method using multiple unmanned aerial vehicles (UAVs) in accordance with claim 5, characterized in that, After obtaining the real-time traffic condition monitoring results based on the fused traffic condition data of each of the cooperative drones, the method further includes: Based on the real-time traffic condition monitoring results, congested road sections and traffic flow on congested road sections are determined, and traffic anomaly alerts are sent to other drones within a preset range of the traffic anomaly area through the cooperative drone. After receiving the traffic anomaly alert, the other drones determine the traffic status data of the connecting road segments of the congested road segment; A driving route that bypasses the congested road section is generated based on the traffic status data of the connected road sections.

7. An apparatus for traffic condition monitoring based on the multi-UAV collaborative method according to any one of claims 1-6, characterized in that, The device includes: The drone scheduling module is used to conduct inspections within a preset inspection area using multiple drones; wherein each drone inspects a sub-area within the inspection area based on a preset inspection route; The information recognition module is used to identify whether there are any abnormalities in the traffic conditions within the corresponding sub-region using the trained deep learning model carried by each drone. When the information identification module detects an abnormal traffic condition, the drone scheduling module is also used to select a cooperating drone based on the real-time status of each drone and assign each cooperating drone to the area with abnormal traffic conditions, so as to collect traffic condition data collaboratively through multiple cooperating drones. The information fusion module is used to enable each of the cooperative drones to merge traffic status data from other cooperative drones and then output the merged traffic status data. The information processing module is used to obtain real-time traffic condition monitoring results based on the fused traffic condition data of each of the cooperative drones.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the steps of the method as described in any one of claims 1 to 6.

9. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method as described in any one of claims 1 to 6.

Images