Multi-unmanned aerial vehicle cooperative traffic state monitoring method, device, equipment and medium

Abstract

The embodiment of the invention discloses a multi-unmanned aerial vehicle cooperative traffic state monitoring method, device and equipment and a medium, and relates to the technical field of multi-unmanned aerial vehicle cooperative control, and the method comprises the steps: carrying out the routing inspection in a preset routing inspection region through a plurality of unmanned aerial vehicles; identifying whether the traffic state is abnormal or not through the trained deep learning model; when it is recognized that the traffic state is abnormal, cooperative unmanned aerial vehicles are selected, the cooperative unmanned aerial vehicles are distributed to the traffic state abnormal areas, and traffic state data are cooperatively collected through the cooperative unmanned aerial vehicles; each cooperative unmanned aerial vehicle fuses the traffic state data of other cooperative unmanned aerial vehicles and then outputs fused traffic state data; and obtaining a real-time traffic state monitoring result based on the fused traffic state data of the cooperative unmanned aerial vehicle. According to the method, the same scene is perceived from different angles through the multiple unmanned aerial vehicles, data of multiple view angles can be fused, the accuracy of accident detection is improved, and data support can be provided for related departments to deal with traffic anomalous events.

Description

Technical Field

[0001] The present application relates to the technical field of multi-UAV cooperative control, and in particular to a multi-UAV cooperative traffic status monitoring method, device, equipment and medium. Background Art

[0002] With the economic development and social progress of our country, the number of vehicles has increased rapidly, and traffic congestion and violations have gradually become hot issues in social development. At present, vehicles on the road are usually monitored by monitoring equipment installed on the road. Based on the monitoring information, the traffic volume can be counted, the speed can be measured, and the traffic conditions can be identified. However, the monitoring equipment installed on the road has the following disadvantages:

[0003] The field of vision is limited, the monitoring area can only cover individual nodes, and there are monitoring blind spots, which makes it impossible to obtain macro traffic data of a certain traffic node in a long-distance, multi-section accident scene. The response is delayed and needs to rely on manual analysis. It is difficult to respond based on limited information in a short period of time, and the accident handling efficiency is low; the layout and construction costs are high, requiring a lot of initial construction and subsequent maintenance investment; data transmission and communication bottlenecks, the amount of data generated by the monitoring system is large, and the bandwidth is limited in remote roads and other environments, which cannot meet the needs of real-time transmission and response, and it is difficult to achieve real-time monitoring and processing of traffic conditions.

[0004] In order to address the defects of traditional traffic status monitoring methods, relevant technologies can use drones to monitor traffic status. Compared with traditional monitoring methods, drones have the advantages of flexibility, wide field of view, and quick deployment. However, the use of drones in relevant technologies has not fundamentally solved the above technical problems.

[0005] Although the related technology uses drones to replace traditional monitoring equipment, it still requires staff to determine the traffic status and whether a traffic accident has occurred. For traffic accidents with a wide impact, it is difficult to provide an evacuation plan in a short period of time to alleviate the large-scale congestion caused by traffic accidents. Therefore, the use of drones by related technologies has not changed the occupation of manpower and material resources in traffic monitoring. In addition, the coverage and viewing angle of a single drone are limited. Multiple drones can collect data comprehensively from multiple angles, but the coordination of multiple drones is difficult. Traditional coordination methods usually require each drone to transmit all perception data. Although this retains complete scene information, it also leads to low bandwidth utilization and excessive communication load.

[0006] Therefore, there is currently a lack of a method that can use drones to achieve real-time and efficient traffic status monitoring. Summary of the invention

[0007] The embodiments of the present application provide a multi-UAV coordinated traffic status monitoring method, device, equipment and medium to solve the defects of the above-mentioned related technologies. The technical solution is as follows:

[0008] In a first aspect, an embodiment of the present application provides a multi-UAV coordinated traffic status monitoring method, comprising:

[0009] Conducting inspections in a preset inspection area by using a plurality of drones; wherein each drone inspects a sub-area within the inspection area based on a preset inspection route;

[0010] Through the trained deep learning model carried by each drone, it is possible to identify whether there are any abnormalities in the traffic status in the corresponding sub-area.

[0011] When an abnormal traffic state is identified, a cooperative drone is selected based on the real-time state of each drone, and each cooperative drone is assigned to an abnormal traffic state area, so that traffic state data can be collected collaboratively by multiple cooperative drones;

[0012] Each of the cooperative drones outputs fused traffic status data after fusing the traffic status data of other cooperative drones;

[0013] A real-time traffic status monitoring result is obtained based on the fused traffic status data of each of the cooperative UAVs.

[0014] In an optional solution of the first aspect, each of the drones identifies whether the traffic status in the corresponding sub-area is abnormal through a trained deep learning model, including:

[0015] The image data in the corresponding sub-area is collected by an image acquisition device carried by the drone;

[0016] Inputting the image data into the trained deep learning model, extracting features from the image data through the trained deep learning model, and identifying whether there are abnormal traffic status features;

[0017] When an abnormality in the traffic status is identified, point cloud data in the corresponding sub-area is collected by the laser radar carried by the drone, and the location range of the abnormal traffic status is determined based on the point cloud data, thereby determining the abnormal traffic status area.

[0018] In an optional solution of the first aspect, the selecting a cooperative drone based on the real-time status of each drone and assigning each cooperative drone to an area with abnormal traffic status includes:

[0019] Determine the distance from the real-time position coordinates of each drone to the abnormal traffic condition area;

[0020] Based on the distance, the remaining power of each drone after reaching the abnormal traffic state area is estimated, and if the remaining power is greater than a first power threshold, the corresponding drone is selected as a cooperative drone;

[0021] The inspection route of the cooperative drone to the abnormal traffic condition area is updated, and each cooperative drone is assigned to the abnormal traffic condition area.

[0022] In an optional solution of the first aspect, the method further includes:

[0023] If the remaining power of any drone is less than or equal to a second power threshold, update the return route of the low-power drone with a remaining power less than or equal to the second power threshold to the drone nest, and generate a return instruction to the low-power drone, so that the low-power drone executes the return instruction and returns to the drone nest along the return route;

[0024] When generating a return instruction to the low-battery drone, the real-time location and the inspected sub-area of ​​the low-battery drone are obtained, a flight route from the standby drone to the real-time location of the low-battery drone is generated, and a control instruction is generated to the standby drone, so that the standby drone executes the control instruction and flies to the real-time location of the low-battery drone according to the flight route, and continues to inspect the sub-area originally inspected by the low-battery drone.

[0025] In an optional solution of the first aspect, each cooperative UAV generates a first feature distribution matrix based on the collected feature map, and generates a second feature distribution map that is negatively correlated with the first feature distribution matrix;

[0026] After each of the cooperative drones fuses the traffic status data of other cooperative drones, the output of the fused traffic status data includes:

[0027] The first cooperative UAV receives the second feature distribution map of the second cooperative UAV, and constructs a feature selection matrix based on the first feature distribution matrix of the first cooperative UAV and the second feature distribution map of the second cooperative UAV;

[0028] Determine a selection weight of each feature in a first feature distribution matrix of the first cooperative UAV based on the feature selection matrix, select a corresponding feature based on the value of the selection weight, generate a sparse priority feature map, and send the sparse priority feature map to the second cooperative UAV;

[0029] fusing the feature map of the second cooperative UAV with the sparse priority feature map, and outputting the fused feature map as the fused traffic state data;

[0030] Wherein, each element in the first feature distribution matrix represents the confidence of the traffic status data at each position.

[0031] In an optional solution of the first aspect, obtaining a real-time traffic status monitoring result based on the fused traffic status data of each of the cooperative drones includes:

[0032] Modeling is performed based on the fused traffic status data of each of the cooperative drones to determine the vehicle distribution on each road and the average speed of the vehicles, and output real-time traffic status monitoring results.

[0033] In an optional solution of the first aspect, after obtaining the real-time traffic status monitoring result based on the fused traffic status data of each of the cooperative drones, the method further includes:

[0034] Determine the congested road section and the traffic volume of the congested road section based on the real-time traffic status monitoring result, and send a traffic abnormality prompt to other drones within a preset range of the abnormal traffic status area through the cooperative drone;

[0035] After receiving the traffic anomaly prompt, the other drones determine the traffic status data of the connected road sections of the congested road section;

[0036] A driving route that avoids the congested road section is generated based on the traffic status data of the connected road sections.

[0037] In a second aspect, the embodiment of the present application further provides a multi-UAV coordinated traffic status monitoring device, comprising:

[0038] A drone dispatching module is used to perform inspections in a preset inspection area by using multiple drones; wherein each drone inspects a sub-area in the inspection area based on a preset inspection route;

[0039] The information recognition module is used to identify whether the traffic status in the corresponding sub-area is abnormal through the trained deep learning model carried by each drone;

[0040] When the information recognition module recognizes that the traffic status is abnormal, the drone scheduling module is further used to select a cooperative drone based on the real-time status of each drone, and assign each cooperative drone to an abnormal traffic status area, so as to collaboratively collect traffic status data through multiple cooperative drones;

[0041] An information fusion module, configured to enable each of the cooperative UAVs to fuse the traffic status data of other cooperative UAVs and output the fused traffic status data;

[0042] The information processing module is used to obtain real-time traffic status monitoring results based on the fused traffic status data of each of the cooperative drones.

[0043] In a third aspect, an embodiment of the present application further provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the method provided in the first aspect of the embodiment of the present application or any one of the implementation methods of the first aspect is implemented.

[0044] In a fourth aspect, the present application also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the method provided by the first aspect of the embodiment of the present application or any one of the implementations of the first aspect.

[0045] The beneficial effects brought about by the technical solutions provided by some embodiments of the present application include at least:

[0046] The embodiment of the present application provides a method, device, equipment and medium for monitoring traffic status by coordinating multiple drones. The embodiment of the present application first collects information in the corresponding sub-area through a single drone, thereby realizing large-scale monitoring and being able to quickly determine the approximate area where the traffic status is abnormal, and then more cooperative drones can be allocated to the abnormal traffic status area, which is conducive to saving the computing resources of the drone. The drone can move freely and cover a wide area, breaking through the limitations of fixed monitoring equipment. Further, in the abnormal traffic status area, a method of cooperative perception of multiple drones is adopted. Through multiple drones perceiving the same scene from different angles, the system can integrate data from multiple perspectives and improve the accuracy of accident detection. After detecting an accident, the embodiment of the present application can generate traffic status monitoring results in real time, and the traffic status monitoring results can be transmitted to the server through the communication network, which can provide users with efficient traffic abnormality warning information, and can also provide real-time and efficient data support for relevant departments to handle traffic abnormal events, which is conducive to shortening the response time of relevant departments to accidents. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] In order to more clearly illustrate the technical solutions in the present application or related technologies, the drawings required for use in the embodiments or related technical descriptions are briefly introduced below. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0048] Figure 1 It is a flow chart of a multi-UAV coordinated traffic status monitoring method provided in an embodiment of the present application;

[0049] Figure 2It is a structural schematic diagram of a multi-UAV coordinated traffic status monitoring device provided in an embodiment of the present application;

[0050] Figure 3 It is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0051] In order to make the purpose, technical solutions and advantages of this application clearer, the technical solutions in this application will be clearly and completely described below in conjunction with the drawings in this application. Obviously, the described embodiments are part of the embodiments of this application, not all of them. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0052] The terms "including" and "having" and any variations thereof in the specification and claims of the present application and the above drawings are intended to cover non-exclusive inclusions. For example, a process, method, system, product or device including a series of steps or modules is not limited to the listed steps or modules, but may optionally include steps or modules that are not listed, or may optionally include other steps or modules that are inherent to these processes, methods, products or devices.

[0053] It should be noted that the terms "first\second" involved in the present application are only used to distinguish similar objects, and do not represent a specific order for the objects. It is understandable that "first\second" can be interchanged with a specific order or sequence where permitted. It should be understood that the objects distinguished by "first\second" can be interchanged where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those described or illustrated herein.

[0054] The present application is described in detail below with reference to specific embodiments.

[0055] Next, combine Figure 1 , introduces a multi-UAV collaborative traffic status monitoring method provided by the embodiment of the present application. For details, please refer to Figure 1 , Figure 1 FIG. 1 is a flow chart of a method for monitoring traffic status using multiple drones in collaboration provided by an embodiment of the present application. Figure 1 As shown, the method comprises the following steps:

[0056] S101, using multiple drones to conduct inspections in a preset inspection area;

[0057] S102, identifying whether the traffic status in the corresponding sub-area is abnormal through the trained deep learning model carried by each drone;

[0058] When it is identified that the traffic status is abnormal, S103 is executed:

[0059] S103, selecting a cooperative drone based on the real-time status of each drone, and assigning each cooperative drone to an area with abnormal traffic status, so as to collaboratively collect traffic status data through multiple cooperative drones;

[0060] S104, each of the cooperative drones fuses the traffic status data of other cooperative drones and outputs the fused traffic status data;

[0061] S105, obtaining a real-time traffic status monitoring result based on the fused traffic status data of each of the cooperative drones.

[0062] Specifically, in S101, each drone inspects a sub-area within the inspection area based on a preset inspection route. There may be overlapping areas between each sub-area, and each sub-area includes at least one drone, which is not limited in the embodiment of the present application.

[0063] Understandably, based on historical data, areas with heavy traffic and accident-prone areas can be identified, and consideration can be given to increasing the number of drone deployments in areas with heavy traffic and accident-prone areas. Multiple drones can increase the frequency of inspections in areas with heavy traffic or accident-prone areas, and be prioritized for monitoring on high-risk or critical sections through a dynamic task allocation mechanism.

[0064] It is understandable that for areas with heavy traffic and accident-prone areas, as well as locations with many obstacles that cause blind spots in drone photography, smaller sub-areas can be set, and / or the number of drones in the corresponding sub-areas can be increased.

[0065] Among them, the inspection area can be an urban area where monitoring tasks need to be performed, and one or more blocks can be designated for monitoring, which is not limited in the embodiments of the present application.

[0066] Specifically, in S102, each drone is respectively equipped with a trained deep learning model. The deep learning model can be used to extract features from the information collected by the drone to quickly identify whether there is an abnormal traffic state. For example, image data in the corresponding sub-area can be collected by an image acquisition device carried by the drone, and the image data can be input into the trained deep learning model. The features in the image data are extracted by the trained deep learning model to identify whether there are abnormal traffic state features. The abnormal traffic state features can be understood as a state where the difference between the traffic state on the road and the traffic state during off-peak hours is large, for example, whether there is a situation where the traffic volume in the lane is greater than the traffic volume during off-peak hours, and / or, accident features such as vehicle rear-end collisions, rollovers, and collisions are monitored.

[0067] Furthermore, if the above-mentioned abnormal traffic status characteristics are detected, the laser radar carried by the drone can be further combined to collect point cloud data in the corresponding sub-area, and the location range where the traffic status is abnormal can be determined based on the point cloud data to determine the abnormal traffic status area.

[0068] In some embodiments, the two-dimensional image can also be combined with the three-dimensional point cloud data through a multimodal feature fusion module, and the information captured by the camera and the lidar can be spatially aligned and semantically fused through a neural network model. To ensure the consistency of multimodal data, the system can project the three-dimensional point cloud data and the two-dimensional image to a unified bird's-eye view through a geometric transformation method before fusion. Specifically, through coordinate mapping and feature point matching, the data of different sensors are spatially overlapped, so that a unified view can be generated, which reduces the dimensional differences between the data and improves the accuracy of subsequent feature extraction and information fusion.

[0069] In some embodiments, the specific method of generating a bird's-eye view includes: first extracting features of a two-dimensional image and a three-dimensional point cloud, and projecting the three-dimensional information into a bird's-eye view. The camera's intrinsic and extrinsic matrices, as well as perspective or orthographic projection transformations, are used to ensure that the three-dimensional point cloud of the lidar maintains a consistent coordinate system with the two-dimensional image. The bird's-eye view formed by fusion can simultaneously have the global spatial layout and local details of the road environment, which is helpful for accident detection and vehicle monitoring tasks.

[0070] It should be noted that the abnormal traffic status area determined by a single drone is only a preliminary identified area. The monitoring perspective and data collection of a single drone are limited, and it is difficult to ensure a comprehensive perception of the accident scene.

[0071] Therefore, it is necessary to introduce more collaborative drones to monitor areas with abnormal traffic conditions from multiple angles.

[0072] In some embodiments, when S102 identifies that the traffic status is abnormal, step S103 is executed to select a cooperative drone and assign it to an area with abnormal traffic status, specifically including:

[0073] Determine the distance from the real-time position coordinates of each drone to the abnormal traffic condition area;

[0074] Based on the distance, the remaining power of each drone after reaching the abnormal traffic state area is estimated, and if the remaining power is greater than a first power threshold, the corresponding drone is selected as a cooperative drone;

[0075] The inspection route of the cooperative drone to the abnormal traffic condition area is updated, and each cooperative drone is assigned to the abnormal traffic condition area.

[0076] Specifically, the distance and the real-time power can be combined for weighted calculation, and weights can be assigned from high to low in order of distance from small to large, and weights can be assigned from high to low in order of real-time power from high to low, and then the order of weights can be determined from high to low, and a list of drones that can be selected as cooperative drones can be determined, from which a preset number of drones are selected as cooperative drones.

[0077] It can be understood that the first power threshold is the minimum power required for the drone to perform inspection tasks after flying to an area with abnormal traffic conditions.

[0078] Specifically, after some drones are selected as cooperative drones, there may be a lack of inspection drones in the original inspection sub-area. When it is determined that there is an abnormal traffic condition, a spare drone can be determined to perform the inspection task in the corresponding sub-area, or a spare drone can be directly selected as a cooperative drone. The embodiment of the present application is not limited to this.

[0079] In some embodiments, considering the power consumption of the drone, it is necessary to allocate the inspection tasks of the drone in combination with the real-time power of the drone, specifically including:

[0080] If the remaining power of any drone is less than or equal to a second power threshold, update the return route of the low-power drone with a remaining power less than or equal to the second power threshold to the drone nest, and generate a return instruction to the low-power drone, so that the low-power drone executes the return instruction and returns to the drone nest along the return route;

[0081] When generating a return instruction to the low-battery drone, the real-time location and the inspected sub-area of ​​the low-battery drone are obtained, a flight route from the standby drone to the real-time location of the low-battery drone is generated, and a control instruction is generated to the standby drone, so that the standby drone executes the control instruction and flies to the real-time location of the low-battery drone according to the flight route, and continues to inspect the sub-area originally inspected by the low-battery drone.

[0082] It can be understood that the second power threshold is the minimum power threshold at which the drone can return to the drone nest. The second power threshold can be dynamically updated in real time. The value of the second power threshold is estimated based on the real-time position of the drone and the path to the nearest drone nest. This application does not limit this.

[0083] Furthermore, in multi-UAV collaborative monitoring, UAVs need to communicate frequently to share data, which results in huge consumption of communication bandwidth. Traditional collaborative methods usually require each UAV to transmit all perception data. Although this retains complete scene information, it also leads to low bandwidth utilization and excessive communication load. In addition, multiple UAVs may collect the same or similar traffic information from different perspectives. Therefore, it is necessary to effectively integrate the perception results of multiple UAVs to avoid data redundancy and communication congestion.

[0084] Based on this, in S104, it is necessary to integrate the traffic status data collected by multiple cooperative drones, including:

[0085] Each cooperative UAV generates a first feature distribution matrix based on the collected feature graphs, and generates a second feature distribution graph that is negatively correlated with the first feature distribution matrix;

[0086] After each of the cooperative drones fuses the traffic status data of other cooperative drones, the output of the fused traffic status data includes:

[0087] The first cooperative UAV receives the second feature distribution map of the second cooperative UAV, and constructs a feature selection matrix based on the first feature distribution matrix of the first cooperative UAV and the second feature distribution map of the second cooperative UAV;

[0088] Determine a selection weight of each feature in a first feature distribution matrix of the first cooperative UAV based on the feature selection matrix, select a corresponding feature based on the value of the selection weight, generate a sparse priority feature map, and send the sparse priority feature map to the second cooperative UAV;

[0089] fusing the feature map of the second cooperative UAV with the sparse priority feature map, and outputting the fused feature map as the fused traffic state data;

[0090] Wherein, each element in the first feature distribution matrix represents the confidence of the traffic status data at each position.

[0091] It can be understood that all cooperative drones can be used as first cooperative drones or second cooperative drones. The first cooperative drones and the second cooperative drones here are only used to distinguish between drones that send the second characteristic distribution map and drones that receive the second characteristic distribution map.

[0092] It can be understood that the first cooperative UAV is used to construct a first feature distribution matrix based on the collected feature map, that is, the traffic status data. The first feature distribution matrix is ​​used to characterize the confidence of the features at each position in the feature map collected by the first cooperative UAV. The first feature distribution matrix is ​​the distribution of the confidence of the features at each position, which can be used to characterize the perceptual importance of different spatial areas and clarify the key areas of the perception task. For example, the area containing the target object has more perceptual value than the background area. The first feature distribution matrix can guide the system to only share the most important perceptual information when the bandwidth is limited, so as to save communication costs and improve perception efficiency. In traditional collaborative systems, information in all spatial areas is often processed and transmitted equally.

[0093] It can be understood that the second feature distribution matrix is ​​negatively correlated with the first feature distribution matrix, that is, the matrix indicates the spatial area where the drone needs to obtain more information from other cooperative drones. It reflects the areas with weaker perception, which may have missing information due to factors such as occlusion and distance. The second feature distribution matrix is ​​generated based on the first feature distribution matrix to indicate the uncertainty of the current drone's perception data in certain areas, so as to request other cooperative drones to provide supplementary data.

[0094] Therefore, the first cooperative drone can receive the second feature distribution matrix of the second cooperative drone, thereby determining the data missing from the second cooperative drone, and can further determine whether to transmit data to the second cooperative drone in response to the second feature distribution matrix of the second cooperative drone.

[0095] Furthermore, the first cooperative UAV can select the critical spatial area of ​​perception from the feature map in combination with the first feature distribution matrix of the first cooperative UAV and the second feature distribution matrix received from the second cooperative UAV to form a sparse priority feature map. Specifically, a feature selection matrix is ​​constructed by the first feature distribution matrix of the first cooperative UAV and the second feature distribution map of the second cooperative UAV, and then it is determined whether each feature needs to be transmitted to the second cooperative UAV according to the value of the feature selection matrix.

[0096] In some embodiments, the weights in the feature selection matrix may be binarized to 0 or 1. If it is 1, it indicates that the corresponding feature needs to be transmitted, otherwise it does not need to be transmitted.

[0097] In some embodiments, the feature selection matrix can be used to characterize the degree of information overlap between the first feature distribution matrix of the first cooperative drone and the second feature distribution map of the second cooperative drone. The higher the value of the feature selection matrix, the higher the information overlap rate between the first feature distribution matrix of the first cooperative drone and the second feature distribution map of the second cooperative drone, indicating that the first cooperative drone has more information required by the second cooperative drone, and it is determined that data can be transmitted to the second cooperative drone. Conversely, if the value of the feature selection matrix is ​​low, a selection threshold can be set. When the value of the feature selection matrix is ​​lower than the selection threshold, it indicates that the first feature distribution map of the first cooperative drone has less data required by the second feature distribution map. At this time, there is no need to transmit data to the second cooperative drone, and the communication overhead is greater than the benefits brought by communication, avoiding unnecessary bandwidth occupation.

[0098] Based on such processing, in some embodiments, between every two cooperative drones, if it is determined according to the feature selection matrix that there is at least one feature that needs to be transmitted, then only the second feature distribution matrix and the sparse priority feature map need to be transmitted to the other cooperative drone. Transmitting only these sparse information can greatly reduce the amount of information and reduce bandwidth occupancy to ensure efficient sharing of key perception information.

[0099] Furthermore, after each cooperative UAV receives the sparse priority feature map of other UAVs, it is necessary to fuse the received information, including:

[0100] Each drone integrates its own feature map with the sparse priority features from other drones to further improve perception accuracy, specifically combining the multi-head attention mechanism for fusion. This fusion process combines the first feature distribution matrix of all drones to accurately identify which areas of features deserve more attention. The multi-head attention mechanism gives greater weight to these high-priority areas during fusion, ensuring that the most critical perception information is effectively transmitted and shared among different drones. Then output the fused traffic status data.

[0101] Further, in S105, the real-time traffic status monitoring result is obtained based on the fused traffic status data of each of the cooperative drones, including:

[0102] Modeling is performed based on the fused traffic status data of each of the cooperative drones to determine the vehicle distribution on each road and the average speed of the vehicles, and output real-time traffic status monitoring results.

[0103] Specifically, the fused traffic status data of each drone can be fed back to the constructed traffic status model respectively, so as to reflect the traffic flow and vehicle speed on each lane, determine the specific location of the traffic accident and the starting and ending points of the congestion caused by the traffic accident, and provide warning information according to the lane direction when the traffic flow in some lanes is large. The embodiments of the present application are not limited to this.

[0104] In some embodiments, after S105, the method further includes:

[0105] Determine the congested road section and the traffic volume of the congested road section based on the real-time traffic status monitoring result, and send a traffic abnormality prompt to other drones within a preset range of the abnormal traffic status area through the cooperative drone;

[0106] After receiving the traffic anomaly prompt, the other drones determine the traffic status data of the connected road sections of the congested road section;

[0107] A driving route that avoids the congested road section is generated based on the traffic status data of the connected road sections.

[0108] Specifically, the system where the drone is located can be linked to the systems of relevant departments and / or third-party map suppliers, and the information collected by the drone in real time can be input into the systems of relevant departments and / or third-party map suppliers to determine the traffic volume of congested sections and sections around the congested sections. The system of the third-party map supplier can be combined to capture the destinations of various users around the congested sections and calculate detour routes. The embodiments of the present application are not limited to this.

[0109] The following are device embodiments of the present application, which can be used to execute the method embodiments of the present application. For details not disclosed in the device embodiments of the present application, please refer to the method embodiments of the present application.

[0110] See next Figure 2 , is a schematic diagram of the structure of a multi-UAV coordinated traffic status monitoring device provided by an exemplary embodiment of the present application. The device can be implemented as all or part of a terminal through software, hardware, or a combination of both, and can also be integrated on a server as an independent module. The multi-UAV coordinated traffic status monitoring device in the embodiment of the present application can be applied to a terminal or a cloud. The device 20 includes a UAV scheduling module 201, an information identification module 202, an information fusion module 203, and an information processing module 204, wherein:

[0111] The drone scheduling module 201 is used to perform inspections in a preset inspection area by using multiple drones; wherein each drone inspects a sub-area in the inspection area based on a preset inspection route;

[0112] The information identification module 202 is used to identify whether the traffic status in the corresponding sub-area is abnormal through the trained deep learning model carried by each drone;

[0113] When the information identification module 202 identifies that the traffic status is abnormal, the drone scheduling module 201 is further used to select a cooperative drone based on the real-time status of each drone, and assign each cooperative drone to an abnormal traffic status area, so as to collaboratively collect traffic status data through multiple cooperative drones;

[0114] The information fusion module 203 is used to enable each of the cooperative drones to fuse the traffic status data of other cooperative drones and output the fused traffic status data;

[0115] The information processing module 204 is used to obtain real-time traffic status monitoring results based on the fused traffic status data of each of the cooperative drones.

[0116] It should be noted that the device 20 provided in the above embodiment only uses the division of the above functional modules as an example when executing the multi-UAV collaborative traffic status monitoring method. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above. In addition, the device provided in the above embodiment and the multi-UAV collaborative traffic status monitoring method embodiment belong to the same concept, and its implementation process is detailed in the method embodiment, which will not be repeated here.

[0117] An embodiment of the present application also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of any of the above-mentioned method embodiments when executing the program.

[0118] See also Figure 3 , which is a structural block diagram of an electronic device provided in an embodiment of the present application.

[0119] like Figure 3 As shown, the electronic device 300 includes: a processor 301 and a memory 302 .

[0120] In the embodiment of the present application, the processor 301 is the control center of the computer system, which can be a processor of a physical machine or a processor of a virtual machine. The processor 301 may include one or more processing cores, such as a 4-core processor, an 8-core processor, etc. The processor 301 may be implemented in at least one hardware form of DSP (Digital Signal Processing), FPGA (Field-Programmable Gate Array), and PLA (Programmable Logic Array).

[0121] The processor 301 may also include a main processor and a coprocessor. The main processor is a processor for processing data in the awake state, also called a CPU (Central Processing Unit); the coprocessor is a low-power processor for processing data in the standby state.

[0122] The memory 302 may include one or more computer-readable storage media, which may be non-transitory. The memory 302 may also include a high-speed random access memory, and a non-volatile memory, such as one or more disk storage devices, flash memory storage devices. In some embodiments of the present application, the non-transitory computer-readable storage medium in the memory 302 is used to store at least one instruction, which is used to be executed by the processor 301 to implement the method in the embodiment of the present application.

[0123] In some embodiments, the electronic device 300 further includes: a peripheral device interface 303 and at least one peripheral device 304. The processor 301, the memory 302 and the peripheral device interface 303 can be connected via a bus or a signal line. Each peripheral device 304 can be connected to the peripheral device interface 303 via a bus, a signal line or a circuit board. Specifically, the peripheral device 304 includes: a display screen, a camera and an audio circuit. The peripheral device interface 303 can be used to connect at least one peripheral device related to I / O (Input / Output) to the processor 301 and the memory 302.

[0124] In some embodiments of the present application, the processor 301, the memory 302, and the peripheral device interface 303 are integrated on the same chip or circuit board; in some other embodiments of the present application, any one or two of the processor 301, the memory 302, and the peripheral device interface 303 can be implemented on a separate chip or circuit board. This embodiment of the present application does not specifically limit this.

[0125] The electronic device structure block diagram shown in the embodiment of the present application does not constitute a limitation on the electronic device 300. The electronic device 300 may include more or fewer components than shown in the figure, or combine certain components, or adopt a different component arrangement.

[0126] The embodiment of the present application also provides a computer-readable storage medium, on which a computer program is stored, and when the program is executed by a processor, the steps of the method of any of the above embodiments are implemented. The computer-readable storage medium may include, but is not limited to, any type of disk, including a floppy disk, an optical disk, a DVD, a CD-ROM, a micro drive, and a magneto-optical disk, a ROM, a RAM, an EPROM, an EEPROM, a DRAM, a VRAM, a flash memory device, a magnetic card or an optical card, a nanosystem (including a molecular memory IC), or any type of medium or device suitable for storing instructions and / or data.

[0127] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, by hardware. Based on this understanding, the above technical solution can be essentially or in other words, the part that contributes to the relevant technology can be embodied in the form of a software product, which can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiment.

[0128] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit it. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A multi-UAV coordinated traffic status monitoring method, characterized in that: include: Conducting inspections in a preset inspection area by using a plurality of drones; wherein each drone inspects a sub-area within the inspection area based on a preset inspection route; Through the trained deep learning model carried by each drone, it is possible to identify whether there are any abnormalities in the traffic status in the corresponding sub-area. When an abnormal traffic state is identified, a cooperative drone is selected based on the real-time state of each drone, and each cooperative drone is assigned to an abnormal traffic state area, so that traffic state data can be collected collaboratively by multiple cooperative drones; Each of the cooperative drones outputs fused traffic status data after fusing the traffic status data of other cooperative drones; A real-time traffic status monitoring result is obtained based on the fused traffic status data of each of the cooperative UAVs.

2. The method for monitoring traffic status by multiple UAVs in collaboration according to claim 1, characterized in that: Each drone uses a trained deep learning model to identify whether the traffic status in the corresponding sub-area is abnormal, including: The image data in the corresponding sub-area is collected by an image acquisition device carried by the drone; Inputting the image data into the trained deep learning model, extracting features from the image data through the trained deep learning model, and identifying whether there are abnormal traffic status features; When an abnormality in the traffic status is identified, point cloud data in the corresponding sub-area is collected by the laser radar carried by the drone, and the location range of the abnormal traffic status is determined based on the point cloud data, thereby determining the abnormal traffic status area.

3. The method for multi-UAV coordinated traffic status monitoring according to claim 2, characterized in that: The selecting a cooperative drone based on the real-time status of each drone and assigning each cooperative drone to an area with abnormal traffic status includes: Determine the distance from the real-time position coordinates of each drone to the abnormal traffic condition area; Based on the distance, the remaining power of each drone after reaching the abnormal traffic state area is estimated, and if the remaining power is greater than a first power threshold, the corresponding drone is selected as a cooperative drone; The inspection route of the cooperative drone to the abnormal traffic condition area is updated, and each cooperative drone is assigned to the abnormal traffic condition area.

4. The method for multi-UAV coordinated traffic status monitoring according to claim 1, characterized in that: The method further comprises: If the remaining power of any drone is less than or equal to a second power threshold, update the return route of the low-power drone with a remaining power less than or equal to the second power threshold to the drone nest, and generate a return instruction to the low-power drone, so that the low-power drone executes the return instruction and returns to the drone nest along the return route; When generating a return instruction to the low-battery drone, the real-time location and the inspected sub-area of ​​the low-battery drone are obtained, a flight route from the standby drone to the real-time location of the low-battery drone is generated, and a control instruction is generated to the standby drone, so that the standby drone executes the control instruction and flies to the real-time location of the low-battery drone according to the flight route, and continues to inspect the sub-area originally inspected by the low-battery drone.

5. A multi-UAV coordinated traffic status monitoring method according to any one of claims 1 to 4, characterized in that: Each cooperative UAV generates a first feature distribution matrix based on the collected feature graphs, and generates a second feature distribution graph that is negatively correlated with the first feature distribution matrix; After each of the cooperative drones fuses the traffic status data of other cooperative drones, the output of the fused traffic status data includes: The first cooperative UAV receives the second feature distribution map of the second cooperative UAV, and constructs a feature selection matrix based on the first feature distribution matrix of the first cooperative UAV and the second feature distribution map of the second cooperative UAV; Determine a selection weight of each feature in a first feature distribution matrix of the first cooperative UAV based on the feature selection matrix, select a corresponding feature based on the value of the selection weight, generate a sparse priority feature map, and send the sparse priority feature map to the second cooperative UAV; fusing the feature map of the second cooperative UAV with the sparse priority feature map, and outputting the fused feature map as the fused traffic state data; Wherein, each element in the first feature distribution matrix represents the confidence of the traffic status data at each position.

6. The method for multi-UAV coordinated traffic status monitoring according to claim 1, characterized in that: The real-time traffic status monitoring result is obtained based on the fused traffic status data of each cooperative UAV, including: Modeling is performed based on the fused traffic status data of each of the cooperative drones to determine the vehicle distribution on each road and the average speed of the vehicles, and output real-time traffic status monitoring results.

7. The method for monitoring traffic status by multiple UAVs in coordination according to claim 6, characterized in that: After obtaining the real-time traffic status monitoring result based on the fused traffic status data of each cooperative UAV, the method further includes: Determine the congested road section and the traffic volume of the congested road section based on the real-time traffic status monitoring result, and send a traffic abnormality prompt to other drones within a preset range of the abnormal traffic status area through the cooperative drone; After receiving the traffic anomaly prompt, the other drones determine the traffic status data of the connected road sections of the congested road section; A driving route that avoids the congested road section is generated based on the traffic status data of the connected road sections.

8. A multi-UAV coordinated traffic status monitoring device, characterized in that: include: A drone dispatching module is used to perform inspections in a preset inspection area by using multiple drones; wherein each drone inspects a sub-area in the inspection area based on a preset inspection route; The information recognition module is used to identify whether the traffic status in the corresponding sub-area is abnormal through the trained deep learning model carried by each drone; When the information recognition module recognizes that the traffic status is abnormal, the drone scheduling module is further used to select a cooperative drone based on the real-time status of each drone, and assign each cooperative drone to an abnormal traffic status area, so as to collaboratively collect traffic status data through multiple cooperative drones; An information fusion module, configured to enable each of the cooperative UAVs to fuse the traffic status data of other cooperative UAVs and output the fused traffic status data; The information processing module is used to obtain real-time traffic status monitoring results based on the fused traffic status data of each of the cooperative drones.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the program, the steps of the method according to any one of claims 1 to 7 are implemented.

10. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

Images