A method for marking and detecting disaster scene information from the perspective of a UAV

Abstract

The application discloses a kind of unmanned plane visual angle disaster scene information marking and detection method, comprising the following steps: (1) the data dictionary of object to be identified is established using cloud edge end cooperation mode;(2) construct neural network MIM-YOLOv5 based on YOLOv5, (3) the image captured by unmanned plane is transmitted into MIM-YOLOv5, through cloud center, front-line command center, handheld terminal and unmanned plane terminal respectively acting as cloud, edge, end node, constitute cloud edge end cooperation system, disaster scene information is automatically marked and target detection using cloud edge end cooperation mode, when disaster label appears in image, neural network automatically detects and alarms, and enable cloud edge end three-party cooperation processing.This application combines cloud edge end technology with disaster detection and dispatch command for the first time, cooperates cloud computing with terminal equipment, and greatly improves emergency disposal efficiency.

Description

TECHNICAL FIELD

[0001] The present application relates to target detection technology, and in particular to a UAV visual angle disaster scene information labeling and detection method. BACKGROUND

[0002] The past decade can be said to be the decade of the development of cloud computing, and the cloud computing market pattern at home and abroad has been initially determined. The bandwidth of the network is limited, and the 5G technology has basically defined the future 10 years of wireless network, but the number of terminals is growing exponentially. According to the prediction of IDC, the number of Internet of Things terminals in 2021 has reached 50 billion, and will reach 100 billion by 2025.

[0003] Under the background of relatively mature cloud computing, limited network bandwidth, and explosive growth of terminals. Edge computing is becoming more and more important. Since 2019, edge computing technology has gradually matured, and the application scenarios have been rich, which has become a development hotspot in the industry, and will usher in a period of 5-10 years of explosion.

[0004] Edge computing will play an important role in low latency demand, reducing cloud technology pressure and network bandwidth cost, improving data security and reliability, but also faces many challenges such as heterogeneous edge nodes, limited resource transmission, edge data heterogeneity, uneven workload allocation, security and privacy.

[0005] In the application scenarios of intelligent manufacturing defect detection, precision manufacturing, medical auxiliary diagnosis and treatment, automatic driving, cloud gaming, ultra-high definition video, etc., cloud-edge collaboration is the inevitable choice of technical path. Similarly, target detection and disaster labeling are also an application prospect of cloud-edge collaboration.

[0006] Simply put, cloud-edge collaboration is a unified data interaction mode, unified scheduling and management. It is a layered solution, and each layer of service has different processing logic and responsibilities, and layers work together.

[0007] From the detection method, in recent years, the use of deep convolutional neural networks in target detection tasks has made great progress, greatly promoting the development of target detection applications. However, most of the previous deep convolutional neural networks are designed for natural scene images. Directly applying the previous model to process the target detection task in the disaster scene captured by the UAV has three main problems. First, due to the large change in the flight height of the UAV, the object scale changes greatly. Second, the image captured by the UAV contains a high density of objects, which will cause occlusion between objects. Third, when the UAV is shooting, considering the smoke block in the disaster site, the pictures taken are often poor in recognition. The above three problems make the target detection of UAV captured images very challenging.

[0008] From the scene annotation, the disaster scene is complex, the data set is difficult to annotate, and the data set is small. The special disaster scene changes complexly, and the corresponding data set is lacking. The annotation method is mostly plotted by C / S end. This way has limitations in flexibility and timeliness, data sharing is difficult, and it is difficult to feedback the on-site information in the first time. SUMMARY

[0009] In view of the problems in the prior art, the unmanned aerial vehicle visual angle disaster scene information annotation and detection method is provided to solve the problems of low accuracy of unmanned aerial vehicle object recognition and complex disaster scene.

[0010] In order to achieve the above purpose, the following technical solutions are adopted:

[0011] An unmanned aerial vehicle visual angle disaster scene information annotation and detection method comprises the following steps:

[0012] (1) a cloud edge end cooperative mode is used to establish a data dictionary of the object to be identified;

[0013] (2) a neural network MIM-YOLOv5 based on YOLOv5 is constructed, the MIM-YOLOv5 neural network adopts a combination of transformer and Conv, wherein the Swin-MIM module adopts a self-supervised mode to expand the data set; the target detection algorithm adopted is an improved model MIM-YOLOv5, the MIM-YOLOv5 adopts Swin-MIM, CSPDarknet53 and path aggregation network as the backbone and neck, and simultaneously reduces the output size of the original three prediction heads respectively used for detecting small, medium and large objects by one half to obtain three detection modules respectively used for detecting micro, small and medium objects to adapt to the unmanned aerial vehicle captured scene; and a convolution block attention module CBAM is used to generate an attention map along the channel and spatial dimensions in sequence;

[0014] (3) the unmanned aerial vehicle captured image is transmitted into the MIM-YOLOv5, the cloud center, the front-line command center, the handheld terminal and the unmanned aerial vehicle terminal respectively serve as the cloud, the edge and the end node to constitute a cloud edge end cooperative system, the cloud edge end cooperative mode is used to automatically annotate and detect the disaster scene information, when the disaster label appears in the image, the neural network automatically detects and alarms, and the cloud edge end three-party cooperative processing is enabled.

[0015] Further, the specific steps of step 1) for establishing the data dictionary of the object to be identified in a cloud edge end cooperative mode are as follows: the handheld terminal is directly used as the end to label and enter the new disaster image data on the terminal, then the data are reported to the nearby deployed command center, the cloud server data dictionary is periodically summarized, the overall data dictionary is updated, and finally the data dictionary of the label and the disaster image is formed.

[0016] Further, the Swin-MIM in the step 2) sets the mask probability to 0.15 during training and sets the mask probability to 0 during actual testing.

[0017] Further, the specific operation of the step 3) for automatically labeling and target detecting the disaster scene information in a cloud-edge-end collaborative manner is as follows: data is collected by a terminal and uploaded, and the edge end dynamically labels the newly collected data and uploads the labeled disaster image data to the cloud center; the cloud server first collects existing open-source disaster data and matches the local data set, and a cosine similarity calculation value based on image pixels is used as the matching algorithm, and a standard value λ is set to measure the similarity of the data; if the calculated similarity is greater than λ, the image is not updated or replaced; if the calculated similarity is less than λ, the data is added to the cloud data to dynamically expand the cloud data set; then, the expanded data is handed over to cloud center professionals for real-time data labeling; the cloud center server aggregates the labeled data and returns the integrated rescue data to the on-site rescue personnel, forming a cloud-edge-end collaborative labeling system.

[0018] Further, in the step 3), the cloud center can provide API services or use a model repository to distribute images to designated edge nodes, and then deploy them into APIs, which run in the edge nodes, i.e., the Node of the cloud center, to manage all services.

[0019] All sensors in the cloud-edge-end collaborative system are uniformly connected to the gateway of the edge node to realize unified distribution and improve data transmission efficiency; after obtaining all the original data, the cloud end sends the rule engine module to the analysis model service to realize model reasoning and finally output data insight.

[0020] Compared with the prior art, the present application has the following technical effects:

[0021] 1. The present application first combines cloud-edge-end technology with disaster detection, and collaborates cloud computing with terminal devices to greatly improve emergency disposal efficiency.

[0022] 2、Because the model parameters required by the simple transformer structure are too large, and a large training set is needed to train it to fit, the current stage of unmanned aerial vehicle vision cannot meet its requirements, and the data set is difficult to expand like other images. Through experiments, it is found that the mask autoencoding module in the first part is more suitable for training a model that adapts to small targets, so the application adopts the combination of Swin-MIM and convolution, and the design of mask autoencoding enables self-supervised training of images, while data augmentation enables the model to have better generalization ability.

[0023] 3、Solved the problem that the previous unmanned aerial vehicle recognition object accuracy is not high, and the data set is difficult to label for complex disaster scenes.

[0024] 4、The application does not need to modify any traditional equipment, and can realize edge computing by uploading data, which is low in cost; in addition, because the edge node is deployed close to the data source, the response is very timely in processing, so the efficiency is higher; and human resources can be maximized to form a good command system. BRIEF DESCRIPTION OF DRAWINGS

[0025] Figure 1 The structure diagram of the Swin-MIM module based on transformer involved in the embodiment of the application is shown in the figure.

[0026] Figure 2 The overall structure diagram of MIM-YOLOv5 involved in the embodiment of the application is shown in the figure.

[0027] Figure 3 The cloud edge end collaborative labeling and command system diagram involved in the embodiment of the application is shown in the figure.

[0028] Figure 4 The cloud computing diagram involved in the embodiment of the application is shown in the figure.

[0029] Figure 5 The cloud collaboration diagram involved in the embodiment of the application is shown in the figure. DETAILED DESCRIPTION

[0030] The specific content of the application will be further explained and described in detail below in combination with embodiments.

[0031] REFERENCE Figures 1-5 The embodiment provides a kind of unmanned aerial vehicle visual angle disaster scene information labeling and detection method, comprising the following steps:

[0032] 1) Adopting cloud edge end collaborative way to establish data dictionary of the object to be identified; Specifically: directly label the newly added disaster image data on the terminal with the handheld terminal as the end, then report to the nearby deployed command center, regularly summarize to the cloud server data dictionary, update the overall data dictionary, and finally form the data dictionary of the label and disaster image;

[0033] 2) Constructing the neural network MIM-YOLOv5 based on YOLOv5, the MIM-YOLOv5 neural network adopts the way of combining transformer with Conv, wherein the Swin-MIM module adopts the self-supervised way to expand the data set, the mask probability is set to 0.15 during training, and the mask probability is set to 0 during actual test; The target detection algorithm adopted is the improved model MIM-YOLOv5, which adopts Swin-MIM, CSPDarknet53 and path aggregation network as the backbone and neck, and at the same time, the output size of the original three prediction heads respectively used for detecting small, medium and large objects is reduced by one half to obtain three detection modules respectively used for detecting micro, small and medium objects to adapt to the unmanned aerial vehicle capture scene; And the convolution block attention module CBAM generates attention map along the channel and spatial dimension sequence;

[0034] 3) The unmanned aerial vehicle captured image is transmitted into MIM-YOLOv5, through cloud center, front-line command center, handheld terminal and unmanned aerial vehicle terminal, the three parties respectively act as cloud (cloud center), edge (front-line command center), end (handheld terminal and unmanned aerial vehicle terminal) nodes, to constitute a cloud edge end collaborative system, and adopt cloud edge end collaborative way to automatically label and detect disaster scene information, specifically: the on-site personnel hold the terminal, dynamically collect and upload disaster image data, the data will be transmitted to the command center deployed near the disaster scene, the command center dynamically labels the newly collected disaster image data and uploads the labeled disaster image data to the cloud center, the cloud server first collects the existing open source disaster data and matches with the local data set, the matching algorithm adopts the cosine similarity calculation value based on image pixels, and sets a standard value λ, preferably, λ = 50%, which is used to measure the similarity of data. If the calculated similarity is greater than λ, the image is not updated and replaced; if the calculated similarity is less than λ, the data is added in the cloud data to dynamically expand the cloud data set, and then the expanded data is handed over to the cloud center professional personnel for real-time data labeling, the cloud center server aggregates the labeled data, and returns the integrated rescue data to the on-site rescue personnel, forming a cloud edge end collaborative labeling system. At the same time, corresponding models can be established in related disasters to realize automatic labeling of on-site data. When disaster labels appear in the image, the neural network automatically detects and alarms, and the cloud edge end three-party collaborative processing is enabled.

[0035] The cloud center can provide API service or utilize a model warehouse to distribute images to specified edge nodes, and then deploy into API, run in edge nodes, i.e. Node of the cloud center, for managing all services;

[0036] All sensors in the cloud edge end cooperative system are uniformly accessed to the gateway of the edge node, unified distribution is realized, and data transmission efficiency is improved; the cloud end sends a rule engine module to the analyzed model service after obtaining all original data, model reasoning is realized, and finally data insight is output.

[0037] In the application, data labeling is in a cloud edge end cooperative labeling mode, considering that field personnel need to participate in rescue operations, the field personnel only need to collect data using a terminal and upload the data, the edge end dynamically labels newly collected data and uploads the labeled disaster image data to the cloud center, the cloud server first collects existing open source disaster data and matches with a local data set, a matching algorithm adopts cosine similarity calculation value based on image pixels, and a standard value lambda is set to measure data similarity.If the calculated similarity is greater than lambda, the image is not updated and replaced; if the calculated similarity is less than lambda, the data is added in the cloud end data to dynamically expand the cloud end data set, and then the expanded data is handed over to cloud center professionals for real-time data labeling, the cloud center server gathers the labeled data, returns integrated rescue data to field rescue personnel, and forms a cloud edge end cooperative labeling system. Meanwhile, corresponding models can be established in related disasters to realize automatic labeling of field data.

Claims

1. A method for labeling and detecting disaster scene information from the perspective of unmanned aerial vehicles (UAVs), characterized in that, Includes the following steps: (1) A data dictionary for the object to be identified is established using a cloud-edge-device collaborative approach; (2) A neural network MIM-YOLOv5 based on YOLOv5 is constructed. The MIM-YOLOv5 neural network adopts a combination of transformer and Conv. The Swin-MIM module expands the dataset in a self-supervised manner. The target detection algorithm used is the improved model MIM-YOLOv5. The MIM-YOLOv5 uses Swin-MIM, CSPDarknet53 and path aggregation network as the backbone and neck. At the same time, the output size of the original three prediction heads used to detect small, medium and large objects is reduced by half to obtain three detection modules used to detect small, small and medium objects to adapt to the UAV capture scenario. The convolutional block attention module CBAM is used to generate attention maps sequentially along the channel and spatial dimensions. (3) The images captured by the drone are transmitted to MIM-YOLOv5. The cloud center, the front-line command center, the handheld terminal and the drone terminal act as cloud, edge and terminal nodes respectively to form a cloud-edge-terminal collaborative system. The cloud-edge-terminal collaborative method is used to automatically label and detect targets in disaster scene information. When a disaster label appears in the image, the neural network automatically detects and alarms, and enables cloud-edge-terminal three-party collaborative processing.

2. The method for labeling and detecting disaster scene information from a drone's perspective as described in claim 1, characterized in that, The specific steps of step 1) in establishing the data dictionary of the objects to be identified using a cloud-edge-device collaborative approach are as follows: the newly added disaster image data is directly labeled on the handheld terminal, and then reported to the nearby command center. The data dictionary is periodically summarized and updated to the cloud server data dictionary, and finally a data dictionary of labels and disaster images is formed.

3. The method for labeling and detecting disaster scene information from a drone's perspective as described in claim 1, characterized in that, In step 2), the mask probability of Swin-MIM is set to 0.15 during training and 0 during actual testing.

4. The method for labeling and detecting disaster scene information from a drone's perspective as described in claim 1, characterized in that, Step 3) involves the automatic labeling and target detection of disaster scene information using a cloud-edge-device collaborative approach. The specific operation is as follows: Data is collected and uploaded using a terminal. The edge device dynamically labels newly collected data and uploads the labeled disaster image data to the cloud center. The cloud server first collects existing open-source disaster data and matches it with the local dataset. The matching algorithm uses cosine similarity calculation based on image pixels and sets a standard value λ to measure data similarity. If the calculated similarity is greater than λ, the image is not updated or replaced; if the calculated similarity is less than λ, the data is added to the cloud dataset to dynamically expand the cloud dataset. The expanded data is then handed over to cloud center professionals for real-time data labeling. The cloud center server aggregates the labeled data and returns the integrated rescue data to on-site rescue personnel, forming a cloud-edge-device collaborative labeling system.

5. The method for labeling and detecting disaster scene information from a drone's perspective as described in claim 1, characterized in that, In step 3), the cloud center can provide API services or use the model repository to distribute the image to the designated edge node, and then deploy it as an API to run on the edge node, i.e., the cloud center's Node, for managing all services. In the cloud-edge-device collaborative system, all sensors are uniformly connected to the gateway of the edge node to achieve unified distribution and improve data transmission efficiency. After acquiring all the raw data, the cloud sends the rule engine module to the analysis model service to realize model inference and finally output the data detection results.

Images