A method, device and equipment for identifying extremely disaster-stricken areas based on UAV data

Abstract

The present invention provides a method, device and equipment for identifying extremely disaster-stricken areas based on unmanned aerial vehicle data. The method comprises: extracting features from a remote sensing image taken by an unmanned aerial vehicle to obtain a feature map; generating a preselection frame based on pixel points on the feature map, and dividing the preselection frame into a foreground and a background according to a priori identification frame; determining a suggestion frame for selecting an extremely disaster-stricken area in the remote sensing image based on the preselection frame and the feature map in combination with a regression algorithm; fine-tuning the suggestion frame based on the regression algorithm to obtain a prediction frame with high selection accuracy; processing the feature map of the extremely disaster-stricken area selected by the prediction frame to generate an extremely disaster-stricken area object mask; inputting the prediction frame and the extremely disaster-stricken area object mask into a geoscientific classifier to at least determine a digital surface model and thermal infrared data corresponding to the extremely disaster-stricken area, and screening out a target prediction frame and a target extremely disaster-stricken area object mask that meet geoscientific conditions through the digital surface model and the thermal infrared data as recognition result outputs.

Description

Technical Field

[0001] The embodiments of the present invention relate to the technical field of unmanned aerial vehicle systems, and particularly to a method, device, and equipment for identifying extremely disaster-stricken areas based on unmanned aerial vehicle data. Background Technique

[0002] With the development of computer science and technology, the improvement of the performance of network software and hardware (such as the graphics processing unit GPU), and the continuous expansion of the fields of image processing technology and image recognition applications, deep learning has been widely applied in many fields such as computer vision, natural language processing, and image processing. Currently, the most popular deep learning network for images is the convolutional neural network (CNN). The convolutional neural network is one of the representative algorithms for deep learning feature extraction. Its proposal is inspired by the fact that the neurons in the visual cortex of the visual system structure in biology receive local information (that is, these neurons only respond to stimuli in certain specific regions). The convolutional neural network realizes the local perception of the image through the local receptive field (convolution window), and gradually synthesizes information through the connection of multiple convolutional layers and other components to simulate the working mode of the neurons in the visual cortex, extracts the feature map, and then forms the feature vector of the image through the fully connected layer for image classification (such as handwritten digit recognition, animal recognition, etc.). With the improvement of application requirements, the region-based convolutional neural network (R-CNN) supplements the object localization ability on the basis of the CNN, that is, it not only identifies the category of the object, but also marks the relative position of the specific object in the image with a rectangular box, so as to realize the regional detection of the target. Currently, in the field of deep learning, representative object detection methods such as Fast R-CNN, Faster R-CNN, Mask R-CNN, YOLO, and SSD have been formed. Among them, Mask R-CNN further realizes the instance segmentation of the object through the mask branch of the fully convolutional network on the basis of object detection and classification (Faster R-CNN), and can be used for the extraction of planar features. Its algorithm scheme is as Figure 1 、 Figure 2 and shown in the following formula:

[0003] The convolution calculation formula is as follows:

[0004] F (i，j) =∑f(t)⊙g(t), 0 < i, j < n

[0005] where: ⊙ is the Hadamard product, and ∑ is the sum of matrix elements;

[0006]

[0007] n is the length and width of the image;

[0008]

[0009] Let \(m\) be the size of the convolution window, where \(\{m = 2k + 1|k\in N\}\) * ;

[0010] As Figure 3 shown, based on the image features extracted by CNN, the rectangular box identifying the object in the picture is used as the feature vector for target localization to train the target localization regression network; The Mask R-CNN object recognition algorithm generates proposal boxes \(P\) according to specific rules, and then uses bounding box regression to correct the positions of the proposal boxes determined as positive samples to obtain more accurate prediction boxes. Bounding box regression refers to the mapping process \(f\) and its parameters (the horizontal and vertical coordinates \(x,y\) of the center and the width and height \(w,h\) of the border) that make the proposal box \(P\) as close as possible to the identification box \(G\) (i.e., ), and the regression result is used as the final prediction result. The mathematical representation of this algorithm is as follows:

[0011] Given \(G=(G x ,G y ,G w ,G h ), \(P=(P x ,P y ,P w ,P h ), then we have:

[0012]

[0013] Transformation

[0014]

[0015] The learning objective of the bounding box regression task is to make the parameters \(d x (P),d y (P),d w (P),d h (P)\) approximately equal to \((t x ,t y ,t w ,t h ) wirelessly. After obtaining the determined classification and localization results, a mask is generated using a fully convolutional network. For details, please refer to Figure 4 shown.

[0016] The existing technologies using CNN for image target recognition and instance segmentation in the fields of remote sensing, geoscience, and GIS have the following defects:

[0017] (1) Only extract features based on the spectral information of the image itself, and cannot utilize the geoscience-related information inherent in the target object as a ground entity in the field of remote sensing, such as:

[0018] ① The spatial structural characteristics of the target, such as the highly uneven surface of the extremely disaster-stricken area;

[0019] ② Spatial correlation characteristics between targets, such as the distribution of buildings and roads that may exist around the disaster area.

[0020] (2) Targets in earthquake-affected areas have similar texture features to ordinary building garbage. After the CNN network extracts features, there are identical or similar features between different targets, which affects the target recognition and classification results. For example, Figure 5 shown.

[0021] (3) Identification of extremely disaster-stricken areas is one of the important tasks in earthquake relief work. Traditional on-site investigation methods are time-consuming and labor-intensive. It is difficult to quickly and effectively identify earthquake damage to buildings in a large area within the 2-hour black box period after the earthquake. It also provides important information support and decision-making basis for government departments to determine key rescue areas and dispatch rescue forces and materials. Summary of the invention

[0022] The present invention provides a method, a device and equipment capable of quickly and accurately identifying extremely disaster-stricken areas based on drone photography data.

[0023] In order to solve the above technical problems, an embodiment of the present invention provides a method for identifying extremely disaster-stricken areas based on drone data, comprising:

[0024] Perform feature extraction on the remote sensing image taken by the drone to obtain a feature map;

[0025] Generate a series of pre-selected boxes based on the pixel points on the feature map, and divide the pre-selected boxes into foreground and background according to the prior identification box;

[0026] Based on the pre-selected frame and the feature map combined with a regression algorithm, a suggestion frame for selecting the extremely disaster-stricken area in the remote sensing image is determined;

[0027] Fine-tuning the suggestion box based on a regression algorithm to obtain a prediction box with a higher selection accuracy than the suggestion box;

[0028] Processing the feature map based on the extremely disaster area selected by the prediction box to generate an extremely disaster area object mask;

[0029] The prediction frame and the extremely disaster area object mask are input into a geological classifier, so that the geological classifier at least determines the digital surface model and thermal infrared data corresponding to the extremely disaster area based on the prediction frame and the extremely disaster area object mask, and screens the digital surface model and thermal infrared data to output the target prediction frame and target extremely disaster area object mask that meet the geological conditions as the recognition result.

[0030] As an optional embodiment, it further includes:

[0031] Based on the geoscience classifier, identify or extract the temperature anomaly points existing in the target prediction box and the target extremely disaster-prone area object mask.

[0032] As an optional embodiment, the extremely disaster-prone area object mask includes the extremely disaster-prone area and other areas in the remote sensing image;

[0033] The step of enabling the geoscience classifier to at least determine the digital elevation model and thermal infrared data corresponding to the extremely disaster-prone area based on the prediction box and the extremely disaster-prone area object mask includes:

[0034] Based on the geoscience classifier, construct the digital elevation model containing the extremely disaster-prone area and the thermal infrared data corresponding to the extremely disaster-prone area by combining the content selected by the prediction box and the content in the extremely disaster-prone area object mask.

[0035] As an optional embodiment, the step of screening the digital elevation model and thermal infrared data to obtain the target prediction box and the target extremely disaster-prone area object mask that meet the geoscience conditions and outputting them as the recognition result includes:

[0036] Resample the digital elevation model and thermal infrared data to the target pixels and then compress them into column vectors;

[0037] Process the column vectors by the fully connected network in the geoscience classifier;

[0038] The classification module in the geoscience classifier determines at least whether the concavity and convexity of the three-dimensional space corresponding to the extremely disaster-prone area meet the geoscience conditions based on the processing result of the fully connected network, and at the same time determines whether the extremely disaster-prone area has temperature anomaly feature points;

[0039] Based on the judgment results, screen multiple prediction boxes and extremely disaster-prone area object masks to obtain the target prediction box and the target extremely disaster-prone area object mask.

[0040] As an optional embodiment, the step of extracting features from the remote sensing image obtained by drone shooting to obtain a feature map includes:

[0041] Construct and train a feature extraction network for identifying the extremely disaster-prone area features in the remote sensing image;

[0042] Based on the feature extraction network, extract features from the remote sensing image obtained by drone shooting to obtain a feature map.

[0043] As an optional embodiment, the step of calculating and determining the proposed box for framing the extremely disaster-prone area in the remote sensing image based on the preselected box and the feature map in combination with the regression algorithm includes:

[0044] Process the feature map based on the feature pyramid network to generate multiple sub-feature maps with different levels, where the number of pixels used to describe feature information in the sub-feature maps at different levels is different;

[0045] Generate multiple preselected boxes with different sizes on the corresponding remote sensing image based on the sub-feature maps;

[0046] Calculate the overlap degree between the preselected boxes and the identification boxes corresponding to the same remote sensing image;

[0047] Determine a target preselected box that frames a foreground area and whose framing accuracy meets the first threshold based on the overlap degree;

[0048] In the case where it is determined that the target preselected box frames the extremely disaster-stricken area, calculate the proposed box based on the regression algorithm for the target preselected box.

[0049] As an optional embodiment, the method further includes:

[0050] Determine a first preselected box that frames a foreground area based on the overlap degree as a positive sample, and determine a second preselected box that frames a background area as a negative sample;

[0051] Use the content framed by the positive sample and the negative sample as training data to train the target classifier architecture to obtain a background classifier that can directly classify positive and negative samples for the preselected box.

[0052] As an optional embodiment, the fine-tuning of the proposed box based on the regression algorithm to obtain a prediction box with a framing accuracy higher than that of the proposed box includes:

[0053] Map the proposed box to the corresponding position on the matching feature map;

[0054] Resample the extremely disaster-stricken area framed by the proposed box, compress it into a column vector, and input it into the fully connected layer of the object segmentation network for classification of the extremely disaster-stricken area;

[0055] Based on the classification result, the position of the proposed box on the matching feature map, and at the same time combine the regression algorithm to fine-tune the proposed box to obtain a prediction box with a selection accuracy higher than that of the proposed box.

[0056] Another embodiment of the present invention also provides an extremely disaster-stricken area recognition device based on UAV data, including:

[0057] An extraction module for extracting features from a remote sensing image obtained by UAV shooting to obtain a feature map;

[0058] A partitioning module, configured to generate a series of preselected boxes based on pixel points on the feature map, and partition the preselected boxes into foreground and background according to prior identification boxes;

[0059] A calculation module, configured to calculate and determine a proposed box for framing the extremely disaster-stricken area in the remote sensing image according to the preselected box and the feature map in combination with a regression algorithm;

[0060] A fine-tuning module, configured to finely tune the proposed box based on a regression algorithm to obtain a prediction box with a framing accuracy higher than that of the proposed box;

[0061] A processing module, configured to process the feature map according to the extremely disaster-stricken area framed by the prediction box to generate an extremely disaster-stricken area object mask;

[0062] An input module, configured to input the prediction box and the extremely disaster-stricken area object mask into a geoscience classifier, so that the geoscience classifier determines at least a digital surface model and thermal infrared data corresponding to the extremely disaster-stricken area based on the prediction box and the extremely disaster-stricken area object mask, and screens out a target prediction box and a target extremely disaster-stricken area object mask that meet geoscience conditions as an identification result for output.

[0063] Another embodiment of the present invention further provides an extremely disaster-stricken area identification device based on unmanned aerial vehicle data, including:

[0064] At least one processor; and,

[0065] A memory communicatively connected to the at least one processor; wherein,

[0066] The memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to implement the extremely disaster-stricken area identification method based on unmanned aerial vehicle data described in any one of the above embodiments.

[0067] Based on the disclosure of the above embodiments, it can be known that the beneficial effects of the embodiments of the present invention include realizing the rapid and effective identification of building earthquake damage in a large area through unmanned aerial vehicle remote sensing images and relevant geoscience knowledge without manual intervention by constructing a deep neural network model for automatically identifying extremely disaster-stricken areas, and providing an effective automated algorithm for the interpretation work of extremely disaster-stricken areas and a computer program for automatic extraction of extremely disaster-stricken areas. The identification method proposed in the embodiments of the present invention can optimize parameters for extremely disaster-stricken area extraction on the basis of existing image target recognition algorithms, and at the same time add geoscience knowledge to the extremely disaster-stricken area to construct a geoscience classification model, and the geoscience classification model realizes the accurate identification of the extremely disaster-stricken area, thereby greatly suppressing the problems of misclassification and misidentification of the current extremely disaster-stricken area by the image target automatic recognition algorithm.

[0068] Other features and advantages of the present application will be described in the subsequent specification, and in part will be obvious from the specification, or will be understood by implementing the present application. The objectives and other advantages of the present application can be achieved and obtained through the structures specifically pointed out in the written specification, claims, and drawings.

[0069] The technical solutions of the present application will be further described in detail below through the drawings and embodiments. Description of the Drawings

[0070] Figure 1 It is a schematic diagram of neighborhood convolution in the prior art.

[0071] Figure 2 It is a schematic diagram of a fully connected layer in the prior art.

[0072] Figure 3 It is a regression schematic diagram of the target positioning box in the prior art.

[0073] Figure 4 It is a schematic diagram of mask generation in the prior art.

[0074] Figure 5 It is a schematic diagram of the similarity between different targets in the prior art.

[0075] Figure 6 It is a flowchart of the method for identifying extremely disaster-stricken areas based on UAV data in an embodiment of the present invention.

[0076] Figure 7 It is an application flowchart of the method for identifying extremely disaster-stricken areas based on UAV data in an embodiment of the present invention.

[0077] Figure 8 It is a schematic structural diagram of the geoscience classifier in an embodiment of the present invention.

[0078] Figure 9 It is a process diagram of generating a feature map in an embodiment of the present invention.

[0079] Figure 10 It is a schematic diagram of the preselected box on the feature map in an embodiment of the present invention.

[0080] Figure 11 It is a schematic structural diagram of the intersection and union between preselected boxes in an embodiment of the present invention.

[0081] Figure 12 It is a structural block diagram of the device for identifying extremely disaster-stricken areas based on UAV data in an embodiment of the present invention. Detailed Embodiments

[0082] Next, specific embodiments of the present invention will be described in detail with reference to the drawings, but it is not a limitation of the present invention.

[0083] It should be understood that various modifications can be made to the embodiments disclosed herein. Therefore, the following description should not be regarded as limiting, but merely as an example of the embodiments. Those skilled in the art will think of other modifications within the scope and spirit of the present disclosure.

[0084] The accompanying drawings, which are included in and constitute a part of this specification, illustrate embodiments of the present disclosure and, together with the general description of the present disclosure given above and the detailed description of the embodiments given below, serve to explain the principles of the present disclosure.

[0085] These and other features of the present invention will become apparent from the following description of the preferred forms of the embodiments given as non - limiting examples with reference to the accompanying drawings.

[0086] It should also be understood that although the present invention has been described with reference to some specific examples, those skilled in the art can surely implement many other equivalent forms of the present invention, which have the features as described in the claims and thus are all within the scope of protection defined thereby.

[0087] When combined with the accompanying drawings, the above - mentioned and other aspects, features, and advantages of the present disclosure will become more apparent in view of the following detailed description.

[0088] Hereinafter, specific embodiments of the present disclosure will be described with reference to the accompanying drawings; however, it should be understood that the disclosed embodiments are merely examples of the present disclosure, which can be implemented in various ways. Well - known and / or repetitive functions and structures are not described in detail to avoid obscuring the present disclosure with unnecessary or redundant details. Therefore, the specific structural and functional details disclosed herein are not intended to be limiting, but merely serve as a basis for the claims and a representative basis for teaching those skilled in the art to use the present disclosure in substantially any suitable detailed structure in a variety of ways.

[0089] This specification may use the phrases "in one embodiment", "in another embodiment", "in yet another embodiment", or "in other embodiments", which may each refer to one or more of the same or different embodiments according to the present disclosure.

[0090] Next, embodiments of the present invention will be described in detail with reference to the accompanying drawings.

[0091] As Figure 6 and Figure 7 shown, an extremely disaster - stricken area recognition method based on UAV data provided by an embodiment of the present invention includes:

[0092] S101: Extract features from a remote - sensing image obtained by UAV shooting to obtain a feature map;

[0093] S102: Generate a series of preselected boxes based on the pixel points on the feature map, and divide the preselected boxes into foreground and background according to the prior identification boxes;

[0094] S103: Based on the preselected boxes and the feature map, combine the regression algorithm to calculate and determine the proposed boxes for framing the extremely disaster-stricken areas in the remote sensing image;

[0095] S104: Fine-tune the proposed boxes based on the regression algorithm to obtain prediction boxes with a framing accuracy higher than that of the proposed boxes;

[0096] S105: Process the feature map based on the extremely disaster-stricken areas framed by the prediction boxes to generate an extremely disaster-stricken area object mask;

[0097] S106: Input the prediction boxes and the extremely disaster-stricken area object mask into the geoscience classifier, so that the geoscience classifier determines at least the digital surface model and the thermal infrared data corresponding to the extremely disaster-stricken areas based on the prediction boxes and the extremely disaster-stricken area object mask, and screen out the target prediction boxes and the target extremely disaster-stricken area object mask that meet the geoscience conditions through the digital surface model and the thermal infrared data as the recognition results for output.

[0098] For example, this embodiment proposes a deep neural network model (system) for automatically identifying extremely disaster-stricken areas, which includes multiple network modules with different functions, such as feature extraction, regression network, geoscience classifier, etc., and is used as the execution subject to respectively execute each step of the above method. The deep neural network model for automatically identifying extremely disaster-stricken areas constructed based on this embodiment can realize the rapid and effective identification of building earthquake damage in large areas through unmanned aerial vehicle remote sensing images and relevant geoscience knowledge without manual intervention, and provide an effective automated algorithm for the interpretation work of extremely disaster-stricken areas and a computer program for automated extraction of extremely disaster-stricken areas. The recognition method proposed in this embodiment can optimize the parameters for the extraction of extremely disaster-stricken areas on the basis of the existing image target recognition algorithm, and at the same time add geoscience knowledge for extremely disaster-stricken areas to construct a geoscience classification model, and the geoscience classification model realizes the accurate recognition of extremely disaster-stricken areas, thereby greatly suppressing the problems of misclassification and misrecognition of the current extremely disaster-stricken areas by the image target automatic recognition algorithm.

[0099] That is to say, this embodiment realizes the integration of the geoscience classifier and the image recognition algorithm, comprehensively utilizes the image spectral information and the geoscience knowledge of extremely disaster-stricken areas, and greatly improves the accuracy of the image recognition algorithm in remote sensing and geoscience target recognition. At the same time, it realizes an automatic recognition algorithm for extremely disaster-stricken areas that integrates geoscience knowledge in the model, making the constructed recognition model greatly improve the interpretation efficiency compared with the traditional manual visual method, which is conducive to the smooth progress of earthquake relief work in a timely manner.

[0100] Furthermore, the method in this embodiment further includes:

[0101] S107: Identify or extract the temperature anomaly points in the target prediction box and the target extremely disaster-stricken area object mask based on the geoscience classifier.

[0102] The above temperature anomaly points can be, for example, whether there are injured people in the extremely disaster-stricken area, whether there is a fire, etc. The geoscience classifier in this embodiment can realize the judgment of the temperature anomaly points, and identify or extract them for the user's reference.

[0103] The target extremely disaster-stricken area object mask in this embodiment includes the extremely disaster-stricken area region and other regions in the remote sensing image. For example, this mask is the mask corresponding to the remote sensing image, which includes the extremely disaster-stricken area region.

[0104] When making the geoscience classifier determine at least the digital surface model and thermal infrared data of the corresponding extremely disaster-stricken area region based on the prediction box and the target extremely disaster-stricken area object mask, it includes:

[0105] Based on the geoscience classifier, combine the content selected by the prediction box and the content in the target extremely disaster-stricken area object mask to construct a digital surface model including the extremely disaster-stricken area region and the thermal infrared data of the corresponding extremely disaster-stricken area region.

[0106] For example, the geoscience classifier takes the content selected by the prediction box and the content in the target extremely disaster-stricken area object mask as the region of interest (ROI) of the input data. Then, based on the obtained content, it correspondingly constructs a panoramic digital surface model of the recorded scene in the mask, or can also only construct the digital surface model (DSM) of the extremely disaster-stricken area region. At the same time, based on the obtained content, the geoscience classifier can also construct the thermal infrared data of the extremely disaster-stricken area region, or the thermal infrared data corresponding to the whole mask panorama, and the specific is not unique. In addition, the thermal infrared data can also be obtained by remote sensing shooting by an unmanned aerial vehicle. In this way, the geoscience classifier only needs to construct the digital surface model.

[0107] Further, in this embodiment, when screening out the target prediction box and the target extremely disaster-stricken area object mask that meet the geoscience conditions from the digital surface model and the thermal infrared data as the recognition result output, it includes:

[0108] S109: Resample the digital surface model and the thermal infrared data to the target pixels and then compress them into column vectors;

[0109] S110: Process the column vectors by the fully connected network in the geoscience classifier;

[0110] S111: The classification module in the geoscience classifier at least judges whether the concavity and convexity of the three-dimensional space corresponding to the extremely disaster-stricken area region meet the geoscience conditions based on the processing result of the fully connected network, and at the same time judges whether the extremely disaster-stricken area region has temperature anomaly characteristic points;

[0111] S112: Screen the multiple prediction boxes and the target extremely disaster-stricken area object mask based on the judgment result to obtain the target prediction box and the target extremely disaster-stricken area object mask.

[0112] For example, as Figure 8 shown, after resampling the DSM (Digital Surface Model) and thermal infrared data processed by the geoscience classifier to 128 * 128 pixels, they are compressed into column vectors and input into the internal fully connected network for processing. Based on the processing results, it is determined whether the concavity and convexity of the extremely disaster-stricken area in the three-dimensional space meet the geoscience conditions. These geoscience conditions are formed based on the characteristics of the extremely disaster-stricken area and are used to identify the extremely disaster-stricken area. The geoscience conditions can be one or multiple, or for different types of extremely disaster-stricken areas, their geoscience conditions are different. For example, the geoscience conditions for extremely disaster-stricken areas on plains and hills are different. At the same time, based on the processing results, it can also be determined whether the extremely disaster-stricken area has temperature anomaly characteristics. As mentioned above, by determining the temperature anomaly, it is determined whether there is a fire in the extremely disaster-stricken area and whether there are a large number of casualties, etc. Then, based on the judgment results, the extremely disaster-stricken area target prediction boxes and extremely disaster-stricken area object masks that meet the geoscience conditions, etc., among the multiple prediction boxes and masks obtained previously are output as the final recognition results. At the same time, the target objects with temperature anomalies are key extracted or marked.

[0113] Further, in this embodiment, when extracting features from the remote sensing image obtained by drone shooting to obtain a feature map, it includes:

[0114] S113: Construct and train a feature extraction network for identifying the features of the extremely disaster-stricken area in the remote sensing image;

[0115] S114: Based on the feature extraction network, extract features from the remote sensing image obtained by drone shooting to obtain a feature map.

[0116] For example, in this embodiment, first construct a feature extraction network. Specifically, a set of pictures with a size of 600 * 600 pixels containing the extremely disaster-stricken area can be used as the training samples of the feature extraction network for the extremely disaster-stricken area, and based on this training sample, the convolution template parameters in the feature extraction network are trained. Then, through non-linear activation by activation functions (such as ReLU, Sigmoid, Tanh, etc.) to complete the training. At this time, a complete feature extraction network can be obtained. After that, the remote sensing image obtained by drone shooting can be input into the trained feature extraction network, and it extracts features to obtain a feature map. The specific process can refer to Figure 9 shown.

[0117] When the feature map is obtained, the extremely disaster-stricken area in the remote sensing image can be boxed and labeled by artificial or machine recognition methods, that is, marking a bounding box on the remote sensing image. Then, the model can calculate and determine the proposed box for boxing the extremely disaster-stricken area in the remote sensing image based on the preselected box and the feature map in combination with the regression algorithm. It specifically includes:

[0118] S115: Process the feature map based on the Feature Pyramid Network to generate multiple sub-feature maps with different levels, where the number of pixels used to describe the feature information in the sub-feature maps at different levels is different;

[0119] S116: Generate multiple preselection boxes with different sizes on the corresponding remote sensing image based on the sub-feature maps;

[0120] S117: Calculate the overlap degree between the preselection boxes corresponding to the same remote sensing image and the identification boxes;

[0121] S118: Determine the target preselection boxes that enclose the foreground area and whose enclosure accuracy meets the first threshold based on the overlap degree;

[0122] S119: In the case where it is determined that the target preselection box encloses the extremely disaster-stricken area, calculate the proposed box based on the regression algorithm for the target preselection box.

[0123] For example, use the identification box in the remote sensing image and the feature map extracted in the previous steps as the training samples of the regression network to obtain a regression network for processing the identification box and the feature map to generate the proposed box. Then, process the feature map according to the Feature Pyramid Network (FPN) to generate multiple, for example, five, sub-feature maps with different levels. The number of pixels used to describe the feature information in the sub-feature maps at different levels is different, that is, the intensity of the sub-feature maps used to describe the feature information is different. After that, based on the sub-feature maps at different levels, generate preselection boxes with a total of 15 different size combinations of [32, 64, 128, 256, 512] pixels and aspect ratios of [1:1, 2:1, 1:2] on the original remote sensing image at different strides. There are multiple preselection boxes on each feature map. Specifically, refer to Figure 10 As shown, for example, there are three preselection boxes on each feature map. The content enclosed by the multiple preselection boxes is not completely the same and has intersections. Further, calculate the overlap degree (as shown in Figure 11 Overlap degree = area of intersection / area of union) between all the generated preselection boxes and the identification boxes corresponding to the enclosed content respectively. Sort the overlap degrees based on the calculation results, and determine the first 128 or other number of preselection boxes as the preselection boxes containing the foreground content of the image. Then, further determine whether the target object, that is, whether the extremely disaster-stricken area, is included in the preselection box. If not, eliminate it. Finally, process and adjust the preselection boxes determined to contain the extremely disaster-stricken area through the regression algorithm to make it as close as possible to the identification box. The adjusted preselection box can be used as the proposed box.

[0124] Optionally, the method in this embodiment further includes:

[0125] S120: Determine the first preselected box that frames the foreground area containing the extremely disaster-stricken area as a positive sample based on the overlap degree, and determine the second preselected box that frames the background area as a negative sample.

[0126] S121: Use the content framed by the positive samples and negative samples as training data to train the target classifier architecture to obtain a background classifier that can directly classify positive and negative samples for the preselected boxes.

[0127] S122: Based on the background classifier, determine whether the preselected box frames the extremely disaster-stricken area.

[0128] For example, take the top 120, 128, etc. preselected boxes as positive samples (the content they frame is the foreground), and randomly select 128 non-overlapping preselected boxes of the same quantity as negative samples (the content framed by this preselected box is the background). Use the pixels in these multiple preselected boxes containing positive and negative samples as training data to train and obtain a background classifier. Based on this background classifier, the image framed by the preselected box can be classified and judged to determine whether it frames the foreground or the background, so as to determine whether the preselected box frames the extremely disaster-stricken area. The framed extremely disaster-stricken area can be a part of the framed area or the entire extremely disaster-stricken area, and it is not unique specifically. For example, framing the characteristic content indicating the extremely disaster-stricken area can also be considered as framing the extremely disaster-stricken area. At the same time, other area content can also be framed while framing the extremely disaster-stricken area. When applying, the framed content of multiple preselected boxes can be input into the background classifier to obtain the preselected box that frames the extremely disaster-stricken area.

[0129] Further, after generating the proposed boxes, since their framing accuracy still cannot meet the requirements, in this embodiment, the proposed boxes need to be fine-tuned based on the regression algorithm to obtain prediction boxes with higher framing accuracy than the proposed boxes. Specifically, it includes:

[0130] S123: Map the proposed box to the corresponding position on the feature map.

[0131] S124: Resample the extremely disaster-stricken area framed in the proposed box, compress it into a column vector, and then input it into the fully connected layer of the object segmentation network for classification of the extremely disaster-stricken area.

[0132] S125: Based on the classification result, the position of the proposed box on the matching feature map, and in combination with the regression algorithm, fine-tune the proposed box to obtain a prediction box with higher selection accuracy than the proposed box.

[0133] For example, after obtaining the proposed bounding boxes of the possible severely affected areas in each remote sensing image, the proposed bounding boxes can be mapped to the matching positions on the specific feature map through the region alignment layer in the classification regression and segmentation network. In the classification and regression branches of the above network, the content selected in the proposed bounding box, i.e., the selected feature map information, is resampled to a size of 7×7 pixels (the specific pixel values are not unique). Then, all the pixels are compressed into a column vector and input into the fully connected layer of the above network for classification of the severely affected areas. At the same time, the RPN network regressor is used again to finely tune the proposed bounding box to make it further approach the corresponding labeled bounding box, and finally the required prediction bounding box is formed.

[0134] In addition, in the mask branch of the above network, the feature map information included in the input proposed bounding box can be resampled to a size of 14×14 pixels, then upsampled to a size of 28×28 pixels, and then pixel-level classification of each category is realized through a fully convolutional network to obtain the severely affected area object mask. Of course, the pixel sizes of the above resampling can be changed and can be adjusted according to actual needs.

[0135] Compared with the general image target recognition method, the image data for target recognition and instance segmentation in the remote sensing field has more complex background information, which will interfere with the recognition of target objects, such as the severely affected areas; at the same time, due to reasons such as the system error of the remote sensing imaging sensor itself, the remote sensing image will also have factors such as distortion and salt-and-pepper noise that affect the manual visual recognition of the severely affected areas. The different illumination intensities of the remote sensing images caused by different imaging times of the sensor will also affect the judgment of visual interpretation workers; in addition, the severely affected areas are not normal ground features but transient by-products of extreme disasters. Identifying the severely affected areas can help government departments determine key rescue areas and provide important information support for the dispatch of rescue forces and materials. The disaster relief work pays great attention to efficiency, while the efficiency of manual visual interpretation is low. The deep learning method proposed in this embodiment can optimize the parameters in large-scale and high-precision remote sensing images to quickly and efficiently extract the information of the severely affected areas automatically, greatly improving the efficiency of identifying the severely affected areas and reducing the degree of manual participation.

[0136] Moreover, the regional attributes of the severely affected areas are complex. Most of them are areas where buildings are severely damaged after an earthquake, and secondary disasters such as fires may be triggered after the disaster. Only from the spectral characteristics of the remote sensing image, it is easy to be confused with construction waste and cannot be accurately distinguished in the existing CNN methods based on the image itself. Therefore, in this embodiment, the geoscience knowledge of the severely affected areas is integrated, and the misclassified target objects are excluded in the form of a geoscience classifier, and the target objects with abnormal temperature are extracted as the primary disaster relief areas. This method is an important extension of the existing deep learning-based image recognition technology for geographical target recognition in remote sensing.

[0137] Such as Figure 12As shown, another embodiment of the present invention also provides a device 100 for identifying extremely disaster-stricken areas based on drone data, comprising:

[0138] An extraction module is used to extract features from remote sensing images taken by drones to obtain feature maps;

[0139] A segmentation module, used to generate a series of pre-selected boxes based on the pixel points on the feature map, and to segment the pre-selected boxes into foreground and background according to the prior identification box;

[0140] A calculation module, used to calculate and determine a suggestion box for selecting the extremely disaster-stricken area in the remote sensing image based on the pre-selected box and the feature map in combination with a regression algorithm;

[0141] A fine-tuning module, used to fine-tune the suggestion box based on a regression algorithm to obtain a predicted box with a higher selection accuracy than the suggestion box;

[0142] A processing module, used for processing the feature map according to the extremely disaster area selected by the prediction box to generate an extremely disaster area object mask;

[0143] An input module is used to input the prediction frame and the extremely disaster area object mask into a geological classifier, so that the geological classifier at least determines the digital surface model and thermal infrared data corresponding to the extremely disaster area based on the prediction frame and the extremely disaster area object mask, and screens the target prediction frame and target extremely disaster area object mask that meet the geological conditions from the digital surface model and thermal infrared data as the recognition result output.

[0144] As an optional embodiment, it also includes:

[0145] The identification module is used to identify or extract the temperature anomaly points existing in the target prediction frame and the target extreme disaster area object mask according to the geoscience classifier.

[0146] As an optional embodiment, the extremely disaster area object mask includes the extremely disaster area and other areas in the remote sensing image;

[0147] The method of causing the geoscience classifier to at least determine a digital surface model and thermal infrared data corresponding to the extremely disaster area based on the prediction frame and the extremely disaster area object mask comprises:

[0148] Based on the geoscience classifier, a digital surface model including the extremely disaster area and thermal infrared data corresponding to the extremely disaster area are constructed based on the content selected in the prediction box and the content in the extremely disaster area object mask.

[0149] As an optional embodiment, the method of screening the digital surface model and the thermal infrared data to obtain a target prediction frame and a target extreme disaster area object mask that meet the geological conditions as the recognition result output includes:

[0150] Resample the digital terrain model and thermal infrared data to target pixels and then compress them into column vectors;

[0151] Process the column vectors by the fully connected network in the geoscience classifier;

[0152] Based on the processing results of the fully connected network, the classification module in the geoscience classifier determines at least whether the concavity and convexity of the three-dimensional space corresponding to the extremely disaster-stricken area meet the geoscience conditions, and at the same time determines whether there are temperature anomaly feature points in the extremely disaster-stricken area;

[0153] Based on the judgment results, screen multiple prediction frames and extremely disaster-stricken area object masks to obtain the target prediction frame and target extremely disaster-stricken area object mask.

[0154] As an alternative embodiment, the feature extraction of the remote sensing image obtained by drone shooting to obtain a feature map includes:

[0155] Construct and train a feature extraction network for identifying the features of the extremely disaster-stricken area in the remote sensing image;

[0156] Based on the feature extraction network, perform feature extraction on the remote sensing image obtained by drone shooting to obtain a feature map.

[0157] As an alternative embodiment, the calculation and determination of the proposed box for framing the extremely disaster-stricken area in the remote sensing image based on the preselected box and the feature map includes:

[0158] Process the feature map based on the feature pyramid network to generate multiple sub-feature maps with different levels, and the number of pixels for describing feature information in the sub-feature maps with different levels is different;

[0159] Generate multiple preselected boxes with different sizes on the corresponding remote sensing image based on the sub-feature maps;

[0160] Calculate the overlap degree between the preselected boxes and the identification boxes corresponding to the same remote sensing image;

[0161] Based on the overlap degree, determine the target preselected box that frames the foreground area and whose framing accuracy meets the first threshold;

[0162] When it is determined that the target preselected box frames the extremely disaster-stricken area, calculate the proposed box based on the regression algorithm for the target preselected box.

[0163] As an alternative embodiment, the device further includes:

[0164] A determination module, configured to determine, according to the overlapping degree, a first pre-selection box enclosing a foreground area as a positive sample, and determine a second pre-selection box enclosing a background area as a negative sample;

[0165] A training module, configured to use the content enclosed by the positive sample and the negative sample as training data to train a target classifier architecture, so as to obtain a background classifier capable of directly classifying positive and negative samples for the pre-selection box.

[0166] As an optional embodiment, the fine-tuning of the proposed box based on a regression algorithm to obtain a prediction box with a higher box selection accuracy than the proposed box includes:

[0167] Mapping the proposed box to a matching feature map position;

[0168] Resampling the extremely disaster-stricken area enclosed by the proposed box, compressing it into a column vector, and inputting it into the fully connected layer of the object segmentation network for classification of the extremely disaster-stricken area;

[0169] Based on the classification result, the position of the proposed box on the matching feature map, and in combination with the regression algorithm, fine-tuning the proposed box to obtain a prediction box with a higher selection accuracy than the proposed box.

[0170] Furthermore, an embodiment of the present invention further provides an extremely disaster-stricken area recognition device based on UAV data, including:

[0171] At least one processor; and,

[0172] A memory communicatively connected to the at least one processor; wherein,

[0173] The memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to implement the extremely disaster-stricken area recognition method based on UAV data as described in any one of the above embodiments.

[0174] Furthermore, an embodiment of the present invention further provides a storage medium, on which a computer program is stored, and when the program is executed by a processor, it implements the extremely disaster-stricken area recognition method based on UAV data as described above. It should be understood that each solution in this embodiment has the corresponding technical effects in the above method embodiment, and will not be elaborated here.

[0175] Furthermore, an embodiment of the present invention further provides a computer program product, the computer program product is tangibly stored on a computer-readable medium and includes computer-readable instructions, and when the computer-executable instructions are executed, at least one processor is caused to execute the extremely disaster-stricken area recognition method based on UAV data such as in the above embodiments.

[0176] It should be noted that the computer storage medium of the present application can be a computer-readable signal medium, a computer-readable storage medium, or any combination of the two. A computer-readable medium can, for example, but is not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination of the above. More specific examples of the computer-readable storage medium can include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access storage medium (RAM), a read-only storage medium (ROM), an erasable programmable read-only storage medium (EPROM or flash memory), an optical fiber, a portable compact disk read-only storage medium (CD-ROM), an optical storage medium, a magnetic storage medium, or any suitable combination of the above. In the present application, the computer-readable storage medium can be any tangible medium that contains or stores a program, which can be used by or in conjunction with an instruction execution system, apparatus, or device. And in the present application, the computer-readable signal medium can include a data signal propagated in a baseband or as part of a carrier wave, which carries the computer-readable program code. Such a propagated data signal can take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. The computer-readable signal medium can also be any computer-readable medium other than the computer-readable storage medium, which can send, propagate, or transmit a program configured to be used by or in conjunction with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium can be transmitted by any appropriate medium, including but not limited to: wireless, antenna, optical cable, RF, etc., or any suitable combination of the above.

[0177] In addition, those skilled in the art should understand that the embodiments of the present application can be provided as a method, a system, or a computer program product. Therefore, the present application can take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware aspects. Moreover, the present application can take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage and optical storage, etc.) that contain computer-usable program code.

[0178] The present application is described with reference to the flowcharts and / or block diagrams of methods, apparatuses (systems), and computer program products according to the embodiments of the present application. It should be understood that each flow and / or block in the flowchart and / or block diagram, and the combination of flows and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to the processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing devices to generate a machine, such that the instructions executed by the processor of the computer or other programmable data processing devices generate means for implementing the processes Figure 1One or more processes and / or boxes Figure 1 A system for the functions specified in one box or more boxes.

[0179] These computer program instructions can also be stored in a computer-readable memory that can direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer-readable memory produce a manufactured article including an instruction system, and the instruction system implements the processes Figure 1 One or more processes and / or boxes Figure 1 The functions specified in one box or more boxes.

[0180] These computer program instructions can also be loaded onto a computer or other programmable data processing device, so that a series of operation steps are executed on the computer or other programmable device to produce a computer-implemented process. Thus, the instructions executed on the computer or other programmable device provide for implementing the processes Figure 1 One or more processes and / or boxes Figure 1 The steps of the functions specified in one box or more boxes.

[0181] Obviously, those skilled in the art can make various changes and modifications to this application without departing from the spirit and scope of this application. Thus, if these modifications and variations of this application fall within the scope of the claims of this application and their equivalent technologies, this application is also intended to include these changes and modifications.

[0182] The above embodiments are only exemplary embodiments of the present invention and are not used to limit the present invention. The protection scope of the present invention is defined by the claims. Those skilled in the art can make various modifications or equivalent replacements within the essence and protection scope of the present invention, and such modifications or equivalent replacements should also be regarded as falling within the protection scope of the present invention.

Claims

1. A method for identifying extremely disaster-stricken areas based on UAV data, characterized in that, Including: Performing feature extraction on a remote sensing image obtained by drone shooting to obtain a feature map; Generating a series of preselection boxes based on pixel points on the feature map, and dividing the foreground and background of the preselection boxes according to prior identification boxes; Calculating and determining a proposal box for framing the extremely disaster-stricken area in the remote sensing image based on the preselection boxes and the feature map in combination with a regression algorithm; Fine-tuning the proposal box based on the regression algorithm to obtain a prediction box with a framing accuracy higher than that of the proposal box; Processing the feature map based on the extremely disaster-stricken area framed by the prediction box to generate an extremely disaster-stricken area object mask; Inputting the prediction box and the extremely disaster-stricken area object mask into a geoscience classifier, so that the geoscience classifier determines at least a digital surface model and thermal infrared data corresponding to the extremely disaster-stricken area based on the prediction box and the extremely disaster-stricken area object mask, and screening out a target prediction box and a target extremely disaster-stricken area object mask that meet geoscience conditions from the digital surface model and the thermal infrared data as an identification result for output.

2. The method for identifying extremely disaster-stricken areas based on UAV data according to claim 1, wherein Further including: Identifying or extracting temperature anomaly points existing in the target prediction box and the target extremely disaster-stricken area object mask based on the geoscience classifier.

3. The method for identifying extremely disaster-stricken areas based on UAV data according to claim 1, wherein, The extremely disaster-stricken area object mask includes the extremely disaster-stricken area and other areas in the remote sensing image; The step of enabling the geoscience classifier to determine at least a digital surface model and thermal infrared data corresponding to the extremely disaster-stricken area based on the prediction box and the extremely disaster-stricken area object mask includes: Constructing a digital surface model including the extremely disaster-stricken area and thermal infrared data corresponding to the extremely disaster-stricken area based on the content framed by the prediction box combined with the geoscience classifier and the content in the extremely disaster-stricken area object mask.

4. The method for identifying extremely disaster-stricken areas based on UAV data according to claim 1, wherein, The step of screening out a target prediction box and a target extremely disaster-stricken area object mask that meet geoscience conditions from the digital surface model and the thermal infrared data as an identification result for output includes: Resampling the digital surface model and the thermal infrared data to target pixels and then compressing them into column vectors; Processing the column vectors by a fully connected network in the geoscience classifier; Based on the processing results of the fully connected network, the classification module in the geoscience classifier determines at least whether the concavity and convexity of the three-dimensional space corresponding to the extremely disaster-stricken area meet the geoscience conditions, and at the same time determines whether the extremely disaster-stricken area has temperature anomaly characteristic points; Screening multiple prediction boxes and extremely disaster-stricken area object masks based on the judgment results to obtain the target prediction box and the target extremely disaster-stricken area object mask.

5. The method for identifying extremely disaster-stricken areas based on UAV data according to claim 1, wherein, The step of performing feature extraction on a remote sensing image obtained by drone shooting to obtain a feature map includes: Constructing and training a feature extraction network for identifying extremely disaster-stricken area features in the remote sensing image; Performing feature extraction on a remote sensing image obtained by drone shooting based on the feature extraction network to obtain a feature map.

6. The method for identifying extremely disaster-stricken areas based on UAV data according to claim 1, wherein The step of calculating and determining a proposal box for framing the extremely disaster-stricken area in the remote sensing image based on the preselection boxes and the feature map in combination with a regression algorithm includes: Processing the feature map based on a feature pyramid network to generate multiple sub-feature maps with different levels, and the number of pixels for describing feature information in the sub-feature maps at different levels is different; Generate multiple preselected boxes of different sizes on the corresponding remote sensing image based on the sub-feature map; Calculate the overlap degree of the preselected boxes and the identification boxes corresponding to the same remote sensing image; Determine a target preselected box that frames a foreground area and whose framing accuracy meets the first threshold based on the overlap degree; In the case where it is determined that the target preselected box frames the extremely disaster-stricken area, calculate the proposed box based on the regression algorithm for the target preselected box.

7. The method for identifying an extremely disaster-stricken area based on UAV data according to claim 6, wherein The method further includes: Determine a first preselected box that frames a foreground area as a positive sample based on the overlap degree, and determine a second preselected box that frames a background area as a negative sample, where the foreground area includes the extremely disaster-stricken area; Train a target classifier architecture using the content framed by the positive samples and negative samples as training data to obtain a background classifier that can directly classify positive and negative samples for the preselected boxes.

8. The method for identifying extremely disaster-stricken areas based on UAV data according to claim 1, wherein, The fine-tuning of the proposed box based on the regression algorithm to obtain a prediction box with a framing accuracy higher than that of the proposed box includes: Map the proposed box to the corresponding position on the feature map; Resample the extremely disaster-stricken area framed in the proposed box, compress it into a column vector, and input it into the fully connected layer of the object segmentation network for classification of the extremely disaster-stricken area; Based on the classification result, the position of the proposed box on the matching feature map, and at the same time combine the regression algorithm to fine-tune the proposed box to obtain a prediction box with a selection accuracy higher than that of the proposed box.

9. An extremely disaster-stricken area recognition device based on drone data, characterized in that, Includes: An extraction module for extracting features from a remote sensing image obtained by drone shooting to obtain a feature map; A division module for generating a series of preselected boxes based on the pixel points on the feature map and dividing the preselected boxes into foreground and background according to the prior identification box; A calculation module for calculating and determining a proposed box for framing the extremely disaster-stricken area in the remote sensing image based on the preselected box and the feature map in combination with the regression algorithm; A fine-tuning module for fine-tuning the proposed box based on the regression algorithm to obtain a prediction box with a framing accuracy higher than that of the proposed box; A processing module for processing the feature map according to the extremely disaster-stricken area framed by the prediction box to generate an extremely disaster-stricken area object mask; An input module for inputting the prediction box and the extremely disaster-stricken area object mask into a geoscience classifier, so that the geoscience classifier determines at least the digital surface model and thermal infrared data corresponding to the extremely disaster-stricken area based on the prediction box and the extremely disaster-stricken area object mask, and filters out the target prediction box and the target extremely disaster-stricken area object mask that meet the geoscience conditions as the recognition result for output.

10. An extremely disaster-stricken area recognition device based on drone data, characterized in that, Includes: At least one processor; And, A memory communicatively connected to the at least one processor; wherein, The memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to implement the method for identifying extremely disaster-stricken areas based on drone data as described in any one of claims 1-8.

Images