Method and system for emergency monitoring of structures based on remote sensing images

Abstract

The present invention belongs to the field of target detection technology and relates to a method and system for emergency monitoring of structures based on remote sensing images. The method comprises: acquiring high-resolution remote sensing images to construct a structure dataset, and dividing the structure dataset into a training set and a test set; constructing a structure scene classifier based on target distribution priors and training it using the training set to obtain a structure scene classifier; constructing a structure rotation target detector based on scene context awareness and training it using the training set to obtain a structure rotation target detector; inputting the test set into the structure scene classifier to locate the image containing the structure scene, and inputting the image containing the structure scene into the structure rotation target detector to extract the structure target and generate a structure mapping result; and fusing the structure mapping result with the disaster mapping result generated by the structure emergency monitoring module to obtain a structure damage mapping result. The present invention can quickly and accurately implement emergency monitoring of structures.

Description

Technical Field

[0001] The present invention belongs to the technical field of target detection, and in particular relates to a method and system for emergency monitoring of structures based on remote sensing images. Background Art

[0002] Remote sensing technology, with its wide-area coverage and high-frequency revisit observation capabilities, has laid a clear and easily identifiable data foundation for structure observation. The use of multi-source remote sensing images for rapid and accurate structure extraction and emergency monitoring can provide spatial distribution statistics of disaster-stricken structures to assist in disaster relief. Structure emergency monitoring is a type of structure mapping problem under conditions of limited computing resources. The key difficulties that need to be overcome include rapid structure extraction under limited resource conditions and accurate structure distinction in dense scenes. Most of the earliest rapid structure extraction methods were based on exponential methods, which are difficult to effectively generalize to different research areas under the complex background of remote sensing. In recent years, the emergence of deep learning target detection technology has effectively alleviated this problem. Its main idea is to crop the large image and perform slice-by-slice reasoning on the cropped image slices, extract the structure mapping results of each image slice, and then merge them into the large image results. For example, a one-dimensional convolutional neural network is introduced for structure extraction, achieving good generalization; for another example, a high-density biased sampler is used to realize dense structure target detection, effectively improving the detection accuracy of dense structures.

[0003] While the aforementioned existing technologies have improved the generalizability of dense structure mapping, they still face challenges in rapidly extracting structures with limited resources and accurately distinguishing structures in dense scenes. Therefore, to address the challenges of rapid and accurate structure mapping in emergency scenarios, a novel structure extraction framework is urgently needed to enable structure mapping and emergency monitoring. Summary of the Invention

[0004] Based on the above-mentioned shortcomings and deficiencies in the prior art, one of the objects of the present invention is to at least solve one or more of the above-mentioned problems in the prior art. In other words, one of the objects of the present invention is to provide a remote sensing image-based emergency monitoring method and system for structures that meets one or more of the above-mentioned needs.

[0005] In order to achieve the above-mentioned object of the invention, the present invention adopts the following technical solutions:

[0006] A method for emergency monitoring of structures based on remote sensing images, characterized by comprising the following steps:

[0007] S1. Collect high-resolution remote sensing images to construct a structure dataset, and divide the structure dataset into a training set and a test set;

[0008] S2. Construct a structure scene classifier based on target distribution prior and train it using the training set. The obtained structure scene classifier is used to locate the structure scene;

[0009] A structure rotation target detector based on scene context perception is constructed and trained using a training set. The obtained structure rotation target detector is used to extract structure targets.

[0010] S3. Input the test set into the structure scene classifier to locate the image where the structure scene is located. The image where the structure scene is located is input into the structure rotation target detector to extract the structure target and generate the structure mapping result; the structure mapping result is integrated with the disaster mapping result generated by the structure emergency monitoring module to obtain the structure disaster mapping result.

[0011] As a preferred solution, in step S2, the construction and training process of the structure rotation target detector includes:

[0012] Step 2a, constructing a building rotation target detector based on scene context perception;

[0013] Deep features are extracted through the convolutional neural network ResNet and the feature pyramid network FPN, and the deep features are input into the region prediction network RPN to generate prediction boxes. The deep features and prediction boxes are then input into the scene context-aware feature extraction module to generate fused features by connecting the foreground target features with the scene context features in the channel dimension. The fused features are used to predict classification results and location information.

[0014] Step 2b: training a building rotation target detector based on scene context perception;

[0015] Input the training set into the structure rotation target detector constructed in step 2a to generate the predicted classification results and predicted position information, and calculate the loss function based on the true label. Train until convergence.

[0016] As a preferred solution, the scene context-aware feature extraction module includes a foreground target feature extraction branch and a scene context feature extraction branch. For the foreground target feature extraction branch, the deep features are combined with the prediction box to obtain the features of the area where it is located, and the rotation box learner of the encoding-decoding structure is input to generate a predicted rotation box for perceiving the foreground target features; for the scene context feature extraction branch, the features of the area where the prediction box is located are generated by combining the deep features and the prediction box, which are used to perceive the background context features; finally, the features output by the two branches are connected in the channel dimension to generate the final fusion feature.

[0017] As a preferred solution, the loss function includes a classification loss function and a position loss function. The classification loss function is based on the predicted classification result and is calculated by the cross entropy loss function; the position loss function is based on the predicted position information and is calculated by the Smooth L1 loss function.

[0018] As a preferred solution, the position information includes a rotation angle, horizontal and vertical coordinates, length and width.

[0019] As a preferred solution, in step S2, the construction and training process of the building scene classifier includes:

[0020] Step S21: constructing pre-training parameters for structure target detection as target distribution priors;

[0021] The training parameters of the convolutional neural network ResNet in the structure rotation target detector are used as the pre-training parameters for structure target detection;

[0022] Step 22: Construct a building scene classifier;

[0023] Deep features are extracted through the convolutional neural network ResNet, which is then fed into a fully connected layer classifier to output a two-dimensional classification vector, with the two dimensions representing the building scene and background categories respectively. A threshold segmentation module is connected after the fully connected layer classifier to determine the final predicted classification result by setting a confidence threshold.

[0024] Step 23: Train the building scene classifier based on the pre-trained parameters of building target detection, input the training set into the building scene classifier constructed in step S22, generate the predicted classification results, and calculate the cross entropy loss function based on the classification results and the true labels, and optimize the network parameters by the gradient descent method until convergence.

[0025] As a preferred solution, step S1 specifically includes the following steps:

[0026] S11. Use drones for aerial photography or high-resolution satellites to collect high-spatial-resolution remote sensing images;

[0027] S12. Perform structure scene classification and structure target extraction on high spatial resolution remote sensing images to construct a structure dataset; wherein the structure scene classification is divided into two types: structure scene and background, where the background is without structures;

[0028] S13. Divide the structure dataset into a training set and a test set.

[0029] As a preferred solution, the step S11 further includes: slicing the high spatial resolution remote sensing image to obtain image slices.

[0030] As a preferred solution, the structure emergency monitoring module is an earth observation satellite, and the corresponding disaster mapping result is a water body distribution map with elevation information.

[0031] The present invention further provides a structure emergency monitoring system based on remote sensing images, which applies the structure emergency monitoring method described in any of the above solutions, and the structure emergency monitoring system includes:

[0032] Acquisition module, used to collect high-resolution remote sensing images;

[0033] A construction module is used to construct a building dataset and divide the building dataset into a training set and a test set; it is also used to construct a building scene classifier based on target distribution prior; it is also used to construct a building rotation target detector based on scene context perception;

[0034] A training module, used to train a building scene classifier and a building rotation target detector using a training set;

[0035] The monitoring module is used to input the test set into the structure scene classifier to locate the image where the structure scene is located, and is also used to input the image where the structure scene is located into the structure rotation target detector to extract the structure target and generate the structure mapping result; it is also used to fuse the structure mapping result with the disaster mapping result generated by the structure emergency monitoring module to obtain the structure disaster mapping result.

[0036] Compared with the prior art, the present invention has the following beneficial effects:

[0037] The remote sensing image-based structure emergency monitoring method and system of the present invention can quickly and accurately implement structure emergency monitoring. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] Figure 1 This is a processing flow chart of a structure scene classifier based on target distribution prior according to embodiment 1 of the present invention;

[0039] Figure 2 This is a processing flow chart of a structure rotating target detector based on scene context perception according to embodiment 1 of the present invention;

[0040] Figure 3 This is a flow chart of the structure emergency monitoring method according to Example 1 of the present invention. DETAILED DESCRIPTION

[0041] To more clearly illustrate the embodiments of the present invention, specific embodiments of the present invention will be described below with reference to the accompanying drawings. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings and other embodiments can be obtained based on these drawings without inventive efforts.

[0042] Example 1: The remote sensing image-based structure emergency monitoring method of this embodiment includes the following steps:

[0043] Step 1: Collect high-resolution remote sensing images to construct a structure dataset, and divide the structure dataset into a training set and a test set;

[0044] Specifically, drone aerial photography or high-resolution satellite series photography is used to obtain the required number of high-spatial-resolution images, and the high-spatial-resolution remote sensing images are sliced to obtain image slices of 512*512 pixels. The greenhouse structures in the farmland plots in the image slices are used as samples, and the image slices with greenhouse structures (i.e., positive samples) and the image slices without greenhouse structures (i.e., negative samples) are used as structure datasets. The positive samples are labeled, and the labeled greenhouse structure position information includes the horizontal coordinate x, the vertical coordinate y, the length h, the width w, and the rotation angle angle (the angle relative to the horizontal direction), such as Figure 1 As shown; finally, the structure dataset is divided into two parts: training set and test set.

[0045] As an example, the training set of this embodiment includes 1,290 satellite earth images, which serve as image slices for greenhouse structure scene classification, with 44,176 greenhouse structure targets annotated for greenhouse structure target extraction; the test set and validation set are 430 and 862 images respectively, containing 14,237 and 32,018 greenhouse structure targets respectively; the dataset contains images and annotations from six provinces and regions across the country.

[0046] Step 2: Build and train a scene context-aware rotating object detector to accurately extract rotating greenhouse structure targets in dense scenes. Figure 1 As shown, the specific process includes the following steps:

[0047] Step 2.1: Build a scene context-aware rotating object detector for structures. Extract deep features using a convolutional neural network (ResNet) and a feature pyramid network (FPN). Then, feed the deep features into a subnetwork consisting of multiple convolutional layers, the region prediction network (RPN), to generate a prediction box (i.e., a horizontal box). The deep features and the prediction box are then fed into a scene context-aware feature extraction module to generate a predicted rotation box that best fits the greenhouse structure within the prediction box, as well as fused features of the region where the predicted rotation box is located. The fused features are specifically obtained by concatenating foreground object features and scene context features along the channel dimension.

[0048] The fused features of this embodiment are used for two purposes, one for predicting the classification result, and the other for predicting the position information to further optimize the position accuracy of the rotation box.

[0049] The scene context-aware feature extraction module generates a predicted rotation box and a fused feature containing foreground object information and scene context information by inputting depth features and prediction boxes. Specifically, the scene context-aware feature extraction module is divided into two branches: a foreground object feature extraction branch and a scene context feature extraction branch.

[0050] For the foreground target feature extraction branch, the deep features are combined with the predicted box to obtain the features of the area where it is located, and this feature is input into a rotation box learner with an encoding-decoding structure to generate a predicted rotation box. The rotation box learner converts the input feature dimension into five dimensions through a fully connected layer, representing the horizontal coordinate, vertical coordinate, length, width, and rotation angle offset respectively. The five-dimensional features are input into a decoder to generate the five position information of the predicted rotation box; the deep features and the predicted rotation box are combined to generate the features within the predicted rotation box area, which are used to perceive the information of the foreground greenhouse structure and obtain the foreground target features.

[0051] For the scene context feature extraction branch, the features of the area where the prediction box is located are generated by combining the deep features and the prediction box to perceive the background context information;

[0052] Finally, the features output by the two branches are connected in the channel dimension to generate the final fusion features;

[0053] Step 2.2: Train a scene context-aware structure rotation object detector.

[0054] The above training set is input into the scene context-aware structure rotation target detector to generate the predicted classification results and predicted position information, and the loss function is calculated in combination with the true label. Among them, the loss function includes the classification loss function and the position loss function. The classification loss function is based on the predicted classification results and is calculated using the cross entropy loss function; the position loss function is based on the predicted position information and is calculated using the Smooth L1 loss function. The expression is:

[0055] ;

[0056] in, Represents the predicted position information of the input, including five position variables: horizontal and vertical coordinates, length and width, and rotation angle.

[0057] Step 3: Construct and train a building scene classifier based on target distribution prior for rapid building scene positioning, such as Figure 2 As shown, the specific process includes the following steps:

[0058] Step 3.1: Construct pre-training parameters for building target detection as target distribution priors;

[0059] Specifically, the network parameters of the convolutional neural network ResNet after the structure rotation target detector training in step 2 are directly used as the pre-training parameters for structure target detection, that is, the training set is input into the structure rotation target detector based on scene context perception, and the structure rotation target detector based on scene context perception is trained until convergence; the target detector after training convergence can be used to determine whether there is a greenhouse structure target in the image slice, that is, to determine whether there is a target in the candidate box result. This feature can be used to assist in the classification of greenhouse structure scenes, where the image slice with the greenhouse structure target is the greenhouse structure scene, and the image slice without the greenhouse structure target is the background; and this feature relies on the automatic learning mechanism of deep learning, and the information is mainly stored in the convolutional neural network ResNet, so the network parameters of the convolutional neural network ResNet are used as pre-training parameters;

[0060] Step 3.2: Construct a building scene classifier;

[0061] Specifically, deep features are extracted through a convolutional neural network (ResNet). These deep features are then fed into a fully connected layer classifier, which outputs a two-dimensional classification vector. The two dimensions represent the greenhouse structure scene and the background, respectively. A threshold segmentation module is then added after the fully connected layer classifier to determine the final predicted classification result by setting a specific confidence threshold.

[0062] Step 3.3: Train the building scene classifier;

[0063] Specifically, based on the pre-trained parameters of structure target detection, the structure scene classifier is trained through fine-tuning, the training set is input into the constructed structure scene classifier to generate predicted classification results, and the cross-entropy loss function is calculated based on the classification results and the true labels. The parameters are optimized by the gradient descent method; through multiple rounds of iterative training, the optimal network parameters are gradually approached until the parameters converge.

[0064] Step 4: Use the target distribution prior-based structure scene classifier and the scene context-aware structure rotation target detector after training convergence to predict the test set, obtain the greenhouse structure mapping results, and integrate the disaster mapping results generated by the structure emergency monitoring module to obtain the structure disaster mapping results to realize structure emergency monitoring.

[0065] This embodiment is specifically described by taking the damage analysis of greenhouse structures in Shouguang City during the flood season as an example. Figure 3 As shown in the figure, the structure emergency monitoring module uses Sentinel-1 and Google Earth satellites. Sentinel-1 acquires synthetic aperture radar (SAR) imagery, both before and during the flood season. Water bodies are extracted using a water index threshold method to generate a map of newly added water bodies in Shouguang City during the flood season. Combined with Google Earth elevation data, a kriging interpolation method is used to generate a 6.7m resolution digital elevation model of Shouguang City. By combining this map with the digital elevation model, a three-dimensional information fusion is performed to generate a map of newly added water bodies that incorporates terrain elevation, known as the disaster mapping result. This disaster mapping result is then fused with the greenhouse structure mapping results to generate a structure damage mapping result.

[0066] The following target detection comparison is performed on the structure scene classifier and structure rotation target detector of this embodiment 1 with the existing target detection models RetinaNet, Faster R-CNN, and Oriented R-CNN. The results are shown in Table 1. The target detection of Example 1 is improved by 21.28%, 6.88% and 1.09% in the mean average precision (mAP) index compared with the existing target detection models RetinaNet, Faster R-CNN, and Oriented R-CNN, respectively, and achieves the best performance of 15.1fps in inference efficiency, which can quickly and accurately realize emergency monitoring of structures.

[0067] Table 1 Results of different target detection methods

[0068] Object detection model mAP (%) Inference efficiency (fps) RetinaNet 22.32 12.2 Faster R-CNN 36.72 12.5 Oriented R-CNN 42.51 12.5 Example 1 43.60 15.1 .

[0069] Based on the above-mentioned structure emergency monitoring method of remote sensing images, the structure emergency monitoring system of this embodiment includes the following functional modules: an acquisition module, a construction module, a training module and a monitoring module;

[0070] The above acquisition module is used to acquire high-resolution remote sensing images;

[0071] The above-mentioned building module is used to construct a building dataset and divide the building dataset into a training set and a test set; it is also used to construct a building scene classifier based on target distribution prior; it is also used to construct a building rotation target detector based on scene context perception;

[0072] The training module is used to train the building scene classifier and the building rotation target detector using the training set;

[0073] The monitoring module is used to input the test set into the structure scene classifier to locate the image of the structure scene, and is also used to input the image of the structure scene into the structure rotation target detector to extract the structure target and generate a structure mapping result; it is also used to fuse the structure mapping result with the disaster mapping result generated by the structure emergency monitoring module to obtain a structure disaster mapping result;

[0074] The specific processing process of the above functional modules can be referred to the detailed description in the above structure emergency monitoring method, which will not be repeated here.

[0075] The above description is only a detailed description of the preferred embodiments and principles of the present invention. For ordinary technicians in this field, based on the ideas provided by the present invention, there may be changes in the specific implementation methods, and these changes should also be considered as the scope of protection of the present invention.

Claims

1. A method for emergency monitoring of structures based on remote sensing images, characterized in that: The following steps are involved: S1. Collect high-resolution remote sensing images to construct a structure dataset, and divide the structure dataset into a training set and a test set; S2. Construct a structure scene classifier based on target distribution prior and train it using the training set. The obtained structure scene classifier is used to locate the structure scene; A structure rotation target detector based on scene context perception is constructed and trained using a training set. The obtained structure rotation target detector is used to extract structure targets. S3. Input the test set into the structure scene classifier to locate the image where the structure scene is located. The image where the structure scene is located is input into the structure rotation target detector to extract the structure target and generate a structure mapping result. The structure mapping result is integrated with the disaster mapping result generated by the structure emergency monitoring module to obtain a structure disaster mapping result. In step S2, the construction and training process of the structure rotation target detector includes: Step 2a, constructing a building rotation target detector based on scene context perception; Deep features are extracted through the convolutional neural network ResNet and the feature pyramid network FPN, and the deep features are input into the region prediction network RPN to generate prediction boxes. The deep features and prediction boxes are then input into the scene context-aware feature extraction module to generate fused features by connecting the foreground target features with the scene context features in the channel dimension. The fused features are used to predict classification results and location information. Step 2b: training a building rotation target detector based on scene context perception; Input the training set into the structure rotation target detector constructed in step 2a to generate the predicted classification results and predicted position information, and calculate the loss function based on the true labels. Train until convergence; The scene context-aware feature extraction module includes a foreground target feature extraction branch and a scene context feature extraction branch. For the foreground target feature extraction branch, the deep features are combined with the prediction box to obtain the features of the area where it is located, and the predicted rotation box is input into the rotation box learner of the encoding-decoding structure to generate a predicted rotation box for perceiving the foreground target features; for the scene context feature extraction branch, the features of the area where the prediction box is located are generated by combining the deep features and the prediction box, which are used to perceive the background context features; finally, the features output by the two branches are connected in the channel dimension to generate the final fusion feature.

2. The remote sensing image structure emergency monitoring method according to claim 1, characterized in that: The loss function includes a classification loss function and a position loss function. The classification loss function is based on the predicted classification result and is calculated by the cross entropy loss function; the position loss function is based on the predicted position information and is calculated by the Smooth L1 loss function.

3. The remote sensing image structure emergency monitoring method according to claim 1, characterized in that: The position information includes a rotation angle, horizontal and vertical coordinates, length and width.

4. The remote sensing image structure emergency monitoring method according to claim 1, characterized in that: In step S2, the construction and training process of the building scene classifier includes: Step S21: constructing pre-training parameters for structure target detection as target distribution priors; The training parameters of the convolutional neural network ResNet in the structure rotation target detector are used as the pre-training parameters for structure target detection; Step 22: Construct a building scene classifier; Deep features are extracted through the convolutional neural network ResNet, which is then fed into a fully connected layer classifier to output a two-dimensional classification vector, with the two dimensions representing the building scene and background categories respectively. A threshold segmentation module is connected after the fully connected layer classifier to determine the final predicted classification result by setting a confidence threshold. Step 23: Train the building scene classifier based on the pre-trained parameters of building target detection, input the training set into the building scene classifier constructed in step S22, generate the predicted classification results, and calculate the cross entropy loss function based on the classification results and the true labels, and optimize the network parameters by the gradient descent method until convergence.

5. The remote sensing image structure emergency monitoring method according to any one of claims 1 to 4, characterized in that: The step S1 specifically includes the following steps: S11. Use drones for aerial photography or high-resolution satellites to collect high-spatial-resolution remote sensing images; S12. Perform structure scene classification and structure target extraction on high spatial resolution remote sensing images to construct a structure dataset; wherein the structure scene classification is divided into two types: structure scene and background, where the background is without structures; S13. Divide the structure dataset into a training set and a test set.

6. The remote sensing image structure emergency monitoring method according to claim 5, characterized in that: The step S11 further includes: slicing the high spatial resolution remote sensing image to obtain image slices.

7. The remote sensing image structure emergency monitoring method according to claim 5, characterized in that: The structure emergency monitoring module is an earth observation satellite, and the corresponding disaster mapping result is a water body distribution map with elevation information.

8. A structure emergency monitoring system based on remote sensing images, applying the structure emergency monitoring method according to any one of claims 1 to 7, characterized in that: The structure emergency monitoring system includes: Acquisition module, used to collect high-resolution remote sensing images; A construction module is used to construct a building dataset and divide the building dataset into a training set and a test set; it is also used to construct a building scene classifier based on target distribution prior; it is also used to construct a building rotation target detector based on scene context perception; A training module, used to train a building scene classifier and a building rotation target detector using a training set; The monitoring module is used to input the test set into the structure scene classifier to locate the image where the structure scene is located, and is also used to input the image where the structure scene is located into the structure rotation target detector to extract the structure target and generate the structure mapping result; it is also used to fuse the structure mapping result with the disaster mapping result generated by the structure emergency monitoring module to obtain the structure disaster mapping result.

Images