A remote sensing disaster scene mapping method based on hierarchical dense architecture search

Abstract

The present application relates to a kind of remote sensing disaster scene mapping method based on hierarchical dense architecture search, for complex multi-disaster scene ground cover mapping and damage assessment, assist disaster emergency response and decision-making.Design efficient hierarchical dense architecture search framework, search stage combines existing various depth encoders, adaptive optimization multi-scale dense decoder architecture parameters, to meet the efficient fusion of various disaster features of current scene, through breadth-first decoding optimal decoding architecture;Training stage is realized by migrating depth encoder model parameters, jointly search decoding structure, efficient model training.The present application can solve the problem of complex disaster scene network architecture design, time cost is expensive and other problems, without artificial design depth interpretation architecture, in the face of various disasters or other complex scene can realize the efficient full automation of data to interpretation result.

Description

Technical Field

[0001] This invention belongs to the field of high-resolution remote sensing image recognition, and specifically relates to a remote sensing disaster scene mapping method based on hierarchical dense architecture search. Background Technology

[0002] High spatial resolution remote sensing imagery provides rich surface details, enabling us to observe the Earth's environment more clearly. To quickly and accurately obtain mapping results for features of interest, numerous studies over the past few decades have proposed various methods. Traditional methods focus on the design of handcrafted features and classifiers. However, these methods rely on human expertise and exhibit low mapping accuracy and weak transferability when faced with massive amounts of data and complex disaster scenarios. Therefore, advanced data-driven algorithms have been extensively studied, represented by various deep fully convolutional neural networks and their variants, and applied to various remote sensing image semantic segmentation and mapping tasks. Although deep learning networks can automatically extract and classify features through end-to-end supervision, the network design remains manual. The complex design process of deep learning networks requires extensive interdisciplinary knowledge, such as computer vision and remote sensing interpretation. Furthermore, due to the large amounts of remote sensing data and model parameters, the adjustment and validation cycle of manually designed network architectures is often very long, failing to meet the rapid mapping needs in disaster scenarios.

[0003] Neural architecture search can automatically design network architectures in a data-driven manner, reducing the need for specialized knowledge. By encoding the architecture with learnable parameters, neural architecture search utilizes specific strategies (gradient descent, reinforcement learning, evolutionary computation, etc.) to optimize the parameters. However, existing architecture search methods suffer from the following problems in disaster scene mapping tasks: 1) They focus on searching encoder architectures, and the search space grows exponentially with the depth of the layers. The huge computational overhead is accompanied by expensive time costs. Therefore, only shallow networks can be searched, or the network can be deepened by repeatedly stacking basic search units. This limits the performance of the model and makes it unsuitable for situations with complex post-disaster terrain features. 2) Their neglect of the decoder leads to weak spatial detail recovery, making it difficult to accurately recall terrain feature details and weak targets in disaster scenes. Summary of the Invention

[0004] To address these issues, we propose a remote sensing disaster scene mapping method based on hierarchical dense architecture search, shifting the focus to the search of the adaptive decoder. By utilizing and fixing existing deep encoder structures, we improve search efficiency and model generalization. This method aims to automatically design the decoder to adapt to existing encoders while generating a set of remote sensing pre-trained weights. This approach enhances network search efficiency, ensures network depth and robustness, and strengthens the decoder's ability to interpret ground feature details. Without requiring manual design of the deep interpretation architecture, it achieves efficient and fully automated data-to-interpretation results for various disasters or other complex scenarios, enabling timely responses to emergency needs.

[0005] The purpose of this invention is to propose a remote sensing disaster scene mapping method based on hierarchical dense architecture search. Given the current mature deep coding architecture, the focus is on decoding architecture search. To efficiently and automatically search remote sensing networks, we designed a cascaded process including hierarchical dense architecture search and model training based on encoder weight transfer. By fixing the encoder architecture, hierarchical dense search based on a large-scale remote sensing dataset can be completed within one day. This framework is compatible with different types of encoders and has high scalability and versatility. After finding the optimal architecture, the model training based on encoder weight transfer is initialized using the encoder weights from the search phase and retrained for downstream disaster scene mapping tasks, thereby achieving transfer for specific mapping tasks.

[0006] This invention provides a remote sensing disaster scene mapping method based on hierarchical dense architecture search, the implementation steps of which are as follows:

[0007] Step 1: Construct a land cover classification image sample library for specific disaster scenarios. Collect high-resolution remote sensing images of disasters such as floods and hurricanes, divide them into training set, validation set and test set according to proportion, normalize the data, and perform data augmentation on the training set.

[0008] Step 2: Construct a hierarchical dense search space for decoding architecture search. The hierarchical dense search space includes a densely connected hierarchical search space and a multi-scale operator set search space. By combining existing deep coding models, a differentiable search for the decoding architecture is achieved on the training set. The multi-scale fusion module is then used to fuse the multi-scale features output by the decoding architecture to obtain the final mapping probability output.

[0009] Step 3: Model training based on encoder weight transfer. Using the decoder architecture parameters searched in Step 2, the optimal decoding architecture is decoded using the breadth-first search algorithm. Combined with the encoder model weights, the model is retrained on a specific disaster task.

[0010] Step 4: Based on the trained encoder-decoder model, perform land cover mapping and damage assessment on unlabeled large-scale disaster scenarios, and predict classification probabilities to obtain mapping results.

[0011] Furthermore, the specific implementation of step 1 includes the following sub-steps:

[0012] Step 1.1: Based on typical natural disasters such as hurricanes, floods, earthquakes, and volcanic eruptions, collect high-resolution remote sensing images of the disaster-stricken areas using drones or satellite platforms;

[0013] Step 1.2: Assess the damage level of the features of interest after the disaster and label them at the pixel level, including but not limited to the following categories: ruins, water bodies, undamaged buildings, moderately damaged buildings, highly damaged buildings, completely destroyed buildings, vehicles, ships, roads, trees, swimming pools, sand, etc.

[0014] Step 1.3: Crops all images and labeled masks, and divides them into training, validation, and test sets. Normalizes all image data, and augments the training set using random flipping, random color perturbation, and random scaling.

[0015] Furthermore, the specific implementation of step 2 includes the following sub-steps:

[0016] Step 2.1: Design a densely connected hierarchical search space to find the optimal feature spatial resolution transfer path in the decoding architecture. Existing deep learning encoders (such as ResNet, Swin-Transformer, etc.) all have feature outputs at four scales: {c1, c2, c3, c4}, with spatial resolutions of {1 / 4, 1 / 8, 1 / 16, 1 / 32} of the original image, respectively. Therefore, how to include the optimal interaction path for features at various scales is the main requirement considered in the design of this space. We design a densely connected hierarchical search space that covers all paths for feature interactions at various scales. The first layer of this space can be represented as: Where α is an architecture parameter, representing the weights or connection strength of the previous layer output. This represents a multi-scale operator fusion strategy, with four nodes at different scales in each layer. For the second or subsequent layers l, the node connection expression formula is as follows:

[0017]

[0018] Where l≥2 represents the depth of the current layer in the search space, and α is normalized based on the softmax function. n i The feature output representing the previous layer node {n i|i=5,6,7,...,4l}, where i represents the output number of the previous layer.

[0019] Step 2.2: Design a multi-scale operator set search space where the feature space resolution remains unchanged. This space is used to search for the micro-level, i.e., the optimal fusion strategy of multi-scale operators in the nodes. The operator selection and fusion strategy for each node are independent, combined from the set of available operators: global average pooling, 3×3 separable convolution, 5×5 separable convolution, and 7×7 separable convolution. Therefore, the search space within a given node is represented as follows:

[0020]

[0021] Where β represents the importance of each operator, x represents the node input feature, and O j This refers to the specific operation operator (such as global average pooling), where j represents the number of the optional search operator. (Appendix) Figure 1 O in p A 3×3 separable convolution is used to preprocess the input features, achieving feature dimensionality reduction. R(·) represents post-processing of the multi-scale fused features, consisting of a 3×3 separable convolution, a normalization layer, and a ReLU activation function.

[0022] Step 2.3: Design a multi-scale fusion module to fuse the multi-scale outputs of the decoding architecture. The outputs {d1, d2, d3, d4} of the decoding architecture are also divided into four scales, which are {1 / 4, 1 / 8, 1 / 16, 1 / 32} of the original image, respectively. For the outputs at the four scales, different numbers of upsampling modules U are used respectively. ×2 (·) Perform a scale transformation. U ×2 (·) contains a 3×3 convolution, a normalized layer, a ReLU activation function, and an optional ×2 bilinear upsampling function. Therefore, {d1, d2, d3, d4} are each composed of one U... ×1 (·) (No upsampling), 1 U ×2 (·), 2 U ×2 (·), 3 U ×2 The image is processed by (·) to unify the scale to 1 / 4 of the original image. Then, the processed features of the same scale are added and fused. Finally, the original image resolution is restored using a ×4 bilinear upsampling function. Then, a 1×1 convolution is used to reduce the channel dimensionality to the number of classes. The segmentation map probability is output using SoftMax.

[0023] Step 2.4: In the densely connected hierarchical search space, all weights and network architecture parameters α and β are differentiable, enabling end-to-end optimization using the gradient descent algorithm.

[0024] Furthermore, the specific implementation of step 3 includes the following sub-steps:

[0025] Step 3.1: Based on the searched architecture parameters α and β, firstly, using the breadth-first search algorithm, based on the connection strength represented by α, retain the c connections with the highest strength in the input of each node, thereby finding the optimal feature space resolution transfer path; based on the importance of the operators represented by β, retain the p most important operators inside each node for feature fusion.

[0026] Step 3.2: Using the found decoding architecture and the encoder from the search process, transfer the network weights obtained during the search process for initialization, and retrain on the training set; the loss function is: L ce = -ylog(p), where y is the training true label and p is the network's predicted output probability.

[0027] Furthermore, for the normalized remote sensing images of the test set, they are input into the retrained and converged network model to obtain the output classification probability. The classification result is obtained by taking the maximum value, thus obtaining the land cover and assessment mapping results for the disaster scene. Faced with new scenes and new datasets, they can be re-searched and retrained; similarly, the existing searched network architecture and training parameters can be transferred to the new dataset for fine-tuning and testing.

[0028] The method of this invention has the following significant effects: (1) By retaining the existing excellent deep encoder architecture and focusing on the search of the decoding architecture, it can simultaneously improve network search efficiency, ensure network depth and robustness, and enhance the decoder's ability to interpret details of ground features; (2) Model training based on encoder weight transfer can reuse existing remote sensing pre-trained parameters, accelerate network model convergence, shorten the training process, and improve the efficiency of downstream task applications. Furthermore, the obtained series of remote sensing network model parameters and network architectures have high generalization and are suitable for various new scenarios and new mapping tasks. Attached Figure Description

[0029] Figure 1 This is the remote sensing disaster scene mapping method designed in steps 2 and 3 of Embodiment 1 of the present invention, which is based on hierarchical dense architecture search.

[0030] Figure 2 This is a schematic diagram of the feature space resolution transfer path obtained in step 3 of embodiment 1 of the present invention, where the thickness of the lines represents their importance.

[0031] Figure 3 It is the result of the disaster mapping of ground features under the hurricane disaster scenario obtained by model reasoning in step 4 of embodiment 1 of the present invention. Detailed Implementation

[0032] The following examples further illustrate the outstanding features and significant advancements of the present invention, which are intended to illustrate the invention but are in no way limiting it.

[0033] Example 1

[0034] (i) The publicly available RescuNet imagery of the region after Hurricane Michael was collected by drones and used as a sample database of disaster scenes. It contains 4,494 3000×4000 images and corresponding semantic segmentation annotations.

[0035] 1.1 After Hurricane Michael, a Category 5 storm, struck the Gulf of Mexico in 2018, DJI drones were used to collect image data of the post-disaster scene in the Gulf of Mexico and surrounding areas by flying at low altitude.

[0036] 1.2 Damage rating is performed based on the features of interest after the disaster, and pixel-level labeling is performed, including 12 categories: ruins, water areas, undamaged buildings, moderately damaged buildings, highly damaged buildings, completely destroyed buildings, vehicles, ships, roads, trees, swimming pools, sand, etc.

[0037] Step 1.3: Crops all images and labeled masks, and divides them into a training set (3595 images), a validation set (449 images), and a test set (449 images). Normalizes all image data, and augments the training set using random flipping, random color perturbation, and random scaling.

[0038] (ii) A hierarchical dense search space is used for decoder architecture search to adapt to various existing deep encoder architectures. Specifically, by designing a densely connected hierarchical search space and a multi-scale operator set search space, the gradient descent algorithm is used to simultaneously optimize network weights and architecture parameters to achieve efficient decoder architecture search.

[0039] 2.1 A densely connected hierarchical search space is designed to find the optimal feature spatial resolution transfer path in the decoding architecture. Utilizing the existing deep learning encoder ResNet50, which has feature outputs at four scales: {c1, c2, c3, c4}, with spatial resolutions of {1 / 4, 1 / 8, 1 / 16, 1 / 32} of the original image, the primary requirement for this space design is to encompass the optimal interaction paths for features at various scales. We design a densely connected hierarchical search space that covers all paths for feature interactions at various scales. The first layer of this space can be represented as: Where α is an architecture parameter, representing the weights or connection strength of the previous layer output. This represents a multi-scale operator fusion strategy, with four nodes at different scales in each layer. For the second or subsequent layers l, the node connection expression formula is as follows:

[0040]

[0041] Where l≥1 represents the depth of the current layer in the search space, and α is normalized based on the softmax function. n i The feature output representing the previous layer node {n i |i=5,6,7,...,4l}, where i represents the output number of the previous layer. During the search process, the layer depth l=2.

[0042] 2.2 Design a multi-scale operator set search space where the feature space resolution remains unchanged. This space is used to search for the optimal fusion strategy of multi-scale operators at the micro-level, i.e., within a node. The operator selection and fusion strategy for each node are independent, and the following are chosen from the set: global average pooling, 3×3 separable convolution, 5×5 separable convolution, and 7×7 separable convolution. Therefore, the search space within a given node is represented as follows:

[0043]

[0044] Where β represents the importance of each operator, x represents the node input feature, and O j This refers to the specific operation operator (such as global average pooling), where j represents the number of the optional search operator. (Appendix) Figure 1 O in p A 3×3 separable convolution is used to preprocess the input features, achieving feature dimensionality reduction. R(·) represents post-processing of the multi-scale fused features, consisting of a 3×3 separable convolution, a normalization layer, and a ReLU activation function.

[0045] Step 2.3: Design a multi-scale fusion module to fuse the multi-scale outputs of the decoding architecture. The outputs {d1, d2, d3, d4} of the decoding architecture are also divided into four scales, which are {1 / 4, 1 / 8, 1 / 16, 1 / 32} of the original image, respectively. For the outputs at the four scales, different numbers of upsampling modules U are used respectively. ×2 (·) Perform a scale transformation. U ×2 (·) contains a 3×3 convolution, a normalized layer, a ReLU activation function, and an optional ×2 bilinear upsampling function. Therefore, {d1, d2, d3, d4} are each composed of one U... ×1 (·) (No upsampling), 1 U ×2 (·), 2 U ×2 (·), 3 U ×2The image is processed by (·) to unify the scale to 1 / 4 of the original image. Then, the processed features of the same scale are added and fused. Finally, the original image resolution is restored using a ×4 bilinear upsampling function. Then, a 1×1 convolution is used to reduce the channel dimensionality to the number of classes. The segmentation map probability is output using SoftMax.

[0046] 2.4 In the densely connected hierarchical search space, all weights and network architecture parameters α and β are differentiable, and end-to-end optimization is achieved using the gradient descent algorithm. During the search process, a stochastic gradient descent optimizer is used for optimization, with an initial learning rate of 0.01, a batch size of 16, and 60,000 search iterations.

[0047] (III) Model training based on encoder weight transfer: using the searched decoder architecture parameters, a breadth-first search algorithm is employed to decode the optimal decoding architecture. For example... Figure 2 As shown, for each node, we retain the three most important input connections. Numbering all nodes from top to bottom and left to right, c1, c2, c3, and c4 are 1, 2, 3, and 4 respectively. Therefore, the input encoding for the four nodes in the first layer is: l1 = [1,3,4], [1,3,4], [0,0,0], [0,0,0], where 0 represents no input. The input encoding for the four nodes in the second layer is: l2 = [1,3,4], [3,4,6], [1,3,4], [3,4,5]. Combining the encoder model weights, model retraining is implemented on the training set.

[0048] 3.1 Based on the searched architecture parameters α and β, the breadth-first search algorithm is first used to retain the three strongest connections in the input of each node based on the connection strength represented by α, thereby finding the optimal feature space resolution transfer path; based on the importance of the operators represented by β, the two most important operators in each node are retained for feature fusion.

[0049] 3.2 Utilizing the discovered decoding architecture, combined with the encoder from the search process, the network weights obtained during the search are transferred for initialization, and retraining is performed on the training set. The initial learning rate is set to 0.01, the batch size is 16, and the number of retraining iterations is 30000. The loss function is: L ce = -ylog(p), where y is the training true label and p is the network's predicted output probability.

[0050] (iv) For the normalized remote sensing images of the test set, input them into the retrained and converged network model to obtain the output classification probability. Take the maximum value to obtain the classification result, thus obtaining the disaster scene land cover and assessment mapping results, such as... Figure 3 .

[0051] The specific embodiments described herein are merely illustrative of the spirit of the invention. Those skilled in the art to which this invention pertains may make various modifications or additions to the described specific embodiments or use similar methods to substitute them, without departing from the spirit of the invention or exceeding the scope defined by the appended claims.

Claims

1. A remote sensing disaster scene mapping method based on hierarchical dense architecture search, characterized in that, Comprising the following steps: Step 1, constructing a disaster scene ground cover classification image sample library, collecting high-resolution remote sensing images after disaster strikes, dividing the training set, validation set and test set according to the proportion, and normalizing all image data, and data augmentation for the training set; Step 2, constructing a hierarchical dense search space for decoding architecture search, which includes a densely connected hierarchical search space and a multi-scale operator set search space; By combining existing excellent deep encoding models, the differentiable search of the decoding architecture is realized on the training set, and the multi-scale features output by the decoding architecture are fused by using the multi-scale fusion module to obtain the final mapping probability output; The specific implementation of step 2 includes the following sub-steps: Step 2.1, design a densely connected hierarchical search space, which is used to find the optimal feature space resolution transfer path in the decoding architecture; The existing deep learning encoders all have four scale feature outputs, respectively: The spatial resolutions are of the original image, respectively. The dense connection hierarchical search space covers all paths of various scale feature interactions, and the first layer connection is represented as: wherein is an architecture parameter, representing the weight or connection strength of the output of the previous layer, represents a multi-scale operator fusion strategy, and each layer has four nodes of different scales; for the second layer or more subsequent layers l , wherein the node connection expression formula is as follows: wherein represents the depth of the current layer in the search space, is normalized based on a softmax function, , characteristic output representing the preceding layer node , i preceding layer output number; Step 2.2, design a multi-scale operator set search space in which the feature space resolution remains unchanged, for searching the micro level, i.e. the optimal fusion strategy of multi-scale operators in nodes The operator selection and fusion strategy of each node are independent of each other, and are selected from a set of optional operators: global average pooling, 3x3 separable convolution, 5x5 separable convolution, and 7x7 separable convolution; Step 2.3, design a multi-scale fusion module to fuse the multi-scale outputs of the decoding architecture; the outputs of the decoding architecture Also divided into four scales, respectively, the original image , for the output of the four scales, respectively, with different numbers of up-sampling modules Scale transformation, contains 3x3 convolution, normalization layer, ReLU activation function and an optional x2 bilinear up-sampling function; therefore respectively by 1 , 1 , 2 , 3 Processing, the scale is unified to the original image ; then add the same scale features after processing to fuse, finally use x4 bilinear up-sampling function to restore to the original image resolution size, then use 1x1 convolution to reduce the channel to the class number, and realize the probability output of the segmentation map through SoftMax; Step 2.4, in the dense connected hierarchical search space, all the weights and network architecture parameters are differentiable, enabling end-to-end optimization using gradient descent algorithms; Step 3, model training based on encoder weight migration, using the decoder architecture parameters searched in step 2, decoding the optimal decoding architecture using the breadth-first algorithm, and combining the encoder model weight to realize model retraining on a specific disaster task; The specific implementation of step 3 includes the following sub-steps, Step 3.1, based on the searched architecture parameters and First, using the breadth-first search algorithm, based on The connection strength represented is retained by the one with the highest strength among the inputs of each node. c By establishing connections, the optimal feature space resolution transfer path can be found. based on The importance of the operator represented by the node is reserved for the most important p operators for feature fusion; Step 3.2, using the searched decoding architecture, combining the encoder in the search process, migrating the network weights obtained in the search process to initialize, retraining on the training set; the loss function is: where y is the training true label, p is the network prediction output probability; Step 4, based on the trained encoding-decoding model, ground cover mapping and damage assessment are performed on unmarked large-scale disaster scenes, and the classification probability is predicted to obtain the mapping result.

2. The remote sensing disaster scene mapping method based on hierarchical dense architecture search according to claim 1, characterized in that: The specific implementation of step 1 includes the following sub-steps, Step 1.1, according to the typical natural disasters: hurricane, flood, earthquake, volcanic eruption, use unmanned aerial vehicle or satellite platform to collect high-resolution remote sensing images of disaster area; Step 1.2, according to the damaged ground objects of interest, perform damage rating and pixel-level labeling, including the following categories: ruins, water area, undamaged building, moderately damaged building, severely damaged building, completely damaged building, vehicle, ship, road, tree, swimming pool, sand; Step 1.3, crop all images and label masks, and divide them into training, validation and test sets, normalize all image data, and use random flipping, random color disturbance and random scale to perform data augmentation on the training set.

3. The remote sensing disaster scene mapping method based on hierarchical dense architecture search according to claim 1, characterized in that: In step 2.2, the internal search space of a node is represented as follows: wherein to indicate the importance of each operator, x to indicate the node input features, is a specific operation operator, j represents the optional search operator number.

4. The remote sensing disaster scene mapping method based on hierarchical dense architecture search according to claim 1, characterized in that: For the normalized remote sensing image of the test set, input it into the retrained encoding-decoding model to get the output classification probability, get the classification result by taking the maximum value, get the disaster scene ground cover and evaluation mapping result. Facing new scenes and new data sets, the encoding-decoding model can be re-searched and trained, or the existing searched encoding-decoding model and training parameters can be migrated to the new data set for fine-tuning test.

Images