Superpixel-pixel multi-level fusion recognition method and system for multi-temporal remote sensing images

Through the superpixel-pixel multi-level fusion recognition method, combined with the three-branch structure of the encoder and decoder and the edge information fusion, the problem of local environmental interference in the recognition of the change region of the multi-time phase remote sensing image is solved, and the recognition accuracy and performance are improved.

CN118762263BActive Publication Date: 2024-11-08HUNAN UNIV
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202411247729.9
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2024-09-06
Publication Date
2024-11-08
Estimated Expiration
2044-09-06

AI Technical Summary

Technical Problem

The existing multi-time phase remote sensing image change area recognition method has failed to effectively reduce the interference of local environmental factors such as shadows, textures, and tree occlusion, affecting the recognition accuracy.

Method used

Using the superpixel-pixel multi-level fusion recognition method, the three-branch encoder and decoder can be constructed to achieve effective fusion between superpixels and pixels, and edge information is introduced for constraints, reducing interference from local environmental factors, and extracting superpixel levels. and pixel-level information difference.

Benefits of technology

The recognition accuracy of the changing areas of the multi-time phase remote sensing image is improved, the adhesion and edge expansion of the changing areas are reduced, the structural feature extraction of the changing areas is enhanced, and the recognition performance is improved.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118762263B_ABST
    Figure CN118762263B_ABST
Patent Text Reader

Abstract

The present invention discloses a multi-temporal remote sensing image super-pixel-pixel multi-level fusion recognition method and system. The present invention comprises a multi-temporal image T a and T b Perform superpixel segmentation and obtain superpixel level features F spa and F spb , and encode the multi-temporal images to obtain pixel-level features F pa and F pb ; Perform feature fusion and difference analysis to obtain super-pixel-level fusion coding feature F sp and pixel-level difference encoding feature F p ; for F sp and F p Decode and obtain the pixel-level change prediction results P p , super-pixel level change prediction result P sp and edge-guided fusion change prediction result P Fuse And fusion is used to obtain the final change area prediction result P. The present invention aims to achieve efficient fusion of superpixel and pixel-level features, reduce the interference of local environmental factors such as shadows, textures, and tree occlusions on change area detection, and improve the recognition accuracy of change areas in multi-temporal remote sensing images.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the technical field of remote sensing image processing, and in particular to a multi-temporal remote sensing image superpixel-pixel multi-level fusion recognition method and system. Background Art

[0002] The identification of changing areas in multi-temporal remote sensing images is a research direction that has attracted much attention in the field of remote sensing. It plays a key role in a wide range of fields such as urban planning, environmental monitoring, and disaster assessment. With the rapid advancement of urbanization and the continuous promotion of economic development, the changing areas in urban areas have shown a frequent and diverse trend. Therefore, accurate and efficient monitoring and analysis of changing areas are key issues that need to be addressed. As a precise remote sensing image analysis technology, the identification of changing areas in multi-temporal remote sensing images can capture subtle changes in surface cover and usage types by comparing remote sensing images at different time points, and has important application value and potential.

[0003] In recent years, the rapid development of deep learning technology has greatly improved the accuracy and efficiency of identifying the changed regions of multi-temporal remote sensing images, opening up broader prospects for remote sensing image analysis and application. Deep learning plays a vital role in the field of remote sensing image recognition. With the help of deep learning models such as convolutional neural networks, the deep features of images can be automatically extracted. Therefore, more and more deep learning algorithms are applied to the task of identifying the changed regions of multi-temporal remote sensing images, which has brought great impetus and development to this field. For example, Rodrigo Caye Daudt et al. proposed three convolutional neural network models, namely FC-EF, FC-Siam-conv and FC-Siam-diff, in the paper "Fully Convolutional Siamese Networks for Change Detection.2018 25th IEEEInternational Conference on Image Processing (ICIP). IEEE, 2018: 4063-4067." for the task of identifying the changed regions of multi-temporal remote sensing images. These models introduced the fully convolutional twin network architecture and realized the ability to automatically detect regional changes in remote sensing images using deep learning and convolutional neural network technology. By adopting a fully convolutional structure and deep feature extraction, these models can adapt to different types of remote sensing image data, improving the accuracy and adaptability of identifying the change area of ​​multi-temporal remote sensing images. The FC-EF, FC-Siam-conv and FC-Siam-diff models are each suitable for different remote sensing image scenarios, providing more targeted solutions for the identification of change areas in multi-temporal remote sensing images. In addition, Peng et al. proposed the MSOF network in "End-to-End Change Detection for High Resolution Satellite Images Using Improved UNet++. Remote Sensing, 2019, 11(11): 1382." MSOF is a multi-scale feature fusion method based on convolutional neural networks and optical flow estimation. It uses pyramid optical flow and improved optical flow equations to capture changes at different scales. It can simultaneously consider spatial and temporal information and improve accuracy in the identification of change areas in multi-temporal remote sensing images.Chen et al. proposed a model called STANet for identifying changed regions in multi-temporal remote sensing images in the paper “A Spatial-Temporal Attention-Based Method and a New Dataset for Remote Sensing Image Change Detection. Remote Sensing, 2020, 12(10): 1662.” STANet is a model for identifying changed regions in multi-temporal remote sensing images that combines a spatial-temporal attention mechanism with multi-layer feature fusion. It uses spatial and temporal attention mechanisms to distinguish changed and non-changed regions, and uses a multi-layer feature fusion method to utilize feature information of different scales to improve the accuracy of identifying changed regions in multi-temporal remote sensing images.

[0004] However, the shortcomings of the above existing methods are that they do not use effective means to reduce the interference of local environmental factors such as shadows, textures, and tree occlusions on the detection of changed areas, and do not effectively enhance the structural features of the changed areas. This results in the detection results being easily disturbed by environmental factors in the process of identifying the changed areas of multi-temporal remote sensing images, affecting the accuracy. Summary of the invention

[0005] Technical problem to be solved by the present invention: In view of the above-mentioned problems in the prior art, a method and system for superpixel-pixel multi-level fusion recognition of multi-temporal remote sensing images are provided. The present invention aims to achieve efficient fusion of complementary information of superpixel / pixel level images, improve phenomena such as adhesion of changed areas and expansion of edges of changed areas, reduce interference of local environmental factors such as shadows, textures, and tree occlusions on detection of changed areas, and improve the recognition accuracy of changed areas in multi-temporal remote sensing images.

[0006] In order to solve the above technical problems, the technical solution adopted by the present invention is:

[0007] A multi-temporal remote sensing image superpixel-pixel multi-level fusion recognition method, comprising:

[0008] S101: record two remote sensing images of the same scene taken at different times as pixel-level images T a and T b , for T a Perform superpixel segmentation to obtain superpixel level features F spa , for T b Perform superpixel segmentation to obtain superpixel level features F spb , F spa and F spb Concatenate along the channel dimension to obtain the concatenated feature F csp ;

[0009] S102, using the encoder to T a , T b and F csp Encoding includes: a Encoded as pixel-level features F pa ; T b Encoded as pixel-level features F pb ; F csp Encoded as super-pixel fusion encoding feature F sp ; Calculate F pa and F pb The pixel-level difference encoding feature F p ;

[0010] S103, using the decoder to decode F p and F sp Decoding includes: p Decoded into pixel-level decoding features E p , and then obtain the pixel-level change prediction result P through 1×1 convolution p ; F sp Decoded into super-pixel level decoding feature E sp , and then obtain the final super-pixel level change prediction result P through 1×1 convolution sp ; E p and E sp The intermediate features obtained by decoding are concatenated along the channel dimension and the decoder is used to obtain the multi-fusion decoding feature E. Fuse And the edge change prediction result P edge , the edge change prediction result P edge And the fusion decoding feature E Fuse After fusion, 1×1 convolution is performed to obtain the edge-guided fusion change prediction result P Fuse , P p , P sp and P Fuse Fusion obtains the prediction result P;

[0011] S104, using a Softmax classifier to obtain a change probability map of the prediction result P, and binarizing the change probability map to obtain a visualization image of the change region prediction result.

[0012] Optionally, the superpixel segmentation in step S101 includes firstly performing superpixel segmentation on the image to be processed using a preset superpixel segmentation method, and then performing a averaging operation on each superpixel block in the superpixel features obtained by the superpixel segmentation to obtain a final superpixel segmentation result image.

[0013] Optionally, the preset superpixel segmentation method is a simple linear iterative clustering algorithm SLIC, and when the simple linear iterative clustering algorithm SLIC is used to perform superpixel segmentation on the image to be processed, a function expression for generating the number of superpixels is defined as follows:

[0014] ,

[0015] In the above formula, is the number of superpixels, T a , T b The common image width for both, T a , T b The common image height of both, is the number of pixels in each superpixel defined.

[0016] Optionally, in step S102, F is calculated pa and F pb The pixel-level difference encoding feature F p The function expression is F p =F pa -F pb .

[0017] Optionally, before step S101, the method further includes the step of training a multi-temporal remote sensing image superpixel-pixel multi-level fusion recognition network composed of an encoder and a decoder, and the loss function used in the training is L Total for:

[0018] L Total = L Focal + L MSE ,

[0019] In the above formula, L Focal is the change area change loss, L MSE is the edge change loss, and:

[0020] ,

[0021] ,

[0022] In the above formula, is the number of training samples, is the weighting factor, is the focusing parameter, P n is the prediction result P corresponding to the nth training sample, is the true image label of the nth training sample, and the true image label value is 0 or 255; is the edge change prediction result corresponding to the nth training sample, is the true edge label corresponding to the nth training sample.

[0023] Optionally, the encoder in step S102 includes three encoding network modules with non-shared parameters, which are respectively used to convert T a Encoded as pixel-level features F pa , T b Encoded as pixel-level features F pb , and F csp Encoded as super-pixel fusion encoding feature F sp The coding network module includes multiple coding layers and is used to convert T a Encoded as pixel-level features F pa Each intermediate coding layer in the coding network module performs feature adaptive fusion according to the following formula:

[0024] ,

[0025] In the above formula, and is the weight parameter;

[0026] For T b Encoded as pixel-level features F pb Each intermediate coding layer in the coding network module performs feature adaptive fusion according to the following formula:

[0027] ,

[0028] In the above formula, and is the weight parameter;

[0029] For F csp Encoded as super-pixel fusion encoding feature F sp Each intermediate coding layer in the coding network module performs feature adaptive fusion according to the following formula:

[0030] ,

[0031] In the above formula, is the feature obtained by feature adaptive fusion, ~ is the weight parameter.

[0032] Optionally, the encoding network module is a Resnet18 network.

[0033] Optionally, the decoder comprises ap Decoded into pixel-level decoding features E p The pixel-level feature decoder is used to transform F sp Decoded into super-pixel level decoding feature E sp The super-pixel feature decoder is used to encode the pixel-level feature E p and super-pixel level decoding features E sp Decoded as fusion decoding feature E Fuse The pixel-level feature decoder and the super-pixel-level feature decoder both include seven cascaded feature decoding modules, each of which includes a 3×3 convolution layer, a batch normalization layer, and a linear correction unit connected in sequence, and the front end of the 3×3 convolution layer of the first five feature decoding modules is also provided with an upsampling module for performing a two-fold upsampling operation on the input feature image, and the input end of the first second to fifth feature decoding modules also inputs the multi-level coded feature fusion; E p and E sp The intermediate features obtained by decoding are concatenated along the channel dimension to obtain multi-level fusion features E Fuse It refers to splicing the features output by the seven feature decoding modules of both the pixel-level feature decoder and the super-pixel-level feature decoder along the channel dimension to obtain the multi-level fusion feature E Fuse The fusion feature decoder includes seven cascaded feature decoding modules, each of which includes a 3×3 convolutional layer, a batch normalization layer, and a linear correction unit connected in sequence. An upsampling module is set before the first four feature decoding modules to perform a double upsampling operation on the feature image, and the input end of the feature decoding module also adds the multi-level fusion feature E Fuse The fusion input, and the sixth feature decoding module outputs the edge decoding feature E edge , edge decoding feature E edge The edge change prediction result P is obtained through 1×1 convolution edge , the features output by the seventh feature decoding module and the edge change prediction result P edge After splicing along the channel dimension, 1×1 convolution is performed to obtain the edge-guided fusion change prediction result P Fuse .

[0034] In addition, the present invention also provides a multi-temporal remote sensing image superpixel-pixel multi-level fusion recognition system, including an interconnected microprocessor and a memory, wherein the microprocessor is programmed or configured to execute the multi-temporal remote sensing image superpixel-pixel multi-level fusion recognition method.

[0035] In addition, the present invention also provides a computer-readable storage medium, in which a computer program is stored, and the computer program is used to be programmed or configured by a microprocessor to execute the multi-temporal remote sensing image superpixel-pixel multi-level fusion recognition method.

[0036] Compared with the prior art, the present invention mainly has the following advantages:

[0037] 1. The present invention adopts a convolutional neural network with superpixel-pixel multi-level fusion, and realizes the effective fusion of superpixels and pixels by constructing a three-branch encoder and decoder. At the same time, edge information is introduced to constrain the fusion features, effectively improving the recognition performance of changing areas of multi-temporal remote sensing images.

[0038] 2. The present invention proposes a multi-level feature fusion encoder, which can effectively reduce the interference of local environmental factors on the changed area by introducing superpixels for guidance. At the same time, it realizes the necessary detail supplement of superpixels through effective fusion means, and jointly extracts superpixel-level information and pixel-level difference information, thereby improving the recognition performance of changed areas in multi-temporal remote sensing images.

[0039] 3. The present invention proposes an edge-guided multi-level information fusion decoder, which enables the superpixel-level-pixel level coding features to maintain their original advantages. At the same time, a fusion decoder is performed to obtain fused coding features, and edge information is introduced to reduce the diffusion of superpixels and the adhesion of changed areas. Finally, the change features are obtained through adaptive weighted fusion, which further improves the performance of identifying changed areas in multi-temporal remote sensing images. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] Figure 1 Schematic diagram of the basic flow of the method of the embodiment of the present invention.

[0041] Figure 2 Schematic diagram of the structure of the encoder in an embodiment of the present invention.

[0042] Figure 3 Schematic diagram of the structure of a decoder in an embodiment of the present invention.

[0043] Figure 4 Schematic diagram of the training principle of the network (SPMFCNN) composed of the encoder and decoder in an embodiment of the present invention. DETAILED DESCRIPTION

[0044] like Figure 1 As shown, the multi-temporal remote sensing image superpixel-pixel multi-level fusion recognition method of this embodiment includes:

[0045] S101: record two remote sensing images of the same scene taken at different times as pixel-level images T a and T b , for T a Perform superpixel segmentation to obtain superpixel level features F spa , for T b Perform superpixel segmentation to obtain superpixel level features F spb, F spa and F spb Concatenate along the channel dimension to obtain the concatenated feature F csp ;

[0046] S102, using the encoder to T a , T b and F csp Encoding includes: a Encoded as pixel-level features F pa ; T b Encoded as pixel-level features F pb ; F csp Encoded as super-pixel fusion encoding feature F sp ; Calculate F pa and F pb The pixel-level difference encoding feature F p ;

[0047] S103, using the decoder to decode F p and F sp Decoding includes: p Decoded into pixel-level decoding features E p , and then obtain the pixel-level change prediction result P through 1×1 convolution p ; F sp Decoded into super-pixel level decoding feature E sp , and then obtain the final super-pixel level change prediction result P through 1×1 convolution sp ; E p and E sp The intermediate features obtained by decoding are concatenated along the channel dimension and the decoder is used to obtain the multi-fusion decoding feature E. Fuse And the edge change prediction result P edge , the edge change prediction result P edge And the fusion decoding feature E Fuse After fusion, 1×1 convolution is performed to obtain the edge-guided fusion change prediction result P Fuse , P p , P sp and P Fuse Fusion obtains the prediction result P;

[0048] S104, using a Softmax classifier to obtain a change probability map of the prediction result P, and binarizing the change probability map to obtain a visualization image of the change region prediction result.

[0049] In order to distinguish the existing network model, in this embodiment, the network composed of the encoder and decoder is a superpixel-pixel multi-level fusion convolutional neural network (SPMFCNN). By constructing a three-branch encoder and decoder, the effective fusion of superpixels and pixels is realized, and the edge information is introduced to constrain the fusion features, which effectively improves the recognition performance of the change area of ​​the multi-temporal remote sensing image. In this embodiment, the encoder is named a multi-level feature fusion encoder (MFFE). By introducing superpixels for guidance, the interference of local environmental factors on the change area can be effectively reduced. At the same time, the necessary details of superpixels are supplemented by effective fusion means, and superpixel-level information and pixel-level difference information are jointly extracted, which improves the recognition performance of the change area of ​​the multi-temporal remote sensing image. The decoder is named an edge-guided multi-level information fusion decoder (EMFD), which can keep the original advantages of the superpixel-level-pixel level coding features, and at the same time, by fusing the decoder, the fusion coding features are obtained, and the edge information is introduced to reduce the diffusion of superpixels and the adhesion of the change area. Finally, the change prediction result is obtained by adaptive weighted fusion, which further improves the recognition performance of the change area of ​​the multi-temporal remote sensing image.

[0050] The superpixel segmentation performed in step S101 of this embodiment includes first using a preset superpixel segmentation method to perform superpixel segmentation on the image to be processed, and then performing a mean value operation in each superpixel block in the superpixel features obtained by the superpixel segmentation to obtain the final superpixel segmentation result. Superpixel segmentation is an existing method, so the preset superpixel segmentation method can adopt a known method as needed. For example, as an optional implementation, the preset superpixel segmentation method in this embodiment is a simple linear iterative clustering algorithm SLIC, and when the simple linear iterative clustering algorithm SLIC is used to perform superpixel segmentation on the image to be processed, the function expression for generating the number of superpixels is defined as follows:

[0051] ,

[0052] In the above formula, is the number of superpixels, T a , T b The common image width for both, T a , T b The common image height of both, is the number of pixels in each defined superpixel. In this embodiment, n_segments is set to 900, the superpixel compactness parameter in the simple linear iterative clustering algorithm SLIC is set to 30, and the pixel values ​​in each superpixel block are averaged, and the output format is: PNG.

[0053] It should be noted that the encoding network module of the encoder and the decoder can select a known encoding network as needed, for example, as an optional implementation, such as Figure 2 As shown, the encoding network module in this embodiment is a Resnet18 network. The Resnet18 network is composed of a series of modules connected in series, which can be divided into a 3×3 convolution module, a convolution module of five stages (stage 1 to stage 5) cascaded in sequence, an average pooling module and a fully connected module, wherein each of the convolution modules of the five stages cascaded in sequence includes four 3×3 convolution modules connected in sequence. The pixel-level extracted features of stage 1 and stage 5 of Resnet18 are directly input into the pixel-level feature decoder after only subtraction operations, and the super-pixel-level extracted features are kept as the original features and input into the super-pixel-level feature decoder. Figure 2 The 1st in represents the input of stage 1, and the subsequent result of resnet18 becomes the input, so they are represented by dotted and solid lines respectively. In stage 2, stage 3 and stage 4 of Resnet18, the super-pixel level feature F is fused on the super-pixel level encoder. sp With pixel-level features F pa 、F pb Perform adaptive weighted fusion to obtain super-pixel fusion coding feature F sp , and at the same time, the pixel-level encoding feature F is adaptively fused p Perform weight fusion and subtraction to obtain pixel-level difference encoding feature F p .

[0054] In step S102 of this embodiment, F is calculated pa and F pb The pixel-level difference encoding feature F p The function expression is F p =F pa -F pb The encoder in step S102 includes three encoding network modules with non-shared parameters, which are respectively used to convert T a Encoded as pixel-level features F pa , T b Encoded as pixel-level features F pb , and F csp Encoded as super-pixel fusion encoding feature F sp The coding network module includes multiple coding layers and is used to convert T a Encoded as pixel-level features F pa Each intermediate coding layer in the coding network module performs feature adaptive fusion according to the following formula:

[0055] ,

[0056] In the above formula, and is a weight parameter, which can be set as required. For example, in this embodiment , ;

[0057] For T b Encoded as pixel-level features F pb Each intermediate coding layer in the coding network module performs feature adaptive fusion according to the following formula:

[0058] ,

[0059] In the above formula, and is a weight parameter, which can be set as required. For example, in this embodiment , ;

[0060] For F csp Encoded as super-pixel fusion encoding feature F sp Each intermediate coding layer in the coding network module performs feature adaptive fusion according to the following formula ( Figure 2 In the expression f 1):

[0061] ,

[0062] In the above formula, is the feature obtained by feature adaptive fusion, ~ is a weight parameter, which can be set as required. For example, in this embodiment , , .

[0063] like Figure 3 As shown, the decoder in this embodiment includes a p Decoded into pixel-level decoding features E p The pixel-level feature decoder is used to transform F sp Decoded into super-pixel level decoding feature E sp The super-pixel feature decoder is used to encode the pixel-level feature E p and super-pixel level decoding features E sp Decoded as fusion decoding feature E FuseThe pixel-level feature decoder and the super-pixel-level feature decoder both include seven cascaded feature decoding modules, each of which includes a 3×3 convolution layer, a batch normalization layer, and a linear correction unit connected in sequence, and the front end of the 3×3 convolution layer of the first five feature decoding modules is also provided with an upsampling module for performing a two-fold upsampling operation on the input feature image, and the input end of the first second to fifth feature decoding modules also inputs the multi-level coded feature fusion; E p and E sp The intermediate features obtained by decoding are concatenated along the channel dimension to obtain multi-level fusion features E Fuse It refers to splicing the features output by the seven feature decoding modules of both the pixel-level feature decoder and the super-pixel-level feature decoder along the channel dimension to obtain the multi-level fusion feature E Fuse The fusion feature decoder includes seven cascaded feature decoding modules, each of which includes a 3×3 convolutional layer, a batch normalization layer, and a linear correction unit connected in sequence. An upsampling module is set before the first four feature decoding modules to perform a double upsampling operation on the feature image, and the input end of the feature decoding module also adds the multi-level fusion feature E Fuse The fusion input, and the sixth feature decoding module outputs the edge decoding feature E edge , edge decoding feature E edge The edge change prediction result P is obtained through 1×1 convolution edge , the features output by the seventh feature decoding module and the edge change prediction result P edge After splicing along the channel dimension, 1×1 convolution is performed to obtain the edge-guided fusion change prediction result P Fuse . Figure 3 In the figure, the ×5 at the bottom of the feature decoding module indicates that 5 feature decoding modules are the same, ×4 indicates that 4 feature decoding modules are the same, ×3 indicates that 3 feature decoding modules are the same, and ×2 indicates that 2 feature decoding modules are the same. For the sake of omission, only one is drawn as an illustration. nd It represents the sixth feature decoding module.

[0064] In step S103 of this embodiment, P p , P sp and P Fuse The function expression of the fusion prediction result P is:

[0065] ,

[0066] In the above formula, , as well as is a weight parameter, which can be used as a learning parameter of the multi-temporal remote sensing image superpixel-pixel multi-level fusion recognition network (SPMFCNN) composed of the encoder and the decoder. The initial value can be set to 1.

[0067] This embodiment also includes the step of training a multi-temporal remote sensing image superpixel-pixel multi-level fusion recognition network composed of an encoder and a decoder before step S101, and the loss function used in the training is L Total for:

[0068] L Total = L Focal + L MSE ,

[0069] In the above formula, L Focal is the change area change loss, L MSE is the edge change loss, and:

[0070] ,

[0071] ,

[0072] In the above formula, is the number of training samples, is the weighting factor, is the focusing parameter, P n is the prediction result P corresponding to the nth training sample, is the true image label of the nth training sample, and the true image label value is 0 or 255; is the edge change prediction result corresponding to the nth training sample, is the true edge label corresponding to the nth training sample. The weighting factor and focusing parameter can be set according to actual needs. For example, in this embodiment Set to 0.25, Set to 2.0.

[0073] like Figure 4 As shown in the figure, when training the multi-temporal remote sensing image superpixel-pixel multi-level fusion recognition network (SPMFCNN) composed of the encoder and decoder, the training parameters are updated by back propagation. The training is stopped after the entire network is trained for 100 epochs, and the weight file of the network model is saved according to the best accuracy obtained on the validation set. In the test phase, the best weight file is loaded, the test data is input, and the change probability map of the prediction result P is obtained by using the Softmax classifier. The probability Figure 2The predicted results of the changed area are visualized by quantization.

[0074] In order to verify the effectiveness of the method (SPMFCNN) in this embodiment, the network framework was trained and tested using the public datasets LEVIR-CD, WHU-CD, and GZ-CD, and compared with other deep learning-based methods for identifying regions of change in multi-temporal remote sensing images. The LEVIR-CD dataset contains 445 sets of training data, 64 sets of validation data, and 128 sets of test data. Each set of data contains two multi-temporal images, and the spatial size of each image is 1024. 1024, the original image spatial resolution is 0.5m. Due to the large image size, the LEVIR-CD dataset is cropped into non-overlapping images, and the spatial size of each image is 256 256, after cropping, a total of 7120 sets of training data, 1024 sets of validation data and 2048 sets of test data are generated. The WHU-CD dataset contains two sets of multi-temporal and spatial size 32207 The original image spatial resolution is 0.075m. Due to the large image size, they are cropped into non-overlapping images in the WHU-CD dataset. The spatial size of each image is 256. 256, after cropping, a total of 5205 sets of training data, 743 sets of validation data and 1486 sets of test data are generated. The GZ-CD dataset contains 19 sets of data of uneven size, with an original spatial resolution of 0.55m. Due to the large image size, the GZ-CD dataset is cropped into non-overlapping images, and the spatial size of each image is 256. 256, after cropping, a total of 808 sets of training data, 232 sets of verification data and 116 sets of test data are generated. The real edge labels of all data sets are generated by processing the real image labels by the Canny edge detection operator and stored as a single-channel grayscale image in PNG format. Optionally, in this embodiment, the Adam optimizer is used to optimize the network parameters, the batch size is set to 32, the learning rate is set to 1e-4, and the number of training times is 100.

[0075] In this embodiment, there are three objective evaluation indicators, namely: overall accuracy (OA), mean intersection over union (mIoU) and F1 score (F1). Based on the above indicators, the method of this embodiment (SPMFCNN) is compared with five existing methods for identifying changed areas of multi-temporal remote sensing images on different data sets. The FE-EF, FC-Siam-conc, and FC-Siam-diff methods are described in detail in the literature: Daudt RC, Le Saux B, Boulch A. Fully convolutionalsiamese networks for change detection[C] / / 2018 25th IEEE InternationalConference on Image Processing (ICIP). IEEE, 2018: 4063-4067, STANet in the literature: Chen H, Shi Z. A spatial-temporal attention-based method and a new datasetfor remote sensing image change detection[J]. Remote Sensing, 2020, 12(10):1662, and MSOF in the literature: Peng D, Zhang Y, Guan H. End-to-end change detection forhigh resolution satellite images using improved UNet++[J]. Remote Sensing,2019, 11(11): 1382. Finally, the specific results of the comparative experiment are shown in Table 1 below.

[0076] Table 1: Comparison of objective evaluation indicators of the test results of the method in this embodiment and the existing method on different data sets:

[0077]

[0078] As shown in Table 1, the method of this embodiment (SPMFCNN) is the best result in all three evaluation indicators, verifying the effectiveness of the algorithm proposed in this embodiment, wherein the F1 score is 1.40% higher than that of the STANet method on the LEVIR-CD dataset, 2.79% higher than that of the MSOF method on the WHU-CD dataset, and 3.33% higher than that of the STANet method on the GZ-CD dataset. Combining the experimental results of the method of this embodiment (SPMFCNN) and 5 existing methods for identifying the changed regions of multi-temporal remote sensing images, it is found that the method of this embodiment (SPMFCNN) has a high accuracy rate in identifying the changed regions in the detection results, and the changed regions have high recognition completeness, clearer edges, and the situation of misidentifying non-changed regions is greatly improved.

[0079] In order to further verify the effectiveness of each module in the algorithm proposed in this embodiment, two groups of ablation experiments are designed: w / o MFFE means that superpixel features are not used for encoding fusion, and only images are input to the network to perform the recognition task of the change area of ​​multi-temporal remote sensing images; w / o EMFD means that fusion decoding is not used, and only superpixel-level decoders and pixel-level decoders are used. Experiments are conducted on the LEVIR-CD dataset. There are 6 objective evaluation indicators, namely precision (P), recall rate (Rec), Kappa coefficient (Kappa), overall accuracy (OA), mean intersection over union (mIoU) and F1 score (F1). The final results are shown in Table 2.

[0080] Table 2: Comparison of objective evaluation indicators of ablation experiments in this embodiment:

[0081]

[0082] It can be seen from Table 2 that the method of this embodiment (SPMFCNN) achieves the best in recall rate (Rec), Kappa coefficient (Kappa), overall accuracy (OA), average intersection over union (mIoU) and F1 score (F1). Compared with the w / o MFFE method, all indicators of the method of this embodiment (SPMFCNN) are optimal. Compared with the w / o EMFD method, the method of this embodiment (SPMFCNN) improves the recall rate (Rec) by 1.07%, the Kappa coefficient (Kappa) by 0.33%, the overall accuracy (OA) by 0.02%, the average intersection over union (mIoU) by 0.28%, and the F1 score (F1) by 0.33%, and only reduces the accuracy (P) by 0.41%, which can effectively verify the effectiveness of the multi-level feature fusion encoder and edge-guided multi-level information fusion decoder proposed by the method of this embodiment (SPMFCNN).

[0083] In addition, this embodiment also provides a multi-temporal remote sensing image superpixel-pixel multi-level fusion recognition system, including a microprocessor and a memory connected to each other, and the microprocessor is programmed or configured to execute the multi-temporal remote sensing image superpixel-pixel multi-level fusion recognition method. In addition, this embodiment also provides a computer-readable storage medium, in which a computer program is stored, and the computer program is used to be programmed or configured by the microprocessor to execute the multi-temporal remote sensing image superpixel-pixel multi-level fusion recognition method.

[0084] Those skilled in the art should understand that the embodiments of the present application can be provided as methods, systems, or computer program products. Therefore, the present application can take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present application can take the form of a computer program product implemented on one or more computer-readable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code. The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the process Figure 1 A process or multiple processes and / or boxes Figure 1 These computer program instructions can also be stored in a computer-readable memory that can guide a computer or other programmable data processing device to work in a specific way, so that the instructions stored in the computer-readable memory produce a product including an instruction device, which implements the functions specified in the process. Figure 1 A process or multiple processes and / or boxes Figure 1 These computer program instructions can also be loaded onto a computer or other programmable data processing device, so that a series of operation steps are executed on the computer or other programmable device to produce a computer-implemented process, so that the instructions executed on the computer or other programmable device provide for implementing the process in the process. Figure 1 A process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0085] The above is only a preferred embodiment of the present invention, and the protection scope of the present invention is not limited to the above embodiments. All technical solutions under the concept of the present invention belong to the protection scope of the present invention. It should be pointed out that for ordinary technicians in this technical field, some improvements and modifications without departing from the principle of the present invention should also be regarded as the protection scope of the present invention.

Claims

1. A superpixel-pixel multi-level fusion recognition method for multi-temporal remote sensing images. It is characterized in that include: S101: record two remote sensing images of the same scene taken at different times as pixel-level images T a and T b , for T a Perform superpixel segmentation to obtain superpixel level features F spa , for T b Perform superpixel segmentation to obtain superpixel level features F spb , F spa and F spb Concatenate along the channel dimension to obtain the concatenated feature F csp ; S102, using the encoder to T a , T b and F csp Encoding includes: a Encoded as pixel-level features F pa ; T b Encoded as pixel-level features F pb ; F csp Encoded as super-pixel fusion encoding feature F sp ; Calculate F pa and F pb The pixel-level difference encoding feature F p ; S103, using the decoder to decode F p and F sp Decoding includes: p Decoded into pixel-level decoding features E p , and then obtain the pixel-level change prediction result P through 1×1 convolution p ; F sp Decoded into super-pixel level decoding feature E sp , and then obtain the final super-pixel level change prediction result P through 1×1 convolution sp ; E p and E sp The intermediate features obtained by decoding are concatenated along the channel dimension and the decoder is used to obtain the multi-fusion decoding feature E. Fuse And the edge change prediction result P edge , the edge change prediction result P edge And the fusion decoding feature E Fuse After fusion, 1×1 convolution is performed to obtain the edge-guided fusion change prediction result P Fuse , P p , P sp and P Fuse Fusion obtains the prediction result P; S104, using a Softmax classifier to obtain a change probability map of the prediction result P, and binarizing the change probability map to obtain a visualization image of the change area prediction result; The encoder in step S102 includes three encoding network modules with non-shared parameters, which are respectively used to convert T a Encoded as pixel-level features F pa , T b Encoded as pixel-level features F pb , and F csp Encoded as super-pixel fusion encoding feature F sp The coding network module includes multiple coding layers and is used to convert T a Encoded as pixel-level features F pa Each intermediate coding layer in the coding network module performs feature adaptive fusion according to the following formula: , In the above formula, and is the weight parameter; For T b Encoded as pixel-level features F pb Each intermediate coding layer in the coding network module performs feature adaptive fusion according to the following formula: , In the above formula, and is the weight parameter; For F csp Encoded as super-pixel fusion encoding feature F sp Each intermediate coding layer in the coding network module performs feature adaptive fusion according to the following formula: , In the above formula, is the feature obtained by feature adaptive fusion, ~ is the weight parameter.

2. The multi-temporal remote sensing image superpixel-pixel multi-level fusion recognition method according to claim 1, It is characterized in that The superpixel segmentation in step S101 includes firstly using a preset superpixel segmentation method to perform superpixel segmentation on the image to be processed, and then performing a averaging operation in each superpixel block in the superpixel features obtained by the superpixel segmentation to obtain a final superpixel segmentation result.

3. The multi-temporal remote sensing image superpixel-pixel multi-level fusion recognition method according to claim 2, It is characterized in that The preset superpixel segmentation method is a simple linear iterative clustering algorithm SLIC, and when the simple linear iterative clustering algorithm SLIC is used to perform superpixel segmentation on the image to be processed, the function expression for generating the number of superpixels is defined as follows: , In the above formula, is the number of superpixels, T a , T b The common image width for both, T a , T b The common image height of both, is the number of pixels in each superpixel defined.

4. The multi-temporal remote sensing image superpixel-pixel multi-level fusion recognition method according to claim 1, It is characterized in that In step S102, F is calculated pa and F pb The pixel-level difference encoding feature F p The function expression is F p = F pa -F pb .

5. The multi-temporal remote sensing image superpixel-pixel multi-level fusion recognition method according to claim 1, It is characterized in that Before step S101, the step of training the multi-temporal remote sensing image superpixel-pixel multi-level fusion recognition network composed of the encoder and the decoder is also included, and the loss function used in the training is L Total for: L Total = L Focal + L MSE , In the above formula, L Focal is the change area change loss, L MSE is the edge change loss, and: , , In the above formula, is the number of training samples, is the weighting factor, is the focusing parameter, P n is the prediction result P corresponding to the nth training sample, is the true image label of the nth training sample, and the true image label value is 0 or 255; is the edge change prediction result corresponding to the nth training sample, is the true edge label corresponding to the nth training sample.

6. The multi-temporal remote sensing image superpixel-pixel multi-level fusion recognition method according to claim 1, It is characterized in that The encoding network module is a Resnet18 network.

7. The multi-temporal remote sensing image superpixel-pixel multi-level fusion recognition method according to claim 5, It is characterized in that The decoder includes a p Decoded into pixel-level decoding features E p The pixel-level feature decoder is used to transform F sp Decoded into super-pixel level decoding feature E sp The super-pixel feature decoder E p and super-pixel level decoding features E sp Decoded as fusion decoding feature E Fuse A fusion feature decoder; both the pixel-level feature decoder and the super-pixel-level feature decoder include seven cascaded feature decoding modules, each of which includes a 3×3 convolution layer, a batch normalization layer, and a linear correction unit connected in sequence, and the front end of the 3×3 convolution layer of the first five feature decoding modules is also provided with an upsampling module for performing a two-fold upsampling operation on the input feature image, and the input end of the first second to fifth feature decoding modules also inputs a fusion of multi-level encoded features; E p and E sp The intermediate features obtained by decoding are concatenated along the channel dimension to obtain multi-level fusion features E Fuse It refers to splicing the features output by the seven feature decoding modules of both the pixel-level feature decoder and the super-pixel-level feature decoder along the channel dimension to obtain the multi-level fusion feature E Fuse The fusion feature decoder includes seven cascaded feature decoding modules, each of which includes a 3×3 convolutional layer, a batch normalization layer, and a linear correction unit connected in sequence. An upsampling module is set before the first four feature decoding modules to perform a double upsampling operation on the feature image, and the input end of the feature decoding module also adds the multi-level fusion feature E Fuse The fusion input, and the sixth feature decoding module outputs the edge decoding feature E edge , edge decoding feature E edge Obtain edge change prediction result P through 1×1 convolution edge , the features output by the seventh feature decoding module and the edge change prediction result P edge After splicing along the channel dimension, 1×1 convolution is performed to obtain the edge-guided fusion change prediction result P Fuse .

8. A multi-temporal remote sensing image superpixel-pixel multi-level fusion recognition system, comprising a microprocessor and a memory connected to each other, It is characterized in that The microprocessor is programmed or configured to execute the multi-temporal remote sensing image superpixel-pixel multi-level fusion recognition method as described in any one of claims 1 to 7.

9. A computer-readable storage medium having a computer program stored therein. It is characterized in that The computer program is used to be programmed or configured by a microprocessor to execute the multi-temporal remote sensing image superpixel-pixel multi-level fusion recognition method as described in any one of claims 1 to 7.