Lightweight semantic enhancement and change integration for remote sensing image semantic change detection method

Abstract

The application provides a lightweight semantic enhancement and change integrated remote sensing image semantic change detection method, comprising collecting double-time-phase remote sensing images and constructing a sample library; a multi-task semantic change detection network model is constructed, and network model training and optimization are performed based on the sample library; a lightweight multi-task weight sharing encoder is used for the encoder part of the multi-task semantic change detection network model, and the multi-task weight sharing encoder simultaneously supports semantic segmentation and binary change detection tasks; the decoder part of the multi-task semantic change detection network model comprises a semantic segmentation decoder corresponding to a first time phase, a semantic segmentation decoder corresponding to a second time phase, and a binary change detection decoder; the images of the first time phase and the second time phase input into the encoder part are respectively processed to generate feature maps with different resolutions, and are transmitted to the decoder part through a jump connection; and the double-time-phase remote sensing images are input into the trained multi-task semantic change detection network model to perform semantic change detection.

Description

Technical Field

[0001] This invention relates to the field of intelligent interpretation of remote sensing images, and in particular to a lightweight semantic enhancement and change integration remote sensing image semantic change detection technology. Background Technology

[0002] Changes on the Earth's surface are ubiquitous and manifest in diverse forms at different scales. Currently, remote sensing change detection is widely used in urban planning, resource and environmental monitoring, and disaster emergency response. Remote sensing change detection methods are mainly divided into binary change detection (BCD) and semantic change detection (SCD). BCD is typically used to detect single targets, such as buildings, floods, or farmland, requiring only the identification of areas where changes have occurred. While widely applied, it struggles to meet the monitoring needs of various land cover types in practical applications. In contrast, SCD can simultaneously identify changed areas and their corresponding land use and land cover (LULC) types, providing "from-to" transition information. Although research on SCD is relatively limited, it demonstrates higher efficiency and value in practical applications, thus necessitating further research into more mature technologies for wider application.

[0003] With the development of computer vision technology, the application of convolutional neural networks in remote sensing image analysis has become increasingly widespread. However, traditional deep learning models often have a large number of parameters and high computational complexity, making them unsuitable for rapid processing in resource-constrained environments. The introduction of lightweight models such as MobileViT has significantly reduced computational costs while maintaining high performance. This innovative model architecture makes efficient and accurate change detection possible in practical applications, driving the development of intelligent interpretation technology for remote sensing images.

[0004] Currently, deep learning-based Semantic Change Detection (SCD) methods typically employ a three-branch structure, integrating Semantic Segmentation (SS) and Binary Change Detection (BCD) tasks into a multi-task framework. The decoding stage usually includes two SS branches and one BCD branch, with the detection accuracy improved through the synergistic effect between the branches. Although existing research has optimized for temporal and task relevance, current methods still face two main challenges: first, insufficient model lightweightness leads to high computational costs and slow inference speed, limiting the efficiency and scalability of SCD in practical applications; second, shortcomings exist in handling the interaction between semantic and change information, resulting in inconsistent outputs from different branches, directly affecting the semantic change detection results. Summary of the Invention

[0005] This invention addresses the shortcomings of existing methods for detecting semantic changes in remote sensing images in the field of intelligent interpretation. By combining deep learning and high-resolution remote sensing images, it provides a lightweight semantic enhancement and change integration method and device for detecting semantic changes in remote sensing images, which can effectively meet the needs of many practical applications such as natural resource monitoring.

[0006] The technical solution provided by this invention is a lightweight semantic enhancement and change integration method for detecting semantic changes in remote sensing images, comprising:

[0007] Collect dual-temporal remote sensing images and construct a sample library;

[0008] A multi-task semantic change detection network model is constructed, and the network model is trained and optimized based on a sample database. The encoder part of the multi-task semantic change detection network model adopts a lightweight multi-task weight-shared encoder, which supports both semantic segmentation and binary change detection tasks. It includes an encoder branch corresponding to the first time phase and an encoder branch corresponding to the second time phase. The decoder part of the multi-task semantic change detection network model includes a semantic segmentation decoder corresponding to the first time phase, a semantic segmentation decoder corresponding to the second time phase, and a binary change detection decoder. The images of the first time phase and the second image input to the encoder part are processed to generate feature maps of different resolutions, and then passed to the decoder part through skip connections to achieve multi-level feature integration. Finally, a binary change map and two semantic segmentation maps are generated, and the semantic change map is extracted by combining mask operations.

[0009] Input dual-temporal remote sensing images into a trained multi-task semantic change detection network model to perform semantic change detection.

[0010] Moreover, in the lightweight multi-task weight-sharing encoder, the encoder branch corresponding to the first time phase and the encoder branch corresponding to the second time phase have the same structure and share weights. Each includes two stages. In the first stage, a lightweight feature extraction network is used to encode the input image, extract deep semantic information, and then output three sets of features at different scales. In the second stage, these three sets of features are respectively input into three receptive field attention convolutional blocks to extract receptive field spatial features and obtain three sets of optimized features.

[0011] Furthermore, each semantic segmentation decoder includes three spatiotemporal semantic enhancement fusion modules connected in series and a depthwise separable residual convolutional block; the binary change detection decoder includes three multi-type temporal change integration transformation modules connected in series and a depthwise separable residual convolutional block.

[0012] Furthermore, after a pair of dual-temporal RGB three-band optical remote sensing images T1 and T2 are processed by the corresponding encoder branches of a lightweight multi-task weight-shared encoder, the outputs feature scales corresponding to T1. and features at different scales corresponding to T2 and It is then passed to the decoder section via a skip connection.

[0013] Moreover, the spatiotemporal semantic enhancement fusion module receives feature inputs from two temporal phases, utilizes skip connections and depthwise separable convolutions to fuse change information and generate richer feature representations; in this module, by performing absolute value difference operations on features, change information is captured, and combined with a directional attention mechanism, the model's ability to express semantics in change regions is significantly enhanced.

[0014] Moreover, the multi-type temporal change integration and conversion module generates differential features by performing various forms of operations on features at different times, while maintaining the contextual information of the features; it further extracts spatial and channel features, generates enhanced spatial and channel weights, integrates features, and enhances the representation of change features in deep features through a fusion transformation mechanism, outputting more accurate binary change results.

[0015] Moreover, the depthwise separable residual convolutional block is a residual block constructed based on the depthwise separable convolution DSConv.

[0016] On the other hand, the present invention provides an electronic device including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the lightweight semantic enhancement and change integration remote sensing image semantic change detection method as described above.

[0017] On the other hand, the present invention provides a non-transitory computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the lightweight semantic enhancement and change integration remote sensing image semantic change detection method as described above.

[0018] On the other hand, the present invention provides a computer program product, including a computer program that, when executed by a processor, implements the lightweight semantic enhancement and change integration remote sensing image semantic change detection method as described above.

[0019] In conclusion, this invention proposes a lightweight semantic enhancement and change integration method and apparatus for detecting semantic changes in remote sensing images. Based on the design concept of a lightweight skeleton and skip connections, a novel decoding mechanism network model is constructed to better address the interaction problem between semantic and change information in SCD tasks. While maintaining high detection performance, this model has a lower number of parameters, lower computational cost, and faster inference speed, better meeting the needs of practical applications such as urban planning, resource and environmental monitoring, and disaster emergency response. Furthermore, the detection model of this invention is efficient and scalable, adaptable to different types of remote sensing image data, and can be fine-tuned as needed to adapt to other change detection tasks. This flexibility makes it widely applicable in fields such as emergency response, agricultural monitoring, and infrastructure management. In summary, this invention provides an efficient, accurate, and scalable solution that significantly improves the efficiency of remote sensing image analysis and application.

[0020] Specifically, the beneficial effects of the technical solution provided by this invention are as follows:

[0021] (1) High performance and low computational cost: By adopting a lightweight network structure and skip connection design, the number of model parameters and computational cost are significantly reduced, greatly improving the inference speed. This efficient design not only reduces the consumption of hardware resources, but also enables the model to achieve real-time processing in resource-constrained environments, better adapting to the needs of practical applications.

[0022] (2) Precise semantic change detection capability: This patent innovatively designs a semantic enhancement fusion module and a change integration transformation module, which effectively improves the interaction capability between semantic information and change information. By enhancing the intra-class similarity and inter-class separability of change regions, the model achieves higher semantic segmentation accuracy and change detection performance in complex scenarios, providing more reliable data support for fields such as urban planning and environmental monitoring.

[0023] (3) Wide applicability and flexible adjustment capability: The model has high adaptability and scalability, and is not only applicable to various types of remote sensing image data, but can also be fine-tuned according to different application needs to meet change detection tasks in multiple fields such as urban planning, resource and environmental monitoring, and disaster emergency response. Its flexibility makes it have great application potential and value in actual production and emergency scenarios. Attached Figure Description

[0024] Figure 1 This is a network model structure diagram of an embodiment of the present invention.

[0025] Figure 2 This is a diagram of the depth-separable residual convolution block structure according to an embodiment of the present invention.

[0026] Figure 3This is a schematic diagram of the spatiotemporal semantic enhancement and fusion module in an embodiment of the present invention.

[0027] Figure 4 This is a schematic diagram of the multi-type temporal phase change integrated conversion module according to an embodiment of the present invention. Detailed Implementation

[0028] The technical solution of the present invention will be described in detail below with reference to the accompanying drawings and embodiments.

[0029] Compared to traditional binary change detection methods, semantic change detection can not only identify changed areas but also simultaneously detect land use and land cover types within these areas, providing transitional information from - to, thus demonstrating higher efficiency and application value in practical production. Addressing the urgent needs in urban planning, resource and environmental monitoring, and disaster emergency response, this invention designs an efficient and accurate remote sensing semantic change detection scheme.

[0030] This invention proposes a lightweight semantic enhancement and change integration method for detecting semantic changes in remote sensing images, comprising:

[0031] 1) Acquire dual-temporal remote sensing images and construct a sample library;

[0032] In practice, two high-resolution remote sensing images of the same area can be acquired, preprocessed, and labeled to generate samples required for subsequent steps.

[0033] 2) Construct a multi-task semantic change detection network model and train and optimize the network model based on a sample library. The encoder part of the multi-task semantic change detection network model adopts a lightweight multi-task weight-shared encoder, which supports both semantic segmentation and binary change detection tasks. It includes an encoder branch corresponding to the first time phase and an encoder branch corresponding to the second time phase. The decoder part of the multi-task semantic change detection network model includes a semantic segmentation decoder corresponding to the first time phase and a semantic segmentation decoder corresponding to the second time phase, as well as a binary change detection decoder. The images of the first time phase and the second image input to the encoder part are processed to generate feature maps of different resolutions, and then passed to the decoder part through skip connections to achieve multi-level feature integration. Finally, a binary change map and two semantic segmentation maps are generated, and the semantic change map is extracted by combining mask operations.

[0034] This invention constructs a semantic change detection network model based on MobileViTv3, a multi-task architecture, and a multi-task decoding module. This includes combining a semantic enhancement fusion and change integration transformation module, and designing a multi-task decoding module to more effectively extract semantic change features from the network model. More preferably, the multi-task network model can be optimized from multiple perspectives using a multi-class cross-entropy loss function, a binary cross-entropy loss function, and a semantic consistency loss function, making it more focused on change regions and semantic categories, thereby improving detection performance.

[0035] 3) Input dual-temporal remote sensing images into the trained multi-task semantic change detection network model to perform semantic change detection.

[0036] By using the trained network model to perform semantic change detection on the dual-temporal remote sensing images to be processed, high-precision detection results can be generated.

[0037] The present invention provides a lightweight semantic enhancement and change integration method for detecting semantic changes in remote sensing images. A further preferred implementation method includes the following steps:

[0038] I. Acquire dual-temporal remote sensing images and construct a sample database

[0039] This invention further proposes a meticulously designed data processing workflow to enhance the change detection capability of dual-temporal high-resolution remote sensing images. First, during image acquisition, images containing a large number of change areas are prioritized to enrich the model's change sample. Subsequently, the images undergo preprocessing, including size standardization, cropping of invalid regions, and noise reduction, to ensure image quality stability. In the annotation stage, specialized tools are used to precisely annotate change areas and their land cover types, providing the model with diverse label data. Next, data augmentation techniques such as affine transformation and tone adjustment are used to expand sample diversity, improving the model's robustness and generalization ability. Finally, the dataset is divided into training, validation, and test sets to provide scientific data support for model learning, parameter tuning, and performance evaluation, ensuring stable performance across different datasets and thus improving overall detection effectiveness.

[0040] As a preferred embodiment, the sample library construction of the embodiment specifically includes the following sub-steps:

[0041] 1) First, regarding the acquisition of dual-temporal high-resolution remote sensing images, this invention emphasizes selecting remote sensing images containing a large number of changing areas during the acquisition process to ensure that the model can obtain rich change samples in subsequent training.

[0042] 2) After acquiring these images, rigorous preprocessing is recommended. These preprocessing steps include resizing all images to a uniform size and format, cropping the images to remove invalid areas, and denoising to eliminate noise interference. These steps improve the quality of the image data and ensure the accuracy and consistency of subsequent analysis.

[0043] 3) During the data annotation stage, it is highly recommended to use professional annotation tools to finely annotate the changed areas in each remote sensing image. First, delineate the changed areas in the dual-temporal images, marking the locations where significant differences occur in different temporal phases.

[0044] Then, for these areas of change, the land cover type for each time phase is labeled in detail, such as categories like buildings, water bodies, roads, and vegetation, to provide the model with diverse labeled data. This meticulous labeling process not only improves the model's ability to identify different types of changes but also provides a solid data foundation for subsequent analysis.

[0045] 4) After annotation, data augmentation is a crucial step in image data processing. By applying affine transformations (including scaling, rotation, and translation), tonal adjustments (such as brightness and contrast changes), random cropping, and flipping, the diversity of the sample data can be significantly expanded. These augmentation methods effectively increase the model's robustness to different environments, lighting conditions, and viewpoint changes, and help reduce overfitting during training, thereby improving the overall model performance and generalization ability.

[0046] 5) Finally, to ensure the dataset can be fully utilized during training, it is divided into training, validation, and test sets based on actual training needs. In the subsequent network model training and optimization phases, the training set is used for model learning and parameter optimization, the validation set is used for parameter tuning and performance evaluation during model training, and the test set is used to evaluate the model's final performance after training. This partitioning method ensures stable model performance across different datasets and effectively avoids overfitting and underfitting, providing solid data support for network model training and evaluation.

[0047] II. Constructing a Multi-Task Semantic Change Detection Network Model Based on MobileViTv3

[0048] This invention designs a novel network structure comprising two encoders with shared weights and three decoders with distinct functions, fully leveraging lightweight design, skip connections, and innovative decoding strategies. The encoder preferably employs MobileViTv3 (a third-generation mobile-friendly visual transformer) and Receptive-Field Attention Convolution (RFAConv) to construct a lightweight shared encoder suitable for both semantic segmentation (SS) and binary change detection (BCD) tasks, ensuring sensitivity to detailed features while reducing computational cost and model complexity. In the decoder section, two functionally identical semantic segmentation decoders are constructed around three temporal-spatial semantic enhancement fusion modules (TSEF) and a residual block based on depthwise separable convolution. These decoders output two semantic segmentation maps and improve the consistency of similar features and the distinguishability of dissimilar features. Conversely, a binary change detection decoder is constructed around three multi-type temporal-spatial change integration transformation modules (TSEF) and a residual block based on depthwise separable convolution. In this decoding branch, the output of the BCD serves as auxiliary information for the semantic segmentation decoder, ensuring the capture of key regions and improving the accuracy of change information processing. In summary, the model receives a pair of images T1 and T2, processes them to generate feature maps of different resolutions, and passes them to the decoder via skip connections to achieve multi-level feature integration. Finally, the model generates accurate binary change maps and two semantic segmentation maps, and combines mask operations to extract high-precision semantic change maps, demonstrating efficient change detection capabilities under limited resource conditions.

[0049] For a detailed implementation of MobileViTv3, please refer to the relevant literature, which will not be elaborated upon in this invention: Wadekar SN, Chaurasia A. Mobilevitv3: Mobile-friendly vision transformer with simple and effective fusion of local, global and input features[J]. arXiv preprint arXiv:2209.15159,2022.

[0050] For a detailed implementation of RFAConv, please refer to the relevant literature, which will not be elaborated here: Zhang X, Liu C, Yang D, et al. RFAConv: Innovating spatial attention and standard convolutional operation[J]. arXiv preprint arXiv:2304.03198,2023.

[0051] As a preferred embodiment, the network model structure of this invention is as follows: Figure 1 As shown, the specific description is as follows:

[0052] At the encoding end, the following processing is included:

[0053] A1) First, input a pair of bi-temporal RGB three-band optical remote sensing images into a lightweight multi-task weight-shared encoder. and The superscript 3 indicates that the input image has 3 bands (it can also be replaced with other multi-band images), and H and W represent the height and width of the input image, respectively. For spatial structure, T1 and T2 will simultaneously encode features along the two shared parameter branches of the lightweight multi-task weight-shared encoder.

[0054] A2) The lightweight multi-task weight-sharing encoder consists of two stages. The first stage is the lightweight feature extraction network MobileViTv3. Taking T1 as an example, MobileViTv3 encodes features from the input image, extracts rich deep semantic information, and then outputs three sets of features at different scales. These three sets of features are of different sizes. M1 contains more spatial detail information, while M2 and M3 contain more semantic information.

[0055] The next three sets of features, M1, M2, and M3, are passed to the second stage of the encoder. In the second stage, these three sets of features are respectively input into three receptive field attention convolutional blocks, RFAConv, to extract receptive field spatial features, obtain more global and local information, and obtain three sets of optimized features.

[0056] Meanwhile, T2 will undergo the same processing as T1 to obtain three sets of optimized features.

[0057] A3) After T1 and T2 are processed by a lightweight multi-task weight-sharing encoder, they will output six sets of features: and The numbers in the upper right corner represent the corresponding time phases, and the numbers in the lower right corner represent which RFAConv block outputs the feature. These features will then be combined with the skip connections input to different nodes of different decoders to play a role.

[0058] During the decoding phase, the embodiment has three decoders: two identical semantic segmentation decoders and one binary change detection decoder. Each semantic segmentation decoder consists of three TSEF modules and one depthwise separable residual convolutional block, these four modules being cascaded in series. The binary change detection decoder consists of three MTCT modules and one depthwise separable residual convolutional block, which are also cascaded in series.

[0059] As a preferred embodiment, the present invention further provides a depth-separable residual block structure, such as... Figure 2 As shown, it is a residual block constructed based on depthwise separable convolution (DSConv):

[0060] Assuming the input features of a depthwise separable residual convolutional block are Output features are C DS H represents the corresponding number of feature channels. DS and W DS These represent the height and width of the corresponding feature, respectively. The input feature K first passes through depthwise separable convolution 1, batch normalization layer 1, activation function layer 1, depthwise separable convolution 2, and batch normalization layer 2 in sequence. Then, the features obtained through the above five steps are added to the input feature K and passed to activation function layer 2 to obtain the final output feature L.

[0061] The DSConv module used in Depthwise Separable Convolution 1 and Depthwise Separable Convolution 2 is an existing technology. For details on its implementation, please refer to relevant literature. This invention will not elaborate further: Chollet F. Xception: Deep learning with depthwise separateable convolutions[C] / / Proceedings of the IEEE conference on computer vision and pattern recognition.2017:1251-1258.

[0062] As a preferred embodiment, the semantic segmentation decoder processes the following:

[0063] B11) Taking the semantic segmentation decoder corresponding to T1 as an example, features and First, the features are passed to the first TSEF module of the semantic segmentation decoder for spatiotemporal semantic enhancement and upsampling to obtain optimized features;

[0064] B12) Then the optimized features obtained from B11) will be combined with those passed through the skip connections. and The features are input together into the second TESF module for spatiotemporal semantic enhancement and upsampling, resulting in enhanced features with further improved resolution.

[0065] B13) Next, the enhanced features obtained in B12) will be combined with those passed through the skip connections. and The features are then fed into the third TESF module for further spatiotemporal semantic enhancement and upsampling to obtain optimized and enhanced features.

[0066] B14) Finally, the optimized and enhanced features obtained in step B14) are independently input into the depthwise separable residual convolutional block for further decoding, extraction, and upsampling to obtain the semantic segmentation result. N represents the number of land cover categories in the semantic segmentation task.

[0067] At the same time, the semantic segmentation decoder corresponding to T2 will also perform the same operations (steps B11, B12, B13 and B14) to obtain the semantic segmentation result. N represents the number of land cover categories in the semantic segmentation task. Specifically:

[0068] B21) Taking the semantic segmentation decoder corresponding to T2 as an example, features and First, the features are passed to the first TSEF module of the semantic segmentation decoder for spatiotemporal semantic enhancement and upsampling to obtain optimized features;

[0069] B22) Then the optimized features obtained from B21) will be passed through the skip connections. and The features are input together into the second TESF module for spatiotemporal semantic enhancement and upsampling, resulting in enhanced features with further improved resolution.

[0070] B23) Next, the enhanced features obtained in B22) will be combined with those passed through the skip connections. and The features are then fed into the third TESF module for further spatiotemporal semantic enhancement and upsampling to obtain optimized and enhanced features.

[0071] B24) Finally, the optimized and enhanced features obtained in step B24) are independently input into the depthwise separable residual convolutional block for further decoding, extraction, and upsampling to obtain the semantic segmentation result. N represents the number of land cover categories in the semantic segmentation task.

[0072] As a preferred embodiment, the processing procedure of the binary change detection decoder is as follows:

[0073] C1) For a binary change detection decoder, the feature and First, it is passed to the first MTCT module of the decoder for temporal integration conversion and upsampling to obtain preliminary change characteristics;

[0074] C2) Then the preliminary change characteristics obtained in step C1) will be passed through the jump connection. and The features are input together into the second MTCT module for temporal integration transformation and upsampling to obtain optimized features with further improved resolution.

[0075] C3) The optimized features obtained in the next step C2) will be combined with those passed through the skip connections. and The inputs are fed into the third MTCT module, where they undergo another phase integration transformation and upsampling to obtain optimized and enhanced variation features.

[0076] C4) Finally, the optimized and enhanced change features obtained in step C3) are independently input into the depthwise separable residual convolutional block for further change decoding and upsampling to obtain the binary change detection result. Y S1 Y S2 and Y C The height and width are consistent with T1 and T2, that is, the original input size has been restored.

[0077] Finally, the two semantic segmentation results Y are... S1 and Y S2 Compare with the binary change detection result Y respectively C The purpose of performing mask multiplication is to retain only the semantic segmentation results of the changed regions, thereby obtaining two final high-precision semantic change detection result images, denoted as semantic change images Y1 and Y2. and Their dimensions are consistent with the input image.

[0078] The network model in this embodiment, under a multi-decoder architecture, provides efficient semantic change detection capabilities even under hardware conditions with limited resources through multi-level integration and expression of feature information. Furthermore, this network model is lightweight, computationally inexpensive, and has a slow inference speed, making it easy to apply in practical applications. In addition, the semantic change detection results output by this network model are highly consistent, providing more reliable semantic change information.

[0079] The following section focuses on the spatiotemporal semantic enhancement fusion module and the multi-type temporal change integration and transformation module proposed in this invention. These two modules work together to effectively improve the ability to represent changes in deep features, providing support for generating more accurate land cover change detection results. This innovative design provides a reliable technical foundation for remote sensing image analysis.

[0080] This invention proposes a Temporal-Spatial Semantic Enhancement Fusion Module (TSEF) to improve the performance of remote sensing image change detection models. The core function of the TSEF module is to receive feature inputs from two temporal phases, utilize skip connections and depthwise separable convolutions to fuse change information and generate richer feature representations. This module captures change information by performing absolute value differencing on the features and, combined with a directional attention mechanism, significantly enhances the model's ability to semantically represent changed regions. This innovative design provides a reliable technical foundation for remote sensing image analysis.

[0081] The Temporal-Spatial Semantic Enhancement Fusion Module (TSEF) is a key component of the Semantic Segmentation (SS) decoder. To enhance the interaction between semantic and change information, this invention innovatively introduces change information into the TSEF. This innovation not only significantly improves the accuracy of the SS task but also enables the network to pay closer attention to the semantic features of changing regions. Furthermore, the construction of the TSEF improves the correlation between the two tasks and enhances the temporal correlation between bi-temporal features. This means that the semantic change detection network model can better understand the semantic change process by leveraging change information.

[0082] As a preferred embodiment, the detailed structure of the TSEF proposed in the embodiment is as follows: Figure 3 As shown, with Figure 1 Taking the second TESF module of the first temporal semantic segmentation decoder as an example, the specific steps of the feature within this structure are as follows.

[0083] Step T1: The TSEF module receives two temporal features optimized by RFAConv from the receptive field attention convolutional block obtained from the encoder as input, namely the first temporal feature and the second temporal feature. Second phase And through jump connections and Figure 1 The output features of the first TESF module of the first temporal semantic segmentation decoder To be shared.

[0084] For ease of general representation, this embodiment uses the first input feature. express Using the second input features express Use the output features of the TESF module at the next layer. Let Q represent the specific process of TSEF, C1 and C2 represent the number of channels for the corresponding features, and H1 and W1 represent the height and width of the corresponding features. It is understandable that in the semantic segmentation decoder of the second phase... Figure 3 RF in 1 This indicates the corresponding characteristics of the second phase transmission, RF 2 This indicates that the corresponding feature transmitted in the first phase is the opposite of the representation in the first phase semantic segmentation decoder.

[0085] Step T2, in the processing flow of the TSEF module, firstly, the first input feature RF is... 1 and the output feature X of the previous TESF module T The components are stitched together along the channel dimension and fused across channels using 1×1 convolution.

[0086] Step T3, based on the results obtained in step T2, then optimizes using 3×3 depthwise separable convolution (DSConv) to obtain a new feature representation. C3 represents the number of channels in the TESF module's output features. This operation helps extract richer feature representations. The process can be represented as:

[0087]

[0088] Where ∪ represents splicing along the channel dimension, Represents a 1×1 convolution. This represents a 3×3 depth separable convolution.

[0089] The flow of the third TSEF module in the semantic segmentation decoder is the same as that of the second TSEF module, as shown in Equation (1). However, the first TSEF module does not have the feature X from the previous module. T Therefore, the first TESF module only has two inputs: RF.1 and RF 2 Furthermore, instead of performing channel concatenation and 1×1 convolution in step T2, the input is directly fed into the 3×3 depthwise separable convolutional block of this step to obtain the first output feature Z1. At this time, the number of channels C3, C1 and C2 are the same. Therefore, in the first TSEF module, this step is represented as shown in equation (2):

[0090]

[0091] Step T4, next, the second input feature RF 2 It is introduced and subjected to absolute value difference operation with the first output feature Z1 to capture change information.

[0092] In step T5, the result obtained in step T4 is processed by the Softmax function to generate a variable weight. This weight is then applied to the first output feature Z1 to obtain enhanced semantic features.

[0093] In step T6, the enhanced semantic features obtained in step T5 are then fed into the first deepest separable residual convolutional block (DS_ResBlock1) constructed based on DSConv to extract deeper second output features. Achieve lightweight feature learning.

[0094] The calculation process for steps T4-T6 can be represented as follows:

[0095]

[0096] Where |.| represents the absolute value difference operation, ρ() represents the Softmax function, and × represents feature multiplication. This represents the first depthwise separable residual convolution block of TSEF.

[0097] In step T7, to enhance the expressive power of spatiotemporal semantics, the TSEF module employs two-dimensional adaptive average pooling on feature Z2, encoding different types of channel weights in both the horizontal and vertical directions. The pooling results in the horizontal and vertical directions are then subjected to convolution-activation-convolution operations, forming a weight-shared twin structure that accurately captures the correlation between channels and enhances spatial information.

[0098] Then, the directional attention weights are obtained through the Sigmoid function. and These correspond to the horizontal and vertical directions, respectively.

[0099] These processes can be represented as:

[0100]

[0101] in This represents the activation operation, and σ() represents the Sigmoid function. and P represents the first and second 1×1 convolutional structures with shared weights, respectively. H and P W These represent horizontal pooling and vertical pooling operations, respectively.

[0102] Step T8, finally, Z2 is simultaneously combined with the attention weight A H and A W The components are multiplied, then further optimized by a second depthwise separable residual convolutional block DS_ResBlock2, and then added to Z1. Finally, the output features are generated by upsampling through transposed convolution. It can be represented as:

[0103]

[0104] in This represents a 2×2 transposed convolution used for upsampling. This indicates the second depthwise separable residual convolution block of TSEF, where × indicates feature multiplication and + indicates feature addition.

[0105] The above describes the workflow of the TSEF module, designed based on the correlation between semantic and change information. In this module, the embodiment innovatively integrates feature change information from two temporal phases within the SS branch, combined with spatiotemporal semantic enhancements based on channel and spatial dimensions. This method significantly improves the quality of the model's representation of change regions, enabling it to more effectively distinguish between changed and unchanged regions and provide a more consistent semantic understanding of similar change regions. This design of the TSEF module allows it to effectively fuse change information within the semantic branch, enhancing the model's feature representation capability in change regions and thus more accurately identifying land cover change types.

[0106] This invention proposes a Multi-type Temporal Change Integration Transformation module (MTCT). The MTCT module generates difference features by performing various operations (such as subtraction, addition, and multiplication) on features from different times, while preserving the contextual information of the features. This module further extracts spatial and channel features, generates enhanced spatial and channel weights, and integrates these features. The aim is to integrate the results of multiple temporal difference representations and enhance the representation of change features in deep features through an efficient fusion transformation mechanism, outputting more accurate binary change results.

[0107] As a preferred embodiment, the detailed structure of the MTCT proposed in the embodiment is as follows: Figure 4 As shown, with Figure 1 Taking the second MTCT module of the corresponding BCD decoder as an example, the specific steps of the feature within this structure are as follows.

[0108] Step M1: The MTCT module receives two temporal features optimized by RFAConv from the receptive field attention convolutional block obtained from the encoder as input, namely the first temporal feature and the second temporal feature. Second phase And through jump connections and Figure 1 Output characteristics of the first MTCT module in the BCD decoder The transmission is performed. For ease of general representation, this embodiment uses... express use express use Let V represent the specific process of MTCT, C1 and C2 represent the number of channels for the corresponding feature, and H1 and W1 represent the height and width of the corresponding feature. Here, RF... 1 and RF 2 It is the same as in step T1.

[0109] Step M2, in the MTCT module, different temporal difference representation methods have different functions, RF 1 and RF 2 Differential features are generated through three methods: feature subtraction, addition, and multiplication, while maintaining the same number of channels.

[0110] Step M3, X M RF 1 and RF 2 After concatenation along the channel dimension, the data is passed to a 1×1 convolutional layer to achieve cross-channel information fusion. Then, it is processed through a 3×3 depthwise separable convolution (DSConv) layer to improve feature representation and obtain the output features. It can be represented as:

[0111]

[0112] Where ∪ represents feature concatenation along the channel dimension. Represents a 1×1 convolution. This represents a 3×3 depthwise separable convolution. This preserves more contextual information and details of variation.

[0113] The flow of the third MTCT module in the BCD decoder is the same as that of the second MTCT module, as shown in equation (6). However, the first MTCT module does not have the feature X from the previous module. MTherefore, the first MTCT module only has two inputs: RF. 1 and RF 2 Therefore, only RF splicing is needed. 1 and RF 2 Then the corresponding equation (6) will become equation (7):

[0114]

[0115] Step M4: The features generated in step M2 are further processed through two branches (spatial branch and channel branch).

[0116] In the spatial branch, each feature representation will be max-pooled and average-pooled in the spatial dimension, and the results will be averaged to improve the attention weight generation effect. Figure 4 The simplified representation is (AvgS + MaxS) / 2. These operations can also be represented as:

[0117]

[0118] Where |.| represents taking the absolute value, - represents subtracting features, and P MaxS and P AvgS These represent max pooling and average pooling operations on the spatial dimension, respectively. × indicates feature multiplication, and + indicates feature addition.

[0119] The resulting pooling features P S1 P S2 P S3 The features are concatenated and integrated through a 7×7 convolutional layer to generate diverse features, and then processed by the Sigmoid function to obtain the final spatial weights. It can be represented as:

[0120]

[0121] Where σ represents the Sigmoid function. ∪ represents a 7×7 convolution, and ∪ represents feature concatenation along the channel dimension.

[0122] The resulting spatial weight A S Cascaded features F obtained in step M3 C After multiplication, the data enters the first depthwise separable residual convolutional block DS_ResBlock1 of MTCT to obtain enhanced features of the spatial branch.

[0123]

[0124] in This indicates the first depthwise separable residual convolution block of MTCT, and × indicates feature multiplication.

[0125] In the channel branch, each feature representation obtained in step M2 extracts channel features through adaptive max pooling and average pooling, and the results are averaged to improve the ability to represent the significance of each channel. Figure 4 In simplified Chinese, this is represented as (AvgC + MaxC) / 2. These operations can also be represented as:

[0126]

[0127]

[0128] Where P MaxC and P AvgC represents adaptive max pooling and average pooling operations on the channel dimension, respectively, where |.| represents taking the absolute value, - represents subtracting features, × represents multiplying features, and + represents adding features.

[0129] The obtained channel features P C1 P C2 P C3 After summing, the channel weights are obtained using the Sigmoid function.

[0130] A C =σ(P C1 +P C2 +P C3 (16)

[0131] Channel weight A C With feature F C The components are multiplied and then fed into the second depthwise separable residual convolution block DS_ResBlock2 of MTCT to obtain enhanced features for the channel branches.

[0132]

[0133] in This indicates the second depthwise separable residual convolution block in MTCT, and × indicates feature multiplication.

[0134] Step M5, the enhanced features E generated by the spatial branch and the channel branch are... S and E C The features are added together for feature integration and information supplementation, and then optimized for feature extraction using the third depthwise separable residual convolutional block DS_ResBlock3 of MTCT, and combined with the cascaded features F. C Perform residual connection.

[0135] Step M6: Finally, the result obtained in step M5 is processed by transposed convolution to improve spatial resolution and feature refinement, generating the final variation decoding features.

[0136]

[0137] in This indicates the third depthwise separable residual convolutional block in MTCT, with the plus sign indicating feature addition. This represents a 2×2 transposed convolution. The output feature Y of this MTCT module... M The size and output feature Y in step T8 T They are the same.

[0138] The above describes the MTCT processing flow proposed to enhance the ability to identify change-decoding branches. First, multiple types of temporal change are fused to generate various differential feature representations in both spatial and channel dimensions. Then, these features are integrated into representative features using concatenation and addition methods, and input into the corresponding attention weight generation component for fusion transformation. The final output is a feature optimized based on different perspectives. Therefore, the MTCT module can effectively enhance the change representation capability in deep features, providing more accurate information on change regions for generating the final SCD result.

[0139] III. Using a multi-task loss function for network model training and optimization

[0140] In deep learning networks, backpropagation of the loss function is used to optimize network weights. However, as network complexity increases, the vanishing gradient problem may occur during parameter updates, leading to unstable optimization and reduced change detection performance. To address this issue, this invention proposes a method combining three loss functions to jointly supervise and optimize network training, specifically for semantic segmentation, binary change detection, and semantic consistency. First, for the semantic segmentation task, a multi-class cross-entropy loss function is used to guide network training. Second, for the binary change detection task, a binary cross-entropy loss function is used to supervise the change detection branch. Furthermore, a semantic consistency loss function is introduced to ensure consistent semantic predictions between two temporal branches within a change-free region. These loss functions are combined to form a total loss function. By balancing the influence of different loss functions, the network model performance is systematically optimized, thereby improving its robustness and efficiency in complex change detection tasks. This method enhances model performance from multiple perspectives and improves the accuracy of change detection.

[0141] As a preferred embodiment, the multi-task loss function of the embodiment is specifically constructed and implemented as follows:

[0142] For the semantic segmentation task (SS task), the example uses a multi-class cross-entropy loss function to guide the training of the network, which is defined as follows:

[0143]

[0144] Where N represents the total number of pixels and C represents the number of categories. It is the true probability that pixel i belongs to category C, while This represents the probability that pixel i is predicted to be of category C, where t represents the time phase. This represents the multi-class cross-entropy loss at time t.

[0145] In this semantic change detection framework, since there are two temporal branches, each corresponding to a semantic segmentation task, the total semantic segmentation loss L... seg Defined as the sum of the losses of the two phases:

[0146]

[0147] in and These represent the losses of the two temporal semantic segmentation branches, respectively.

[0148] For the binary change detection task, the embodiment uses the binary cross-entropy loss function L. bcd To supervise the change detection branch, it is defined as follows:

[0149]

[0150] Where N is the total number of pixels, y i Indicates whether pixel i has changed (1 for change, 0 for no change), p i It is the probability that the pixel will change, as predicted by the model.

[0151] To further improve the stability and accuracy of the model, the implementation example uses a semantic consistency loss function (SCLoss), denoted as L. sc The purpose of this loss function is to reward cases where the two temporal branches have consistent semantic predictions in unchanged regions, thereby ensuring that the semantic information of the images remains consistent across different time points. Its definition is as follows:

[0152]

[0153] Where x1 and x2 refer to the pixel vectors of the two semantic segmentation results, respectively, and y i This indicates whether pixel i has changed (1 for change, 0 for no change).

[0154] To balance the impact of individual loss functions during training, the implementation combines these loss functions into a single total loss function L. total :

[0155]

[0156] By combining semantic segmentation loss, binary change detection loss, and semantic consistency loss, this total loss function systematically optimizes the overall performance of the network model from multiple task objectives, making the model more robust and efficient when handling complex change detection tasks.

[0157] IV. Inputting dual-temporal remote sensing images for semantic change detection

[0158] The semantic change detection model and its optimal weight parameters, obtained through training, ensure performance stability and detection accuracy in subsequent applications. In practice, users only need to input the dual-temporal remote sensing image data to be detected into the pre-trained model to automatically generate high-precision semantic change detection maps. These maps visually display the areas in the remote sensing image that have changed, accurately locating the location of the changes and their category information. The generated change detection results can not only be saved but also used for downstream analysis tasks such as urban planning, land use management, ecological monitoring, and natural resource management. For example, in urban planning, change detection maps help identify urban expansion, changes in built-up land, and reductions in green space, thereby enabling the formulation of scientific and reasonable development plans. In natural resource monitoring, the detection results effectively track dynamic changes such as deforestation and wetland changes, supporting environmental protection and resource management.

[0159] In the preceding steps, the trained semantic change detection model and its optimal weight parameters ensure the model's performance stability and detection accuracy during subsequent use. In practical applications, when semantic change detection is required, simply input the dual-temporal remote sensing image data to be detected into the pre-trained network model, and high-precision semantic change detection maps will be automatically generated. These change detection maps can visually display the areas in the remote sensing image that have changed, accurately locating the position and category of the changes.

[0160] Furthermore, the generated semantic change detection results can not only be directly saved but also used in downstream analysis tasks, such as urban planning, land use management, ecological environment monitoring, and natural resource management. Specifically, in urban planning, semantic change detection maps can help planners identify issues such as urban expansion areas, changes in construction land, and reductions in green space, thereby formulating more scientific and rational development plans. In the field of natural resource monitoring, the detection results can effectively track the dynamic changes in natural resources such as deforestation, wetland changes, and water body distribution, supporting the optimization of environmental protection decisions and resource management strategies.

[0161] Building upon this foundation, the detection model of this invention also possesses high efficiency and scalability. It can adapt to different types of remote sensing image data and allows for fine-tuning of the network model as needed to suit other change detection tasks. This flexibility makes the model widely applicable in various scenarios, such as emergency disaster response, agricultural monitoring, and infrastructure management. In summary, this invention provides an efficient, accurate, and scalable solution in the field of semantic change detection, significantly improving the efficiency of remote sensing image analysis and applications.

[0162] In specific implementation, the method proposed in the technical solution of this invention can be automatically executed by those skilled in the art using computer software technology. System devices for implementing the method, such as computer-readable storage media storing the corresponding computer program of the technical solution of this invention and computer equipment including the computer program running the corresponding computer program, should also be within the protection scope of this invention.

[0163] The lightweight semantic enhancement and change integration remote sensing image semantic change detection device provided by the present invention is described below. The lightweight semantic enhancement and change integration remote sensing image semantic change detection device described below can be referred to in correspondence with the lightweight semantic enhancement and change integration remote sensing image semantic change detection method described above.

[0164] In another embodiment, the present invention provides an electronic device that may include a processor, a communications interface, a memory, and a communication bus, wherein the processor, the communications interface, and the memory communicate with each other via the communication bus. The processor can invoke logical instructions in the memory to execute a lightweight semantic enhancement and change integration remote sensing image semantic change detection method.

[0165] Furthermore, the logical instructions in the aforementioned memory can be implemented as software functional units and sold or used as independent products, and can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0166] In another embodiment, the present invention also provides a computer program product, the computer program product including a computer program that can be stored on a non-transitory computer-readable storage medium, and when the computer program is executed by a processor, the computer is able to execute the lightweight semantic enhancement and change integration remote sensing image semantic change detection method provided by the above methods.

[0167] In another embodiment, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the lightweight semantic enhancement and change integration remote sensing image semantic change detection method provided by the methods described above.

[0168] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0169] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0170] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A lightweight semantic enhancement and change integration remote sensing image semantic change detection method, comprising: collecting double-time-phase remote sensing images and constructing a sample library; constructing a multi-task semantic change detection network model, and training and optimizing the network model based on the sample library; the encoder part of the multi-task semantic change detection network model adopts a lightweight multi-task weight sharing encoder, which simultaneously supports semantic segmentation and binary change detection tasks, and includes an encoder branch corresponding to a first time phase and an encoder branch corresponding to a second time phase; the decoder part of the multi-task semantic change detection network model includes a semantic segmentation decoder corresponding to the first time phase and a semantic segmentation decoder corresponding to the second time phase, and a binary change detection decoder; the images of the first time phase and the second time phase input into the encoder part are respectively processed to generate feature maps of different resolutions, and are transmitted to the decoder part through a skip connection to realize multi-level feature integration, finally generate a binary change map and two semantic segmentation maps, and extract a semantic change map by combining a mask operation; each semantic segmentation decoder includes three spatio-temporal semantic enhancement fusion modules and a depth separable residual convolution block connected in sequence; the binary change detection decoder includes three multi-type time phase change integration conversion modules and a depth separable residual convolution block connected in sequence; the spatio-temporal semantic enhancement fusion module receives feature inputs from two time phases, uses a skip connection and a depth separable convolution to fuse change information and generate more rich feature representations; in this module, the feature is subjected to an absolute value difference operation to capture change information, and a directional attention mechanism is combined to significantly enhance the semantic expression ability of the model to the change area; the multi-type time phase change integration conversion module performs a plurality of forms of operation on features of different times to generate difference features and maintain the context information of the features; spatial and channel features are further extracted to generate enhanced spatial and channel weights, and feature integration is performed, and the change feature representation in the deep feature is enhanced through a blending conversion mechanism to output a more accurate binary change result; inputting the double-time-phase remote sensing images into the trained multi-task semantic change detection network model to perform semantic change detection.

2. The method of claim 1, wherein the method further comprises: In the lightweight multi-task weight sharing encoder, the encoder branch corresponding to the first time phase and the encoder branch corresponding to the second time phase have the same structure and share weights, and each includes two stages; in the first stage, a lightweight feature extraction network is used to encode the input image to extract deep semantic information, and then output three groups of features of different scales; in the second stage, the three groups of features are input into three receptive field attention convolution blocks to extract receptive field spatial features and obtain three groups of optimized features.

3. The method of claim 2, wherein the method further comprises: A pair of dual temporal RGB three-band optical remote sensing images and After passing through the respective encoder branches of a lightweight multi-task weight-shared encoder, the outputs are concatenated with the respective features of different scales , , and the respective features of different scales , and and passed to the decoder part through skip connections.​ 4. The method of claim 1, wherein the method further comprises: The depth separable residual convolution block is a residual block constructed based on a depth separable convolution DSConv.

5. An electronic device comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, characterized in that: The processor implements the lightweight semantic enhancement and change integration remote sensing image semantic change detection method according to any one of claims 1 to 4 when executing the program. 6.A non-transitory computer-readable storage medium having stored thereon a computer program. The computer program is executed by a processor to implement the lightweight semantic enhancement and change integrated remote sensing image semantic change detection method according to any one of claims 1 to 4.

7. A computer program product comprising a computer program, characterized in that: The computer program is executed by a processor to implement the lightweight semantic enhancement and change integrated remote sensing image semantic change detection method according to any one of claims 1 to 4.

Images