Object-level change detection method based on semantic flow feature alignment and global attention fusion

Abstract

The invention discloses an object level change detection method based on semantic flow feature alignment and global attention fusion, and belongs to the field of change detection. The method comprises the following steps: inputting a double-time-phase image and carrying out preprocessing; constructing a twin feature encoder to extract multi-scale features; a self-global cross attention module is introduced in a coding stage to realize global semantic fusion and coarse registration of double-temporal image features; a semantic flow alignment module is further designed, and pixel-level alignment of a feature level is realized by generating a semantic flow field, so that view angle difference and structure offset are compensated; in the decoding stage, a feature optimization and jump connection fusion mechanism is adopted, shallow spatial information and deep semantic features are integrated, and boundary expression of a change region is enhanced; and finally, outputting an object level change detection result through a detection head. Experiments show that the method has object-level change detection capability in scenes with visual angle differences, weak textures and the like.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of computer vision and artificial intelligence, and in particular to an object-level change detection method based on semantic flow feature alignment and global attention fusion. BACKGROUND

[0002] Change detection is an important research direction in the field of computer vision and image analysis, and its core goal is to automatically identify the changed regions or objects in the same scene images taken at different times by comparing them. This technology can effectively reflect the structural, semantic or state changes of the scene over time, and has important application value in remote sensing monitoring, urban planning, video security, autonomous driving and disaster assessment. Traditional change detection methods are mostly based on image differencing, optical flow estimation or manual feature matching. Such methods are simple in structure, but are easily disturbed by noise in the case of light changes, view angle differences or complex backgrounds, and the detection results are unstable. With the development of deep learning, convolutional neural networks have been widely used in change detection tasks. The typical approach is to use a twin network structure to extract features from double-time images, and use feature differencing, splicing or attention mechanisms to identify changed regions. This method significantly improves the automation and accuracy of detection, and has become the current mainstream of change detection.

[0003] However, there are still several challenges in existing deep learning-based change detection: on the one hand, due to the differences in shooting conditions, viewpoints and resolutions between double-time images, insufficient feature alignment can cause detection results to be misaligned or blurred; on the other hand, the lack of global context modeling capability makes it difficult for the model to accurately distinguish between changed and unchanged regions in non-textured or weak-textured areas. In addition, the fusion of deep semantic features and shallow spatial features is still insufficient, and the model is difficult to balance between preserving spatial details and capturing semantic information.

[0004] In recent years, to improve the semantic level and interpretability of the detection results, researchers have begun to explore the expansion from pixel-level change detection to object-level change detection. Object-level methods output change results in the form of detection boxes or masks by introducing target detection or instance segmentation mechanisms, which not only reflect the change location, but also identify the target class and morphological features, and have higher application value. However, at the same time, object-level methods are still constrained by cross-time registration accuracy and feature alignment capability, and when there are view angle differences or inconsistent lighting between images, the detection effect is easily affected, and the detection effect in weak-textured environments is also not ideal. SUMMARY

[0005] To solve the above problems, the present application provides an object-level change detection method based on semantic flow feature alignment and global attention fusion, which has the ability to detect target-level changes in scenes with view angle differences, weak textures, etc.

[0006] Step 1: Data Acquisition and Preprocessing. Obtain the dual-temporal image pairs and their corresponding change masks. Crop all images and adjust them to a resolution of 256×256 to ensure consistent model input. To enhance the model's generalization ability, data augmentation operations are applied to the training set, including color jitter, random affine transformation, rotation, and random cropping. These operations effectively simulate various changing scenes under real-world shooting conditions.

[0007] Step 2: Overall Model Structure. The overall framework uses a Siamese UNet network, consisting of two parameter-sharing encoding branches and one decoding branch. Two encoders take the original image and a reference image as inputs, respectively, to extract multi-level features across two time phases. During the encoding stage, shallow features retain more spatial details, while deeper features contain stronger semantic information, providing a foundation for subsequent feature fusion.

[0008] Step 3: Design of a self-global cross-attention module. A self-global cross-attention module is introduced into the encoder to enhance the model's global semantic modeling capability. This module consists of two parts: a self-attention unit and a cross-attention unit. The self-attention unit captures long-distance dependencies between features through a global attention mechanism, improving the model's ability to perceive textureless regions. The cross-attention unit calculates a weighted mapping based on the feature correlation between the original image and the reference image, achieving coarse registration of features across time phases. After processing by this module, feature representations with higher alignment accuracy can be obtained.

[0009] Step 4: Semantic Flow Alignment Module Construction. To further improve registration accuracy, a semantic flow alignment module is introduced before the decoder. This module consists of an optical flow enhancement submodule and a semantic flow field generation submodule: the optical flow enhancement submodule enhances the smoothness and robustness of feature representation through 1×1 convolution and the GELU activation function; the semantic flow field generation submodule generates a two-dimensional semantic flow field using 3×3 convolution and performs pixel-level alignment of the registered features through bilinear interpolation, thereby achieving fine fusion of dual-temporal features. This design effectively alleviates the boundary blurring and misalignment problems that occur in traditional optical flow estimation.

[0010] Step 5: Feature Optimization and Skip Connection Fusion. A feature optimization module is set up between the encoder and decoder to improve the efficiency of shallow and deep feature fusion. This module adopts a depthwise separable convolutional structure and combines it with residual connections to prevent the gradient vanishing problem. The optimized features and the semantically aligned features are concatenated along the channel dimension and passed to the decoder through skip connections, enabling the decoder to simultaneously possess spatial detail and semantic expressive capabilities.

[0011] Step 6: Detection Head Design and Result Output. The decoder output features are input into the CenterNet-based detection head. The detection head predicts the center point location, width and height parameters, and category information of each changed target, and outputs the object-level change results in the form of bounding boxes. Compared to pixel-level mask output, this result has a higher semantic level and interpretability, facilitating subsequent analysis or statistics.

[0012] Step 7: Model Training and Testing. The COCO-Inpainted dataset is used for model training and testing. The encoder backbone is a ResNet-50 network pre-trained on ImageNet, combined with the Adam optimizer and dynamic learning rate adjustment mechanism to ensure the stability and efficiency of model training.

[0013] This invention achieves fine-grained alignment of cross-temporal features and global semantic enhancement through the collaborative design of a global cross-attention module and a semantic flow alignment module. This effectively suppresses background interference and highlights truly changing areas in complex scenes. Experimental results demonstrate that this invention possesses the capability to detect target-level changes in scenes with different viewpoints. Attached Figure Description

[0014] To more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings are briefly described below. It should be noted that the following drawings are only used to illustrate some embodiments of the present invention. Those skilled in the art can still design other types of drawings based on these drawings without innovative effort. The contents of the drawings are as follows:

[0015] Figure 1 A flowchart illustrating an object-level change detection method based on semantic flow feature alignment and global attention fusion in an embodiment of the present invention is shown.

[0016] Figure 2 This invention illustrates the overall framework of the object-level change detection method in an embodiment of the present invention.

[0017] Figure 3 A schematic diagram of the global cross-attention and semantic flow alignment module in an embodiment of the present invention is shown. Detailed Implementation

[0018] The present invention will be further described below with reference to the accompanying drawings and embodiments, but this is not intended to limit the scope of the invention.

[0019] Example:

[0020] Reference Figure 1 An object-level change detection method based on semantic flow feature alignment and global attention fusion includes the following steps:

[0021] (1) First, the dual-temporal image pairs and their corresponding change masks in the dataset are uniformly cropped and adjusted to a resolution of 256×256 pixels. To enhance the model's adaptability to different environments and shooting conditions, image enhancement operations such as color jitter, affine transformation, and random rotation are performed on the training set.

[0022] (2) Feature extraction from twin encoders, please refer to Figure 2 The overall network framework diagram is based on the UNet architecture, with an added encoder forming a dual-branch architecture. Both branches perform the same processing on the dual-temporal images, and their weights are shared. During each feature extraction process, the feature maps obtained from the two branches are merged through channel concatenation and then fed into the global cross-attention and semantic flow alignment module for further processing. The processed feature maps are then jointly input into the decoder on the right. Simultaneously, the original skip connections of UNet are retained, and the feature maps at each level of the two sub-networks are concatenated through channel concatenation and then connected to the corresponding parts of the decoder, achieving effective fusion of multi-scale features.

[0023] (3) For construction of the global attention module, please refer to [link / reference]. Figure 3 For the features extracted by the dual-branch encoder and We employ multi-head self-attention (MSA) to perform global semantic modeling on single-phase images. Furthermore, we replace full self-attention with linear self-attention to significantly reduce the model's computational complexity while maintaining performance. The formula is as follows (using...). (For example):

[0024]

[0025]

[0026]

[0027]

[0028] Where SA(·) denotes the self-attention operation applied to these features. and These are learnable parameters that transform input features into query and keyword embeddings, denoted as Q and K, respectively. Similarly, Fq and From The derived projection feature representation, and This corresponds to the value features carrying semantic information. The attention matrix A is calculated using the dot product similarity between Q and K, and then softmax normalization is performed. Finally, the output feature map is generated by aggregating the value embeddings of A into V. It encodes global contextual relevance and enhances feature discriminability, especially in textureless regions. Simultaneously, to accurately predict changing regions within the original image features, its feature mapping needs to fuse semantic information from reference image features. To achieve this cross-image information interaction, after implementing a self-attention mechanism, a Global Collaborative Attention (GCA) mechanism is employed to perform feature-level registration and fusion of features from both temporal images, obtaining the initially registered features. , This achieves implicit alignment.

[0029] (4) Semantic flow alignment module construction. After processing by the SGCA module, the two images have initially obtained feature information from the other image. However, due to the limitations of SGCA, accurate image registration is still not possible. In order to calibrate the displacement between the images, we propose an optical flow alignment module (SFAM).

[0030] First, the features F1 are input into an optical flow enhancement module (ENFlow) consisting of 1 × 1 convolutions and Gaussian error linear units to enhance the representation of the initial features. The network, composed of convolutional layers, generates the differential flow field. This process can be represented as follows:

[0031]

[0032]

[0033]

[0034] By adding a differential flow field to the original feature point p Obtain the transformed coordinates Furthermore, a differentiable bilinear sampling mechanism is employed to compensate for feature loss caused by positional offset, ultimately generating features. This enables explicit feature alignment.

[0035] (5) Feature optimization and skip connection fusion: The semantically aligned features are concatenated with the shallow features of the encoder along the channel dimension and then input into the feature optimization module for fusion. This module uses a depthwise separable convolutional structure and residual connections to reduce computation and maintain feature continuity. The optimized features are passed to the decoder stage through skip connections to achieve multi-scale fusion of spatial details and semantic information. Through this structure, the model can enhance the semantic consistency and boundary clarity of features while maintaining resolution.

[0036] (6) Detection Head and Result Generation: The feature map output by the decoder is input into the detection head based on the CenterNet structure. The detection head consists of three parallel branches, which predict the target center point heatmap, width and height parameters, and offset, respectively. The model finally outputs the object-level change detection results in the form of bounding boxes, including the location, size, and category information of the changed target. Compared with the traditional pixel-level change masking method, this output format has a higher semantic level and interpretability, and is more suitable for subsequent change target statistics and scene understanding.

[0037] (7) Model training: We divided the Coco-inpainted dataset into training, validation, and test sets at 70%, 10%, and 20% ratios (the test set was divided into three classes based on the size of the changing objects (small, medium, large)). Simultaneously, we tested Synthtext-Change, VIRAT-STD, and Kubric-Change. We used a ResNet-50 encoder backbone pre-trained on ImageNet, along with its detection head. All images were cropped and resized to 256 × 256 pixels, and data augmentation was performed on the training set using color jitter and affine transformation. We trained the model for 200 epochs using a batch size of 16 and the AdamW optimizer with a weight decay of 0.0005. The learning rate was set to 0.0001. Validation was performed after each training epoch, and the best model on the validation set was used to evaluate the test set.

[0038] (8) Real-time inference and result output: To verify the performance of the algorithm of this invention in text detection and recognition tasks, we conducted tests and comparisons with the Cyws method on four datasets. We used AP value (mean precision, which is an indicator that comprehensively evaluates the detection performance of the model by calculating the area under the precision-recall curve) as the evaluation index for comparative analysis. The prediction accuracy results are shown in Table 1:

[0039] Method \ Dataset Coco-inpainted Synthtext-Change VIRAT-STD Kubric-Change Cyws 0.63 0.76 0.54 0.89 Our 0.66 0.79 0.58 0.90

[0040] As shown in Table 1, the object-level change detection method based on semantic flow feature alignment and global attention fusion of the present invention has a high accuracy.

[0041] The above description discloses only one preferred embodiment of the present invention, and should not be construed as limiting the scope of the present invention. Those skilled in the art will understand that all or part of the processes of the above embodiments can be implemented, and equivalent changes made in accordance with the claims of the present invention are still within the scope of the invention.

Claims

1. A method for object-level change detection based on semantic flow feature alignment and global attention fusion, characterized in that, The method comprises the following steps: Step 1: Obtain image training data and pre-process: collect a pair of dual-phase images (original image and reference image) and corresponding change masks, crop and adjust all image pairs and change masks to a uniform pixel size, and perform image enhancement on the training set images by using color jittering and affine transformation; Step 2: Construct an image object-level change detection network model: based on a twin UNet network, extract original image features and reference image features through two branch encoders; Step 3: Construct a self-global cross-attention module, fuse global context information to enhance feature expression in non-texture areas, and realize rough registration of dual-phase image features through cross-attention to obtain registered image features; Step 4: Introduce a semantic flow alignment module to estimate the semantic flow field of the registered image features and the original image features, perform pixel-level alignment on the registered image features based on the semantic flow field, and obtain the aligned image features; Step 5: Embed a feature optimization module in the skip connection between the encoder and the decoder, channel splice the aligned image features and the original image features, and input them into the decoder through the skip connection; Replace the original segmentation head of the twin UNet network with a detection head, input the decoding result into the detection head, and output the object-level change detection result in the form of a target box; Step 6: Model training, use a public change detection dataset to supervise the training and testing of the constructed change detection network model, and obtain the trained image object-level change detection model.

2. The object-level change detection method based on semantic stream feature alignment and global attention fusion according to claim 1, characterized in that, The self-global cross-attention module in step 3 comprises a self-attention unit and a cross-attention unit, the self-attention unit introduces global context information through a linear attention mechanism to enhance feature representation ability, and the cross-attention unit calculates a weighted mapping according to the similarity between the features of the two images to realize rough registration of dual-phase features.

3. The object-level change detection method based on semantic stream feature alignment and global attention fusion according to claim 1, characterized in that, The semantic flow alignment module in step 4 comprises an optical flow enhancement submodule and a semantic flow field generation submodule, the optical flow enhancement submodule is composed of a 1×1 convolution layer and a Gaussian error linear unit (GELU) and is used to enhance the initial feature expression; The semantic flow field generation submodule generates a two-dimensional semantic flow field by using a 3×3 convolution, and realizes pixel-level alignment of the registered features by using bilinear interpolation.

4. The object-level change detection method based on semantic stream feature alignment and global attention fusion according to claim 1, characterized in that, The feature optimization module in step 5 uses a depth separable convolution structure to improve the fusion effect of shallow and deep features, and uses a residual connection to prevent gradient disappearance; and uses a detection head based on an anchor-free detection structure of CenterNet to output the change detection result in the form of a target box by predicting the center point, width and height, and class information of the target.

5. The object-level change detection method based on semantic stream feature alignment and global attention fusion according to claim 1, characterized in that, The step six, using COCO-Inpainted dataset for training and testing, using ResNet-50 pre-trained on ImageNet as the encoder backbone network, keeping its detection head structure; all input images are cropped and adjusted to 256x256 pixels, the training batch size is 16, the optimizer uses AdamW, the weight decay is set to 0.0005, the learning rate is 0.0001, the model training is performed for a total of 200 epochs, and the verification is performed after the end of each training period, and the model with the optimal performance on the verification set is selected for test set evaluation.

Images