A method for detecting changes in image targets based on feature registration and optical flow alignment

Abstract

This invention belongs to the field of change detection technology and discloses an image target-level change detection method based on feature registration and optical flow alignment. The method includes: acquiring image training data and preprocessing the data; constructing an image target-level change detection network model based on feature registration and optical flow alignment; training the change detection model; and inputting images from different time phases into the trained change detection model to obtain the final change detection result. Experiments show that this invention has the capability of target-level change detection in scenes with different viewpoints.

Description

Technical Field

[0001] This invention relates to the fields of computer vision and image processing technology, specifically a method for detecting changes in image targets based on feature registration and optical flow alignment. Background Technology

[0002] The primary goal of change detection is to identify distinct regions between two comparing images. These regions may be formed due to the appearance or disappearance of objects, background motion, or changes in shape. Change detection has significant applications in many fields, including urban development planning, real-time environmental monitoring, autonomous driving, and disaster loss assessment. However, in practical applications, imaging conditions often vary due to differences in shooting equipment and perspectives. Furthermore, change detection systems require rapid inference, all of which present significant challenges to change detection.

[0003] One pressing issue is the difference in viewpoints between images. Achieving accurate image registration for change detection remains a challenge, both in traditional machine learning methods and deep learning-based approaches. Currently, almost all commonly used pixel-based change detection methods require aligning two compared images before detecting a change, or assume that multi-temporal images are strictly registered. This reliance on precise registration is often difficult to implement in practical applications, especially in situations with significant viewpoint differences or dynamic backgrounds.

[0004] Another important issue is change detection at the target level. While high-precision pixel-level segmentation can provide detailed change information, it may lead to information redundancy, thereby increasing computation and inference time. In some practical applications, such as security monitoring and traffic monitoring, such high precision is not always necessary. Target-level change detection can reduce computational complexity and improve the real-time performance and efficiency of the system while maintaining detection accuracy.

[0005] Current mainstream change detection networks are inadequate in handling viewpoint differences between images, making accurate image registration difficult and limiting the accuracy and robustness of change detection. Furthermore, many methods focus on high-precision pixel-level change detection, potentially leading to information redundancy and increased computational burden. In some practical applications, such as security and traffic monitoring, they lack the ability to perform efficient change detection at the target level. Therefore, improving the system's computational efficiency and adaptability to different viewpoint differences while ensuring change detection accuracy is a critical issue that urgently needs to be addressed. Summary of the Invention

[0006] To address the problems existing in the prior art, this invention provides an image target-level change detection method based on feature registration and optical flow alignment, which has the ability to detect target-level changes in scenes with different viewpoints.

[0007] (1) Obtain image training data and preprocess the data: adjust all image pairs and transformation masks to a uniform pixel size, and use color dithering and affine transformation as image enhancement on the training set;

[0008] (2) Constructing an image target-level change detection network model based on feature registration and optical flow alignment: Based on the Siamese UNet network, original image features and reference image features extracted by two branch encoders are used. A feature registration module is constructed to enable the original image features to have the positional information of the reference image features, achieving coarse registration and obtaining registered image features. An optical flow alignment module is added to estimate the optical flow information between the registered image features and the original image features. The registered image features are then aligned at the pixel level using the optical flow information to obtain aligned image features. Finally, the aligned image features and the original image features are concatenated by channels and sent to the decoder via a skip connection method. The original segmentation head is replaced with a detection head, and the decoding result is input into the detection head, outputting the change detection result in the form of a target bounding box.

[0009] (3) Training the change detection model: Using a public change detection dataset, the image target-level change detection network model based on feature registration and optical flow alignment is trained and tested to obtain a trained change detection model.

[0010] (4) Input the images from different time phases into the trained change detection model to obtain the final detection results;

[0011] Furthermore, the feature registration module in step (2) employs a method based on self-attention and collaborative attention. First, a self-attention mechanism is introduced to combine global contextual information to enhance the feature representation capability of textureless regions. Then, a cross-attention mechanism is used to calculate the weighted sum of the feature vectors at positions in the enhanced original image features and the feature vectors at all positions in the enhanced reference image features. The weights are determined by their similarity to positions in the original image features. This can be considered as spatially distorting the feature vectors of an image to obtain the registered image features, thereby achieving the initial registration of the image.

[0012] Furthermore, in step (2), the optical flow alignment module first feeds the registered image features and the original image features into an optical flow enhancement module composed of a 1×1 convolution and a Gaussian error linear unit (GELU) to enhance the expressive power of the initial features. Then, the enhanced registered image features are concatenated with the original image features, and a 3×3 convolution is performed to generate a 2-dimensional optical flow field. Through the optical flow field, bilinear interpolation is used to fine-tune the registered image features at the pixel level, achieving pixel-level alignment of the registered image pixels and obtaining aligned image features, thus realizing pixel-level alignment of the registered image.

[0013] Furthermore, the detection head in step (2) uses a Centerhead detection head, which predicts the center point position, size, and category information of the target to achieve accurate localization and classification of the target bounding box. This detection head can efficiently convert feature maps into target bounding boxes, improving the accuracy and real-time performance of change detection.

[0014] Furthermore, in step (3), the network training method is as follows: calculate the heatmap, width-height and offset losses based on the model's prediction results of the original image and the real labels, summarize these losses into the overall loss, update the model parameters through backpropagation, and train using supervised training.

[0015] Based on the above technical solutions and the technical problems solved, the advantages and positive effects of the technical solution to be protected by this invention are as follows:

[0016] First, this invention replaces the segmentation head of the original twin UNet network with a detection head, inputs the results into the detection head, and outputs the change detection results in the form of target boxes, maintaining detection accuracy while reducing computational complexity and improving real-time performance and efficiency.

[0017] Second, this invention introduces a feature registration module for initial image registration. An optical flow alignment module is then introduced to estimate and calibrate the optical flow field between the registered and original images, explicitly correcting the image offset. This addresses the viewpoint difference problem. Experimental results demonstrate that this invention possesses the capability to detect target-level changes in scenes with viewpoint differences. Attached Figure Description

[0018] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0019] Figure 1This is a schematic diagram illustrating the steps of image target-level change detection based on feature registration and optical flow alignment according to an embodiment of the present invention.

[0020] Figure 2 This is a schematic diagram of the model structure of an image target-level change detection method based on feature registration and optical flow alignment according to an embodiment of the present invention.

[0021] Figure 3 This is a schematic diagram of a feature registration module based on self-attention and collaborative attention according to an embodiment of the present invention.

[0022] Figure 4 This is a schematic diagram of the optical flow alignment module according to an embodiment of the present invention.

[0023] Figure 5 This is a schematic diagram illustrating the detection results of image pairs, their change masks, and models from different viewpoints in an embodiment of the present invention. Detailed Implementation

[0024] The present invention will be further described below with reference to the accompanying drawings and embodiments, but this is not intended to limit the scope of the invention.

[0025] Example:

[0026] Reference Figure 1 A method for detecting changes in image targets based on feature registration and optical flow alignment includes the following steps:

[0027] Step 1: Obtain image training data and preprocess the data.

[0028] Specifically, in this embodiment, image pairs and transformation masks are obtained from the Coco-inpainted dataset, and both are adjusted to 256×256 pixels when the data is loaded. Color dithering and affine transformation are used as image enhancement on the training set.

[0029] Step 2: Construct an image target-level change detection method based on feature registration and optical flow alignment. Please refer to [link to relevant documentation]. Figure 2 Based on the Siamese UNet network, a feature registration module is constructed using feature maps extracted from two branch encoders to give the original image positional information from the reference image, achieving coarse registration. An optical flow alignment module is then added to estimate the optical flow information between the registered and original images, performing pixel-level alignment using this optical flow information. Finally, the channel-stitched feature map is concatenated with the original image and fed into the decoder via a skip connection to obtain the change detection result. The original segmentation head is replaced with a detection head, and the result is input into the detection head, outputting the change detection result as a bounding box.

[0030] Step 2.1: Constructing the Basic Siamese UNet Network: The basic UNet network structure consists of 5 layers, mainly composed of an encoder and a decoder, each containing five convolutional blocks. Each convolutional block consists of two 3×3 convolution operations and a ReLU activation function, followed by downsampling through max pooling with a stride of 2. The input image of size 256×256×1 is processed layer by layer by the encoder. The feature map size is halved after downsampling at each layer, while the number of channels is doubled, ultimately resulting in a high-level feature map of 16×16×512. The decoder consists of five upsampling layers. Each layer doubles the feature map size and reduces the number of channels through transposed convolution. After each upsampling, the feature map of the corresponding layer in the decoder is concatenated with the feature map of the corresponding layer in the encoder, fusing shallow information. Finally, the decoder outputs a 256×256×64 feature map, which is then fused through a 1×1 convolution to generate a 256×256×32 feature map. The basic Siamese UNet network adds an input branch to the basic UNet network and shares the weights of the left-hand encoder between the two inputs. It extracts features from both inputs separately, then concatenates the two feature maps using a concatenation technique before feeding them into the right-hand decoder. Simultaneously, it retains the original skip connections of UNet, concatenating the feature maps from each level of the two sub-networks and connecting them to the corresponding parts of the right-hand decoder using a concatenation technique.

[0031] Step 2.2: Constructing the Feature Registration Module: The feature registration module used in this embodiment employs a self-attention plus collaborative attention mechanism. Please refer to [link to relevant documentation]. Figure 3 Multi-head self-attention is used for the features f1 and f2 extracted by the encoder. Global contextual information is introduced to enhance the feature representation of textureless image regions, resulting in f1s and f2s.

[0032] To predict regions of change in the features of the original image, its feature map must also embed information from the features of the reference image. To allow this information exchange, a collaborative attention mechanism is used in this example to achieve image registration. Specifically, each feature vector at position (x1, y1) in f1s attends to the feature vectors at all positions (x2, y2) in f2s and concatenates them to a weighted sum. This can be viewed as spatially distorting the feature vectors of one image, thus registering the two images. Formally, we obtain the registration feature F' = ψ(f1s, f2s), where ψ(,) is defined as the cross-attention mechanism, defined as:

[0033]

[0034] in

[0035]

[0036] and

[0037] Q = w q f1sK=w k f2sV=f2s

[0038] Among them, w q and w k The `dim` parameter specifies the dimension of the operation, which is a learnable parameter. Therefore, the feature map F' is image-conditioned and contains enough information to locate changes.

[0039] Step 2.3: Construction of the optical flow alignment module includes: the optical flow alignment module used in this embodiment, please refer to [link to relevant documentation]. Figure 4 First, the registered image and the original image are fed into an optical flow enhancement module consisting of a 1x1 convolution and Gaussian error linear units (GELUs) to enhance the expressive power of the initial features. Then, the enhanced registered image features are concatenated with the original image features, followed by a 3x3 convolution to generate a 2-dimensional optical flow field W. p ∈R 2*H*W ΔF stores the offsets of the registered image features to each location in the original image on a standard spatial grid. This results in the registered image F' obtained from feature registration, and the optical flow information W between the two. p The feature map F' is obtained by fine-tuning at the pixel level using bilinear interpolation. wrap ;

[0040]

[0041] Where p w Indicates F' warp The position of the midpoint, N (p) Let w represent the four neighborhood positions of p (top left, bottom left, top right, bottom right). p The bilinear kernel weights are used on the curved spatial grid of optical flow information.

[0042] Step 2.3: Integrate the feature registration module and optical flow alignment module into the Siamese UNet network model. A feature registration module is constructed using two bi-temporal feature maps output by a weight-shared encoder, enabling the original image to possess the positional information of the reference image, achieving coarse registration. An optical flow alignment module is then added to obtain the optical flow fields of the registered image and the original image. The optical flow fields are used to perform pixel-level correction on the initially registered feature maps using bilinear interpolation. Finally, the concatenated channel is used to stitch the image with the original image, and the result is fed into the decoder via a skip connection to obtain the final result.

[0043] Step 2.4: Construct a detection head to replace the original segmentation head, input the results into the detection head, and output the change detection results in the form of target bounding boxes;

[0044] Step 3: Based on the publicly available change dataset, train and test the constructed change detection model. Calculate various losses based on the model's image predictions and the ground truth labels, and summarize these losses into a total loss. Update the model parameters through backpropagation and perform supervised training for object detection. The trained change detection model is then obtained.

[0045] Specifically, in this embodiment, the Coco-inpainted dataset is divided into training, validation, and test sets at a ratio of 70%, 10%, and 20% respectively (the test set is divided into three categories based on the size of the changed object (small, medium, large)). For the network model obtained in step two, various losses are calculated based on the model's predictions of image changes and the ground truth labels. These losses are then summed into a total loss. The model parameters are updated through backpropagation, and supervised training is used. Appropriate batch size, learning rate, and other key hyperparameters are set. Validation is performed after each training iteration, and the best model on the validation set is used to evaluate the test set. The total loss function is:

[0046] L det =L k +λ size L size +λ off L off

[0047] Among them, item L k L size L off These correspond to the heatmap, width-height, and offset, respectively. Use λ. size , λ off To balance the weight of each lost item.

[0048] Step 4: Input images from different time phases into the trained change detection model to obtain the final change detection results. For each image pair to be detected, input each image patch into the network model trained in Step 3, and output the change detection results. Please refer to [link to relevant documentation]. Figure 5 .

[0049] To further verify the advantages of the proposed method, the model from this embodiment was tested on the Coco-Inpainted dataset using both the UNet model and a target-level change detection method based on feature registration and optical flow alignment. The AP value (mean precision, an indicator that comprehensively evaluates model detection performance by calculating the area under the precision-recall curve) was used as the evaluation metric for comparative analysis. The prediction accuracy results are shown in Table 1.

[0050]

[0051] As shown in Table 1, the model of the image target-level change detection method based on feature registration and optical flow alignment of the present invention has a high accuracy in scenes with different viewpoints.

[0052] The above description discloses only one preferred embodiment of the present invention, and should not be construed as limiting the scope of the present invention. Those skilled in the art will understand that all or part of the processes of the above embodiments can be implemented, and equivalent changes made in accordance with the claims of the present invention are still within the scope of the invention.

Claims

1. A method for detecting image target-level changes based on feature registration and optical flow alignment, characterized in that, Includes the following steps: Step 1: Obtain image training data, preprocess the data, crop all image pairs and their corresponding transformation masks, adjust them to a uniform pixel size, and use color dithering and affine transformation to perform image enhancement processing on the images on the training set. Step 2: Construct an image target-level change detection network model based on feature registration and optical flow alignment: Based on the Siamese UNet network, using the original image features and reference image features extracted by the two branch encoders, a feature registration module is constructed to enable the original image features to have the positional information of the reference image features, achieving coarse registration and obtaining registered image features. Then, by adding an optical flow alignment module, the optical flow information between the registered image features and the original image features is estimated. The registered image features are then aligned at the pixel level using the optical flow information to obtain aligned image features. Finally, the aligned image features and the original image features are concatenated by channels and sent to the decoder through a skip connection method, replacing the original segmentation head with the detection head. The decoding result is input into the detection head, and the change detection result is output in the form of a target bounding box. Step 3: Use a publicly available change detection dataset to train and test the constructed change detection model to obtain a trained image target-level change detection model based on feature registration and optical flow alignment. Step 4: Input the images from different time phases into the trained change detection model to obtain the final detection results: The feature registration module in step two adopts a method based on self-attention and collaborative attention. First, a self-attention mechanism is introduced in combination with global context information to enhance the feature representation capability of textureless regions. Then, the cross-attention mechanism is used to calculate the weighted sum of the feature vectors of the enhanced original image features and the feature vectors of all positions in the enhanced reference image features. The weights are determined by the similarity between all positions in the reference image features and the positions in the original image features. This is considered to spatially distort the feature vectors of an image to obtain the registered image features, thereby achieving the initial registration of the image. In step two, the optical flow alignment module first feeds the registered image features and the original image features into an optical flow enhancement module composed of a 1×1 convolution and a Gaussian error linear unit to enhance the expressive power of the initial features. Then, the enhanced registered image features are concatenated with the original image features and a 3×3 convolution is performed to generate an optical flow field with a dimension of 2. By using the optical flow field and bilinear interpolation, pixel-level fine-tuning of the registered image features is performed to achieve pixel-level alignment of the registered image, resulting in aligned image features and thus achieving pixel-level alignment of the registered image.

2. The image target-level change detection method based on feature registration and optical flow alignment according to claim 1, characterized in that, In step two, the original segmentation head is replaced with a detection head. To meet the application requirements of target-level change detection, the model does not include a default segmentation head. Instead, the feature map output by the decoder is input into the detection head. By predicting the center point of the target and its related attributes, the target-level change detection task is achieved.

Images