Image target level change detection method based on feature registration and optical flow alignment

Abstract

The invention belongs to the technical field of change detection, and discloses an image target level change detection method based on feature registration and optical flow alignment, which comprises the following steps: acquiring image training data, and preprocessing the data; constructing an image target level change detection network model based on feature registration and optical flow alignment; the change detection model is trained; and inputting images of different time phases into the trained change detection model to obtain a final change detection result. Experiments show that the method has the capability of target-level change detection in a view angle difference scene.

Description

Technical Field

[0001] The invention relates to the technical field of computer vision and image processing, and in particular to an image object level change detection method based on feature registration and optical flow alignment. Background Art

[0002] The main goal of change detection is to identify different regions between two compared images, which may be formed due to the appearance or disappearance of objects, the movement of the background, or the change of shape. Change detection has important application value in many fields, including urban development planning, real-time environmental monitoring, autonomous driving, and disaster loss assessment. However, in practical applications, due to the differences in shooting equipment and viewing angles, imaging conditions often vary, and change detection systems need to achieve fast reasoning, which together bring significant challenges to change detection.

[0003] One of the most pressing issues to be addressed is the perspective differences between images. Obtaining accurate image registration for change detection remains a challenge in both traditional machine learning and deep learning-based approaches. Currently, almost all commonly used pixel-based change detection methods require aligning the two compared images before detecting changes, or assume that multi-temporal images are strictly registered. This reliance on accurate registration is often difficult to implement in practice, especially in the presence of large perspective differences or dynamic backgrounds.

[0004] Another important issue is change detection at the object level. Although high-precision pixel-level segmentation can provide detailed change information, it may lead to information redundancy, thereby increasing the time of calculation and reasoning. In some practical application scenarios, such as security monitoring and traffic monitoring, such high precision is not always necessary. Object-level change detection can reduce the computational complexity while maintaining detection accuracy, and improve the real-time performance and efficiency of the system.

[0005] The current mainstream change detection network is not good at processing the perspective differences between images, making it difficult to achieve accurate image registration, which limits the accuracy and robustness of change detection. In addition, many methods focus on high-precision pixel-level change detection, which may lead to information redundancy and increased computational burden. In some practical applications, such as security monitoring and traffic monitoring, there is a lack of the ability to perform efficient change detection at the object level. Therefore, how to improve the system's computational efficiency and ability to adapt to different perspective differences while ensuring the accuracy of change detection is a key issue that needs to be solved urgently. Summary of the invention

[0006] In order to solve the above problems existing in the prior art, the present invention provides an image object-level change detection method based on feature registration and optical flow alignment, which has the ability to detect object-level changes in scenes with perspective differences.

[0007] (1) Obtain image training data and preprocess the data: adjust all image pairs and change masks to a uniform pixel size, and use color jitter and affine transformation as image enhancement on the training set;

[0008] (2) Construct an image target-level change detection network model based on feature registration and optical flow alignment: Based on the twin UNet network, use the original image features and reference image features extracted by the two branch encoders. Construct a feature registration module so that the original image features have the position information of the reference image features, achieve rough registration, obtain the registered image features, and estimate the optical flow information of the registered image features and the original image features by adding an optical flow alignment module. Use the optical flow information to align the registered image features at the pixel level to obtain the aligned image features. Finally, the aligned image features are channel-joined with the original image features and sent to the decoder through a jump connection. Replace the original segmentation head with the detection head, input the decoding result into the detection head, and output the change detection result in the form of a target box;

[0009] (3) Training the change detection model: Using a public change detection dataset, the image object-level change detection network model based on feature registration and optical flow alignment is trained and tested to obtain a trained change detection model.

[0010] (4) Input the image pairs at different time phases into the trained change detection model to obtain the final detection results;

[0011] Furthermore, the feature registration module of step (2) adopts a method based on self-attention and collaborative attention, firstly introducing a self-attention mechanism combined with global context information to enhance the feature expression ability of textureless areas. Then, a cross-attention mechanism is used to calculate the weighted sum of the feature vectors of the positions in the enhanced original image features and the feature vectors of all positions in the enhanced reference image features, with the weights determined by their similarity to the positions in the original image features. This can be considered as distorting the feature vectors of an image in space to obtain the registered image features, thereby achieving preliminary image registration.

[0012] Furthermore, in the optical flow alignment module in step (2), the registered image features and the original image features are first sent to an optical flow enhancement module composed of a 1×1 convolution and a Gaussian error linear unit (GELU) to enhance the expressiveness of the initial features. Subsequently, the enhanced registered image features are concatenated with the original image features, and a convolution of size 3×3 is performed to generate an optical flow field of dimension 2. Through the optical flow field, the registered image features are fine-tuned at the pixel level using a bilinear interpolation method to achieve surface alignment of the registered image pixels, obtain the aligned image features, and achieve pixel-level alignment of the registered images.

[0013] Furthermore, the detection head of step (2) adopts Centerhead detection head, which can accurately locate and classify the target frame by predicting the center point position, size and category information of the target. The detection head can efficiently convert feature maps into target frames, improving the accuracy and real-time performance of change detection.

[0014] Furthermore, the network training method in step (3) is as follows: the heat map, width-height and offset losses are calculated based on the model's prediction results for the original image and the true label, and these losses are aggregated into an overall loss, the model parameters are updated through back propagation, and the training is performed using a supervised training method.

[0015] In combination with the above technical solutions and the technical problems solved, the advantages and positive effects of the technical solutions to be protected by the present invention are as follows:

[0016] First, the present invention replaces the segmentation head of the original twin UNet network with a detection head, inputs the results into the detection head, and outputs the change detection results in the form of a target box, while maintaining the detection accuracy, reducing the computational complexity, and improving real-time performance and efficiency.

[0017] Second, the present invention registers images by introducing a feature registration module for preliminary registration between images. The optical flow field between the registered image and the original image is estimated by introducing an optical flow alignment module, and calibration is performed to explicitly calibrate the offset between images. This solves the problem of perspective difference. Experimental results show that the present invention has the ability to detect target-level changes in perspective difference scenes. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0019] Figure 1The present invention is a schematic diagram of the steps of image object-level change detection based on feature registration and optical flow alignment according to an embodiment of the present invention.

[0020] Figure 2 A schematic diagram of the model structure of an image object-level change detection method based on feature registration and optical flow alignment according to an embodiment of the present invention.

[0021] Figure 3 Schematic diagram of a feature registration module based on self-attention and collaborative attention according to an embodiment of the present invention.

[0022] Figure 4 Schematic diagram of an optical flow alignment module according to an embodiment of the present invention.

[0023] Figure 5 Schematic diagram of detection results of image pairs and their change masks and models at different viewpoints in an embodiment of the present invention. DETAILED DESCRIPTION

[0024] The contents of the present invention are further described below in conjunction with the drawings and embodiments, but are not intended to limit the present invention.

[0025] Example:

[0026] Reference Figure 1 , an image object-level change detection method based on feature registration and optical flow alignment, comprising the following steps:

[0027] Step 1: Obtain image training data and preprocess the data.

[0028] Specifically, in this embodiment, image pairs and change masks are obtained from the Coco-inpainted dataset, and are adjusted to 256×256 pixels when the data is loaded, and color jittering and affine transformation are used as image enhancement on the training set.

[0029] Step 2: Build an image object-level change detection method based on feature registration and optical flow alignment. Figure 2 :Based on the twin UNet network, the feature registration module is constructed using the feature maps extracted by the two branch encoders to make the original image have the position information of the reference image, and achieve rough registration. The optical flow information of the registered image and the original image is estimated by adding the optical flow alignment module. The registered image is aligned at the pixel level through the optical flow information, and finally the feature map after channel splicing is spliced ​​with the original image, and sent to the decoder through the jump connection to obtain the change detection result. Replace the original segmentation head with the detection head, input the result into the detection head, and output the change detection result in the form of target box.

[0030] Step 2.1: Build the basic twin UNet network: The basic UNet network structure has a total of 5 layers, mainly composed of two parts: the encoder and the decoder, each of which contains five convolution blocks. Each convolution block consists of two 3×3 convolution operations and a ReLU activation function, followed by downsampling through a maximum pooling with a stride of 2. The input image of size 256×256×1 is processed layer by layer by the encoder. The feature map size is halved after each layer of downsampling, and the number of channels is doubled, and finally a 16×16×512 high-level feature map is obtained. The decoder part consists of five upsampling layers, each of which doubles the feature map size through transposed convolution and reduces the number of channels. After each upsampling, the feature map of the corresponding layer of the decoder is channel-joined with the feature map of the corresponding layer of the encoder to fuse the shallow information. Finally, the decoder outputs a feature map of 256×256×64, and all channels are fused through 1×1 convolution to generate a feature map of 256×256×32. The basic twin UNet network adds an input branch based on the basic UNet network, and shares the weight value of the left half encoder between the two inputs, extracts features from the two inputs respectively, and then concatenates the two extracted feature maps through channels and inputs them to the right half decoder together. At the same time, the original jump connection of UNet is retained, and the feature maps of each level obtained by the two sub-networks are connected to the corresponding part of the right decoder through concatenate channel splicing.

[0031] Step 2.2: Construct feature registration module: The feature registration module used in this embodiment uses a self-attention plus collaborative attention mechanism, see Figure 3 Among them, multi-head self-attention is used for the features f1 and f2 extracted by the encoder, and the feature expression of the texture-free image area is enhanced by introducing global context information to obtain f1s and f2s.

[0032] In order to predict the changed area in the original image features, its feature map must also embed information from the reference image features. In order to allow this information exchange, the collaborative attention mechanism is used in this instance to achieve image registration, that is, each feature vector at position (x1, y1) in f1s pays attention to the feature vectors of all positions (x2, y2) in f2s and connects them to their weighted sum. This can be thought of as spatially distorting the feature vectors of an image so that the two images are registered. Formally, we get the registration feature F'=ψ(f1s,f2s), where ψ(,) is a cross-attention mechanism defined as:

[0033]

[0034] in

[0035]

[0036] and

[0037] Q=w q f1sK=w k f2sV=f2s

[0038] Among them, w q and w k is a learnable parameter and the dim parameter specifies the dimension in which to operate. Therefore, the feature map F' is conditioned on the image and contains enough information to localize the changes.

[0039] Step 2.3: Construction of the optical flow alignment module includes: The optical flow alignment module used in this embodiment, please refer to Figure 4 First, the registered image and the original image are sent to the optical flow enhancement module composed of 1*1 convolution and Gaussian error linear unit (GELU) to enhance the expressiveness of the initial features. Then, the enhanced registered image features are concatenated with the original image features and a 3×3 convolution is performed to generate an optical flow field W with a dimension of 2. p ∈R 2*H*W ; ΔF stores the offset between the registered image features and each position in the original image on the standard spatial grid. The registered image F' obtained by feature registration and the optical flow information W between the two are obtained. p , fine-tune the pixel level by bilinear interpolation to obtain the feature map F' wrap ;

[0040]

[0041] where p w Indicates F' warp The midpoint position, N (p) represents the four neighboring positions of p (upper left, lower left, upper right, lower right), w p Bilinear kernel weights on a curved space grid for optical flow information.

[0042] Step 2.3: Integrate the feature registration module and the optical flow alignment module into the twin UNet network model. The feature registration module is constructed by using the two bi-temporal feature maps output by the weight-sharing encoder to give the original image the position information of the reference image, and to achieve rough registration. The optical flow alignment module is added to obtain the optical flow field of the registered image and the original image. The optical flow field is used to perform pixel-level correction on the preliminarily registered feature map using the bilinear interpolation method, and finally the concatenate channel is used to splice with the original image through the jump connection to send it to the decoder to obtain the result.

[0043] Step 2.4: Build a detection head to replace the original segmentation head, input the result into the detection head, and output the change detection result in the form of target box;

[0044] Step 3: Based on the public change dataset, train and test the constructed change detection model, calculate various losses based on the model's prediction results for the image and the true label, and summarize these losses into the overall loss, update the model parameters through back propagation, and use supervised training to perform target detection training. Obtain a trained change detection model.

[0045] Specifically, in this embodiment, the Coco-inpainted dataset is divided into a training set, a validation set, and a test set at a ratio of 70%, 10%, and 20% (the test set is divided into 3 categories according to the size of the changed object (small, medium, and large)), and the network model obtained in step 2 is calculated according to the model's prediction results of image changes and the true label, and these losses are summarized as the overall loss. The model parameters are updated through back propagation, and supervised training is used for training. Set appropriate key hyperparameters such as Batch Size and learning rate. Verification is performed after each training, and the test set is evaluated using the best model on the validation set. The total loss function is:

[0046] L det =L k +λ size L size +λ off L off

[0047] The term L k , L size , L off Corresponding to heatmap, width-height and offset respectively. size ,λ off to balance the weight of each lost item.

[0048] Step 4: Input the images of different time phases into the trained change detection model to obtain the final change detection result. For the image pair to be detected, input each image block into the network model trained in step 3 and output the change detection result. Figure 5 .

[0049] In order to further verify the advantages of the method proposed in the present invention, the model of this embodiment is used to test the Coco-Inpainted dataset using the UNet model and the model of the target-level change detection method based on feature registration and optical flow alignment, and the AP value (average precision, which is an indicator for comprehensively evaluating the model detection performance by calculating the area under the precision and recall curves) is used as an evaluation index for comparative analysis. The results of prediction accuracy are shown in Table 1:

[0050]

[0051] It can be concluded from Table 1 that the model of the image object-level change detection method based on feature registration and optical flow alignment of the present invention has a higher accuracy in the perspective difference scene.

[0052] What is disclosed above is only a preferred embodiment of the present invention, and it certainly cannot be used to limit the scope of rights of the present invention. Ordinary technicians in this field can understand that all or part of the processes of the above embodiment and equivalent changes made according to the claims of the present invention still fall within the scope of the invention.

Claims

1. An image object-level change detection method based on feature registration and optical flow alignment, characterized in that: The following steps are involved: Step 1: Obtain image training data and preprocess the data. All image pairs and their corresponding change masks are cropped and resized to a uniform pixel size, and image enhancement techniques such as color jitter and affine transformation are used to process the images on the training set. Step 2: Construct an image target-level change detection network model based on feature registration and optical flow alignment: Based on the twin UNet network, use the original image features and reference image features extracted by the two branch encoders. Construct a feature registration module so that the original image features have the position information of the reference image features, achieve rough registration, and obtain the registered image features. Add an optical flow alignment module to estimate the optical flow information of the registered image features and the original image features. Use the optical flow information to align the registered image features at the pixel level to obtain the aligned image features. Finally, the aligned image features are channel-joined with the original image features and sent to the decoder through jump connections. Replace the original segmentation head with the detection head, input the decoding result into the detection head, and output the change detection result in the form of a target box; Step 3, using a public change detection dataset, training and testing the constructed change detection model to obtain a trained image object-level change detection model based on feature registration and optical flow alignment; Step 4: Input the image pairs of different phases into the trained change detection model to obtain the final detection results.

2. The image object level change detection method based on feature registration and optical flow alignment according to claim 1, characterized in that: In step 2, the original segmentation head is replaced with the detection head. In view of the application requirements of object-level change detection, the model does not include a default segmentation head, but inputs the feature map output by the decoder into the detection head, and realizes the object-level change detection task by predicting the center point of the object and its related attributes.

3. The object-level change detection method based on feature registration and optical flow alignment according to claim 1, characterized in that: The feature registration module in step 2 adopts a method based on self-attention and collaborative attention. First, the self-attention mechanism is introduced to combine with global context information to enhance the feature expression ability of texture-free areas. Then, the cross-attention mechanism is used to calculate the weighted sum of the feature vectors of the positions in the enhanced original image features and the feature vectors of all positions in the enhanced reference image features. The weights are determined by their similarity to the positions in the original image features. This can be considered as distorting the feature vectors of an image in space to obtain the registered image features, thereby achieving preliminary image registration.

4. The object-level change detection method based on feature registration and optical flow alignment according to claim 1, characterized in that: In the step 2, the optical flow registration module first sends the registered image features and the original image features to an optical flow enhancement module composed of 1×1 convolution and Gaussian error linear unit (GELU) to enhance the expressiveness of the initial features. Subsequently, the enhanced registered image features are concatenated with the original image features, and a convolution of size 3×3 is performed to generate an optical flow field of dimension 2. Through the optical flow field, the registered image features are fine-tuned at the pixel level using bilinear interpolation to achieve surface alignment of the registered image pixels, obtain the aligned image features, and achieve pixel-level alignment of the registered images.

Images