An image difference recognition method based on convolution and attention mechanism

Abstract

The application discloses an image difference recognition method based on convolution and attention mechanism, wherein a template image and a contrast image are respectively input into an encoder, and after being encoded by multiple layers of query-encoding layers composed of query blocks and encoding blocks, the template image and the contrast image are output to a decoder; in a training stage, a model is trained by inputting corresponding labels of sample pairs and setting a loss function; and in an inference stage, a prediction result is output by the model decoder; and the content difference recognition between multiple images can be automatically realized by using a computer program and an image sensor, so that a robust intelligent method for recognizing sensitive content by image comparison in practical application is provided.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of intelligent recognition of artificial intelligence machine vision, and particularly relates to an image difference recognition method based on convolution and attention mechanism. BACKGROUND

[0002] It is a common requirement to record pictures of the same scene at different times using image sensors such as cameras, and to compare these pictures to identify changes in the scene. For example, from the ground change images of the same place at different times taken by a remote sensing satellite, the evolution of various observable ground objects such as buildings, soil, vegetation, and important facilities can be identified. Or, through a microscope, images of different times of cells, colonies, etc. can be observed to observe the progress of the experiment. Or, by recording pictures of equipment operation at different times, the damage and aging of the equipment can be inferred. In most of these scenarios, the changes in the content cannot be automatically identified by simply changing the image color and pixel value, especially when the camera pose, shooting environment, shooting parameters, and other conditions cannot be guaranteed to be consistent when shooting in reality. Industrial applications urgently need a robust method that can intelligently identify image content differences under certain shooting errors. SUMMARY

[0003] The present application provides an image difference recognition method based on convolution and attention mechanism that can resist changes in camera angle, pose, parameters, and lighting environment.

[0004] The technical scheme adopted by the present application to overcome the technical problems is:

[0005] An image difference recognition method based on convolution and attention mechanism, comprising the following steps:

[0006] a) establishing an image difference recognition model, which is composed of a query encoder, a decoder, and a classifier;

[0007] b) inputting a template image and a comparison image into the encoder;

[0008] c) inputting the output result of the encoder into the decoder for decoding to output a predicted result image.

[0009] Further, step b) comprises the following steps:

[0010] b-1) the encoder is stacked with N query encoding layers, the input of each query encoder comes from the output of the previous query encoder, N is a positive integer greater than or equal to 2, and each query encoder is composed of a query block and an encoding block;

[0011] b-2) input the template image into the query block in the first query encoding layer, the query block encodes the template image by using the convolution operation, and outputs a feature map;

[0012] b-3) input the feature map output by the query block in the first query encoding layer into the query block in the second query encoding layer, and output a feature map;

[0013] b-4) repeat step b-3) until the query block in the Nth query encoding layer outputs a feature map;

[0014] b-5) cut the feature map output by the query block in the first query encoding layer into multiple sub-regions of the same size, and input the cut sub-regions after linear projection into the encoding block in the first query encoding layer together with the encoding of the contrast image after segmentation and linear projection, the encoding block re-encodes by using the multi-head attention module, and outputs the latest encoding;

[0015] b-6) duplicate the output of the encoding block in the first query encoding layer twice as the key and value input into the encoding block in the second query encoding layer, and cut the feature map output by the query block in the second query encoding layer into multiple sub-regions of the same size, and input the cut sub-regions after linear projection into the encoding block in the second query encoding layer, and output the feature map after segmentation and linear projection;

[0016] b-7) repeat step b-6) until the encoding block in the Nth query encoding layer outputs an encoded feature.

[0017] Further, step c) comprises the following steps:

[0018] c-1) convert the segmented output of the feature map output by the encoding block in the Nth query encoding layer into a feature matrix of the same size as the feature map output by the query block in the Nth query encoding layer, and perform a concatenation operation on the two feature matrices to obtain a multi-channel feature matrix;

[0019] c-2) the decoder is composed of M stacked convolution layers, and a upsampling layer is arranged after each convolution layer, and M is a positive integer greater than or equal to 2;

[0020] c-3) input the multi-channel feature matrix into the first convolution layer in the decoder, and output a feature map;

[0021] c-4) input the feature map into the first upsampling layer in the decoder, and output an upsampled feature map;

[0022] c-5) repeat steps c-3) to c-4) until the feature map output by the Mth upsampling layer is obtained, which is used as a predicted result image.

[0023] Further, the following steps are further included after step c):

[0024] d-1) Collect a template data set, the template data set contains multiple template images, label the template images, and use polygons to cover the meaningful entities in the labeled template images;

[0025] d-2) Cut out the labeled meaningful entity area, paste the cut-out image to the template image to obtain the contrast image of the template image;

[0026] d-3) Record the position of each meaningful entity when pasted to the template image, obtain the change area of the contrast image relative to the template image, the change area is the dynamic area, and the other areas on the template image are the static areas, create a blank image with the same size as the template image as a label, set the value of the static area to 0 and the value of the dynamic area to 1.

[0027] The beneficial effects of the present application are: using computer programs and images captured by image sensors, the content difference recognition between multiple images can be automatically realized, and a robust intelligent method for recognizing sensitive content through image comparison in real application is provided. BRIEF DESCRIPTION OF DRAWINGS

[0028] Figure 1 The model structure diagram of the present application. DETAILED DESCRIPTION

[0029] The following will be described in conjunction with the accompanying Figure 1 Further description of the present application.

[0030] An image difference recognition method based on convolution and attention mechanism, comprising the following steps:

[0031] a) Establish an image difference recognition model, the model is composed of a query encoder, a decoder and a classifier.

[0032] b) Input the template image and the contrast image into the encoder.

[0033] c) Input the output result of the encoder into the decoder for decoding, and output to obtain a prediction result image.

[0034] The template image and the contrast image are respectively input into the encoder, and after being encoded by multiple layers of query encoding layers composed of query blocks and encoding blocks, they are output to the decoder. In the training stage, the input sample pair is labeled and the loss function is set to train the model. In the inference stage, the model decoder outputs the prediction result. The method can rely on a computer program to automatically identify two images with the same main content but local differences. The method has high robustness and does not require pixel value correspondence in the same content area between images. It can resist changes in camera angle, pose, parameters, and lighting environment.

[0035] Embodiment 1:

[0036] Step b) comprises the following steps:

[0037] b-1) The N query encoding layers are stacked, and the input of each query encoder comes from the output of the previous query encoder, N is a positive integer greater than or equal to 2, and each query encoder is composed of a query block and an encoding block.

[0038] b-2) The template image is input into the query block in the first query encoding layer, and the query block encodes the template image using convolution operation to obtain a feature map.

[0039] b-3) The feature map output by the query block in the first query encoding layer is input into the query block in the second query encoding layer, and a feature map is output.

[0040] b-4) Repeat step b-3) until the query block in the Nth query encoding layer outputs a feature map.

[0041] b-5) The feature map output by the query block in the first query encoding layer is cut into multiple sub-regions of the same size, and after linear projection, the cut multiple sub-regions are input into the encoding block in the first query encoding layer together with the encoding of the contrast image after segmentation and linear projection. The encoding block uses a multi-head attention module for re-encoding, and outputs the latest encoding.

[0042] b-6) The output of the encoding block in the first query encoding layer is copied twice and input into the encoding block in the second query encoding layer as key and value respectively. The feature map output by the query block in the second query encoding layer is cut into multiple sub-regions of the same size, and after linear projection, the cut multiple sub-regions are input into the encoding block in the second query encoding layer, and a feature map is output.

[0043] b-7) Repeat step b-6) until the encoding block in the Nth query encoding layer outputs an encoding feature.

[0044] Embodiment 2:

[0045] Step c) comprises the following steps:

[0046] c-1) converting the segmented output of the encoding block output feature map of the Nth query encoding layer into a feature matrix with the same size as the query block output feature map of the Nth query encoding layer, and splicing the two feature matrices to obtain a multi-channel feature matrix.

[0047] c-2) the decoder is composed of M stacked convolutional layers, and a upsampling layer is arranged after each convolutional layer, and M is a positive integer greater than or equal to 2.

[0048] c-3) inputting the multi-channel feature matrix into the first convolutional layer in the decoder to output a feature map.

[0049] c-4) inputting the feature map into the first upsampling layer in the decoder to output an upsampling feature map.

[0050] c-5) repeating steps c-3) to c-4) until the feature map output by the Mth upsampling layer is obtained, which is used as the predicted result image.

[0051] Example 3:

[0052] In order to train the above model, the method proposes a semi-automatic method for constructing a data set. A sample pair available for training is composed of three parts, including a template image, a contrast image and a label between them, the label includes the category of each pixel, static class or dynamic class. 0 can represent static class and 1 can represent static class. Specifically, the following steps are performed after step c):

[0053] d-1) collect a template data set containing multiple template images, and label the template images by using a polygon to cover the meaningful entities in the labeled template images.

[0054] d-2) cut out the labeled meaningful entity area, and paste the cut-out image to the template image to obtain a contrast image of the template image.

[0055] d-3) record the position of each meaningful entity when pasted to the template image to obtain the change area of the contrast image relative to the template image, the change area as a dynamic area, and other areas on the template image as static areas, create a blank image with the same size as the template image as a label, set its value to 0 in the static area, and set its value to 1 in the dynamic area. In order to increase the robustness and generalization of the model, perturb the generated contrast image, including non-displacement class perturbations such as color and brightness, and displacement class perturbations such as translation, rotation and perspective transformation, and apply these perturbations to the label at the same time when performing displacement class perturbations.

[0056] In order to more conveniently construct the training data set, a public semantic segmentation image data set can be used as a template data set, such as the COCO data set, and additional template image labeling is omitted. Using computer programming, based on a deep learning model programming framework, such as PyTorch, etc., the image difference recognition model in step a) is constructed.

[0057] Finally, it should be noted that: the above only for the preferred embodiments of the present application, and not for limiting the present application, although the foregoing detailed description of the present application is made with reference to the foregoing embodiments, for those skilled in the art, it still can be modified, or equivalent replacement of part of the technical features recorded in the foregoing embodiments. Any modification, equivalent replacement, improvement, etc. within the spirit and principles of the present application shall be included within the scope of protection of the present application.

Claims

1. An image difference recognition method based on convolution and attention mechanisms, characterized in that, Includes the following steps: a) Establish an image difference recognition model, which consists of an encoder, a decoder, and a classifier; b) Input the template image and the comparison image into the encoder; c) Input the encoder's output into the decoder for decoding, and output the predicted image; Step b) includes the following steps: b-1) The encoder is composed of N query coding layers stacked together. The input of each query coding layer comes from the output of the previous query coding layer. N is a positive integer greater than or equal to 2. Each query coding layer consists of a query block and a coding block. b-2) The template image is input into the query block in the first query encoding layer. The query block encodes the template image using convolution operations and outputs a feature map. b-3) Input the feature map output from the query block in the first query coding layer into the query block in the second query coding layer, and output the feature map; b-4) Repeat step b-3) until the query block of the Nth query coding layer outputs the feature map; b-5) The feature map output by the query block in the first query coding layer is cut into multiple sub-regions of the same size. The multiple sub-regions after being cut are linearly projected and then fed into the coding block in the first query coding layer together with the coding of the comparison image after segmentation and linear projection. The coding block is re-encoded using a multi-head attention module and outputs the latest coding. b-6) Copy the output of the encoding block of the first query encoding layer twice, and use them as keys and values ​​respectively to input the encoding block of the second query encoding layer. Cut the feature map output of the query block of the second query encoding layer into multiple sub-regions of the same size. Linearly project the multiple sub-regions into the encoding block of the second query encoding layer, and output the linear projection of the feature map segmentation. b-7) Repeat step b-6) until the Nth query coding layer outputs the coding features of the coding block; Step c) includes the following steps: c-1) Convert the output of the output feature map of the Nth query coding layer after segmentation into a feature matrix of the same size as the output feature map of the query block of the Nth query coding layer, and concatenate the two feature matrices to obtain a multi-channel feature matrix. The c-2) decoder consists of M stacked convolutional layers, with an upsampling layer after each convolutional layer, where M is a positive integer greater than or equal to 2; c-3) Input the multi-channel feature matrix into the first convolutional layer in the decoder, and output the feature map; c-4) Input the feature map into the first upsampling layer in the decoder, and output the upsampled feature map; c-5) Repeat steps c-3) to c-4) until the feature map output by the Mth upsampling layer is obtained, which is used as the prediction result image.

2. The image difference recognition method based on convolution and attention mechanisms according to claim 1, characterized in that, This also includes performing the following steps after step c): d-1) Collect a template dataset containing multiple template images, annotate the template images, and use polygons to cover meaningful entities in the annotated template images; d-2) Extract the meaningful entity regions marked, paste the extracted image onto the template image, and obtain a comparison image of the template image; d-3) Record the position of each meaningful entity when pasted into the template image, and obtain the change area of ​​the comparison image relative to the template image. This change area is used as the dynamic area, and other areas on the template image are used as the static area. Create a blank image of the same size as the template image as a label, set its value to 0 in the static area, and set its value to 1 in the dynamic area.

Images