An image recognition method based on causal interference attention

Abstract

An image recognition method based on causal interference attention includes the following steps: S1. Data preprocessing: processing the image used as training data into the size and format required by the model and enhancing it; S2. The preprocessed training data is fed into a backbone benchmark network to obtain a feature map of the image; S3. The feature map is divided into four parts in the channel dimension, which is regarded as causal interference on the feature map; S4. In each feature map, category-spatial attention is implemented; S5. Based on the category-spatial attention of each part, dot product attention is implemented; S6. The four feature maps after passing through the two-layer attention module are spliced ​​together in the channel dimension to obtain the final feature; S6. The final feature is input into a binary classifier to obtain a prediction result. The method of the present invention can remove the interference caused by context offset on image recognition, thereby improving the accuracy and robustness of image recognition, and solving the problem that the existing attention mechanism aggravates context offset.

Description

Technical Field

[0001] The present invention relates to the field of image processing, and more particularly, to an image recognition method based on causal interference attention. Background Art

[0002] The attention mechanism is essentially a weighting algorithm. Its input is a signal sequence, and it selects and highlights the most informative and effective parts of the signal sequence. The output is a weighted signal. Initially, attention modules were primarily applied to naturally sequential tasks such as natural language processing. In recent years, attention mechanisms have also been widely used in computer vision. Both the original traditional attention mechanism and the more recent self-attention mechanism have achieved excellent results across various tasks. However, the attention mechanism also has its own challenges. The selection and weighting of the input signal is unsupervised, and it only highlights what it "self-deems" to be important signal segments, making it prone to reinforcing incorrect components. Specifically, when there is a significant context shift in the training set (for example, most instances of the "knife and fork" category are found in the context of a dining table), the attention mechanism will mistakenly assume that a table is a prerequisite for the presence of "knife and fork" and thus prioritize the table over the knife and fork in the image. This misweighting can lead to incorrect predictions of the presence of a knife and fork in an image of a table without one. In fact, although the attention mechanism can improve the performance in most cases, when the data distribution of the training set and the test set is different, the performance of the model with the attention module will be worse.

[0003] Causal inference is a theory that explores the cause-and-effect relationship between things. It is widely used in fields such as medicine, political science, and sociology. In the field of artificial intelligence, causal methods are often used to eliminate the impact of bias in models and data sets. The problem with causal inference is that the context of the text (such as the dining table) is a typical confounding factor. Confounding factors are factors with A B C structure relationships, such as movement Older Cancer and the confounding effect of "age" can cause models to believe there's a causal relationship between exercise and cancer. In image recognition tasks, there are many foreground elements and their frequently co-occurring backgrounds. These background elements act as confounding factors, obscuring the causal relationship between the foreground and the recognition result. For example, misleading a model into believing there's a causal relationship between the background of a table and the appearance of a knife and fork. This misleading causal relationship leads to non-robust predictions. For example, when presented with images of a table without a knife and fork, or images of a table without a knife and fork, conventional models often make incorrect predictions. The attention mechanism, however, cannot distinguish whether the causal relationship derived by the model is correct or incorrect; it only reinforces existing causal relationships (i.e., dataset shifts). Therefore, models with attention modules perform worse when the training and test sets have different distributions. To address this issue and ensure the full effectiveness of the attention module, it's necessary to remove the influence of confounding factors and obtain the correct causal relationship. In causal inference, removing confounding effects is typically accomplished through causal interference, the most common of which is backdoor interference (do operator): , where X and Y are the cause and effect we want to solve, and C is the confounding factor in the model.

[0004] The above problems and drawbacks are very difficult to solve because context shift is ubiquitous in datasets and difficult to quantify. Currently, there is no comprehensive solution to this problem.

[0005] The significance of solving the above problems is to help the model learn more essential object features and obtain more robust predictions, thereby achieving better results in difficult scenarios such as inconsistent distribution of application scenario data and training data. Summary of the Invention

[0006] The present invention provides an image recognition method based on causal interference attention, which can remove the interference of context offset on image recognition, give full play to the effectiveness of the attention mechanism, thereby improving the accuracy and robustness of image recognition, and solving the problem that the existing attention mechanism aggravates context offset.

[0007] The technical solutions of the present invention are as follows:

[0008] An image recognition method based on causal interference attention of the present invention includes the following steps: S1. Data preprocessing: processing the image as training data into the size and format required by the model and enhancing it; S2. The preprocessed training data enters the backbone reference network to obtain the feature map of the image; S3. The feature map is divided into four parts in the channel dimension, called four heads, which are regarded as causal interference on the feature map; S4. In each feature map, the first layer of attention: category-spatial attention is implemented; S5. Based on the category-spatial attention of each part, the second layer of attention: dot product attention is implemented; S6. The four feature maps after passing through the two-layer attention module are spliced ​​together in the channel dimension to obtain the final feature; S6. The final feature is input into a binary classifier to obtain a prediction result.

[0009] Preferably, in the above-mentioned image recognition method based on causal interference attention, in step S1, the image used as training data is scaled to a uniform size according to the subsequent backbone benchmark network, and then data enhancement is performed, and the image is flipped horizontally or vertically with a random probability, and then rotated and distorted with a random probability.

[0010] Preferably, in the above-mentioned image recognition method based on causal interference attention, in step S2, the backbone reference network is a convolutional architecture of a CNN that can output a spatial feature map or a transformer skeleton without convolution.

[0011] Preferably, in the above-mentioned image recognition method based on causal interference attention, in step S4, the first layer of attention first classifies the features in the spatial dimension, and then multiplies the classification probability of each category back to the spatial features, and finally obtains the features of each category in the four heads respectively.

[0012] Preferably, in the above-mentioned image recognition method based on causal interference attention, in step S5, the second-layer attention weights the results in the four heads of the first-layer attention. The weighting method is to multiply the query Q and the value K and multiply the normalized result by the value V. What is obtained is the feature of each category in each head that integrates the information of other heads, and its dimension is the same as that of the previous layer.

[0013] Preferably, in the above-mentioned image recognition method based on causal interference attention, in step S6, the output sequence of the dot product attention is first decomposed to obtain the category features of the four heads, and then the category features of the four heads are spliced ​​together in the channel dimension in the order of segmentation in step S3 to obtain the classification features of each category.

[0014] Preferably, in the above-mentioned image recognition method based on causal interference attention, in step S6, after obtaining the pre-classification score, a layer of Softmax is passed to obtain a category-based probability distribution, and based on this probability, the category-based mixed set is weighted and summed to obtain the context feature.

[0015] According to the technical solution of the present invention, the beneficial effects produced are:

[0016] This method removes the confounding factor of contextual bias through causal interference, establishing a correct causal relationship between the object and its prediction, allowing the attention mechanism to function effectively. This method can make robust predictions for the target object in varying contexts, improving recognition accuracy.

[0017] The framework for improving the attention mechanism proposed by the method of the present invention is not limited to the category-spatial attention in the method, but can be used to improve and enhance any other attention modules, such as the well-known CBAM and SENET.

[0018] The method is based on the theory of causal reasoning and is applicable to any skeleton and probabilistic model, and can be flexibly transferred to different methods. At the same time, the method is also applicable to various visual recognition tasks, including single classification, multi-classification, detection, etc.

[0019] In order to better understand and illustrate the concept, working principle and effect of the present invention, the present invention is described in detail below with reference to the accompanying drawings and through specific embodiments: BRIEF DESCRIPTION OF THE DRAWINGS

[0020] In order to more clearly illustrate the specific implementation of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for the specific implementation or the description of the prior art.

[0021] Figure 1 is a flow chart of the image recognition method based on causal interference attention of the present invention;

[0022] Figure 2 It is a flow chart of the network framework of the image recognition method based on causal interference attention of the present invention. DETAILED DESCRIPTION

[0023] To make the purpose, technical methods and advantages of the present invention more clear, the present invention will be further described in detail below with reference to the accompanying drawings and specific examples. These examples are merely illustrative and not intended to limit the present invention.

[0024] like Figure 1 and Figure 2 As shown, the image recognition method based on causal interference attention of the present invention includes the following steps:

[0025] S1. Data preprocessing: The images used as training data are processed into the size and format required by the model and augmented.

[0026] Specifically, in this step, the training images are first scaled to a uniform size, the specific size of which is determined by the subsequent backbone network. Data augmentation is then performed, with the images randomly flipped horizontally or vertically, and then rotated and distorted with random probabilities. For example, during data preprocessing, the images are scaled to a standard 448*448 size and fed into the backbone network after random flipping and rotation.

[0027] S2. The training data enters the backbone benchmark network to obtain the feature map of the image.

[0028] The backbone benchmark network can use any network model, including convolutional CNN architectures and non-convolutional transformer backbones, as long as it can output spatial feature maps. The residual network (ResNet) and sliding window transformer (SwinT) are used as examples. For ResNet, the input image size is 448*448, and the output feature map is a 2048*14*14 matrix. For SwinT, the input image size is 384*384, and the output feature map is a 1024*12*12 matrix. For simplicity, the following uniform ResNet dimensions are used as examples. 2048 is the channel dimension, and 14*14 is the width and height of the feature map, corresponding to the spatial dimension.

[0029] S3. Split the feature map into four parts along the channel dimension, called four heads, and consider it as causal interference on the feature map.

[0030] Splitting the feature map along the channel dimension into four parts, called four heads, is a way to model the backdoor adjustment formula (a type of causal interference). Processing the split features separately is equivalent to performing causal interference on the features. Specifically, split the feature map obtained above into four parts along the channel dimension, each with a size of 512*14*14, called four heads. The formula after splitting is: , this formula is an approximation of the backdoor adjustment formula, so the split operation is regarded as a one-step causal interference on the feature, where X and Y represent the object features and the prediction results respectively, K represents the number of heads, and w represents the weight of the classifier. Set to 0.03125. Then, compress the last two dimensions into one dimension, resulting in four 512*196 matrices, where 196 corresponds to the spatial dimension.

[0031] S4. Implement the first level of attention in each feature map: Category-Spatial Attention (SCA).

[0032] The first layer of attention first classifies the features in the spatial dimension, then multiplies the classification probability of each class back to the spatial features, and finally obtains the features of each class in the four heads. Specifically, the feature map is first classified through a layer of convolution, and the channel dimension is converted into the number of categories. Assuming there are 80 classes, the result is 80*196, where the dimension 80 is the classification probability of each class; then a Softmax operation is performed on the dimension 196 so that the sum of the 80 probabilities is 1; then, these probabilities are used to perform a weighted average on the original feature map in the spatial dimension, that is, the 80*196 probability matrix is ​​multiplied by the previous 512*196 feature map matrix to obtain the channel features of each category (E1, E2, ..., E 80 ), with a size of 80*512. This operation is performed in all four heads to obtain four 80*512 category feature matrices, called multi-head category-spatial attention (MHSCA).

[0033] S5. Based on each class-spatial attention, implement the second level of attention: dot product attention (DPA).

[0034] Based on the category-spatial attention, dot product attention is implemented, taking a weighted average of the results from the four heads. The weighting is achieved by multiplying the query Q by the value K, normalizing the result, and then multiplying it by the value V. This yields a per-category feature for each head that integrates information from all other heads, with the same dimensionality as the previous layer. Specifically, the category features from the four heads are stacked together to form a sequence of size 80*4*512, where the sequence length is the number of heads, four. This sequence is then used to generate the query values ​​Q, K, and V through three 512*512 projections, each with a size of 80*4*512. Matrix multiplication is then performed on Q and K to obtain a sequence weight of size 80*4*4. A softmax is applied to this weight so that the weight sums to 1. Finally, this weight is multiplied by the value V to obtain the output of the dot product attention, which is size 80*4*512.

[0035] S6. Concatenate the four feature maps after the two-layer attention module in the channel dimension to obtain the final feature.

[0036] After obtaining the pre-classification scores, a Softmax layer is applied to obtain a category-based probability distribution. Based on this probability, the category-based mixed set is weighted and summed to obtain the contextual features. Specifically, the output sequence of the dot product attention is first decomposed to obtain the category features of four heads, with a size of 80*512. Then, according to the splitting order in step S3, the category features of the four heads are spliced ​​together along the channel dimension to obtain the category features of each category, with a size of 80*2048.

[0037] S7. Input the final features into the binary classifier to obtain the prediction results. The final binary classifier also uses the interference classifier, but with different parameters from the pre-classifier. Specifically, the 2048-dimensional features of 80 categories are input into 80 classifiers respectively, and the size of each classifier is also 2048. The classification probabilities of 80 categories are obtained, as shown in the figure. Figure 2 The probability of a “person” appearing in the image is 0.99, the probability of a car appearing is 0.87, and the probability of a “cat” appearing is 0.06. The image is then identified based on the probability.

[0038] The test results of the present invention are shown in Tables 1 and 2 below. It can be seen from Tables 1 and 2 that the image recognition method based on causal interference attention of the present invention can remove the interference caused by context offset on image recognition, give full play to the effectiveness of the attention mechanism, and thus improve the accuracy and robustness of image recognition.

[0039] Table 1

[0040]

[0041] The quantitative comparison results of the proposed method and other well-known methods are shown in Table 1, where IDA(L) and IDA(H) are the lightweight model and heavyweight model of the proposed method (which repeatedly expands the second layer of attention multiple times), respectively. WH is the resolution, mAP is the main objective average indicator, and CP, CR, CF1, OP, OR, and OF1 are objective evaluation indicators.

[0042] Table 2

[0043]

[0044] Table 2 shows the ablation test results for each module of the proposed method. SCA represents the original class-spatial attention model, Multi-head represents MHSCA obtained by adding multiple heads to the class-spatial attention model, DPA represents adding dot product attention to the multi-head model, and Trans represents a heavyweight model obtained by multi-layer expansion of the second dot product attention layer. Parameters represents the number of parameters (in M).

[0045] The above description is the best embodiment based on the concept and working principle of the invention. The above embodiment should not be understood as limiting the scope of protection of the present claims. Other embodiments and combinations of implementations of the present invention are also within the scope of protection of the present invention.

Claims

1. An image recognition method based on causal interference attention, characterized in that: The following steps are involved: S1. Data preprocessing: Process the images used as training data into the required size and format and perform augmentation. S2. The preprocessed training data is fed into the backbone benchmark network to obtain image feature maps. S3. Splitting the feature map along the channel dimension into four parts, referred to as four heads, is considered to have performed causal interference on the feature map; S4. Implement the first level of attention in each feature map: category-spatial attention. S5. Based on each class-spatial attention, implement the second level of attention: dot product attention. S6. Concatenate the four feature maps after the two attention layers along the channel dimension to obtain the final feature. S6. Input the final features into the binary classifier to obtain the prediction results.

2. The image recognition method based on causal interference attention according to claim 1, characterized in that: In step S1, the image used as training data is scaled to a uniform size according to the subsequent backbone benchmark network, and then data enhancement is performed. The image is flipped horizontally or vertically with a random probability, and then rotationally distorted with a random probability.

3. The image recognition method based on causal interference attention according to claim 1, characterized in that: In step S2, the backbone benchmark network is a convolutional architecture of a CNN that can output a spatial feature map or a transformer skeleton without convolution.

4. The image recognition method based on causal interference attention according to claim 1, characterized in that: In step S4, the first layer of attention first classifies the features in the spatial dimension, then multiplies the classification probability of each class back to the spatial features, and finally obtains the features of each class in the four heads.

5. The image recognition method based on causal interference attention according to claim 1, characterized in that: In step S5, the second-layer attention weights the results of the four heads of the first-layer attention. The weighting method is to multiply the query Q and the value K and multiply the normalized result by the value V. The result is the features of each category in each head that integrate the information of other heads, and its dimension is the same as the previous layer.

6. The image recognition method based on causal interference attention according to claim 1, characterized in that: In step S6, the output sequence of the dot product attention is first decomposed to obtain the category features of the four heads, and then the category features of the four heads are spliced ​​together in the channel dimension according to the order of segmentation in step S3 to obtain the classification features of each category.

7. The image recognition method based on causal interference attention according to claim 1, characterized in that: In step S6, after obtaining the pre-classification scores, a Softmax layer is passed to obtain a category-based probability distribution. Based on the probability, the category-based mixed set is weighted and summed to obtain the context feature.

Images