Image false correlation removal method based on causal inference

Abstract

The application relates to the technical field of computer vision, and proposes an image false correlation removal method based on causal inference, which comprises the following steps: inputting a picture to be classified into a classification model used for classifying the input picture, wherein the classification model outputs a first classification result; extracting a background from the picture to be classified, and inputting the background into the classification model, wherein the classification model outputs a second classification result; converting the first classification result into an unbiased classification result by using the second classification result and a causal effect analysis debiasing model for removing the causal effect of image false correlation based on causal inference.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computer vision, and more particularly, to an image false correlation removal method based on causal inference. BACKGROUND

[0002] In recent years, the vigorous development of deep learning has made breakthrough progress in image recognition, speech recognition, video, image generation and other aspects, enabling people to achieve many previously difficult tasks. However, the essence of the network model training of the current hot deep learning neural network is only a high-level statistical learning method. No matter how complex it becomes a black box, deep learning ultimately only stays at the level of "observation" and tries to find some correlation from known observation data to do data fitting work. Therefore, it is inevitable to need a large amount of data to train the model to ensure its generalization, and it is also impossible to obtain the causal relationship between variables, which will cause errors when the model encounters data that has not been seen. For example, due to the deep learning network's tendency to learn simple backgrounds, it does not learn well about complex foregrounds (the part of the picture excluding the background). This leads to errors in the model's predictions when the input image background changes. Deep learning networks may capture some shortcuts in the data set to improve accuracy, and such shortcuts lead to false correlations that people do not need.

[0003] The prior art discloses an infrared weak target detection method based on counterfactual causal learning. First, an infrared image is obtained; then data augmentation is performed to obtain an infrared data set; an causal structure model is constructed using infrared target features; counterfactual causal relationships output by the causal structure model are used to extract infrared target generalization features; the infrared target generalization features are input into a target detection network structure for training, and a pre-trained target detection network is obtained after training; the infrared data set is resampled to construct a causal relationship between the infrared data context and the class label; the resampled data is used to perform feedforward adjustment on the pre-trained target detection network parameters to obtain the final target detection network.

[0004] The prior art has the defects of requiring the introduction of additional data and retraining, resulting in large consumption of computing resources, and being unable to be plugged in and used. SUMMARY

[0005] To overcome the defects of large consumption of computing resources and inability to be plugged in and used in the prior art, the present application provides an image false correlation removal method based on causal inference with small consumption of computing resources and the ability to be plugged in and used.

[0006] To solve the above technical problems, the technical solution of the present application is as follows:

[0007] input a picture to be classified into a classification model for classifying the input picture, the classification model outputting a first classification result;

[0008] extract a background from the picture to be classified, input the background into the classification model, the classification model outputting a second classification result;

[0009] convert the first classification result into an unbiased classification result by using the second classification result and a causal-effect analysis debiasing model based on causal inference for removing image spurious correlations.

[0010] The application further provides an image spurious correlation removal system based on causal inference, which is used for implementing the image spurious correlation removal method based on causal inference.

[0011] A first classification module, which is configured with a classification model for classifying an input picture, is used for inputting a picture to be classified into the classification model, and the classification model outputs a first classification result;

[0012] A second classification module, which is configured with a classification model for classifying an input picture, is used for extracting a background from the picture to be classified, inputting the background into the classification model, and the classification model outputs a second classification result;

[0013] A causal inference debiasing module is used for converting the first classification result into an unbiased classification result by using the second classification result and a causal-effect analysis debiasing model based on causal inference for removing image spurious correlations.

[0014] The application further provides a computer device, which comprises a memory and a processor, the memory is stored with computer readable instructions, and the computer readable instructions are executed by the processor to enable the processor to execute the steps of the image spurious correlation removal method based on causal inference.

[0015] Compared with the prior art, the beneficial effects of the technical scheme of the application are as follows:

[0016] The first classification result is converted into an unbiased classification result by using the second classification result and a causal-effect analysis debiasing model based on causal inference for removing image spurious correlations, the image spurious correlations are removed based on causal inference, no additional data and retraining are needed, the application can be used immediately, and the consumption of computing resources is reduced. BRIEF DESCRIPTION OF DRAWINGS

[0017] Figure 1 A flowchart of the image spurious correlation removal method based on causal inference provided for embodiment 1 is shown in the figure.

[0018] Figure 2 A causal graph under the perspective of a front-background image provided for embodiment 1 is shown in the figure.

[0019] Figure 3 A TIE causal effect debiasing example graph proposed for example 2;

[0020] Figure 4 A causal effect analysis debiasing model framework diagram proposed for example 2;

[0021] Figure 5 A result example graph obtained by using a causal effect analysis debiasing model proposed for example 2;

[0022] Figure 6 A whole framework diagram of an image false correlation removal system based on causal inference proposed for example 3. DETAILED DESCRIPTION

[0023] The accompanying drawings are only for illustrative purposes and should not be understood as a limitation on the embodiments;

[0024] In order to better illustrate the embodiments, some components in the drawings may be omitted, enlarged or reduced, and do not represent the actual size of the product;

[0025] It is understandable that some well-known structures and their descriptions in the drawings may be omitted for those skilled in the art.

[0026] The technical solutions of the present application will be further described below in combination with the drawings and embodiments.

[0027] Example 1

[0028] The present embodiment proposes a method for removing image false correlation based on causal inference, Figure 1 A flowchart of the method for removing image false correlation based on causal inference of the present embodiment;

[0029] The method for removing image false correlation based on causal inference proposed in the present embodiment comprises the following steps:

[0030] S1: input the picture to be classified into a classification model for classifying the input picture, and the classification model outputs a first classification result;

[0031] S2: extract the background from the picture to be classified, and input the background into the classification model, and the classification model outputs a second classification result;

[0032] S3: convert the first classification result into an unbiased classification result by using the second classification result and a causal effect analysis debiasing model for removing image false correlation based on causal inference.

[0033] In the implementation process, the first classification result is converted into an unbiased classification result by using the second classification result and a causal-effect analysis debiasing model based on causal inference to remove false correlations of the image. The causal-effect analysis debiasing model based on causal inference does not need to introduce additional data and retraining, can be used immediately, and reduces the consumption of computing resources.

[0034] In an optional embodiment, the classification model comprises a binary classification model or a multi-classification model.

[0035] Expressions of the first classification result and the second classification result respectively comprise:

[0036] Rs1={[a1,b1],[a2,b2],…,[a i ,b i ],…,[a N ,b N ]}

[0037] Rs2={[a1,c1],[a2,c2],…,[a i ,c i ],…,[a N ,c N ]}

[0038] 1≤i≤N

[0039] In the formula, Rs1 represents the first classification result, Rs2 represents the second classification result, a i represents the i-th label, b i represents the probability that the label of the picture to be classified calculated by the classification model is a i , N represents the total number of picture labels, and c i represents the probability that the label of the background of the picture to be classified calculated by the classification model is a i .

[0040] As an example, when the classification model is a binary classification model, N=2, that is, expressions of the first classification result and the second classification result are respectively:

[0041] Rs1={[a1,b1],[a2,b2]}

[0042] Rs2={[a1,c1],[a2,c2]}.

[0043] In an optional embodiment, the causal-effect analysis debiasing model comprises the following steps when converting the first classification result into an unbiased classification result:

[0044] If argmax(Rs1)≠argmax(Rs2), the label in argmax(Rs1) is regarded as the unbiased classification result.

[0045] wherein argmax(·) represents a function of finding the argument value corresponding to the maximum value of the function, argmax(Rs1) and argmax(Rs2) each contain two elements, the elements of argmax(Rs1) include: the maximum probability in Rs1 and the label corresponding to the maximum probability; the elements of argmax(Rs2) include: the maximum probability in Rs2 and the label corresponding to the maximum probability.

[0046] If argmax(Rs1)=argmax(Rs2), the image spurious correlation is removed by causal inference to obtain an unbiased classification result, and the expression of the unbiased classification result is:

[0047] argmax(TIE)=[Rs f ,e]

[0048] TIE=TE-NDE

[0049] =f(Rs1-Rs2)

[0050] ={[a1,d1],[a2,d2],…,[a i ,d i ],…,[a N ,d N ]}

[0051] d i =b i -c i

[0052] In the formula, TIE represents the result after removing the image spurious correlation, TE represents the total effect of causal inference, and NDE represents the natural direct effect of causal inference; f(Rs1-Rs2) represents that b i in Rs1 is subtracted from c i in Rs2, and a i remains unchanged; d i represents the probability that the label of the picture to be classified after removing the image spurious correlation is a i ; e represents the maximum probability in TIE, and Rs f represents the label corresponding to the maximum probability e, which also represents the unbiased classification result.

[0053] As an example, when the label corresponding to the maximum probability in Rs1 is equal to the label corresponding to the maximum probability in Rs2, argmax(Rs1)=argmax(Rs2), otherwise, argmax(Rs1)≠argmax(Rs2).

[0054] In the optional embodiment, if the label corresponding to the maximum probability in the second classification result corresponding to the background in the picture to be classified is equal to the label corresponding to the maximum probability in the first classification result corresponding to the picture to be classified, it indicates that the classification model uses more the background in the picture to be classified for classification, and there is false correlation. The image false correlation is removed by using causal inference, and thus an unbiased classification result is obtained.

[0055] In an optional embodiment, the classification model comprises a binary classification model.

[0056] The expressions of the first classification result and the second classification result respectively comprise:

[0057] Rs3=a

[0058] Rs4=c

[0059] In the formula, Rs3 represents the first classification result, Rs4 represents the second classification result; a represents the label of the picture to be classified output by the classification model; c represents the label of the background of the picture to be classified output by the classification model, wherein a is a positive example label or a negative example label, and c is a positive example label or a negative example label.

[0060] In an optional embodiment, when the first classification result is converted into an unbiased classification result by the causal effect analysis debiasing model, the following steps are included:

[0061] If a≠c, a is regarded as an unbiased classification result;

[0062] If a=c and a is a positive example label, a negative example label is regarded as an unbiased classification result;

[0063] If a=c and a is a negative example label, a positive example label is regarded as an unbiased classification result.

[0064] In the optional embodiment, for an input picture, the prediction result of the binary classification model has only two possibilities: the prediction label of the input picture is a positive example label, or the prediction label of the input picture is a negative example label. If a=c, it indicates that the prediction label obtained by the binary classification model is the label of the background of the input picture, rather than the label of the foreground of the input picture. Therefore, when a is a positive example label, a negative example label is regarded as an unbiased classification result, and vice versa.

[0065] In an optional embodiment, when the background is extracted from the picture to be classified, a colored mask is used to block the foreground of the picture to be classified to obtain the background.

[0066] As an exemplary illustration, the colored mask comprises a white mask.

[0067] In an optional embodiment, when extracting the background from the picture to be classified, the foreground of the picture to be classified is identified by using an image segmentation model, and a mask is added to the identified foreground to obtain the background.

[0068] Embodiment 2

[0069] Based on the image false correlation removal method based on causal inference proposed in Embodiment 1, the following example of the causal inference process is proposed:

[0070] Taking the medical field as an example, the causally defined effect traditionally represents the potential result Y of receiving treatment x and the result under different treatments The difference. Without considering the medium M, the total effect (TE) of treatment x relative to treatment x * can be expressed as the equation:

[0071]

[0072] In the formula, M x represents the medium corresponding to treatment x, represents the medium corresponding to treatment x * , Y x , M x represents the effect of treatment x when the medium is M x , and Y represents the effect of treatment x when the medium is * .

[0073] The total effect TE can be decomposed into two basic components: natural direct effect (NDE) and total indirect effect (TIE). When the treatment is converted from treatment x * to treatment x, NDE quantifies the expected change in Y, assuming remains unchanged. It focuses on how x affects Y while keeping the medium M unchanged, as shown in the equation:

[0074]

[0075] In the formula, Y x , represents the effect of treatment x when the medium is , Y represents the effect of treatment x when the medium is * .

[0076] TIE represents the difference between TE and NDE:

[0077]

[0078] In an embodiment, TIE is described as a result of causal factors, while NDE represents a result of correlation factors.

[0079] Figure 2 A causal graph under the foreground-background image view is proposed for this embodiment, where the foreground-background image represents an image with only foreground left after removing the background image; as Figure 2 shown, X is an input picture with both causal and correlation factors, X->R represents the causal feature representation learned by the model that is truly used for judgment, and X->C represents the remaining correlation factors such as the impact of the background, which emphasizes that the prediction result C is jointly affected by the causal feature R and the remaining mixed correlation factors such as the background in X. When the background is used as a potential false correlation reason, it is crucial to remove the effect of the background. The goal of this embodiment is to remove the causal effect of X->C, because the causal effect of X->C can lead to false correlation. The classification result C = c is jointly determined by R = f(x) and X = x, as in the equation:

[0080] Y x ,Y x(c) = Y(c, X = x, R = f(x))

[0081] Without loss of generality, after removing C, the equation is simplified as:

[0082] Y x ,R x = Y(X = x, R = f(x))

[0083] As Figure 2 shown, there are two paths leading to C: X->R->C and X->C. In this case, the TE formula can be transformed into the following equation:

[0084]

[0085] In the equation, the symbol x * represents the value of x under the condition of no intervention, i.e., x * represents the case of no input picture.

[0086] NDE can be represented as the following equation:

[0087]

[0088] Since R x is set to The causal effect of X->R->C is effectively blocked, and the spurious correlation generated by the path X->C is preserved, so the NDE captures the effect of the correlation factor. This effect can be achieved through the masking foreground of counterfactuals. By masking the foreground of the picture to be classified using a colored mask (as in Example 1), the effect of the background of the correlation factor on classification is preserved, and the model cannot use the causal feature representation, blocking the path of the causal factor.

[0089] In addition, Figure 3 The TIE causal effect debiasing example graph proposed in this embodiment can achieve the reduction of the correlation factor in the following way, as shown in the figure: Figure 3 Subtract NDE from TE, as shown in the equation:

[0090]

[0091] In the equation, TIE provides the result after removing spurious correlation, effectively eliminating background spurious correlation. This allows us to solve the problem of spurious correlation when determining its impact on X->C as background.

[0092] The framework of this embodiment includes two key components: a classification model and a causal effect analysis debiasing (CEAD) model. As an exemplary illustration, the classification model includes: ViT (Visual Transformer) or ResNet50 (Residual Network 50); initially, as an exemplary illustration, a controllable proportion of white masks are used to mask the foreground of the image, thereby creating an estimated background image. Subsequently, the input image and the estimated background image are fed into the classification model respectively to obtain their respective results, thereby finding the background that generates spurious correlation. Finally, these results are input into the causal effect analysis debiasing model to generate the final prediction.

[0093] The causal effect analysis debiasing model mainly focuses on incorporating the causal effect and counterfactual concept into the model to eliminate serious spurious correlation and improve the robustness of the model. The key factor that leads to poor model robustness is the background spurious correlation, represented by X->C. This spurious correlation is derived from the causal effect of natural direct effect (NDE). Therefore, by reducing the impact of the causal effect NDE of serious background spurious correlation, an approximately unbiased estimation result can be obtained. According to the shortcut learning concept, in image recognition scenarios with significant background spurious correlation, the model always pays more attention to the background. However, it should be noted that not all backgrounds will bring serious spurious correlation. Once the background is used as a source of serious spurious correlation, the model's prediction for the original image input will be consistent with the prediction of the background input, such as inputting a plant background, and the probability of getting an insect prediction is high.

[0094] Figure 4 A schematic diagram of the causal effect analysis debiasing model framework proposed in this embodiment; as Figure 4 shown, the causal effect analysis debiasing module includes two main components, the first part is committed to accurately identifying the existence of serious background spurious correlation. In this case, this embodiment compares the results of natural direct effect (NDE) with the results of total effect (TE), note that at this time the natural direct effect (NDE) and total effect (TE) are the logit output by the classification model. When argmax(NDE) == argmax(TE), it indicates that there is serious spurious correlation in the background, and the influence caused by the background needs to be eliminated. On the contrary, when argmax(NDE)!= argmax(TE), it indicates that the influence of background spurious correlation is not great, and there is no need to remove the relevant factors. Based on this, the judgment logic of the causal effect analysis debiasing model is designed.

[0095] The second part of the process involves applying the equation TIE to eliminate NDE. In this case, TE represents the prediction result of the input image, and NDE is the prediction result of the estimated background. The estimated background is created by masking the counterfactual thinking of the foreground. The TIE obtained by subtracting the two represents the unbiased prediction result obtained after removing the spurious correlation of the background.

[0096] As an exemplary illustration, in the specific implementation process, any picture that needs to be classified, such as a picture of a cow on the beach, is input into the classification model, and the causal effect analysis debiasing model examines whether the single beach background is consistent with the result of the input picture by analyzing the causal effect. If it is consistent, the causal effect analysis debiasing model eliminates the influence caused by the background. On the contrary, if it is not consistent, the causal effect analysis debiasing model does not need to remove the relevant factors. Figure 5 An example diagram of the result obtained by using the causal effect analysis debiasing model proposed in this embodiment, as Figure 5 shown, the result obtained by using the causal effect analysis debiasing model has high accuracy.

[0097] Embodiment 3

[0098] This embodiment proposes an image spurious correlation removal system based on causal inference, which is used to implement the image spurious correlation removal method based on causal inference proposed in embodiment 1.

[0099] Figure 6 A schematic diagram of the overall framework of the image spurious correlation removal system based on causal inference of this embodiment.

[0100] The image spurious correlation removal system based on causal inference includes:

[0101] A first classification module, on which a classification model for classifying an input picture is configured, configured to input a picture to be classified into the classification model, and the classification model outputs a first classification result;

[0102] A second classification module, on which a classification model for classifying an input picture is configured, configured to extract a background from a picture to be classified, input the background into the classification model, and the classification model outputs a second classification result;

[0103] A causal inference debiasing module, configured to convert the first classification result into an unbiased classification result by using the second classification result and a causal inference debiasing model based on causal effect analysis for removing false correlation of an image.

[0104] In an optional embodiment, the causal inference debiasing module is inserted into the classification model in the form of a plug-in.

[0105] The embodiment provides a computer device, including a memory and a processor, the memory stores computer readable instructions, and the computer readable instructions are executed by the processor to make the processor execute the steps of the image false correlation removing method based on causal inference in the embodiment 1.

[0106] It can be understood that the image false correlation removing system and the computer device based on causal inference in the embodiment improve the method in the embodiment 1, and the optional items in the above embodiment 1 are also applicable to the embodiment, and thus are not repeatedly described herein.

[0107] The same or similar reference signs correspond to the same or similar components;

[0108] The terms for describing the positional relationship in the drawings are only used for example and illustration, and cannot be understood as a limitation to the embodiment;

[0109] Obviously, the above embodiment of the application is only an example for clearly illustrating the application, and is not a limitation to the embodiment of the application. Any modification, equivalent replacement and improvement made on the basis of the above description for the ordinary skilled in the art should be included in the protection scope of the claims of the application.

Claims

1. A method for removing false correlations in images based on causal inference, characterized in that, The method comprises the following steps: inputting a picture to be classified into a classification model for classifying the input picture, the classification model outputting a first classification result; extracting a background from the picture to be classified, inputting the background into the classification model, and the classification model outputting a second classification result; transforming the first classification result into an unbiased classification result by using the second classification result and a causal-effect analysis debiasing model based on causal inference for removing false correlations of images; wherein the causal-effect analysis debiasing model comprises the following steps when transforming the first classification result into the unbiased classification result: If , then the labels in are considered as unbiased classification results; wherein, denotes a function that finds the argument value of the function at which the maximum value is obtained, and both contain two elements, the elements of include the maximum probability in and the label corresponding to the maximum probability; the elements of include the maximum probability in and the label corresponding to the maximum probability; If Then, the spurious correlations of the image are removed by using causal inference, and the unbiased classification results are obtained, and the expression of the unbiased classification results is: wherein, represents the result after removing image spurious correlations, represents the total effect of causal inference, represents the natural direct effect of causal inference; represents the subtraction of in from in , remains unchanged; represents the probability that the label of the picture to be classified after removing image spurious correlations is ; represents the maximum probability in , represents the maximum probability corresponds to the label, also represents the unbiased classification result.

2. The method for removing false correlation of image based on causal inference according to claim 1, characterized in that, the classification model comprises a binary classification model or a multi-classification model; expressions of the first classification result and the second classification result respectively comprise: In the formula, denotes the first classification result, denotes the second classification result; denotes the first label, denotes the probability that the label of the picture to be classified calculated by the classification model is , denotes the total number of picture labels, denotes the probability that the label of the background of the picture to be classified calculated by the classification model is .

3. The method for removing false correlation of image based on causal inference according to claim 1, characterized in that, the classification model comprises a binary classification model; expressions of the first classification result and the second classification result respectively comprise: In the formula, represents the first classification result, represents the second classification result; represents a label of a picture to be classified output by the classification model; represents a label of a background of the picture to be classified output by the classification model, wherein, is a positive example label or a negative example label, is a positive example label or a negative example label.

4. The method of claim 3, wherein, the causal-effect analysis debiasing model comprises the following steps when transforming the first classification result into the unbiased classification result: If then, the classification result is considered unbiased; If , and is a positive example label, then the negative example label is considered as an unbiased classification result; If , and is a negative example label, then the positive example label is considered as an unbiased classification result.

5. The method of claim 1-4, wherein, when extracting the background from the picture to be classified, a colored mask is used to shield the foreground of the picture to be classified to obtain the background.

6. The method of claim 1-4, wherein, when extracting the background from the picture to be classified, an image segmentation model is used to identify the foreground of the picture to be classified, and a mask is added to the identified foreground to obtain the background.

7. A system for removing false correlations in images based on causal inference, for implementing the method for removing false correlations in images based on causal inference according to any one of claims 1 to 6, characterized in that, comprise: a first classification module, which is configured with a classification model for classifying an input picture, and is used to input a picture to be classified into the classification model, the classification model outputting a first classification result; a second classification module, which is configured with a classification model for classifying an input picture, and is used to extract a background from the picture to be classified, input the background into the classification model, and the classification model outputting a second classification result; a causal inference debiasing module, which is used to transform the first classification result into an unbiased classification result by using the second classification result and a causal-effect analysis debiasing model based on causal inference for removing false correlations of images.

8. The system for removing false correlations in images based on causal inference of claim 7, wherein, The causal inference debiasing module is inserted into the classification model in the form of a plug-in. 9.A computer device, comprising a memory and a processor, wherein the memory stores computer readable instructions, and the computer readable instructions are configured to cause the processor to perform the method according to any one of claims 1-8. The computer-readable instructions, when executed by the processor, cause the processor to perform the steps of the image false correlation removal method based on causal inference according to any one of claims 1-6.

Images