Out-of-distribution image classification method and system based on sample balanced causal reasoning

Abstract

The application belongs to the field of image classification, and provides a distribution out-of-distribution image classification method and system based on sample balance causal reasoning. The application automatically generates balance subsets with different background features through a balance method, and then performs causal intervention on the data through the subsets with different backgrounds, so that the model can correctly focus on the causal features of the image in different backgrounds. The training difficulty of different categories is balanced through a class difficulty balance method, and the learning of the categories is gradually strengthened in order from difficult to easy. The application balances the data from the aspects of quantity and difficulty, thereby reducing the bias of the model when extracting features and promoting the learning of causal features.

Description

TECHNICAL FIELD

[0001] The present application belongs to the field of image classification, and particularly relates to an out-of-distribution image classification method and system based on sample balanced causal reasoning. BACKGROUND

[0002] The statements in this section merely provide background information related to the present application and do not necessarily constitute the prior art.

[0003] Image classification algorithms are usually based on the independent and identically distributed (i.i.d.) assumption, but in reality, there is a widespread out-of-distribution (OOD) generalization problem, i.e., the training and testing do not satisfy the same distribution, and the context of the images on the test set is usually unseen in training.

[0004] In this case, the existing models trained under the i.i.d. assumption are limited in generalization ability. Causal reasoning is an important method to learn invariant causal relationships in different environments, thereby improving the generalization ability of the model. However, the existing causal intervention backdoor adjustment method usually needs to divide the data set into multiple different environments, and then learn the cross-environment invariant causal features from different environments. However, the current environment division method is either directly divided according to the background, which is relatively rigid, or many data sets are not annotated with backgrounds. To solve this problem, CaaM proposes a feature-level fine-grained division, but this division lacks constraints, resulting in an imbalance in the number of divided environments, which brings another bias. SUMMARY

[0005] In order to solve at least one of the technical problems in the background art, the present application provides an out-of-distribution image classification method and system based on sample balanced causal reasoning, which divides the data set into subsets of different backgrounds without manual annotation, and maintains the balance in the number. At the same time, the number difference of different classes in the training data is also analyzed, and the training of the few-sample class is increased, so as to improve the quality of causal feature extraction.

[0006] In order to achieve the above purpose, the present application adopts the following technical solutions:

[0007] The first aspect of the present application provides an out-of-distribution image classification method based on sample balanced causal reasoning, comprising the following steps:

[0008] performing feature extraction on the image data set to obtain causal features and confounding features;

[0009] using the confounding features to divide the image data set into multiple balanced subsets of different environments;

[0010] The training sequence is set according to the classification accuracy of different categories in the image data set, and the categories are trained in an incremental manner in the order from low to high classification accuracy;

[0011] According to the determined training sequence, the learning of the causal feature is performed, the knowledge learned from the balanced subsets of different environments is fused to learn the causal feature invariant across environments through the causal intervention on the balanced subsets of different environments;

[0012] The classification is performed based on the knowledge learned from the balanced subsets of different environments to obtain a classification result.

[0013] The second aspect of the present application provides a distribution out-of-distribution image classification system based on sample balanced causal reasoning, comprising:

[0014] The feature extraction module is configured to perform feature extraction on the image data set to obtain the causal feature and the confusion feature;

[0015] The balanced division module is configured to divide the image data set into a plurality of balanced subsets of different environments using the confusion feature;

[0016] The class difficulty balancing module is configured to set a training sequence according to the classification accuracy of different categories in the image data set, and train the categories in an incremental manner in the order from low to high classification accuracy;

[0017] The causal intervention module is configured to learn the causal feature according to the determined training sequence, fuse the knowledge learned from the balanced subsets of different environments through the causal intervention on the balanced subsets of different environments, and learn the causal feature invariant across environments;

[0018] The classification module is configured to perform classification based on the knowledge learned from the balanced subsets of different environments to obtain a classification result.

[0019] The third aspect of the present application provides a computer readable storage medium.

[0020] A computer readable storage medium having a computer program stored thereon, the program being executed by a processor to implement the steps of the distribution out-of-distribution image classification method based on sample balanced causal reasoning as described above.

[0021] The fourth aspect of the present application provides a computer device.

[0022] A computer device comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, wherein the processor executes the program to implement the steps of the distribution out-of-distribution image classification method based on sample balanced causal reasoning as described above.

[0023] Compared with the prior art, the present application has the beneficial effects that:

[0024] The present application automatically generates balanced subsets with different background characteristics by a balancing method, and then performs causal intervention on the data through the subsets with different backgrounds, so that the model can correctly focus on the causal characteristics of the image in different backgrounds, and the training difficulty of different classes is balanced through a class difficulty balancing method, and the learning of the classes is gradually strengthened in order from difficult to easy. Therefore, the present application balances the data from the aspects of quantity and difficulty, thereby reducing the bias of the model in extracting features and promoting the learning of causal characteristics. BRIEF DESCRIPTION OF DRAWINGS

[0025] The accompanying drawings, which form a part of this specification, are included to provide a further understanding of the application and are incorporated in and constitute a part of this specification. The embodiments of these drawings are set forth to explain the present application and are not intended to limit the present application in any way.

[0026] Figure 1 is the overall framework of the out-of-distribution image classification method based on sample balanced causal reasoning provided by the embodiment of the present application;

[0027] Figure 2 is a schematic diagram of the unsupervised fine-grained balanced division method provided by the embodiment of the present application;

[0028] Figure 3 is a comparison diagram of the balanced division effect of the method of the present application and the existing method. DETAILED DESCRIPTION

[0029] The present application will be further described below in conjunction with the drawings and embodiments.

[0030] It should be noted that the following detailed description is exemplary and is intended to provide further explanation of the present application. Unless otherwise specified, all technical and scientific terms used herein have the same meaning as generally understood by those skilled in the art to which the present application belongs.

[0031] It should be noted that the terms used herein are only for the purpose of describing specific embodiments and are not intended to limit the exemplary embodiments according to the present application. As used herein, the singular form is intended to include the plural form unless the context clearly indicates otherwise, and furthermore, it should be understood that when the terms "comprise" and / or "include" are used in the specification, there is a feature, step, operation, device, component and / or combination thereof.

[0032] TERMINOLOGY

[0033] 1. OOD generalization:

[0034] To handle the problem that training data and test data do not meet the same distribution in machine learning, people have proposed the concept of out-of-distribution (OOD) generalization. OOD generalization problem aims to solve the challenging setting where the test distribution is unknown and different from the training distribution. OOD problems exist in many fields, such as computer vision, natural language processing, recommendation systems, etc. For example, visual models may rely on context or local texture for recognition, rather than image subject or global shape cues; in language models, the model may only predict by some words, rather than focusing on the meaning of the entire sentence; in recommendation systems, the model may focus on a larger number of interactions, and some niche content may never be recommended. This will introduce bias in model recognition, affecting the performance of the model.

[0035] In the field of computer vision, there are some common challenging OOD settings, which are: debiasing problem, domain generalization problem, and long-tail classification problem. Model recognition is often disturbed by some bias, such as the background and texture of the picture, and debiasing method separates the subject and bias by data augmentation or introducing bias model. In the domain generalization problem, the model often faces different styles of images from different domains, such as real photos and cartoon images. It achieves robust recognition of different domains by learning the common points in different images. In the long-tail classification problem, there is a great imbalance in the number of images of different categories, which greatly affects the recognition of tail categories. The model promotes tail class learning by establishing balance between head and tail classes while maintaining head class learning.

[0036] Several common methods for handling OOD generalization problems are: domain generalization / adaptation methods, causal learning methods, and stable learning methods, etc. FSDR method generalizes between different domains by decoupling images into domain-invariant and domain-varying frequency components (FCs). GCM-CF method generates counterfactual samples by generating causal models, without strict decoupling, thus promoting zero-shot and open-set recognition. StableNet method suggests eliminating dependencies between features by learning the weights of training samples, and focusing on the true relationship between discriminative features and labels. GrOD method reduces the regularized loss to experience loss minimization by improving the training process of regularized deep learning.

[0037] To promote the research of distribution generalization, many datasets have appeared in the field of computer vision in recent years: NICO, NICO++, CMNIST, PACS, ImageNet-LT, CIFAR-LT, etc. NICO dataset helps the model better distinguish the subject and environment of the image by adding context labels to the image. PACS dataset provides images of several different styles to help learn domain adaptation. ImageNet-LT and CIFAR-LT construct long-tail datasets to solve the long-tail problem.

[0038] 2. Causal reasoning

[0039] Causal reasoning is an effective means to solve the OOD problem. Causal reasoning is divided into three levels: association, intervention, and counterfactual.

[0040] Association is to find the correlation between variables by observing data, but only the correlation between data can be known, not the causal relationship between them. This is the association learned by classical machine learning algorithms. For example, if the model judges the picture of the dog on the grass as a dog, it can only know that this picture is a dog, but it cannot judge whether the machine learning model predicts this image by the dog or the grass.

[0041] Causal intervention and counterfactual are effective tools to achieve causal reasoning. The second level of causal reasoning is intervention, that is, when changing an event, it is judged whether another event will change. If it changes, it means that there is a causal relationship between the two, and if it does not change, it means that it is a confounding variable. Through causal intervention, the model can remain robust to changes in different environments, as the environment is a confounding variable. In general, front-door adjustment or back-door adjustment is an effective means to intervene to reduce the impact of confounding variables. Arjovsky divides the dataset into subsets of different environments and designs the IRM loss to learn the optimal classifier in all environments, thereby intervening on confounding features and focusing on causal features that remain unchanged in different environments. The CaaM method performs more fine-grained automatic partitioning and prevents over-intervention by constantly updating to achieve back-door adjustment. However, this method is usually limited by the unbalanced partitioning of the dataset. Covariate balancing is also an intervention method. ICP and a series of subsequent work improve the stability of generalization by controlling the covariance of different subsets. The CRLR method achieves the similarity of covariates by reweighting samples. Lin et al. intervene in training to distinguish between model and bias, use back-door adjustment to eliminate the influence of contrast distribution bias, and use weighting strategy to eliminate the influence of spatial distribution bias.

[0042] The third level of causal reasoning is counterfactual, which infers possible outcomes through conditions that did not occur. Counterfactual reasoning usually helps the model identify subjects by generating counterfactual samples. The GCM-CF method proposes to generate causal models to produce counterfactual samples, effectively alleviating the visible / invisible class imbalance in zero-shot learning and open set identification. Tang et al. use counterfactual reasoning to detect and eliminate the causal effect of momentum terms in the test phase.

[0043] In order to solve the technical defects of the current out-of-distribution image classification mentioned in the background of the present application, the present application proposes a sample balanced causal learning framework (BCL: Balanced Causal Learning, BCL), which automatically generates balanced subsets with different background features through an unsupervised fine-grained sample balancing method (GB). For example Figure 2 Without manual annotation, the data set is divided into subsets with different backgrounds and maintains balance in quantity. Then, through the background different subsets, the data is intervened causally, so that the model can correctly focus on the causal features of the image in different backgrounds. Through the class difficulty balancing (CDB) method, the training difficulty of different classes is balanced, and the learning of the class is gradually strengthened in order from difficult to easy. The present application balances the data from the quantity and difficulty aspects, thereby reducing the bias of the model in extracting features and promoting the learning of causal features. Finally, the experiments on the OOD data sets NICO and NICO++ show that the BCL realizes stable prediction on the OOD data. We also found that compared with existing causal reasoning methods, the model using BCL focuses more accurately on the subject of the image and ignores the background, thereby effectively improving the generalization ability.

[0044] Embodiment one

[0045] As Figure 1 shown, the present embodiment provides an out-of-distribution image classification method based on sample balanced causal reasoning, comprising the following steps:

[0046] Step 1: Feature extraction: performing feature extraction on the obtained image data to obtain causal features F c , confusion features F s and mixed features F x ;

[0047] The features of the input image are decoupled into causal features, confusion features and mixed features by embedding the CBAM module, which are used for different training processes respectively.

[0048] Before training, first randomly select two samples from the training samples for random weighted summation, and the labels of the samples are also correspondingly weighted and summed to augment the data and improve the generalization ability of the model.

[0049]

[0050]

[0051] where λ ∈ [0, 1] is a probability value, λ ~ Beta(α, α), i.e., λ follows Beta distribution with both parameters being α.

[0052] Then the image features are decoupled into causal features F c , confusing features F s and mixed features F x by using attention modules. c s x , specifically including:

[0053] The CBAM attention module is embedded into each block of the ResNet network, and the causal features F c and the confusing features F s are decoupled in each block.

[0054]

[0055]

[0056] where Sigmoid(-z) = 1-Sigmoid(z), and represents dot product. Then the two features are fused to obtain mixed features F x :

[0057] F x = Conv(F c ) + Conv(F s )

[0058] where Conv(·) represents the remaining convolutional layers in the basic block of ResNet after the CBAM attention module.

[0059] By stacking multiple basic blocks, the three decoupled features can be obtained, and the next step can be performed.

[0060] The above scheme obtains causal features, confusing features and mixed features through feature extraction. Then, the mixed features are used for overall learning, the confusing features are used to automatically balance the data set into subsets of different environments, and the causal features are used to further train the divided subsets. The ability of the trained model to extract causal features improves its generalization ability and robustness.

[0061] Step 2: Balanced division: use the confusing features to train the division index matrix, and use a balanced method such as GB to enhance the balance of the division, and divide the data set into subsets with different background features.

[0062] After the confusing features, the data set is divided, specifically including:

[0063] An un-supervised de-training a partition matrix using the confusion features to automatically partition the dataset into subsets of different environments.

[0064] First, train a classifier h for each matrix, and get the predicted value c = h(F s ).

[0065] Minimize the ERM loss of the classifier:

[0066]

[0067] Where is the training data, h is a linear classifier, F s is the confusion features from the previous module, and y is the label.

[0068] Through the above method, the model can more accurately extract confusion features, which generally represent the characteristics of the environment, so that the dataset can be more accurately partitioned into different environments.

[0069] Then freeze the classifier and the feature extractor trained in the previous round, and train a partition matrix θ under the constraint of an IRM loss to update the partition of the dataset in a fine-grained manner:

[0070]

[0071] Where, is the current data partition, represents the partition is determined by θ ∈ R k×m , K is the total number of training samples, m is the number of partitioned subsets, is the loss in subset t i , h is the classifier trained in the previous step, w = 1.0 is a scalar and a fixed "virtual" classifier, the gradient norm penalty is used to measure the optimality of the virtual classifier in each subset t, and γ ∈ [0, ∞] is a regularizer that balances the ERM term and the invariance of the predictor 1·h.

[0072] However, the partition obtained only by the above training has a large imbalance.

[0073] This embodiment adopts three methods to balance the partition, including GB (Aggregation Balance), LB (Loss Balance) and MB (Manual Balance) methods. The role of the balancing method is to balance the difference in the number of images between different partitions.

[0074] The process of balancing partitioning by GB method includes:

[0075] The degree of imbalance is alleviated by training multiple partition matrices.

[0076] Training only one partition matrix θ can lead to a huge imbalance of subsets. In order to alleviate the imbalance, in this embodiment, multiple matrices are trained, and the probability distribution of multiple training is combined, and then the final partition is determined:

[0077]

[0078] Where p(k n ,m j ) represents the probability of n th images being partitioned into j th partitions. For θ final ∈R K×m , the split index to be partitioned is:

[0079]

[0080] The process of balancing partitioning by LB method includes:

[0081] The LB method is to add a balance loss in the process of training the partition matrix. The number of images in m partitions is calculated, and the value of adjacent partitions is subtracted two by two, which is used as a balance loss to intervene the training of the partition matrix, so that the dataset can be balancedly partitioned. As shown in the formula:

[0082]

[0083] Where N i represents the number of samples in i th segmentation.

[0084] The process of balancing partitioning by MB method includes:

[0085] The MB method is to balance different subsets by redistributing part of the images. This method can be used after the GB and LB methods. After the GB and LB methods, there is still a slight imbalance, and the MB method can be used to fine-tune the subsets to which the images are partitioned to achieve further balance.

[0086] In this embodiment, balance experiments are carried out in multiple ways, including balancing classes (MB_CLS) and contexts (MB_CTX, MB_MTC). Since the context of each class follows a long-tail distribution, this embodiment uses two methods to balance the context: one is to balance all contexts (MB_CTX), and the other is to only balance the first two contexts which account for the vast majority of all images (MB_MTC).

[0087] Step 3: Class difficulty balance: train different classes in the image dataset in order from difficult to easy, improve the training balance of different difficulty classes, consolidate simple classes while strengthening the training of difficult classes.

[0088] Through the balancing method, the balance of image quantity between different divisions is achieved in this part of data division.

[0089] Further, for the training process, the training difficulty of the class is balanced. That is, the class with low classification accuracy and difficult to identify is increased.

[0090] According to the accuracy rate of the last round of model judgment, the difficult training samples are trained first, and then the normal training process is continued.

[0091]

[0092] where C q represents the accuracy of class q th g i represents the classifier of the i-th division. Then, C is sorted from low to high according to the accuracy.

[0093]

[0094] In this embodiment, the classes are trained in an incremental manner. First, the two classes with the lowest training accuracy (C1, C2) are trained. After a division matrix update process, the trained classes are updated to the next two classes (C3, C4). In order to prevent the model from forgetting the classes learned in the last update, an incremental method is adopted, that is, some samples of the last trained classes are retained and new samples of the new classes are added, instead of replacing the samples of the last trained classes with only the samples of the newly added classes. That is, if the last training uses class 1 and class 2 samples, this time we will retain some class 1 and class 2 samples and add class 3 and class 4 samples, instead of directly replacing all class 1 and class 2 samples with class 3 and class 4 samples.

[0095] The process is as follows:

[0096]

[0097]

[0098] …

[0099]

[0100] where φ x represents the feature extractor, F cq represents the causal feature of class q th Pq represents q th The prediction of the class. By balancing the difficulty, the classes with greater differences can be better balanced in the OOD environment.

[0101] It should be noted that here After splitting, it is still split because it needs to learn causal features from different environments, but the included classes are different.

[0102] Step 4: Causal intervention: By implementing causal intervention on subsets of different backgrounds, the knowledge learned from different environment data subsets is fused to distinguish the invariance of causal feature environments.

[0103] After dividing the dataset and deciding the training order, further learning of causal features is carried out, and causal reasoning is achieved through backdoor adjustment.

[0104]

[0105] where P(Y|X,t) represents the prediction result of the classifier trained in partition t, and P(t): = 1 / m.

[0106] do(·) represents the do operator, which is a tool for causal intervention. do(X) is to intervene on X, exclude the relationship between image X and context, and further exclude the spurious correlation between context and prediction result. Therefore, the model is trained on the data of different environments in the balanced partition, so that the model can exclude the relationship between the environment and the prediction result, so that the model can focus on the subject of the image in any environment, thereby achieving causal reasoning.

[0107] First, use ERM loss to constrain the feature extraction part and the classifier, so that the model can accurately extract features.

[0108]

[0109] where m is the number of partitions, t i represents a specific partition, g i is a linear classifier for t i , F c is the causal feature.

[0110] Then, by dividing the data of different environments and training multiple classifiers, and using IRM Loss to make the weights of these classifiers consistent, an optimal classifier in all environments can be obtained. With this classifier, the interference of the context can be reduced, and the model can better focus on the causal features.

[0111]

[0112] After multiple division updates and training, the backdoor adjustment formula can be gradually approached to realize causal reasoning. For mixed features, the embodiment is trained on all data sets, and the ERM loss is calculated to update the model. The loss is as follows:

[0113]

[0114] wherein represents a classifier for mixed features learned by all images.

[0115] There are multi-class loss constraints in BCL, and a phased training method is adopted. The first phase trains the feature extraction, which is jointly constrained by the experience risk loss from different data subsets and the invariant risk loss of aligning the classifier weights in different environments, and is also constrained by the experience risk loss from the entire data set. In order to make the model extract better features, the three losses are minimized:

[0116]

[0117] wherein η is the coefficient of invariant loss, which is currently set to 5e5.

[0118] The second phase is training data division. First, train a biased classifier using the experience risk loss, and then use the classifier to constrain data division by invariant risk loss. Minimize the experience loss to improve the model's ability to distinguish confusing features, but maximize the invariant loss to divide the data into m data divisions according to the differences in confusing features, so as to achieve the purpose of dividing different environmental data subsets:

[0119]

[0120] In order to verify the technical effect of the present application, the following experiments are carried out, specifically:

[0121] Obtain the NICO and NICO++ data sets; by comparing the performance of the BCL of the present application and the existing image classification methods on the NICO and NICO++ data sets, the performance of the BCL of the present application is compared with the existing image classification methods.

[0122] The tested methods are simply divided into three categories: traditional methods, debiased methods and causal methods. The traditional methods include baseline ResNet18, attention module CBAM, and two data augmentation methods Cutout and Mixup. The debiased methods include ordinary and debiased Rebias, and also include the RUBi method. The causal methods include CNBB, IRMCon, CaaM and the BCL method of the present application. As shown in Table 1, the results of the comparison are as follows.

[0123] Table 1 Performance comparison of BCL and existing algorithms on NICO and NICO++ datasets

[0124]

[0125] From Table 1, we can see that:

[0126] Due to the effect of confounding factors, traditional deep learning models such as ResNet18 will be affected in performance when facing complex OOD environments. Simply adding attention mechanism CBAM may cause attention to focus on the wrong area, such as the background of the image, in OOD environment. This leads to the performance of the model even worse than the baseline algorithm.

[0127] Image enhancement methods Cutout and Mixup can effectively expand the distribution of context. After enhancing the learning ability of the model through the method of causality, these two can make the training of the model more sufficient, see Table 2 for details.

[0128] By reducing the model's dependence on data distribution bias, the de-biasing method makes the model's identification more stable, achieving better results than the baseline.

[0129] The performance of the Causal algorithm is better than the baseline algorithm, and also better than the attention method and the de-biasing method. CNBB uses ResNet18 here to replace the baseline in the NICO paper. For CaaM, random partition means using a random matrix conforming to the normal distribution to perform the first round of training. Pre-partition means using the model trained by the baseline to perform partition first, and then using this partition to perform the first round of training. It learns the invariant causal features in different environments through the partition of the dataset, and maintains stability in prediction in different environments. However, the partition of the dataset lacks constraints, which loses part of the performance.

[0130] When BCL divides the dataset into different environmental subsets, it strengthens the balance between subsets by adding constraint conditions in the partition process. At the same time, the classes are balanced in difficulty to strengthen the training of difficult classes. This method is better than the baseline and the use of attention mechanism alone in OOD conditions, with about 5%-6% performance improvement.

[0131] Table 2 Impact of each module of the algorithm on performance.

[0132]

[0133] The effectiveness of the proposed algorithm is investigated. ResNet18 is selected as the baseline in the experiment. As shown in Table 2, adding the attention mechanism CBAM will directly affect the performance, because the attention mechanism may capture false correlations as the reason for prediction under the OOD background. On the basis of CBAM, a causal relationship method is added to alleviate the problem of attention focus error under the OOD environment. However, at this time, there is no restriction on the division of the data set, which is easy to cause the problem of unbalanced data division.

[0134] Different balancing methods are tried to constrain the division. In the GB method, the imbalance of the division is alleviated to a certain extent while the intervention on the division is minimal, and the CDB method balances the training difficulty of different categories, so the best performance is obtained.

[0135] In addition, it is found that mixup as an effective data augmentation method also has good effect under OOD. The effect of three balancing methods after adding mixup is tested, and the GB+CDB method achieves the best performance.

[0136] (1) Comparison of balancing strategies

[0137] The present application tests three quantity balancing methods LB, MB, GB and difficulty balancing method CDB. There are three implementation ways for the MB method to balance the data set. The present application shows the test results of these methods to explain the reason for selecting GB and CDB as the final method. Table 3 is the comparison results of different balancing methods in performance, and Table 4 is the performance comparison results of different MB methods.

[0138] Table 3 Comparison of different balancing methods in performance

[0139]

[0140] From Table 3, it can be seen that compared with unbalanced data division, the effect of balancing method is improved, whether or not to join the data enhancement method, the GB and CDB methods have achieved better results than other methods. This is because the GB method can reduce the degree of imbalance while retaining the original division to the maximum. And the CDB method can focus on the training difficulty of different categories in the OOD environment.

[0141] Table 4 Performance comparison of different MB methods

[0142]

[0143]

[0144] From Table 4, it is shown that balancing the dataset from different perspectives, including balancing classes, balancing contexts, and balancing the two most numerous contexts. Among these methods, it can be found that balancing classes has the best effect, which may be because the number of classes in different subsets is very different, causing the model to not be able to fully learn the invariant features in different environments.

[0145] Although the GB method can alleviate the degree of imbalance, there will still be some imbalance. Therefore, by training the three branches of the GB method and the MB method, and also training the difficulty balancing method CDB, it is verified whether they will have good effects. As shown in Table 5.

[0146] Table 5 Effect comparison of various balancing methods

[0147]

[0148] For the MB_MTC+GB method, it can be found that different results are obtained on the two datasets. On the NICO dataset, the two methods are used at the same time, which is better than the simple GB method. However, on the NICO++ dataset, it can be found that the effect of the GB+MB_MTC method can exceed the simple GB method. The possible reason is that the complexity of the environment and the difficulty of class recognition in NICO are higher than in the selected NICO++ dataset. The NICO dataset contains 10 animal classes, and the NICO++ dataset contains animal classes and other classes such as vehicles, which is one of the complexities of NICO. In addition, the NICO dataset contains 10 classes and a total of 33 contexts, while the NICO++ dataset contains only 6 contexts. Therefore, the complexity of the environment makes the balancing of the environment division of the NICO dataset more destructive, and the hard balancing will bring performance loss.

[0149] For the GB+CDB method, good results are obtained on both datasets, which shows that a more balanced environment division and a more balanced class difficulty in the OOD environment can effectively improve the stability and robustness of the model.

[0150] (3) Analysis of balanced division

[0151] As shown in Figure 3 , the actual effect of BCL in subset balancing division, using the division of the CaaM method Figure 3 , there is obvious imbalance between different divisions, which greatly restricts the ability of the model to learn invariant features from different subsets. Using the subsets in the BCL division Figure 3 (a), the subsets are relatively more balanced and very stable in different divisions, effectively improving the effect of the model learning from causal features.

[0152] Example Two

[0153] The embodiment provides a distribution out-of-distribution image classification system based on sample balance causal reasoning, which comprises the following steps:

[0154] a feature extraction module, which is configured to perform feature extraction on the image dataset to obtain causal features and confusion features;

[0155] a division balance module, which is configured to divide the image dataset into balance subsets of different environments using the confusion features;

[0156] a class difficulty balance module, which is configured to set a training sequence according to classification accuracies of different classes in the image dataset, and train the classes in an incremental manner in a sequence from low to high according to the classification accuracies;

[0157] a causal intervention module, which is configured to learn the causal features according to the determined training sequence, perform causal intervention on the balance subsets of different environments, and fuse knowledge learned from the balance subsets of different environments to learn causal features that are invariant across environments;

[0158] a classification module, which is configured to perform classification based on the knowledge learned from the balance subsets of different environments to obtain a classification result.

[0159] Embodiment three

[0160] The embodiment provides a computer readable storage medium, which stores a computer program, and the program is executed by a processor to implement steps in the distribution out-of-distribution image classification method based on sample balance causal reasoning.

[0161] Embodiment four

[0162] The embodiment provides a computer device, which comprises a memory, a processor, and a computer program stored in the memory and executable on the processor, and the processor implements steps in the distribution out-of-distribution image classification method based on sample balance causal reasoning when executing the program.

[0163] Those skilled in the art should understand that the embodiments of the present application can be provided as a method, a system, or a computer program product. Therefore, the present application can be in the form of a hardware embodiment, a software embodiment, or an embodiment combining software and hardware aspects. Moreover, the present application can be in the form of a computer program product implemented on one or more computer usable storage media (including but not limited to disk storage and optical storage, etc.) containing computer usable program code.

[0164] The embodiments of methods, apparatuses (systems) and computer program products according to the present application can be described in flowchart and / or block diagram form in accordance with embodiments of the application. It will be understood that each and every flow and / or block in the flowcharts and / or block diagrams, and combinations of flows and / or blocks in the flowcharts and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general purpose computer, special purpose computer, embedded processing system or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions specified in the flowcharts and / or block diagrams block or blocks. Figure 1 one or more flows and / or blocks. Figure 1 one or more flows and / or blocks.

[0165] These computer program instructions can also be stored in a computer-readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable memory produce an article of manufacture including instructions which implement the function specified in the flowcharts and / or block diagrams block or blocks. Figure 1 one or more flows and / or blocks. Figure 1 one or more flows and / or blocks.

[0166] These computer program instructions can also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowcharts and / or block diagrams block or blocks. Figure 1 one or more flows and / or blocks. Figure 1 one or more flows and / or blocks.

[0167] It can be understood by those skilled in the art that all or part of the above-mentioned embodiment methods can be implemented by computer program instructions to instruct related hardware, and the program can be stored in a computer-readable storage medium. When the program is executed, it can include the flow of each method embodiment as described above. The storage medium can be a magnetic disc, an optical disc, a read-only memory (ROM) or a random access memory (RAM), etc.

[0168] The above only describes the preferred embodiments of the present application and is not intended to limit the present application. For those skilled in the art, the present application can have various modifications and changes. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application shall be included in the protection scope of the present application.

Claims

1. An out-of-distribution image classification method based on balanced causal reasoning, characterized in that, Includes the following steps: Feature extraction is performed on the image dataset to obtain causal features, confounding features, and mixed features; The system utilizes fusion features for holistic learning, trains the fusion features on all datasets, and uses confusion features to divide the image dataset into multiple balanced subsets of different environments. The training order is set according to the classification accuracy of different categories in the image dataset. The categories are trained incrementally in order of classification accuracy from low to high. First, the two categories with the lowest classification accuracy are trained. After a partition matrix update process, the trained categories are updated to the next two categories. An incremental method is adopted, which retains some samples from the previous training categories and adds samples from the new categories. According to the predetermined training sequence, causal features are learned. By intervening in the equilibrium subsets of different environments, the knowledge learned from the equilibrium subsets of different environments is integrated to learn cross-environment invariant causal features. The classification results are obtained by classifying based on knowledge learned from equilibrium subsets of different environments.

2. The out-of-distribution image classification method based on sample-balanced causal reasoning as described in claim 1, characterized in that, When using confusion features to divide an image dataset into balanced subsets of different environments, the specific steps include: A partition index matrix is ​​trained using obfuscated features. The dataset is then partitioned based on the partition index matrix to obtain subsets with different background features. A balancing method is then used to further enhance the partitioning, and the partitioning results are used as the final subsets with different environments.

3. The out-of-distribution image classification method based on sample-balanced causal reasoning as described in claim 2, characterized in that, When using obfuscated features to train and partition the index matrix, the following is included: For each matrix, a classifier is trained to obtain the predicted value. During training, the current classifier and the feature extractors trained in previous rounds are frozen. Under the constraint of loss, the partition matrix is ​​trained, and the partitions of the dataset are updated step by step in a fine-grained manner to obtain the trained partition matrix.

4. The out-of-distribution image classification method based on sample-balanced causal reasoning as described in claim 1, characterized in that, When learning causal features, causal reasoning is achieved through backdoor adjustments to eliminate spurious correlations between context and prediction results.

5. The out-of-distribution image classification method based on sample-balanced causal reasoning as described in claim 1, characterized in that, When training the image classifier, multi-class loss constraints are used for phased training. The first phase is the extraction of training features, which includes the common constraints of empirical risk loss for different data subsets, invariant risk loss for aligning classifier weights under different environments, and empirical risk loss for the entire dataset. The second phase is the partitioning of training data. An empirical risk loss is used to train a biased classifier, and then the classifier is used to constrain the data partitioning through invariant risk loss.

6. The out-of-distribution image classification method based on sample-balanced causal reasoning as described in claim 1, characterized in that, Before training the image classification model, two samples are randomly selected from the training samples and summed using random weights. The labels of the samples are also summed using corresponding weights to augment the data.

7. An out-of-distribution image classification system based on balanced causal reasoning, characterized in that, include: The feature extraction module is used to extract features from the image dataset to obtain causal features, confusion features, and mixed features; Holistic learning is performed using fusion features, which are trained on all datasets. The partitioning balance module is used to divide the image dataset into multiple balanced subsets of different environments using confusion features; The class difficulty balancing module is used to set the training order based on the classification accuracy of different categories in the image dataset. The classes are trained incrementally in order of classification accuracy from low to high. First, the two classes with the lowest classification accuracy are trained. After a partition matrix update process, the training classes are updated to the next two classes. An incremental method is adopted, which retains some samples from the classes trained in the previous training and adds samples from the new classes. The causal intervention module is used to learn causal features according to a predetermined training order. By intervening in the equilibrium subsets of different environments, it integrates the knowledge learned from the equilibrium subsets of different environments to learn cross-environment invariant causal features. The classification module is used to classify data based on knowledge learned from balanced subsets of different environments, and obtain the classification results.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it implements the steps in the out-of-distribution image classification method based on sample-balanced causal reasoning as described in any one of claims 1-6.

9. A computer device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the steps in the out-of-distribution image classification method based on sample-balanced causal reasoning as described in any one of claims 1-6.

Images