Inter-class gap adaptive causal inference semi-supervised image classification method and device
By constructing a loss function based on the propensity score of image samples and a dynamic margin threshold, the problem of class imbalance in semi-supervised learning is solved, improving the accuracy and robustness of image classification and making it applicable to various imbalanced datasets.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- NAT UNIV OF DEFENSE TECH
- Filing Date
- 2023-11-21
- Publication Date
- 2026-05-12
AI Technical Summary
Existing semi-supervised learning image classification methods assume that the label distribution of unlabeled data is balanced when dealing with imbalanced datasets. This makes them ineffective in handling long-tailed datasets in the real world, leading to predicted pseudo-labels biased towards the head class, which affects classification accuracy and robustness.
By calculating the propensity score of image samples, a loss function is constructed by introducing a dynamic interval threshold, and the image classification model is optimized to generate a robust classification model, severing the spurious association between the category and the missing variable, and eliminating the influence of imbalance factors.
It improves the accuracy and robustness of image classification in imbalanced scenarios, is applicable to dynamic open scenarios, simplifies the training process, and has good theoretical guarantees and interpretability.
Smart Images
Figure CN117611887B_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of image classification technology, and in particular to a class-spaced adaptive causal reasoning semi-supervised image classification method and apparatus. Background Technology
[0002] Existing image classification methods based on deep neural networks often rely on large-scale, high-quality manually labeled datasets for their good performance. Compared to a large amount of reliable labeled data, unlabeled samples are inexpensive. Image classification methods based on semi-supervised learning introduce a large amount of unlabeled data to assist the classification model in training with a small amount of labeled data, which is an effective way to reduce the cost of manual labeling.
[0003] However, existing semi-supervised learning-based image classification methods typically assume that the label distribution of unlabeled data is balanced, meaning that the number of samples in each class is almost equal. Many real-world datasets, however, exhibit a long-tailed distribution, where the number of samples in the head class is significantly greater than that in the tail class. This contradiction between ideal data assumptions and practical applications presents both theoretical and performance challenges to the effectiveness and robustness of these methods in class-imbalanced image classification scenarios. When general deep semi-supervised methods process this type of data, the predicted pseudo-labels tend to favor the head class. As training iterates, this bias is gradually amplified, eventually manifesting as a severe deterministic bias that negatively impacts the final classification accuracy and robustness.
[0004] Semi-supervised image classification with imbalanced class distribution presents a greater challenge due to the unknown prior class distribution. Related research and technical solutions can be categorized into three types: example re-sampling, pseudo-label alignment, and example re-weighting.
[0005] Sample resampling methods attempt to resample training samples to construct a balanced dataset. The CReST method is a typical resampling approach, assuming that labeled and unlabeled data have similar class distributions and proposing to estimate the proportion of unlabeled samples based on the distribution of the number of labeled samples. Similar to the CReST sampling technique, the CoSSL method decouples the feature representation from the classifier during training and resamples samples with low recall based on the proportion of labeled samples by class. The difference is that this method uses random interpolation to augment the feature layer of the minority class and mixes pseudo-labels from different classes to reduce the overall bias of pseudo-labels. Similarly, the Auxiliary Balanced Classifier (ABC) method adds an auxiliary classifier to the original semi-supervised classifier. It uses a mask sampled according to a Bernoulli distribution that is positively correlated with the proportion of labeled sample classes to ensure the class balance of the auxiliary classifier; for unlabeled samples, it uses a consistent regularization loss with a class mask applied, and the mask probability is estimated based on the Bernoulli distribution between the number of pseudo-label classes and the number of labeled sample classes. The drawback of this type of method is that it assumes that labeled and unlabeled samples have similar class distributions, making it unable to handle class imbalance scenarios where the distributions are dissimilar. To alleviate this problem, the Adsh method argues that selecting pseudo-labels based on a fixed threshold is the main cause of class imbalance. It proposes setting a threshold variable for each class and using bi-level optimization to solve for the pseudo-label threshold and optimize the parameters of the classification model step by step.
[0006] Pseudo-label alignment methods eliminate the cumulative deterministic bias of the model by matching pseudo-labels with the expected distribution. Kim J et al. treated pseudo-label comparison as a convex optimization problem and achieved this by minimizing the KL divergence between the predicted pseudo-labels and the true class distribution. However, this method is based on the assumption that labeled and unlabeled samples have similar distributions and uses a confusion matrix to estimate the true class distribution of unlabeled samples. To overcome the assumption of similarity between labeled and unlabeled samples in the above methods, Oh et al. found that the class bias of semantic pseudo-labels caused by imbalanced datasets is opposite to that of pseudo-labels predicted by the classifier. They proposed the DASO method, which re-matches pseudo-labels based on the difference between semantic labels and predicted pseudo-labels. This method is applicable to imbalanced datasets with similar distributions as well as imbalanced datasets with dissimilar distributions or even inversely matched distributions. In addition, there are margin-based methods, which are often used in logit adjustment and pseudo-label smoothing. Logit adjustment adds a class margin to the SoftMax exponentially normalized probability output to balance the weights of different classes in gradient training. Pseudo-label smoothing uses an approximate direct causal effect to remove the cumulative deterministic bias of the model, thereby obtaining re-matched pseudo-labels. The ACR method dynamically adjusts the interval weights in the pseudo-label matching process based on the degree of matching between the distributions of labeled and unlabeled samples in the dataset, thereby adapting to datasets with different balance conditions and expanding the application of interval-based methods.
[0007] Sample reweighting aims to assign different weights to unlabeled samples to reduce the impact of class imbalance. Lai et al. analyzed the importance of class weighting and used the sum of pseudo-labels of all unlabeled samples to estimate class weights, which were then used to weight the consistency regularization loss. The Retrieve method employs a two-layer optimized sample reweighting strategy: in the outer loop, sample weights are fixed to optimize the semi-supervised model, while in the inner loop, the first-order gradient approximation of labeled samples is used to estimate the binary discretized sample weights. Recently, some reweighting methods based on propensity score estimation have been proposed. These methods construct a causal relationship between labels and missing labels based on missing data theory, and use inverse probability weighting to sever the association between labels and missing labels, ensuring consistency between the empirical and theoretical risks of class imbalance semi-supervised learning methods. For example, the Self-Mask method uses inverse probability weighting (IPW) to improve the robustness of semi-supervised image classification methods on imbalanced datasets. Zhang Hanwang's research group proposed a Double Robust (DR) class of imbalanced semi-supervised image classification method, CADR. It uses inverse probability weighting to sever the association between labels and missing labels on the structural causal graph and uses a threshold to filter out unreliable pseudo-labels, thus ensuring the robustness of the CADR method.
[0008] In summary, existing semi-supervised learning image classification methods typically assume that the class distribution of the data is balanced, which cannot effectively handle the class imbalance problem that is prevalent in unlabeled data in dynamic open scenes. Summary of the Invention
[0009] Therefore, it is necessary to provide a semi-supervised image classification method and apparatus with adaptive causal reasoning based on class intervals to address the aforementioned technical problems.
[0010] A semi-supervised image classification method based on adaptive causal reasoning, the method comprising:
[0011] Based on the class conditional distribution of labeled image samples and the class margin distribution of unlabeled image samples, a propensity score related to the class margin distribution of the image sample is calculated.
[0012] The propensity score of the corresponding category is introduced into the category probability output by the image classification model as a dynamic interval threshold related to the category, and the loss function is constructed accordingly.
[0013] Optimize the loss function to obtain a trained image classification model for target image classification.
[0014] A semi-supervised image classification device with adaptive causal reasoning, the device comprising:
[0015] The propensity score calculation module is used to calculate the propensity score related to the class margin distribution of the image sample based on the class conditional distribution of labeled image samples and the class margin distribution of unlabeled image samples.
[0016] The loss function construction module is used to introduce the propensity score of the corresponding class into the class probability output by the image classification model, as a class-related dynamic margin threshold, and to construct the loss function accordingly.
[0017] The loss function optimization module is used to optimize the loss function to obtain a trained image classification model for target image classification. Attached Figure Description
[0018] Figure 1 This is a flowchart illustrating a semi-supervised image classification method with class-spaced adaptive causal reasoning in one embodiment.
[0019] Figure 2 This is a schematic diagram of the overall design of this method.
[0020] Figure 3 This is an internal structural diagram of a computer device in one embodiment. Detailed Implementation
[0021] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.
[0022] In one embodiment, such as Figure 1 As shown, a semi-supervised image classification method based on class-spaced adaptive causal reasoning is provided, including the following steps:
[0023] Step 102: Based on the class conditional distribution of labeled image samples and the class margin distribution of unlabeled image samples, calculate the propensity score related to the class margin distribution to which the image sample belongs.
[0024] Step 104: Introduce the propensity score of the corresponding category into the category probability output by the image classification model as a dynamic interval threshold related to the category, and construct the loss function accordingly.
[0025] Step 106: Optimize the loss function to obtain a trained image classification model for target image classification.
[0026] In the above-mentioned class-margin adaptive causal reasoning semi-supervised image classification method, propensity score adaptive margining is used for the training process. That is, a class-related dynamic margin threshold is constructed to cut off the spurious association between the class and the missing variable on the structural causal graph. This eliminates the influence of imbalanced factors in the data on model optimization, and finally obtains a robust image classification model. This solves the dual challenges of accuracy and robustness in image classification in class-imbalanced scenarios in both theory and performance.
[0027] It should be understood that, although Figure 1 The steps in the flowchart are shown sequentially as indicated by the arrows, but these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified herein, there is no strict order in which these steps are executed, and they can be performed in other orders. Figure 1 At least some of the steps in the process may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be executed in turn or alternately with other steps or at least some of the sub-steps or stages of other steps.
[0028] In one embodiment, the step of estimating the class edge distribution of unlabeled image samples includes:
[0029] To obtain Stable and accurate estimation results are achieved by using an exponential moving average (EMA) to weight historical class edge distributions, as shown in the following formula, to estimate the class edge distributions of unlabeled image samples:
[0030]
[0031] in, This represents the edge distribution of labels corresponding to class c in the normalized unlabeled image samples. Image sample x belonging to class c i The corresponding output of the logit layer after inputting the image classification model. Let λ0 be the class margin distribution of unlabeled image samples of class i, λ0 be the average scaling factor, m be the number of unlabeled image samples, and C be the total number of classes.
[0032] In one embodiment, based on the class conditional distribution of labeled image samples and the class margin distribution of unlabeled image samples, the propensity score related to the class margin distribution to which the image sample belongs is calculated as follows:
[0033]
[0034] in, This represents the propensity score related to the marginal distribution of the label in the image sample corresponding to the c-th class, where r is a binary indicator of missing labels; r=1 indicates labeled, and r=0 indicates unlabeled or missing labels. This represents the edge distribution of labels corresponding to class c in unlabeled image samples. This represents the conditional distribution of labels in labeled image samples corresponding to class c.
[0035] In one embodiment, based on the class conditional distribution of labeled image samples and the class margin distribution of unlabeled image samples, a propensity score related to the class margin distribution to which the image sample belongs is calculated, including:
[0036] The class conditional distribution of labeled image samples is set to a uniform distribution, and then the propensity score related to the marginal distribution of the class to which the image sample belongs is calculated:
[0037]
[0038] In one embodiment, training an image classification model includes:
[0039] Estimate the unbiased pseudo-labels of unlabeled image samples, use them as labels for the unlabeled image samples, and train the image classification model together with labeled image samples to obtain an intermediate image classification model.
[0040] In one embodiment, the unbiased pseudo-labels of unlabeled image samples are estimated as follows:
[0041]
[0042] in, The unbiased pseudo-label representing an unlabeled image sample, z k z c When the input is x k x c At that time, the logit output of the image classification model, i.e., z k =f θ (x k ), z c =f θ (x c ), where θ represents the parameters to be optimized in the image classification model, and λ1 is a manually set debiasing weight parameter. Let be the estimated class marginal probabilities of unlabeled data in classes k and c.
[0043] In one embodiment, the loss function is expressed as follows:
[0044]
[0045] Where θ represents the parameters of the image classification model. This represents the propensity score associated with the marginal distribution of the image's class, and for any k∈C, it holds. r is a binary indicator of missing labels, r = 1 indicates labeled, r = 0 indicates unlabeled or missing labels, and C represents the total number of image categories; (x i ,y i ) represents the i-th training image sample and x i Let y be the feature vector corresponding to this image sample. i The label is the corresponding label for the image sample; n is the number of labeled image samples, and m is the number of unlabeled image samples; This represents the unsupervised learning loss term corresponding to the debiased pseudo-label estimation for unlabeled image samples. This indicates that the supervised learning loss term for labeled image samples uses the negative logarithmic loss form. This indicates the intervalization of propensity scores, where λ is a hyperparameter.
[0046] In one embodiment, the loss function can be formalized as follows:
[0047]
[0048] Where C is the total number of categories in the sample dataset, z k z cWhen the input is x k x c At that time, the logit output of the image classification model, i.e., z k =f θ (x k ), z c =f θ (x c ), Corresponding to input samples x k x c The propensity score is calculated based on the input sample x; when the input sample x is labeled data, p(y=k|x) represents its corresponding label; when the input sample x is unlabeled data... This indicates the corresponding pseudo-tag.
[0049] like Figure 2 As shown, a schematic diagram of the overall design of this method is provided. It mainly includes two functional modules: propensity score intervalization and biased pseudo-label estimation. First, if... Figure 2 As shown in (b), during the inverse probability weighting process, the propensity scores related to categorical variables are spaced out, severing the spurious associations between categorical and missing variables on the structural causal graph, thus eliminating the influence of imbalanced factors in the data on model optimization. Secondly, if Figure 2 As shown in (c), in the process of pseudo-label prediction and model inference, counterfactual reasoning is used to remove the deterministic bias between categories accumulated during the training process of the model, so as to generate class-unbiased pseudo-labels, which are used as labels for unlabeled samples and trained together with labeled data to form an image classification model. Finally, a robust semi-supervised image classification model for class imbalance can be obtained. Figure 2 The confounding factors in the data are equivalent to missing variables.
[0050] In summary, this invention introduces causal reasoning technology into the training framework of image classification models based on semi-supervised learning. It also incorporates the theory of missing data, assuming a spurious association between class imbalance and labels, and uses inverse probability weighting to remove the influence of confounding factors on model performance. Furthermore, counterfactual reasoning is used to remove class bias accumulated by momentum to generate debiased pseudo-labels. This invention has the following advantages: (1) It unifies the class bias estimation in the propensity score and pseudo-label generation process under the perspective of class interval, which not only improves the stability of training but also simplifies the training process and has strong versatility; (2) Based on a weak class prior assumption, this method is applicable not only to imbalanced scenarios where labeled and unlabeled samples have consistent class distributions but also to scenarios where they are inconsistent or even inversely distributed; (3) It has good theoretical guarantees and interpretability, and possesses variance reduction properties.
[0051] In one embodiment, a class-margin adaptive causal reasoning semi-supervised image classification device is provided, comprising: a propensity score calculation module, a loss function construction module, and a loss function optimization module, wherein:
[0052] The propensity score calculation module is used to calculate the propensity score related to the class margin distribution of the image sample based on the class conditional distribution of labeled image samples and the class margin distribution of unlabeled image samples.
[0053] The loss function construction module is used to introduce the propensity score of the corresponding class into the class probability output by the image classification model, as a class-related dynamic margin threshold, and to construct the loss function accordingly.
[0054] The loss function optimization module is used to optimize the loss function to obtain a trained image classification model for target image classification.
[0055] Specific limitations regarding the class-spaced adaptive causal reasoning semi-supervised image classification device can be found in the limitations of the class-spaced adaptive causal reasoning semi-supervised image classification method above, and will not be repeated here. Each module in the aforementioned class-spaced adaptive causal reasoning semi-supervised image classification device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device, or stored in the memory of a computer device as software, so that the processor can call and execute the operations corresponding to each module.
[0056] In one embodiment, a computer device is provided, which may be a terminal, and its internal structure diagram may be as follows: Figure 3 As shown, the computer device includes a processor, memory, network interface, display screen, and input devices connected via a system bus. The processor provides computational and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The network interface is used to communicate with external terminals via a network connection. When executed by the processor, the computer program implements a class-interval adaptive causal reasoning semi-supervised image classification method. The display screen can be a liquid crystal display (LCD) or an e-ink display. The input devices can be a touch layer covering the display screen, buttons, a trackball, or a touchpad mounted on the computer device casing, or an external keyboard, touchpad, or mouse.
[0057] Those skilled in the art will understand that Figure 3The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.
[0058] In one embodiment, a computer device is provided, including a memory and a processor, the memory storing a computer program, the processor executing the computer program to implement the steps of the method described above.
[0059] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon, which, when executed by a processor, implements the steps of the method described above.
[0060] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. Any references to memory, storage, databases, or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), Rambus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.
[0061] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.
[0062] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the invention patent. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this patent application should be determined by the appended claims.
Claims
1. A semi-supervised image classification method based on adaptive causal reasoning, characterized in that, The method includes: Based on the class conditional distribution of labeled image samples and the class margin distribution of unlabeled image samples, a propensity score related to the class margin distribution of the image sample is calculated. The propensity score of the corresponding category is introduced into the category probability output by the image classification model as a dynamic interval threshold related to the category, and the loss function is constructed accordingly. Optimize the loss function to obtain a trained image classification model for target image classification; The steps for estimating the class edge distribution of unlabeled image samples include: An exponential moving average is used to weight the historical class margin distribution, thereby estimating the class margin distribution of unlabeled image samples: ； in, This indicates that the label in the normalized unlabeled image sample corresponds to the first... class edge distribution, Indicates belonging to the first Image samples of the class The corresponding output of the logit layer after inputting the image classification model. For the first i Class edge distribution of unlabeled image samples of a class. The average scaling factor. m This represents the number of unlabeled samples, where C is the total number of categories.
2. The method according to claim 1, characterized in that, Based on the class conditional distribution of labeled image samples and the class margin distribution of unlabeled image samples, the propensity score related to the class margin distribution of the image sample is calculated as follows: ； in, This indicates that the label in the image sample corresponds to the first... The marginal distribution of the class is related to the propensity score. r This represents a binary indicator variable indicating missing labels. r = 1 indicates that it has been marked. r = 0 indicates no annotation or missing label. This indicates that the label in the unlabeled image sample corresponds to the first... class edge distribution, This indicates that the label in the labeled image sample corresponds to the first... The conditional distribution of the class.
3. The method according to claim 2, characterized in that, Based on the class conditional distribution of labeled image samples and the class margin distribution of unlabeled image samples, a propensity score related to the class margin distribution of the image sample is calculated, including: The class conditional distribution of labeled image samples is set to a uniform distribution, and then the propensity score related to the marginal distribution of the class to which the image sample belongs is calculated: 。 4. The method according to claim 1, characterized in that, Training an image classification model includes: Estimate the unbiased pseudo-labels of unlabeled image samples, use them as labels for the unlabeled image samples, and train the image classification model together with labeled image samples to obtain an intermediate image classification model.
5. The method according to claim 4, characterized in that, The estimated unbiased pseudo-labels for unlabeled image samples are: ； in, Unbiased pseudo-labels representing unlabeled image samples When the input is At that time, the logit output of the image classification model is... , , This represents the parameters to be optimized and determined in the image classification model. The bias removal weight parameters are set manually. For the estimated first k , c The class marginal probability of unlabeled data of a class.
6. The method according to claim 1, characterized in that, The expression for the loss function is as follows: ； in, For image classification model parameters, This represents a propensity score related to the marginal distribution of the image's class, and for any... have , For a binary indicator variable with missing labels, r = 1 indicates that it has been marked. r = 0 indicates no annotation or missing label. C This represents the total number of image categories; Indicates the first i training image samples and This is the feature vector corresponding to the image sample. Assign a label to the image sample; n The number of labeled image samples. m The number of unlabeled image samples; This represents the unsupervised learning loss term corresponding to the debiased pseudo-label estimation for unlabeled image samples. This indicates that the supervised learning loss term for labeled image samples uses the negative logarithmic loss form. This indicates that the biased scores have been spaced out. This is a hyperparameter.
7. The method according to claim 6, characterized in that, The loss function can be formalized as follows: ； in, C The total number of categories contained in the sample dataset. When the input is At that time, the logit output of the image classification model is... , , , Corresponding to the input samples respectively The propensity score at the time; when the input sample When there is labeled data, This indicates its corresponding label; when input sample When the data is unlabeled, This indicates the corresponding pseudo-tag.
8. A semi-supervised image classification device with adaptive causal reasoning based on class-interval, characterized in that, The apparatus comprising the method according to any one of claims 1 to 7, wherein the apparatus includes: The propensity score calculation module is used to calculate the propensity score related to the class margin distribution of the image sample based on the class conditional distribution of labeled image samples and the class margin distribution of unlabeled image samples. The loss function construction module is used to introduce the corresponding class propensity score into the class probability output by the image classification model as a class-related dynamic margin threshold, and to construct the loss function accordingly. The loss function optimization module is used to optimize the loss function to obtain a trained image classification model for target image classification.