A domain adaptation image classification method based on self-training and centroid alignment

Abstract

The application discloses a domain self-adaption image classification method based on self-training and centroid alignment, comprising the following steps: obtaining data sets of a source domain and a target domain and performing simple data preprocessing; through a self-training process, the network obtains simple classification capability; through a consistency regularization loss, the consistency of outputs of two classification modules is improved; the distance between target domain data and corresponding class centroids of the source domain is shortened, and the target domain data features are promoted to move to the direction of the source domain class centroids, so that the features are aligned at a structure level; and in the learning process, the classification model effectively aligns the knowledge of different domains gradually. The application combines feature information and structure information, considers two angles of distance and direction, guides the model to learn domain-invariant features, and effectively solves the problem of performance decline of an image classification network caused by large differences between data distribution of the source domain and the target domain.

Description

Technical Field

[0001] This invention belongs to the field of image classification technology, and particularly relates to a domain-adaptive image classification method based on self-training and centroid alignment. Background Technology

[0002] Image classification, as one of the important tasks in computer vision, is widely used in various scenarios, such as face recognition, person re-identification, and video detection. In recent years, deep learning-based image classification tasks have been extensively explored and researched. These methods learn and train image classification networks from large amounts of data, achieving satisfactory accuracy and performance on datasets with the same source.

[0003] The effectiveness of image classification models depends on two conditions: first, a large amount of labeled dataset is needed for training; second, the training and test data must come from datasets with the same distribution. However, these conditions are difficult to meet in most real-world scenarios. First, creating a large number of labeled datasets is extremely time-consuming and labor-intensive, and manual annotation inevitably introduces human bias. Second, due to the diversity of real-world scenarios, when the distributions of image data in the target and source domains differ significantly, a classification model trained in the source domain may struggle to achieve effective performance in the target domain.

[0004] To address the issues of scarce labeled data and inconsistent data distribution in the target domain, domain adaptation methods are currently the mainstream approach. These methods learn domain-invariant features shared by the source and target domains using labeled source domain data and unlabeled target domain data, resulting in good performance for classification tasks on the target domain. Summary of the Invention

[0005] To address the aforementioned technical problems, this invention proposes a domain-adaptive image classification method based on self-training and centroid alignment, thereby resolving the issues present in the prior art.

[0006] To achieve the above objectives, this invention provides a domain-adaptive image classification method based on self-training and centroid alignment, comprising:

[0007] Step S1: Construct an image classification network, which includes a feature extraction module, a first linear classification module, and a second linear classification module;

[0008] Step S2: Perform self-training using source domain image data and target domain image data;

[0009] Step S3: Perform consistency regularization processing on the outputs of the first linear classification module and the second linear classification module of the source domain image data;

[0010] Step S4: Calculate the centroids of the source domain image data output from the first linear classification module and the target domain image data output from the second linear classification module; align the centroids of the source and target domains.

[0011] Step S5: Repeat steps S2-S4 to train the image classification network to obtain a domain-adaptive image classification model.

[0012] Step S6: Obtain the image to be processed and classify the image to be processed based on the domain adaptive image classification model.

[0013] Optionally, the self-training process in step S2 includes:

[0014] The source domain image data and the target domain image data are preprocessed;

[0015] The preprocessed source domain image data is sent to the feature extraction module to obtain the first output feature; the first output feature is then input to the first linear classification module for training.

[0016] The preprocessed target domain image data is sent to the feature extraction module to obtain the second output feature; the second output feature is input to the first linear classification module to obtain several output vectors, and pseudo-labels of the target domain image data are filtered from the labels of the output vectors according to the confidence level.

[0017] The pseudo-label is used as the data label, and the second output feature is input into the second linear classification module for training.

[0018] Optionally, during the self-training process in step S2, the source domain image data and the target domain image data are trained using the cross-entropy loss function; the vector with the highest confidence is selected as the pseudo-label, and the pseudo-label is filtered by the confidence threshold.

[0019] Optionally, the overall loss function of step S1 is as follows:

[0020]

[0021] Among them, y′ t It is the label of the maximum probability of the output vector of the target domain image data in the image classification network, x t It is the target domain image data, x s It is source domain image data, y s is the true label of the source domain image data, and μ is the set confidence threshold.

[0022] Optionally, the consistency regularization process in step S3 includes: inputting the first output feature into the first linear classification module to obtain the first output vector; inputting the first output feature into the second linear classification module to obtain the second output vector; and calculating the output loss of the first linear classification module and the second linear classification module on the source domain image data based on the first output vector and the second output vector using the mean squared error loss.

[0023] Optionally, the calculation process of various centroids in step S4 includes: calculating various centroids of the source domain based on the output of the source domain image data generated by the first linear classification module; and calculating various centroids and data clusters of the target domain based on the output of the target domain image data generated by the clustering method and the second linear classification module.

[0024] Optionally, the centroid alignment in step S4 includes alignment from both distance and direction angles.

[0025] Optionally, the centroid alignment process in step S4 includes: taking the pseudo-label of the target domain image data closest to the class centroid as the centroid label of the target domain class; performing centroid alignment based on distance and angle according to Euclidean distance; calculating the direction of movement of the output vector of each data point of the target domain image data compared to the output vector of the previous data point; and performing centroid alignment based on direction and angle by combining cosine similarity loss.

[0026] Compared with the prior art, the present invention has the following advantages and technical effects:

[0027] This invention first employs a self-training process decoupled from a feature extraction module and two classification modules to learn basic classification knowledge. Secondly, it improves classifier consistency by bringing the outputs of the two classification modules to the source domain closer together. Then, by calculating the centroid of the source domain and the clusters of the target domain, during model training, it reduces the distance between the target domain features and the source domain centroids from both distance and direction perspectives, prompting the target domain features to move towards the source domain centroids. This invention combines feature information and structural information, considering both distance and direction, to guide the model in learning domain-invariant features, effectively solving the problem of performance degradation in image classification networks caused by significant differences in data distribution between the source and target domains. Attached Figure Description

[0028] The accompanying drawings, which form part of this application, are used to provide a further understanding of this application. The illustrative embodiments and descriptions of this application are used to explain this application and do not constitute an undue limitation of this application. In the drawings:

[0029] Figure 1 This is a flowchart of the domain adaptive image classification model training method according to an embodiment of the present invention;

[0030] Figure 2 This is a flowchart illustrating the self-training process of an embodiment of the present invention;

[0031] Figure 3 This is a flowchart illustrating the centroid alignment process according to an embodiment of the present invention.

[0032] Figure 4 This is a diagram illustrating the overall framework of the domain-adaptive image classification model according to an embodiment of the present invention. Detailed Implementation

[0033] It should be noted that, unless otherwise specified, the embodiments and features described in this application can be combined with each other. This application will now be described in detail with reference to the accompanying drawings and embodiments.

[0034] It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions, and although a logical order is shown in the flowchart, in some cases the steps shown or described may be executed in a different order than that shown here.

[0035] Example 1

[0036] The success of deep learning image classification networks typically relies on two assumptions: (i) accurate model fitting requires a large labeled dataset, and (ii) the training and testing data are independent and identically distributed. For tasks where the target domain lacks sufficient labeled data, conventional learning methods are affected by variations in data distribution, potentially leading to a significant degrade in model performance. While data can be labeled for the target domain, labeling large-scale datasets typically requires substantial time and effort.

[0037] like Figure 1-2 As shown, this embodiment provides a domain-adaptive image classification method based on self-training and centroid alignment. It considers multiple perspectives, including data features and structure, distance from the centroid, and direction of movement towards the centroid, to guide the model in learning domain-invariant features, including:

[0038] Step S1: Use source and target domain image data to perform self-training to learn a basic classification network.

[0039] Step S101: Perform simple data preprocessing on the source domain image data (Xs, Ys) and the target domain image data (Xt), and then feed them into the image classification network for training;

[0040] It should be noted that simple image preprocessing is performed on the source and target domain image data, mainly including image cropping, size normalization, and regularization. The image classification network consists of a feature extraction module G and two linear classification modules C1 and C2. ResNet50 is chosen as the feature extraction module, and a combination of linear layers is chosen as the classification module. Specifically, the ResNet50 network structure is as follows: one conv1 convolutional layer, three conv2_x (three convolutional layers), four conv3_x (three convolutional layers), twenty-three conv4_x (three convolutional layers), three conv5_x (three convolutional layers), and one average pooling layer. The first layer is a convolution with a stride of 2, and the last layer is a fully connected layer. The residual blocks are of four different sizes: conv2_x (64 kernels; 64 kernels; 256 kernels), conv3_x (128 kernels; 128 kernels; 512 kernels), conv4_x (256 kernels; 256 kernels; 1024 kernels), and conv5_x (512 kernels; 512 kernels; 2048 kernels). The classification module structure is as follows: one linear layer, where the input dimension is the same as the output dimension of the feature extraction module, and the output dimension is the same as the number of categories in the classification task.

[0041] Step S102: The source domain image data Xs is sent to the feature extraction module G, and the feature output Fs (first output feature) is sent to the first linear classification module C1. The source domain data has the correct label, and the cross-entropy loss is used to learn the classifier G→C1 on the source domain; G is the feature extraction module, and C is the linear classification module, which can be represented as "C1(G(·))".

[0042] Step S103: The target domain image data Xt is fed into the feature extraction module G, and the feature output Ft (second output feature) is fed into the first linear classification module C1. The output vector of C1 is then used. The label with the highest confidence level is used as the pseudo-label for the target domain image data;

[0043] Step S104: Filter out fake labels using a pre-set confidence threshold;

[0044] Step S105: The feature Ft generated by the feature extraction module G from the target domain image data Xt is sent to the second linear classification module C2. The cross-entropy loss is calculated using pseudo-labels, and the classifier G→C2 on the target domain is learned, which can be represented as “C2(G(·))”.

[0045] It should be noted that in step S1, the classification network G→C1, trained from the source domain image data, generates pseudo-labels for the target domain image data. These pseudo-labels are then used to train the classification network G→C2, giving it basic classification capabilities. Specifically, the source domain image data training uses cross-entropy loss L... ce The pseudo-labels are obtained from the labels with the highest confidence max(C1(G(Xt))). The target domain image data uses the pseudo-labels as data labels and employs cross-entropy loss L. ce train.

[0046] Due to the distribution differences between the source and target domains, the pseudo-labels generated by the classification network trained in the source domain have significant noise. This noise interferes with network training, causing the network to learn incorrect knowledge. Furthermore, with each training iteration, the erroneous knowledge introduced by the noise accumulates, leading to a sharp decline in network performance. Therefore, this embodiment sets a confidence threshold μ to filter pseudo-labels. Labels with a confidence level exceeding the threshold are included in the loss calculation, while labels with a confidence level below the threshold are excluded from the loss calculation. This eliminates low-confidence pseudo-labels and reduces erroneous noise affecting the network. The overall loss function for step S1 is as follows:

[0047]

[0048] Among them, y′ t The label x is the label with the maximum probability of the output vector of the target domain image data in the classification network G→C1 (i.e., C1(C(·))). t It is the target domain image data, x s It is source domain image data, y s is the true label of the source domain image data, and μ is the set confidence threshold.

[0049] Step S2: Apply consistency loss regularization to the output vectors of the source domain image data in the two classification modules to ensure output consistency.

[0050] Step S201: The features Fs generated by the feature extraction module G from the source domain image data Xs are sent to the second linear classification module C2.

[0051] Step S202, bring C1 output closer (First output vector) and C2 output The distance between the second output vector and the classifier improves the consistency of the classifier.

[0052] It should be noted that in step S2, this embodiment aims to ensure that the predicted outputs of the two classification models are consistent for the same input under different perturbations, and that the network should have the ability to resist interference from small perturbations. Consistency regularization loss is used to narrow the gap between the outputs of the two classification modules, thereby improving their consistency. Specifically, the mean squared error (MSE) loss is used to calculate the output loss of the C1 and C2 classifier modules for the source domain image data. The loss function expression is as follows:

[0053] L mse =MSE(C1(G(x) s )),C2(G(x s )))

[0054] Step S3: Calculate the centroids of various classes of the source domain output of the first linear classification module and the target domain output of the second linear classification module, and narrow down the feature space of the two domains from the perspectives of structural hierarchy, distance and orientation.

[0055] Step S301: Calculate the centroids of various types of the source domain using the source domain image data output generated by G→C1, and calculate the centroids and data clusters of the target domain using the clustering method based on the target domain image data output generated by G→C2;

[0056] Step S302: Estimate the category to which the data clusters belong after clustering the target domain;

[0057] Step S303: Use Euclidean distance to bring the target domain image data and the source domain centroid of the corresponding category closer together;

[0058] Step S304: Calculate the direction vector of the target domain image data output movement, calculate the direction vector of the target domain image data towards the centroid of the corresponding class in the source domain, and use cosine similarity loss to promote the movement of the target domain image data towards the centroid of the corresponding class in the source domain.

[0059] It should be noted that since the class centroid represents the structural center of the class in the feature space, and domain differences are particularly significant for samples at the classification boundary, while having a smaller impact on samples near the class centroid, assuming that the sample points near the class centroid in the target domain image data are correctly classified, the class centroid can represent the average feature of that class in the target domain. Based on this assumption, the output vectors of the source and target domains during training are stored. The average output vector of each class in the source domain as G→C1 is calculated as the centroid of each class in the source domain. K-means clustering is used to calculate the output of the target domain as G→C2, obtaining the centroids and data clusters of each class in the target domain, and centroid alignment is performed. The centroid expression is as follows:

[0060]

[0061]

[0062] in, Denotes the centroid of class k in the source domain. This represents the output vector of the k-th class of data in the source domain in the first linear classification module. y represents the centroid of class k in the target domain. t This represents the output vector of the target domain in the second linear classification module.

[0063] It should be noted that instead of using the target domain pseudo-labels obtained in step S1 to calculate the average vector of each output class in the target domain as the centroids of each class, K-means clustering is used to obtain the centroids and data clusters of the target domain output from the output structure level. The source domain and target domain centroids are updated after a complete iteration.

[0064] Centroid alignment includes reducing the distance between the target domain image data and the source domain centroid from a distance perspective, and promoting the movement of target domain image data features toward the corresponding source domain centroid from a directional perspective.

[0065] Specifically, Euclidean distance is used as a loss to reduce the distance between the target domain image data and various centroids of the source domain.

[0066]

[0067] in, Denotes the centroid of class k in the source domain. The output vector represents the data i that is clustered into k classes in the target domain.

[0068] For centroid movement, during training, the output vector of the target domain image data is stored. By calculating the direction of movement of the target domain image data points relative to the previous data points, cosine similarity loss is used to promote the movement of the target domain image data towards the centroid of the corresponding source domain class. Cosine similarity is calculated by the cosine of the angle between two vectors, reaching its maximum value when the two vectors are in the same direction. The specific function expression is as follows:

[0069]

[0070] in, This represents the output vector of data i that is clustered into k classes in the target domain. This represents the previous output vector for this data. This represents the centroid of class k in the source domain.

[0071] It should be noted that, since the data clusters obtained by K-means clustering do not have labels on the target domain, it is impossible to know the label of the data cluster to which the target domain image data belongs. Therefore, the pseudo-label of the target domain image data closest to the centroid is used to represent the centroid label of the target domain.

[0072] Step S4: Repeat steps S1-S3 to train the model and obtain a domain-adaptive image classification model.

[0073] It should be noted that this invention utilizes knowledge from the source domain to obtain pseudo-labels for the target domain, and by improving the consistency of the source domain outputs and the centroid shifting directions of the target and source domain classes, it narrows the feature spaces between the domains, ultimately resulting in a cross-domain image classification network model. The final loss function of the model is:

[0074] L = L1 + L mse +L2+L cos

[0075] The above are merely preferred embodiments of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A domain-adaptive image classification method based on self-training and centroid alignment, characterized in that, Includes the following steps: Step S1: Construct an image classification network, which includes a feature extraction module, a first linear classification module, and a second linear classification module; Step S2: Perform self-training using source domain image data and target domain image data; The self-training process in step S2 includes: The source domain image data and the target domain image data are preprocessed; The preprocessed source domain image data is sent to the feature extraction module to obtain the first output feature; the first output feature is then input to the first linear classification module for training. The preprocessed target domain image data is sent to the feature extraction module to obtain the second output feature; the second output feature is input to the first linear classification module to obtain the output vector; and pseudo-labels of the target domain image data are filtered from the labels of the output vector according to the confidence level. The pseudo-label is used as the data label, and the second output feature is input into the second linear classification module for training. Step S3: Perform consistency regularization processing on the outputs of the first linear classification module and the second linear classification module of the source domain image data; The process of consistency regularization in step S3 includes: inputting the first output feature into the first linear classification module to obtain the first output vector; inputting the first output feature into the second linear classification module to obtain the second output vector; and calculating the output loss of the first linear classification module and the second linear classification module on the source domain image data based on the first output vector and the second output vector using the mean squared error loss. Step S4: Calculate the centroids of the source domain image data output from the first linear classification module and the target domain image data output from the second linear classification module; align the centroids of the source and target domains. The centroid alignment in step S4 includes alignment from both distance and direction angles; The centroid alignment process in step S4 includes: taking the pseudo-label of the target domain image data closest to the class centroid as the centroid label of the target domain class; performing centroid alignment based on Euclidean distance and angle; calculating the direction of movement of the output vector of each data point of the target domain image data compared to the output vector of the previous data point; and performing centroid alignment based on direction and angle by combining cosine similarity loss. Step S5: Repeat steps S2-S4 to train the image classification network to obtain a domain-adaptive image classification model. Step S6: Obtain the image to be processed and classify the image to be processed based on the domain adaptive image classification model.

2. The domain adaptive image classification method based on self-training and centroid alignment according to claim 1, characterized in that, In step S2, during the self-training process, the source domain image data and the target domain image data are trained using the cross-entropy loss function; the vector with the highest confidence is selected as the pseudo-label, and the pseudo-label is filtered by the confidence threshold.

3. The domain adaptive image classification method based on self-training and centroid alignment according to claim 2, characterized in that, The overall loss function for step S1 is as follows: Among them, It is the label of the maximum probability of the output vector of the target domain image data in the image classification network. It is the target domain image data. It is source domain image data. is the true label of the source domain image data, and μ is the set confidence threshold.

4. The domain adaptive image classification method based on self-training and centroid alignment according to claim 1, characterized in that, The calculation process of various centroids in step S4 includes: calculating various centroids of the source domain based on the output of the source domain image data generated by the first linear classification module; and calculating various centroids and data clusters of the target domain based on the output of the target domain image data generated by the clustering method and the second linear classification module.

Images