Picture black box field adaptive classification method, electronic device and medium

Abstract

This invention discloses an image black-box domain adaptive classification method, electronic device, and medium, comprising: inputting target domain image data to be classified into a pre-trained target domain classifier to predict the output label result; wherein, the training process of the target domain classifier includes: acquiring a first image and a second image; calculating the labeled image classification loss based on the output of the first image and the second image in the target domain classifier and their corresponding labels; constructing a feature similarity matrix based on the image and its corresponding enhanced copy, and constructing a label relevance matrix based on the latent labels of the image, thereby calculating the contrastive loss; calculating the guessed label of the unlabeled image, calculating the unlabeled image classification loss based on the target domain predicted label of the unlabeled image and the guessed label; and using the sum of the labeled image classification loss, the contrastive loss, and the unlabeled image classification loss as a loss function, and training the target domain classifier based on this loss function.

Description

Technical Field

[0001] This invention relates to the field of machine learning, and more particularly to an adaptive classification method for image black-box domains, electronic devices, and media. Background Technology

[0002] Unsupervised domain adaptation (UDA) is a popular research area in deep learning. It enables classifiers trained on a source domain to generalize well to target domains with different data distributions without accessing the label information of the target domain images. While UDA has been widely adopted, it requires access to source domain data during training, which may pose a threat to data privacy and security. Furthermore, when the source domain contains a large amount of data, using UDA methods can lead to significant storage and computational costs. Therefore, in recent years, many researchers have begun to design source-free data-free (SFDA) methods, which do not require access to source domain data during the training of the target domain classifier. However, this method is highly dependent on white-box source domain classifiers, so privacy concerns and computational resource limitations remain. To address this issue, researchers aim to train unlabeled target domains using only the source domain's API, thus completely alleviating the privacy and computational resource problems. This is known as black-box domain adaptation (BDA).

[0003] Currently, only a limited number of studies have investigated black-box domain adaptive classification methods. The key to black-box domain adaptive classification lies in the alignment between the source and target domains, consistent with ordinary unsupervised domain adaptation. Existing black-box domain adaptive classification methods identify a fixed proportion of easily adapted images in the target domain and allow these images to dominate the training of the unadapted images in the target domain. However, these methods ignore the dynamic process of domain adaptation, where some unadapted images gradually align, forming highly adapted images that can then be used to further dominate the training of the remaining unadapted images and improve classification accuracy. Furthermore, these methods only perform domain alignment from the perspective of labels, neglecting the mining of the rich feature structures of the target domain. Summary of the Invention

[0004] To address the shortcomings of existing technologies, this invention provides an adaptive classification method, electronic device, and medium for image black-box domains.

[0005] In a first aspect, embodiments of the present invention provide an image black-box domain adaptive classification method, the method comprising:

[0006] Input the target domain image data to be classified into a pre-trained target domain classifier to predict the output label result;

[0007] The training process for the target domain classifier includes:

[0008] Calculate the source domain loss for each image in the target domain, set a source domain loss threshold, and use images with a loss less than the source domain loss threshold as the first image; set a target domain predicted label confidence threshold, and use images with a target domain predicted label confidence higher than the target domain predicted label confidence threshold as the second image;

[0009] Set the label of the first image as the source domain predicted label, and set the label of the second image as the target domain predicted label; calculate the labeled image classification loss based on the output of the first and second images in the target domain classifier and their corresponding labels;

[0010] Data augmentation is performed on each image in the target domain to obtain a corresponding augmented copy; a feature similarity matrix is ​​constructed based on the images and their corresponding augmented copies; a label relevance matrix is ​​constructed based on the latent labels of the images; and a contrastive loss is calculated based on the feature similarity matrix and the label relevance matrix.

[0011] Obtain the k nearest neighbor images of the unlabeled image in the feature space in the target domain, take the average label of the k nearest neighbor images output by the target domain classifier as the guessed label of the unlabeled image, and calculate the classification loss of the unlabeled image based on the target domain predicted label and the guessed label of the unlabeled image.

[0012] The sum of the labeled image classification loss, contrastive loss, and unlabeled image classification loss is used as the loss function, and the target domain classifier is trained based on this loss function.

[0013] Secondly, embodiments of the present invention provide an electronic device, including a memory and a processor, wherein the memory is coupled to the processor; wherein the memory is used to store program data, and the processor is used to execute the program data to implement the above-described image black-box domain adaptive classification method.

[0014] Thirdly, embodiments of the present invention provide a computer-readable storage medium having a program stored thereon, which, when executed by a processor, is used to implement the above-described image black-box domain adaptive classification method.

[0015] Compared with the prior art, the beneficial effects of the present invention are:

[0016] This invention discloses a black-box domain adaptive classification method for images. It fully explores images with good adaptability in the source domain and makes full use of the topological structure of the image feature space. It uses a semi-supervised learning approach to select the first image (i.e., an easily adaptable image) and the second image (i.e., a highly adaptable image) as labeled images to guide domain adaptation. It uses contrastive learning to achieve domain alignment of unadapted images in the feature space. Attached Figure Description

[0017] To more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings used in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0018] Figure 1 This is a flowchart illustrating an image black-box domain adaptive classification method provided in an embodiment of the present invention.

[0019] Figure 2 Example diagram of black-box domain adaptive classification provided in embodiments of the present invention;

[0020] Figure 3 This is a schematic diagram of an electronic device provided in an embodiment of the present invention. Detailed Implementation

[0021] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0022] It should be noted that, unless otherwise specified, the features in the following embodiments and implementation methods can be combined with each other.

[0023] like Figure 1 As shown, this embodiment of the invention provides an image black-box domain adaptive classification method, the method comprising the following steps:

[0024] Input the target domain image data to be classified into a pre-trained target domain classifier to predict the output label result;

[0025] Input the target domain image data x to be tested into the trained target domain classifier h. T The prediction is performed within parentheses, and the label result is output to achieve black-box adaptive classification. The expression is as follows:

[0026]

[0027] In the formula, h T (·) is the j-th element of the vector, where L represents the probability distribution of the image being predicted as each category. This indicates the classification result of the target domain image to be tested.

[0028] The training process for the target domain classifier includes:

[0029] Step S1: Calculate the source domain loss for each image in the target domain, set a source domain loss threshold, and use images with a loss less than the source domain loss threshold as the first image; set a target domain prediction label confidence threshold, and use images with a target domain prediction label confidence higher than the target domain prediction label confidence threshold as the second image.

[0030] For an image x in the target domain i Its source domain predicted label is z i =h S (x i The target domain predicted label is p. i =h T (x i ), where h S () represents the source domain classifier of the black box, h T () represents the target domain classifier;

[0031] Let L be the total number of categories, then the source domain predicts the label z. i And target domain predicted label p i These are all L-dimensional normalized vectors, representing the probability distribution of an image being predicted as each category;

[0032] Subsequently, the source domain loss of an image is defined as the cross-entropy loss between the output of the target domain classifier and its predicted source domain label, as expressed below:

[0033]

[0034] in, Indicates the source domain predicted label z i One-hot form of a vector, p ij Indicates the target domain predicted label p i The j-th element of the vector Indicates the source domain predicted label z i The j-th element in the vector is then used as the first image, selecting the top δ% images with the minimum loss from the target domain images. The first image is an easily adaptable set, defined as follows:

[0035]

[0036] in, This represents the set of source domain losses for the δ% target domain images that have the minimum source domain loss;

[0037] Subsequently, images with target domain predicted label confidence scores higher than a threshold τ are selected as second images. These second images are highly adapted images, and their set is defined as follows:

[0038]

[0039] Step S2: Set the label of the first image as the source domain predicted label and the label of the second image as the target domain predicted label; calculate the labeled image classification loss based on the output of the first and second images in the target domain classifier and their corresponding labels.

[0040] Let the label of the first image be the source domain predicted label, the label of the second image be the target domain predicted label, and the remaining images be unlabeled images. Then, the set of labeled images can be represented as the union of the first image set and the second image set:

[0041]

[0042] Each image x i Corresponding tags It can be represented as:

[0043]

[0044] in Indicates the target domain predicted label p i One-hot form of a vector

[0045] make If the set represents all images in the target domain, then the set of unlabeled images can be represented as the set of the remaining samples:

[0046]

[0047] Subsequently, the classification loss for labeled images is defined as the cross-entropy loss between the output of the classifier in the target domain and its corresponding label, as expressed below:

[0048]

[0049] In the formula, Let L represent the loss for labeled image classification, where L represents the probability distribution of an image being predicted as each category. express The j-th element of the vector This represents the label vector of a labeled image. For the first image, we have... Regarding the second image: in p i One-hot form of a vector, p ij Indicates the target domain predicted label p i The j-th element of the vector.

[0050] Step S3: Perform data augmentation on each image in the target domain to obtain a corresponding augmented copy; construct a feature similarity matrix based on the image and its corresponding augmented copy; construct a label relevance matrix based on the latent labels of the image; calculate the contrast loss based on the feature similarity matrix and the label relevance matrix.

[0051] For each image x in the target domain i The RandAugment data augmentation method is applied to generate an augmented copy.

[0052] Subsequently, projection models g() and g′() are introduced, where g′() is a copy of g() updated in a momentum manner. The projection model g() is related to the target domain classifier h. T () Share the same backbone network; send image x i and Enhanced Dungeon By inputting the projection model g() and the replica projection model g′() respectively, we obtain the feature q. i and enhanced copy feature q i ′, where q i and q i ′ are all vectors of dimension d;

[0053] Given a batch of images of size B in the training set of the target domain, construct a B×B dimension feature similarity matrix. as follows:

[0054]

[0055] Where t is the temperature scaling parameter, q′ j Image x j Enhanced copy The replica features obtained from the input replica projection model;

[0056] Subsequently, a B×B dimension label relevance matrix was constructed. as follows:

[0057]

[0058] in, and They represent the image x i and x j The latent label is λ, which represents a scalar between 0 and 1; for a labeled sample, its latent label is the corresponding label. For unlabeled samples, their latent labels are the labels predicted by the target domain classifier, expressed as follows:

[0059]

[0060] Subsequently, row normalization was performed on the feature similarity matrix and the label relevance matrix respectively to obtain... and The expression is as follows:

[0061]

[0062] The contrastive loss is defined as the cross-entropy loss between the normalized feature similarity matrix and the label relevance matrix, to constrain the features of each image to be similar to the features of its enhanced copy, and also similar to the features of images with the same latent label. The expression is as follows:

[0063]

[0064] In the formula, This indicates a comparative loss.

[0065] Step S4: Obtain the k nearest neighbor images of the unlabeled image in the feature space in the target domain. Use the average label output by the k nearest neighbor images on the target domain classifier as the guessed label of the unlabeled image. Calculate the classification loss of the unlabeled image based on the target domain predicted label and the guessed label.

[0066] For a feature q of an unlabeled image i Find the top k most similar enhanced copy features in its batch, and let Representing the image indices corresponding to these k enhanced replica features, calculate the guessed label s for the unlabeled image. i :

[0067]

[0068] In the formula, p j Image x j Predict labels for the target domain;

[0069] Subsequently, the unlabeled image classification loss is defined as the cross-entropy loss between the output of the unlabeled image on the target domain classifier and its guessed label, as expressed below:

[0070]

[0071] In the formula, s represents the unlabeled image classification loss. ij Indicates guessing the tag s i The j-th element of the vector.

[0072] Step S5: The sum of the classification loss for labeled images, the contrastive loss, and the classification loss for unlabeled images is used as the loss function, and the target domain classifier is trained based on this loss function.

[0073] The final loss is obtained by summing the classification losses for labeled images, contrastive images, and unlabeled images.

[0074]

[0075] The parameters of the target domain classifier are obtained using the stochastic gradient descent method.

[0076] Example 1

[0077] In this example, the label sets for both the source and target domain datasets are set to {airplane, bridge, seagull, aircraft carrier, fishing boat, castle, whale}. Figure 2 (A) in the dataset is a sample from the source domain (clip art). Figure 2 In the dataset, (B) is a sample from the target domain (real) dataset. This sample is used as the test sample, and its pixel values ​​are used as the input image x, labeled "aircraft carrier," meaning the image's label is y=4. This image is then fed into a pre-trained target domain classifier to predict the output label.

[0078] Assuming the target domain classifier output h is obtained T (x) = [0.001, 0.002, 0.042, 0.933, 0.020, 0.001, 0.001], and then obtain the category corresponding to the maximum output value of the target domain classifier. Given that the true label of the image is y=4, and that the predicted label matches the true label, the prediction is accurate.

[0079] Accordingly, this application also provides an electronic device, including: one or more processors; a memory for storing one or more programs; when the one or more programs are executed by the one or more processors, the one or more processors implement the image black-box domain adaptive classification method as described above. Figure 3 The diagram shown illustrates a hardware structure of any device with data processing capabilities for implementing the image black-box domain adaptive classification method provided in this embodiment of the invention, except... Figure 3 In addition to the processor, memory, and network interface shown, any data processing device in the embodiment may also include other hardware depending on the actual function of the data processing device, which will not be described in detail here.

[0080] Accordingly, this application also provides a computer-readable storage medium storing computer instructions thereon, which, when executed by a processor, implement the image black-box domain adaptive classification method described above. The computer-readable storage medium can be an internal storage unit of any data-processing device as described in any of the foregoing embodiments, such as a hard disk or memory. The computer-readable storage medium can also be an external storage device, such as a plug-in hard disk, smart media card (SMC), SD card, flash card, etc., equipped on the device. Furthermore, the computer-readable storage medium can include both internal storage units of any data-processing device and external storage devices. The computer-readable storage medium is used to store the computer program and other programs and data required by the data-processing device, and can also be used to temporarily store data that has been output or will be output.

[0081] Other embodiments of this application will readily occur to those skilled in the art upon consideration of the specification and practice of the disclosure herein. This application is intended to cover any variations, uses, or adaptations of this application that follow the general principles of this application and include common knowledge or customary techniques in the art not disclosed herein. The specification and embodiments are to be considered exemplary only.

[0082] It should be understood that this application is not limited to the precise structure described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope.

Claims

1. A black-box domain adaptive classification method for images, characterized in that, The method includes: Input the target domain image data to be classified into a pre-trained target domain classifier to predict the output label result; The training process for the target domain classifier includes: Calculate the source domain loss for each image in the target domain, set a source domain loss threshold, and use images with a loss less than the source domain loss threshold as the first image; set a target domain predicted label confidence threshold, and use images with a target domain predicted label confidence higher than the target domain predicted label confidence threshold as the second image; Set the label of the first image as the source domain predicted label, and set the label of the second image as the target domain predicted label; calculate the labeled image classification loss based on the output of the first and second images in the target domain classifier and their corresponding labels; Data augmentation is performed on each image in the target domain to obtain a corresponding augmented copy; a feature similarity matrix is ​​constructed based on the images and their corresponding augmented copies; a label relevance matrix is ​​constructed based on the latent labels of the images; and a contrastive loss is calculated based on the feature similarity matrix and the label relevance matrix. Obtain the k nearest neighbor images of the unlabeled image in the feature space in the target domain, take the average label of the k nearest neighbor images output by the target domain classifier as the guessed label of the unlabeled image, and calculate the classification loss of the unlabeled image based on the target domain predicted label and the guessed label of the unlabeled image. The sum of the labeled image classification loss, contrastive loss, and unlabeled image classification loss is used as the loss function, and the target domain classifier is trained based on this loss function.

2. The image black-box domain adaptive classification method according to claim 1, characterized in that, Calculate the source domain loss for each image in the target domain, set a source domain loss threshold, and use images with a loss less than the source domain loss threshold as the first image; Set a target domain predicted label confidence threshold, and include images whose target domain predicted label confidence is higher than the target domain predicted label confidence threshold as the second image: For an image x in the target domain i Its source domain predicted label is z i =h S (x i The target domain predicted label is p. i =h T (x i ), where h S () represents the source domain classifier of the black box, h T () represents the target domain classifier; Let L be the total number of categories, then the source domain predicts the label z. i And target domain predicted label p i These are all L-dimensional normalized vectors, representing the probability distribution of an image being predicted as each category; The source domain loss for each image in the target domain is calculated using the following expression: in, Indicates the source domain predicted label z i One-hot form of a vector; Indicates the source domain predicted label z i The j-th element in the vector; p ij Indicates the target domain predicted label p i The j-th element of the vector; The top δ% images with the minimum source domain loss in the target domain are selected as the first images, and their set is denoted as . Set the target domain predicted label confidence threshold τ; Images whose target domain predicted label confidence is higher than the target domain predicted label confidence threshold τ are designated as the second images, and their set is denoted as […].

3. The image black-box domain adaptive classification method according to claim 1, characterized in that, Set the label of the first image as the source domain predicted label, and set the label of the second image as the target domain predicted label; Based on the output of the target domain classifier for the first and second images and their corresponding labels, the labeled image classification loss is calculated as follows: The labeled image is the union of the first image and the second image; Set the label of the first image as the source domain predicted label, and set the label of the second image as the target domain predicted label; Based on the output of the target domain classifier for the first and second images and their corresponding labels, the labeled image classification loss is calculated as follows: In the formula, Let L represent the loss for labeled image classification, where L represents the probability distribution of an image being predicted as each category. express The j-th element of the vector This represents the label vector of a labeled image. For the first image, we have... Regarding the second image: in p i One-hot form of a vector, p ij Indicates the target domain predicted label p i The j-th element of the vector.

4. The image black-box domain adaptive classification method according to claim 1, characterized in that, Data augmentation is performed on each image in the target domain to obtain a corresponding augmented copy; Construct a feature similarity matrix based on the image and its corresponding enhanced copy; Construct a label relevance matrix based on the latent labels of images; The contrastive loss is calculated based on the feature similarity matrix and the label relevance matrix, including: For each image x in the target domain i Perform data augmentation to obtain the corresponding augmented copy. Construct the projection model g() and the copy projection model g′(); Image x i and Enhanced Dungeon By inputting the projection model g() and the replica projection model g′() respectively, we obtain the feature q. i and enhanced copy feature q i ′; Construct a feature similarity matrix based on the image and its corresponding enhanced copy. The expression is as follows: Where t is the temperature scaling parameter, q′ j Image x j Enhanced copy The replica features obtained from the input replica projection model; Construct a label relevance matrix based on the latent labels of images. The expression is as follows: in, and They represent the image x i and x j The potential label, λ, represents a scalar between 0 and 1; For labeled samples Its potential tags are the corresponding tags. For unlabeled samples Its latent label is the label predicted by the target domain classifier. The expression is as follows: Row normalization is performed on the feature similarity matrix and the label relevance matrix respectively to obtain and Based on the normalized feature similarity matrix Normalized label relevance matrix Calculate contrast loss The expression is as follows: In the formula, B represents the size of the current training batch.

5. The image black-box domain adaptive classification method according to claim 4, characterized in that, Data augmentation for each image in the target domain includes: The RandAugment data augmentation method is used to augment the data of each image in the target domain.

6. The image black-box domain adaptive classification method according to claim 2 or 4, characterized in that, Projection model g() and target domain classifier h T () share the same backbone network.

7. The image black-box domain adaptive classification method according to claim 1, characterized in that, Obtain the k nearest neighbor images of the unlabeled image in the feature space within the target domain. Use the average label output by the k nearest neighbor images on the target domain classifier as the guessed label for the unlabeled image. Calculate the classification loss for the unlabeled image based on the target domain predicted label and the guessed label, including: For a feature q of an unlabeled image in the target domain i Find the top k most similar enhanced copy features in its batch, and let Representing the image indices corresponding to these k enhanced replica features, calculate the guessed label s for the unlabeled image. i The expression is as follows: In the formula, B represents the size of the current training batch, p j Image x j Predict labels for the target domain; Predicting the label p of the target domain based on unlabeled images ij And guessing the tag s ij The loss for classifying unlabeled images is calculated using the following expression: In the formula, L represents the probability distribution of an image being predicted as each category, and s ij Indicates guessing the tag s i The j-th element of the vector.

8. The image black-box domain adaptive classification method according to claim 1, characterized in that, The target domain image data to be classified is input into a pre-trained target domain classifier, and the predicted output label results include: The target domain image data x to be tested is input into the pre-trained target domain classifier h. T Predict the label results in parentheses and output the label results. The expression is as follows: In the formula, h T The (·) vector is the j-th element, and L represents the probability distribution of the image being predicted as each category.

9. An electronic device comprising a memory and a processor, characterized in that, The memory is coupled to the processor; wherein the memory is used to store program data, and the processor is used to execute the program data to implement the image black-box domain adaptive classification method according to any one of claims 1-8.

10. A computer-readable storage medium having a program stored thereon, characterized in that, When executed by the processor, the program is used to implement the image black-box neighborhood adaptive classification method as described in any one of claims 1-8.

Images