Domain adaptive image classification method, device, storage medium and program product

Abstract

The present invention provides a domain adaptive image classification method, device, storage medium and program product. The domain adaptive image classification includes: determining a source domain image and a target domain image; the target domain image includes image pollution; the image pollution includes at least one of blur pollution, noise pollution, weather pollution and image quality change pollution; obtaining an image classification result, inputting the source domain image and the target domain image into an image classification model, and obtaining an image classification result output by the image classification model; wherein the image classification model is constructed by constructing a target pollution sample through domain difference information between a source domain sample image and a target domain sample image, and calculating the contrast loss of a teacher-student model based on the target pollution sample for training; wherein the source domain sample image and the target domain sample image do not include image pollution.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and in particular to a domain-adaptive image classification method, device, storage medium and program product. Background Art

[0002] Domain adaptation is a very promising technology that transfers knowledge from a labeled source domain to an unlabeled target domain classification learning, aiming to solve the negative transfer problem caused by domain transfer. Domain adaptation is defined as follows: given labeled source domain data and unlabeled target domain data, the two domains have different distributions, but the real label space is the same (e.g. the source domain is cats and dogs in the real world, and the target domain is cats and dogs in the comics domain), the task of domain adaptation is to classify the target domain with the help of the given data.

[0003] In addition to the domain difference between the source domain and the target domain, images in real scenes are often full of image pollution. For example, image pollution can be blur pollution, noise pollution, weather pollution, image quality change pollution, etc. However, most domain adaptation methods only consider the accuracy under ideal clean data sets, and the existing domain adaptation methods are not robust enough for domain adaptive image classification of images with image pollution. Summary of the invention

[0004] The present invention provides a domain adaptive image classification method, device, storage medium and program product, which are used to solve the defect that the existing domain adaptive method has poor robustness in performing domain adaptive image classification on images including image pollution.

[0005] The present invention provides a domain adaptive image classification method, comprising:

[0006] Determine a source domain image and a target domain image; the target domain image includes image pollution; the image pollution includes at least one of blur pollution, noise pollution, weather pollution, and image quality change pollution;

[0007] Obtaining an image classification result, inputting the source domain image and the target domain image into an image classification model, and obtaining an image classification result output by the image classification model; wherein the image classification model is constructed by constructing a target contaminated sample through domain difference information between a source domain sample image and a target domain sample image, and calculating the contrast loss of a teacher-student model based on the target contaminated sample for training;

[0008] The source domain sample image and the target domain sample image do not include image pollution.

[0009] According to a domain adaptive image classification method provided by the present invention, before determining the source domain image and the target domain image, it also includes the step of training the image classification model;

[0010] The training of the image classification model comprises:

[0011] Determine a source domain sample image and a target domain sample image; the source domain sample image and the target domain sample image do not include image pollution;

[0012] Construct a target contaminated sample based on the domain difference information between the source domain sample image and the target domain sample image;

[0013] Construct a teacher-student model; wherein the teacher model and the student model have the same original domain adaptation model structure;

[0014] The contrast loss of the teacher-student model is calculated based on the target contaminated sample, the student model is trained according to the contrast loss, and the student model is used as the image classification model.

[0015] According to a domain adaptive image classification method provided by the present invention, the target contaminated sample is constructed based on domain difference information between a source domain sample image and a target domain sample image, comprising:

[0016] Obtaining the migration loss of the source domain sample image and the target domain sample image after they pass through the feature extractor of the student model respectively;

[0017] A projected gradient descent process is performed based on the migration loss to obtain a target contaminated sample.

[0018] According to a domain adaptive image classification method provided by the present invention, the projected gradient descent processing based on the migration loss is performed to obtain a target contaminated sample, comprising:

[0019] Repeat the following formula until the set number of times threshold is reached to obtain the target contaminated sample:

[0020]

[0021]

[0022]

[0023] in, represents the target contaminated sample; x s represents the source domain sample image, x t represents the target domain sample image, f represents the feature extractor of the student model; δ represents the preset adjustment range parameter, and η represents the update step size.

[0024] According to a domain adaptive image classification method provided by the present invention, the migration loss is obtained by a maximum mean difference method or an adversarial attack method.

[0025] According to a domain adaptive image classification method provided by the present invention, the contrast loss of the teacher-student model is calculated based on the target contaminated sample, the student model is trained according to the contrast loss, and the student model is used as the image classification model, including:

[0026] Get the original domain adaptation model loss;

[0027] Inputting the target contaminated sample into the input end of the feature extractor of the student model; calculating the contrast loss of the student model based on the target contaminated sample;

[0028] The feature extractor parameters and the classifier parameters in the student model are updated based on the original domain adaptation model loss and the contrast loss; and the feature extractor parameters and the classifier parameters of the updated student model are used as the feature extractor parameters and the classifier parameters of the image classification model.

[0029] According to a domain adaptive image classification method provided by the present invention, the contrast loss is expressed by the following formula:

[0030]

[0031] Among them, l con is the contrast loss, and the similarity function l sim is the similarity function, Z (stu) and Z (tea) are the features extracted by the student model and the teacher model respectively. is the characteristic of the i-th sample in the student model, is the feature of the i-th sample in the teacher model, and N is the number of samples in a batch.

[0032] The present invention also provides a domain-adaptive image classification device, comprising:

[0033] A first determination module is used to determine a source domain image and a target domain image; the target domain image includes image pollution; the image pollution includes at least one of blur pollution, noise pollution, weather pollution, and image quality change pollution;

[0034] A result acquisition module is used to obtain image classification results, input the source domain image and the target domain image into an image classification model, and obtain the image classification results output by the image classification model; wherein the image classification model is constructed by constructing a target contaminated sample through domain difference information between the source domain sample image and the target domain sample image, and is trained by calculating the contrast loss of the teacher-student model based on the target contaminated sample; wherein the source domain sample image and the target domain sample image do not include image contamination.

[0035] The present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the steps of any of the above-mentioned domain adaptive image classification methods are implemented.

[0036] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of any of the above-described domain adaptive image classification methods.

[0037] The present invention also provides a computer program product, comprising a computer program, wherein when the computer program is executed by a processor, the steps of any of the above-mentioned domain adaptive image classification methods are implemented.

[0038] The domain adaptive image classification method, device, storage medium and program product provided by the present invention construct a target contaminated sample through the domain difference information between the source domain sample image and the target domain sample image; thereby simulating unknown image contamination by constructing the target contaminated sample; then calculating the contrast loss of the teacher-student model based on the target contaminated sample, and then training the image classification model according to the contrast loss. The present invention calculates the contrast loss of the teacher-student model based on the target contaminated sample and trains the image classification model according to the contrast loss, thereby shortening the feature distance between the target contaminated sample and the original sample of the target domain sample image, thereby improving the robustness of domain adaptive image classification of images including image contamination. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0040] Figure 1 This is one of the flow charts of the domain adaptive image classification method provided by the present invention;

[0041] Figure 2This is a flowchart of the domain adaptive image classification method provided by the present invention;

[0042] Figure 3 This is the third flow chart of the domain adaptive image classification method provided by the present invention;

[0043] Figure 4 It is a structural schematic diagram of the image classification model of the present invention;

[0044] Figure 5 It is a schematic diagram comparing the image classification accuracy of the classic domain adaptation models CDAN+TN, DCAN, the currently best pollution robustness method AugMix, and the method TSCL-WC of the present invention on the target domain images including polluted images;

[0045] Figure 6 It is a schematic diagram of the robustness comparison of the classic domain adaptation models CDAN+TN, DCAN, the best pollution robustness method AugMix at present, and the method TSCL-WC of the present invention on the target domain image including the pollution image;

[0046] Figure 7 It is a structural schematic diagram of the domain adaptive image classification device provided by the present invention;

[0047] Figure 8 It is a structural schematic diagram of the electronic device provided by the present invention. DETAILED DESCRIPTION

[0048] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the drawings of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0049] Domain adaptation is a very promising technology that transfers knowledge from a labeled source domain to unlabeled target domain classification learning, aiming to solve the negative transfer problem caused by domain transfer. Neighborhood adaptation methods are increasingly required to have good robustness to contamination in the target domain. However, most domain adaptation methods only consider the accuracy under an ideal clean dataset, and existing domain adaptation methods are not robust enough for domain adaptive image classification of images that include image contamination.

[0050] To solve the problem that the current domain adaptive methods have poor robustness for domain adaptive image classification of images including image pollution, the difficulty lies in that the current domain adaptive methods lack consideration of improving robustness to image pollution, and the current methods for improving robustness to image pollution cannot be directly applied to domain adaptive methods. Specifically, previous domain adaptive methods did not consider the problem of robustness of the target domain to image pollution, and most of the current methods for improving robustness to image pollution are only applicable to supervised learning scenarios, that is, the training data are labeled, so these methods cannot be directly used in the unlabeled target domain in domain adaptation. In view of this, the present invention proposes a domain adaptive image classification method to solve the defect that the existing domain adaptive methods have poor robustness for domain adaptive image classification of images including image pollution.

[0051] Combine the following Figure 1-Figure 6 The domain-adaptive image classification method of the present invention is described.

[0052] Please refer to Figure 1 , the domain-adaptive image classification method of the present invention comprises:

[0053] Step 200: determine a source domain image and a target domain image; the target domain image includes image pollution; the image pollution includes at least one of blur pollution, noise pollution, weather pollution, and image quality change pollution.

[0054] The source domain image and the target domain image are obtained by an electronic device, wherein the source domain image and the target domain image are images belonging to different domains, for example, the source domain is images of cats and dogs in the real world, and the target domain is images of cats and dogs in the comics domain.

[0055] The present invention is to test the robustness of image classification of images including image pollution. The target domain image adopts an image including image pollution. The image pollution includes at least one of blur pollution, noise pollution, weather pollution and image quality change pollution.

[0056] Specifically, the noise pollution includes at least one of Gaussian noise pollution, shot noise pollution, and pulse noise pollution; the blur pollution includes at least one of defocus blur, motion blur, zoom blur, and glass blur; the weather pollution includes at least one of fog pollution, frost pollution, and snow pollution; the image quality change pollution includes at least one of elastic transformation pollution, contrast change pollution, brightness change pollution, JPEG compression pollution, and pixelation pollution.

[0057] Step 300, obtaining an image classification result, inputting the source domain image and the target domain image into an image classification model, and obtaining an image classification result output by the image classification model; wherein the image classification model is obtained by constructing a target contaminated sample through domain difference information between a source domain sample image and a target domain sample image, and calculating the contrast loss of a teacher-student model based on the target contaminated sample for training; wherein the source domain sample image and the target domain sample image do not include image contamination.

[0058] The image classification result is obtained by an electronic device. The source domain sample image is a clean source domain sample image with a label, and the target domain sample image is a clean target domain sample image without a label. The "clean" means that the image does not contain image pollution.

[0059] The embodiment of the present invention constructs a target contamination sample through domain difference information between a source domain sample image and a target domain sample image, thereby simulating unknown image contamination. The domain difference information is used to generalize the image classification model to unknown contamination during training.

[0060] The image classification model is trained by calculating the contrast loss of the teacher-student model based on the target contaminated samples. Thus, the present invention draws on the idea of ​​knowledge distillation and proposes a teacher-student architecture to solve the problems of domain invariance and improving robustness to image contamination in steps.

[0061] Specifically, the original domain adaptation model is first trained as the teacher model, and the domain invariance of the original domain adaptation model is learned. Then, a student model with the same structure as the teacher model is constructed. It is assumed that the test contamination is known during training, and the target contaminated samples are added to the feature extractor of the student model as data augmentation. A contrast loss constraint is added during the training process, that is, the features of the target contaminated samples through the student model must be similar to the features of the original samples of the target domain sample image through the teacher model, thereby learning features that improve the robustness to image contamination.

[0062] The target contaminated sample is constructed through the domain difference information between the source domain sample image and the target domain sample image; thereby, unknown image contamination is simulated by constructing the target contaminated sample; then, the contrast loss of the teacher-student model is calculated based on the target contaminated sample, and then the image classification model is trained according to the contrast loss. The present invention calculates the contrast loss of the teacher-student model based on the target contaminated sample and trains the image classification model according to the contrast loss, thereby shortening the feature distance between the target contaminated sample and the original sample of the target domain sample image, thereby improving the robustness of domain adaptive image classification of images including image contamination.

[0063] Please refer to Figure 2In some feasible implementations, according to a domain adaptive image classification method provided by the present invention, before step 200, determining the source domain image and the target domain image, further includes:

[0064] Step 100: training an image classification model.

[0065] Please refer to Figure 3 , step 100, training the image classification model comprises:

[0066] Step 110: determining a source domain sample image and a target domain sample image; the source domain sample image and the target domain sample image do not include image contamination;

[0067] The source domain sample images and the target domain sample images are obtained through an electronic device. The source domain sample images are labeled source domain images, and the target domain sample images are unlabeled target domain images. Both the source domain sample images and the target domain sample images are clean images in a general dataset. For example, the source domain sample images are clean images of cats and dogs in the real world, and the target domain sample images are clean images of cats and dogs in the comics domain.

[0068] Step 120: construct a target contaminated sample based on domain difference information between the source domain sample image and the target domain sample image;

[0069] Since both the source domain sample image and the target domain sample image are “clean” images without image contamination, the electronic device can construct a target contamination sample through the domain difference information between the source domain sample image and the target domain sample image to simulate unknown image contamination.

[0070] In some embodiments, step 120, constructing a target contaminated sample based on domain difference information between a source domain sample image and a target domain sample image, includes:

[0071] Step 121, obtaining the migration loss of the source domain sample image and the target domain sample image after passing through the feature extractor of the student model respectively;

[0072] Among them, since the student model and the teacher model in the embodiment of the present invention are both constructed based on the original domain adaptation model, the transfer loss can be defined by the original domain adaptation model. The transfer loss can be determined based on the maximum mean difference (MMD) method or the adversarial attack method.

[0073] Step 122: Perform a projected gradient descent process based on the migration loss to obtain a target contaminated sample.

[0074] Repeat the following formula until the set number of times threshold is reached to obtain the target contaminated sample:

[0075]

[0076]

[0077]

[0078] in, represents the target contaminated sample; x s represents the source domain sample image, x t represents the target domain sample image, f represents the feature extractor of the student model; δ represents the preset adjustment range parameter, and η represents the update step size.

[0079] Specifically, in a typical embodiment, the target contaminated sample construction algorithm of the embodiment of the present invention inputs a source domain sample image x s , target domain sample image x t , feature extractor f, preset adjustment range parameter δ, update step length η and update step number n; output target contaminated sample First, perform initialization: set i=0,

[0080] Then, repeat the following formula (1) to formula (4) until i=n:

[0081]

[0082]

[0083]

[0084] i=i+1; Formula (4)

[0085] return

[0086] The update step number n can be set according to the actual situation. The purpose of formula (3) is to convert the target contaminated sample Scaled to the target domain sample image x t As the center of the circle, the preset adjustment range parameter δ is the radius of the circle. This shortens the characteristic distance between the target contaminated sample and the original sample in the target domain sample image. hour, when hour,

[0087] Step 130: construct a teacher-student model; wherein the teacher model and the student model have the same original domain adaptation model structure;

[0088] The teacher-student model is constructed by electronic devices. Both the teacher model and the student model are constructed based on the original domain adaptation model. The original domain adaptation model has a model structure of f+c (i.e., feature extractor + classifier). Then the structure of the teacher model is f (tea) +c (tea) ; The structure of the student model is f (stu) +c (stu) .

[0089] Step 140: Calculate the contrast loss of the teacher-student model based on the target contaminated sample, train the student model according to the contrast loss, and use the student model as the image classification model.

[0090] The electronic device calculates the contrast loss of the teacher-student model based on the target contaminated sample, trains the student model according to the contrast loss, and uses the student model as the image classification model. Specifically, step 140, calculating the contrast loss of the teacher-student model based on the target contaminated sample, training the student model according to the contrast loss, and using the student model as the image classification model, includes:

[0091] Step 141, obtaining the original domain adaptation model loss;

[0092] The original domain adaptation model loss is obtained by the electronic device. In some embodiments, the original domain adaptation model loss can be expressed by the following formula:

[0093] in represents the classification loss of the original domain adaptation model, MMD(Z s , Z t ) represents the loss function obtained by the original domain adaptation model based on the maximum mean difference (i.e., Maximum Mean Discrepancy, MMD).

[0094] Step 142: input the target contaminated sample into the input end of the feature extractor of the student model; and calculate the contrast loss of the student model based on the target contaminated sample.

[0095] The electronic device constructs a target contamination sample by projecting the gradient ascent through the migration loss to simulate the unknown contamination, and adds the target contamination sample back to the input of the feature extractor of the student model, and calculates the contrast loss of the student model based on the target contamination sample.

[0096] In some embodiments, the contrast loss is expressed by the following formula:

[0097]

[0098] Among them, l conis the contrast loss, and the similarity function l sim is the similarity function, Z (stu) and Z (tea) are the features extracted by the student model and the teacher model respectively. is the characteristic of the i-th sample in the student model, is the feature of the i-th sample in the teacher model, and N is the number of samples in a batch.

[0099] Step 143: Update the feature extractor parameters and classifier parameters in the student model based on the original domain adaptation model loss and the contrast loss; use the updated feature extractor parameters and classifier parameters of the student model as the feature extractor parameters and classifier parameters of the image classification model.

[0100] Updating the feature extractor parameters and classifier parameters in the student model based on the original domain adaptation model loss and the contrast loss can be expressed by the following formula: ori (x s , x t， y s )

[0101]

[0102] l total = l ori +λl con ; Formula (7), where l ori That is, the original domain adaptation model loss, l con That is, contrast loss, based on l ori and l con The total loss l is obtained by weighting total Update the feature extractor parameters and classifier parameters in the student model.

[0103] Specifically, the contrast loss l con Its definition is as follows:

[0104] Given a batch of samples , first use the feature extractor f of the student model mentioned in step 140 (stu) and the teacher model’s f (tea) To get 2N feature vector representation:

[0105]

[0106] Further definition:

[0107]

[0108] Then the similarity between two feature vectors is calculated as:

[0109]

[0110] Among them, the sim() function is a similarity distance function, and the present invention adopts cosine distance.

[0111] The final contrast loss is defined as:

[0112]

[0113] Specifically, in some embodiments, calculating the contrast loss of the teacher-student model based on the target contaminated sample, and training the student model according to the contrast loss includes:

[0114] Repeat the following steps until convergence:

[0115] Step 1. From (X s , Y s ), X t Take out a batch of data (x s ,y s ), x t , let x={x s , x t};

[0116] Step 2: Construct the target domain sample image x t Target contaminated samples

[0117] Step 3: Update f based on the following loss (stu) +c (stu) ;

[0118] l ori (x s , x t ,y s )

[0119]

[0120] l total = l ori +λl con ;

[0121] Return f (stu) +c (stu) ;

[0122] Among them, X S represents the source domain sample image, X t Represents the target domain sample image and the source domain label Y s ; Original domain adaptation model f+c (feature extractor + classifier); f (stu) +c (stu)Represent the feature extractor and classifier of the student model after being updated by the original domain adaptation model loss and the contrast loss, and use the updated feature extractor and classifier of the student model as the feature extractor and classifier of the image classification model.

[0123] The embodiment of the present invention draws on the idea of ​​knowledge distillation and proposes a teacher-student architecture to solve domain invariance and improve robustness step by step. First, the teacher model learns domain invariance, and then the student model further distills the teacher model to learn robustness, thereby improving the robustness of domain adaptive image classification for images including image pollution.

[0124] The following is an explanation through a typical experiment. Figure 4 , Figure 4 The schematic diagram of the structure of the image classification model of the present invention is shown. The loss of the image classification model includes the source domain classification loss and the two-domain migration loss. In the embodiment of the present invention, the migration loss is added back to the input of the image classification model corresponding to the target domain sample image by projected gradient descent (PGD) to construct the target contaminated sample. And the contrast loss is used to constrain the original sample (or original image) of the target domain sample image and the target contaminated sample. Finally, the feature extractor and classifier of the image classification model are updated based on the contrast loss and the original domain adaptation model loss. Among them, X in the figure s represents the source domain sample image, X t represents the sample image of the target domain, Represents the feature vector obtained by the feature extractor of the source domain sample image after passing through the student model; Represents the feature vector obtained by the feature extractor of the student model for the target domain sample image; Represents the feature vector obtained by the feature extractor of the teacher model after the sample image of the target domain passes through it.

[0125] There are 15 types of image pollution used in the experiment of the target domain image of the present invention, including: Gaussian noise pollution, shot noise pollution, impulse noise pollution, defocus blur pollution, motion blur pollution, zoom blur pollution, fog pollution, frost pollution, snow pollution, elastic transformation pollution, contrast change pollution, brightness change pollution, JPEG compression pollution, pixelation pollution and glass blur pollution.

[0126] During the test, the image classification accuracy of the target domain images including the contaminated images was compared with the classic domain adaptation models CDAN+TN, DCAN, the best pollution robustness method AugMix and the TSCL-WC of the present invention. Figure 5 , Figure 5This is a comparison of the original classification accuracy of the classic domain adaptation model CDAN+TN, DCAN, the best pollution robustness method AugMix, and the TSCL-WC of the present invention in various tasks of the commonly used domain adaptation dataset Office-Home. Figure 5 It can be seen that AVG (↑) in the figure represents the average value of the original classification accuracy. The present invention can significantly improve the original classification accuracy of domain adaptive image classification for target domain images including image pollution, while the previous classic domain adaptation models CDAN+TN, DCAN and the current best pollution robustness method AugMix cannot have good original classification accuracy for target domain images including image pollution.

[0127] CDAN+TN is a method, namely Conditional Adversarial Domain Adaptation + Transferable normalization; AugMix is ​​a simple data processing method to improve robustness and uncertainty; DCAN is a Domain conditioned adaptation network.

[0128] Also, please refer to Figure 6 , Figure 6 The robustness of the target domain images including the polluted images is compared with the classic domain adaptation models CDAN+TN, DCAN, the best corruption robustness method AugMix and the TSCL-WC of the present invention under the commonly used domain adaptation dataset Office-Home. The present invention uses the mean corruption error (mCE) ​​commonly used in the robustness field to measure the robustness of various types of corruption. Figure 6 It can be seen that AVG (↓) in the figure represents the average value of the average contamination error rate. The present invention can significantly improve the robustness of domain adaptive image classification for target domain images including image contamination, while the previous classic domain adaptation model (CDAN+TN), DCAN, and the best contamination robustness method (AugMix) cannot have good robustness for target domain images including image contamination.

[0129] It should be noted that the images in the Office-Home dataset are divided into four domains, namely, the art domain (Ar), the paper-cutting domain (Cl), the product domain (Pr, i.e., images with very clean backgrounds on commercial websites), and the real domain (Rw, real photos). The present invention trains in one domain and tests in another domain. Figure 5 or Figure 6 In Ar->C1, training is conducted in the art domain and testing is conducted in the paper-cutting painting domain.

[0130] In summary, the above experiments illustrate that the embodiments of the present invention can not only maintain or even improve the image classification accuracy for target domain images containing image pollution, but also greatly improve the robustness for target domain images containing image pollution.

[0131] Please refer to Figure 7 , the domain adaptive image classification device provided by the present invention is described below. The domain adaptive image classification device described below and the domain adaptive image classification method described above can be referred to each other.

[0132] The domain-adaptive image classification device provided by the present invention comprises:

[0133] The first determination module 201 is used to determine a source domain image and a target domain image; the target domain image includes image pollution; the image pollution includes at least one of blur pollution, noise pollution, weather pollution, and image quality change pollution;

[0134] The result acquisition module 202 is used to obtain the image classification result, input the source domain image and the target domain image into the image classification model, and obtain the image classification result output by the image classification model; wherein the image classification model is constructed by constructing a target contaminated sample through the domain difference information between the source domain sample image and the target domain sample image, and is trained by calculating the contrast loss of the teacher-student model based on the target contaminated sample; wherein the source domain sample image and the target domain sample image do not include image contamination.

[0135] The domain adaptive image classification device of the present invention constructs a target contaminated sample through the domain difference information between the source domain sample image and the target domain sample image; thereby simulating unknown image contamination by constructing the target contaminated sample; then calculating the contrast loss of the teacher-student model based on the target contaminated sample, and then training the image classification model according to the contrast loss. The present invention calculates the contrast loss of the teacher-student model based on the target contaminated sample and trains the image classification model according to the contrast loss, thereby shortening the feature distance between the target contaminated sample and the original sample of the target domain sample image, thereby improving the robustness of domain adaptive image classification of images including image contamination.

[0136] Based on the above embodiments, as an optional embodiment, the domain adaptive image classification device further includes a model training module; the model training module includes:

[0137] A second determination module is used to determine a source domain sample image and a target domain sample image; the source domain sample image and the target domain sample image do not include image pollution;

[0138] A target contaminated sample construction module is used to construct a target contaminated sample based on domain difference information between a source domain sample image and a target domain sample image;

[0139] A teacher-student model building module, used to build a teacher-student model; wherein the teacher model and the student model have the same original domain adaptation model structure;

[0140] A training module is used to calculate the contrast loss of the teacher-student model based on the target contaminated sample, train the student model according to the contrast loss, and use the student model as the image classification model.

[0141] Based on the above embodiments, as an optional embodiment, the target contaminated sample construction module includes:

[0142] A migration loss acquisition module, used to obtain the migration loss of the source domain sample image and the target domain sample image after they pass through the feature extractor of the student model respectively;

[0143] The target contaminated sample acquisition module is used to perform a projected gradient descent process based on the migration loss to obtain the target contaminated sample.

[0144] Based on the above embodiments, as an optional embodiment, the base target contaminated sample acquisition module includes:

[0145] Repeat the following formula until the set number of times threshold is reached to obtain the target contaminated sample:

[0146]

[0147]

[0148]

[0149] in, represents the target contaminated sample; x s represents the source domain sample image, x t represents the target domain sample image, f represents the feature extractor of the student model; δ represents the preset adjustment range parameter, η represents the update step size, and n represents the number of update steps.

[0150] Based on the above embodiments, as an optional embodiment, the migration loss is obtained by a maximum mean difference method or an adversarial attack method.

[0151] Based on the above embodiments, as an optional embodiment, the training module includes:

[0152] The original domain adaptation model loss acquisition module is used to obtain the original domain adaptation model loss;

[0153] A contrast loss acquisition module, used for inputting the target contaminated sample into the input end of the feature extractor of the student model; and calculating the contrast loss of the student model based on the target contaminated sample;

[0154] A sub-training module is used to update the feature extractor parameters and classifier parameters in the student model based on the original domain adaptation model loss and the contrast loss; and use the updated feature extractor parameters and classifier parameters of the student model as the feature extractor parameters and classifier parameters of the image classification model.

[0155] Based on the above embodiments, as an optional embodiment, the contrast loss is expressed by the following formula:

[0156]

[0157] Among them, l con is the contrast loss, and the similarity function l sim is the similarity function, Z (stu) and Z (tea) are the features extracted by the student model and the teacher model respectively. is the characteristic of the i-th sample in the student model, is the feature of the i-th sample in the teacher model, and N is the number of samples in a batch.

[0158] Figure 8 An example of a physical structure diagram of an electronic device is shown in FIG. Figure 8As shown, the electronic device may include: a processor 810, a communication interface 820, a memory 830 and a communication bus 840, wherein the processor 810, the communication interface 820 and the memory 830 communicate with each other through the communication bus 840. The processor 810 may call the logic instructions in the memory 830 to execute the domain adaptive image classification method, which includes: determining a source domain image and a target domain image; the target domain image includes image pollution; the image pollution includes at least one of blur pollution, noise pollution, weather pollution and image quality change pollution; obtaining an image classification result, inputting the source domain image and the target domain image into an image classification model, and obtaining an image classification result output by the image classification model; wherein the image classification model is obtained by constructing a target pollution sample through the domain difference information between the source domain sample image and the target domain sample image, and calculating the contrast loss of the teacher-student model based on the target pollution sample for training; wherein the source domain sample image and the target domain sample image do not include image pollution.

[0159] In addition, the logic instructions in the above-mentioned memory 830 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when it is sold or used as an independent product. Based on such an understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk and other media that can store program codes.

[0160] On the other hand, the present invention also provides a computer program product, which includes a computer program, and the computer program can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the domain adaptive image classification method provided by the above methods, and the method includes: determining a source domain image and a target domain image; the target domain image includes image pollution; the image pollution includes at least one of blur pollution, noise pollution, weather pollution, and image quality change pollution; obtaining an image classification result, inputting the source domain image and the target domain image into an image classification model, and obtaining an image classification result output by the image classification model; wherein the image classification model is constructed by constructing a target pollution sample through the domain difference information between the source domain sample image and the target domain sample image, and calculating the contrast loss of the teacher-student model based on the target pollution sample for training; wherein the source domain sample image and the target domain sample image do not include image pollution.

[0161] On the other hand, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, is implemented to execute the domain adaptive image classification method provided by the above-mentioned methods, the method comprising: determining a source domain image and a target domain image; the target domain image includes image pollution; the image pollution includes at least one of blur pollution, noise pollution, weather pollution, and image quality change pollution; obtaining an image classification result, inputting the source domain image and the target domain image into an image classification model, and obtaining an image classification result output by the image classification model; wherein the image classification model is obtained by constructing a target pollution sample through the domain difference information between the source domain sample image and the target domain sample image, and calculating the contrast loss of the teacher-student model based on the target pollution sample for training; wherein the source domain sample image and the target domain sample image do not include image pollution.

[0162] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.

[0163] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0164] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A domain-adaptive image classification method, It is characterized in that include: Determine a source domain image and a target domain image; the target domain image includes image pollution; the image pollution includes at least one of blur pollution, noise pollution, weather pollution, and image quality change pollution; Obtaining an image classification result, inputting the source domain image and the target domain image into an image classification model, and obtaining an image classification result output by the image classification model; The image classification model is obtained by constructing a target contaminated sample through domain difference information between a source domain sample image and a target domain sample image, and calculating the contrast loss of a teacher-student model based on the target contaminated sample for training; wherein the source domain sample image and the target domain sample image do not include image contamination; The method of constructing a target contaminated sample by using domain difference information between a source domain sample image and a target domain sample image includes: Obtaining the migration loss of the source domain sample image and the target domain sample image after they pass through the feature extractor of the student model respectively; Performing a projected gradient descent process based on the migration loss to obtain a target contaminated sample; The projected gradient descent process based on the migration loss is performed to obtain a target contaminated sample, including: Repeat the following formula until the set number of times threshold is reached to obtain the target contaminated sample: in, represents the target contaminated sample; x s represents the source domain sample image, x t represents the target domain sample image, f represents the feature extractor of the student model; δ represents the preset adjustment range parameter, and η represents the update step size.

2. The domain adaptive image classification method according to claim 1, It is characterized in that Before determining the source domain image and the target domain image, the step of training the image classification model is also included; The training of the image classification model comprises: Determine a source domain sample image and a target domain sample image; the source domain sample image and the target domain sample image do not include image pollution; Construct the target contaminated sample through the domain difference information between the source domain sample image and the target domain sample image; Construct a teacher-student model; wherein the teacher model and the student model have the same original domain adaptation model structure; The contrast loss of the teacher-student model is calculated based on the target contaminated sample, the student model is trained according to the contrast loss, and the student model is used as the image classification model.

3. The domain adaptive image classification method according to claim 1, It is characterized in that The migration loss is obtained by a maximum mean difference method or an adversarial attack method.

4. The domain adaptive image classification method according to claim 2, It is characterized in that The step of calculating the contrast loss of the teacher-student model based on the target contaminated sample, training the student model according to the contrast loss, and using the student model as the image classification model includes: Get the original domain adaptation model loss; Inputting the target contaminated sample into the input end of the feature extractor of the student model; calculating the contrast loss of the student model based on the target contaminated sample; The feature extractor parameters and the classifier parameters in the student model are updated based on the original domain adaptation model loss and the contrast loss; and the feature extractor parameters and the classifier parameters of the updated student model are used as the feature extractor parameters and the classifier parameters of the image classification model.

5. The domain adaptive image classification method according to claim 4, It is characterized in that The contrast loss is expressed by the following formula: Among them, l con is the contrast loss, l sim is the similarity function, Z (stu) and Z (tea) are the features extracted by the student model and the teacher model respectively. is the characteristic of the i-th sample in the student model, is the feature of the i-th sample in the teacher model, and N is the number of samples in a batch.

6. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, It is characterized in that When the processor executes the program, the steps of the domain adaptive image classification method according to any one of claims 1 to 5 are implemented.

7. A non-transitory computer-readable storage medium having a computer program stored thereon, It is characterized in that When the computer program is executed by a processor, the steps of the domain adaptive image classification method according to any one of claims 1 to 5 are implemented.

8. A computer program product comprising a computer program, It is characterized in that When the computer program is executed by a processor, the steps of the domain adaptive image classification method according to any one of claims 1 to 5 are implemented.

Images