Image classification method and system based on online domain adaptive deep learning

Through the online clustering algorithm based on transportation problems, the negative migration problem in the existing technology under high class similarity and uneven distribution is solved, the classification accuracy of the target domain is improved, and the calculation complexity is simplified, thus achieving large-scale Effective processing of scale data.

CN113936168BActive Publication Date: 2024-11-01INST OF COMPUTING TECH CHINESE ACAD OF SCI
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202111088084.5
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2021-09-16
Publication Date
2024-11-01
Estimated Expiration
2041-09-16

AI Technical Summary

Technical Problem

The existing cluster-based unsupervised domain adaptive algorithm has negative migration problems when the category similarity is high and the distribution is uneven, resulting in poor performance of the model in the target domain and unable to effectively process large-scale data.

Method used

Using an online clustering algorithm based on transportation problems, the features of the source domain and the target domain are extracted through the feature extraction network, the category clustering center is obtained, and the feature extraction network and clustering center are updated using the relative entropy distance to solve the transportation problem to obtain the target domain sample. class allocation, thereby improving the classification accuracy of the model in the target domain.

Benefits of technology

This reduces the negative migration situation, improves the accuracy of the algorithm in the target domain, simplifies the computational complexity, and improves the computational accuracy of the model.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN113936168B_ABST
    Figure CN113936168B_ABST
Patent Text Reader

Abstract

The present invention proposes an image classification method and system based on online domain adaptive deep learning, comprising extracting features of source domain image data and target domain image data through a feature extraction network to obtain source domain features and target domain features; explicitly expressing the probability of conditional distribution of categories within the domain based on the distances from the source domain features and the target domain features to the clustering center; updating the feature extraction network and the category clustering center by minimizing the relative entropy distance between the source domain data label and the category conditional distribution probability within the domain; obtaining the category assignment of target domain samples by solving the transportation problem, updating the feature extraction network and the clustering center again by minimizing the relative entropy distance between the category assignment and the category conditional distribution probability within the domain, inputting the image to be classified into the feature extraction network that has been updated again to obtain the image features of the image to be classified, calculating the probability of the image features and all category centers in the category clustering center that has been updated again, and taking the category with the largest probability as the classification result.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the fields of AI application technology and image classification technology, and in particular to an image classification method and system based on online domain adaptive deep learning. Background Art

[0002] The deep convolutional neural network method is based on the assumption that the training dataset and the application scenario have the same distribution. When there is a domain shift between the actual scenario and the training dataset, the performance of the model will be greatly reduced. The existing solution is to fine-tune the model using the data of the target domain, but this method requires a large amount of manually labeled target domain data. The unsupervised domain adaptation (UDA) method uses unlabeled target domain data to solve the performance degradation caused by domain shift. Existing clustering-based unsupervised domain adaptation algorithms rely on traditional offline clustering methods such as K-Means to cluster in the target domain, and use the cluster labels for subsequent domain-consistent feature space learning.

[0003] Existing technical methods suffer from serious negative transfer when category similarity is high and unevenly distributed, and the model performs poorly in the target domain; offline clustering is required and large-scale data cannot be processed. Summary of the invention

[0004] The present invention designs an online clustering algorithm based on the transportation problem, forming a method and system for performing tasks using domain-adaptive deep learning to solve the problem of negative transfer.

[0005] Specifically, the present invention proposes an image classification method based on online domain adaptive deep learning, which includes:

[0006] Step 1: The features of the source domain image data and the target domain image data can be extracted through the feature extraction network to obtain the source domain features and the target domain features;

[0007] Step 2: Obtain the category cluster center, and explicitly express the conditional distribution probability of the category within the domain based on the distance from the source domain feature and the target domain feature to the cluster center;

[0008] Step 3: Update the feature extraction network and category clustering center by minimizing the relative entropy distance between the source domain data label and the conditional distribution probability of the category in the domain;

[0009] Step 4: Solve the transportation problem by the following formula to obtain the category distribution of the target domain samples:

[0010]

[0011] Where γ * Assign to this category, where represents the joint probability space of target domain samples and categories, q represents the category, is the cluster center of all categories, c(z t ,q) represents the sample z t The cost of transporting to category q, γ(z t ,q) represents the target domain sample z under the current transportation strategy γ t The size of transport to category q, α is the sample measure, and β is the category measure β;

[0012] Step 5: Update the feature extraction network and cluster center again by minimizing the relative entropy distance between the category assignment and the conditional distribution probability of the category in the domain;

[0013] Step 6: Input the image to be classified into the updated feature extraction network to obtain the image features of the image to be classified, calculate the distance between the image features and all category centers in the updated category clustering center, and express the probability that the image to be classified belongs to each category. Take the category with the largest probability as the image classification result of the image to be classified.

[0014] The image classification method based on joint clustering domain adaptation, wherein step 1 includes extracting features of source domain and target domain image data using the feature extraction network:

[0015]

[0016] in To input image data, Denote the parameter as θ R The neural network, Represents the output features.

[0017] The image classification method based on joint clustering domain adaptation, wherein step 2 comprises:

[0018]

[0019] in represents the feature cluster center of category j pictures, p(y j |x i ,q) represents the image sample x i Belongs to category y j The conditional distribution probability p of the category within the domain.

[0020] In the above-mentioned image classification method based on joint clustering domain adaptation, the relative entropy distance between the source domain data label and the conditional distribution probability of the category in the domain is minimized in step 3 as follows:

[0021]

[0022] Where N represents the number of samples in the source domain, Represents the sample x in the source domain i The category label of Represents the source domain sample x i Through neural network The extracted features.

[0023] In the image classification method based on joint clustering domain adaptation, the relative entropy distance between the category assignment of the target domain data and the conditional distribution probability of the category within the domain that is minimized in step 5 is specifically:

[0024]

[0025] where γ i,j represents the size of the transport from i to j, Represents the source domain sample x j The category assignment of Represents the source domain sample x i Through neural network The extracted features.

[0026] The present invention also proposes an image classification system based on online domain adaptive deep learning, which includes:

[0027] Module 1 is used to extract features of source domain image data and target domain image data respectively through a feature extraction network to obtain source domain features and target domain features;

[0028] Module 2 is used to obtain the category cluster center and explicitly express the conditional distribution probability of the category within the domain based on the distance between the source domain feature and the target domain feature and the cluster center;

[0029] Module 3 is used to update the feature extraction network and the category clustering center by minimizing the relative entropy distance between the source domain data label and the conditional distribution probability of the category in the domain;

[0030] Module 4 is used to solve the transportation problem and obtain the category assignment of the target domain samples through the following formula:

[0031]

[0032] Where γ * Assign to this category, where represents the joint probability space of target domain samples and categories, q represents the category, is the cluster center of all categories, c(z t ,q) represents the sample z t The cost of transporting to category q, γ(z t ,q) represents the target domain sample z under the current transportation strategy γ tThe size of transport to category q, α is the sample measure, and β is the category measure β;

[0033] Module 5 is used to update the feature extraction network and cluster center again by minimizing the relative entropy distance between the category assignment and the conditional distribution probability of the category in the domain;

[0034] Module 6 is used to input the image to be classified into the feature extraction network that has been updated again, obtain the image features of the image to be classified, calculate the distance between the image features and all category centers in the category clustering center that has been updated again, represent the probability that the image to be classified belongs to each category, and take the category with the largest probability as the image classification result of the image to be classified.

[0035] The image classification system based on joint clustering domain adaptation, wherein the module 1 includes extracting features of source domain and target domain image data using the feature extraction network:

[0036]

[0037] in To input image data, Denote the parameter as θ R The neural network, Represents the output features.

[0038] The picture classification system based on joint clustering domain adaptation, wherein the module 2 includes:

[0039]

[0040] in represents the feature cluster center of category j pictures, p(y j |x i ,q j ) represents the image sample x i Belongs to category y j The conditional distribution probability p of the category within the domain.

[0041] In the image classification system based on joint clustering domain adaptation, the relative entropy distance between the source domain data label and the conditional distribution probability of the category in the domain is minimized in module 3 as follows:

[0042]

[0043] Where N represents the number of samples in the source domain, Represents the sample x in the source domain i The category label of Represents the source domain sample x i Through neural network The extracted features.

[0044] In the image classification system based on joint clustering domain adaptation, the relative entropy distance between the category assignment of target domain data and the conditional distribution probability of the category within the domain that is minimized in module 5 is specifically:

[0045]

[0046] where γ i,j represents the size of the transport from i to j, Represents the source domain sample x j The category assignment of Represents the source domain sample x i Through neural network The extracted features.

[0047] It can be seen from the above scheme that the advantages of the present invention are:

[0048] The negative transfer of the clustering-based domain adaptation algorithm is reduced, and the accuracy of the algorithm in the target domain is improved; the clustering problem in the target domain is solved using the Katrovich relaxation problem to improve the calculation accuracy and reduce the calculation complexity. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] Figure 1 Schematic diagram of the transportation process between the source domain and the target domain. DETAILED DESCRIPTION

[0050] Traditional offline clustering methods ignore the original category conditional probability distribution in the target domain, which will lead to the deviation of the target domain category distribution in the domain consistent feature space. At the same time, when the category conditional probability distribution of the target domain samples is uneven and concentrated at the edge of the source domain category decision surface, the traditional offline clustering method will lead to serious negative transfer effects. And due to its non-differentiable characteristics, the defects of negative transfer are difficult to compensate by objective function design. Therefore, this method solves the problem of negative transfer by designing an online clustering algorithm based on the transportation problem, forming a method and system for performing tasks using domain adaptive deep learning.

[0051] The present invention includes the following key technical points:

[0052] Key point 1, the clustering problem is defined as finding a transport mapping from sample measure to category measure;

[0053] Key point 2: The clustering problem in the source domain is converted into a Mongo problem;

[0054] Key point 3: For the clustering problem in the target domain, relax its discrete assignment constraints and convert it into a Katrovich relaxation problem.

[0055] In order to make the above features and effects of the present invention more clearly understood, embodiments are given below and described in detail with reference to the accompanying drawings.

[0056] In order to solve the limitations of traditional offline clustering methods in the field of domain adaptation, such as Figure 1 As shown, the present invention proposes an online domain adaptive clustering method based on transportation problem.

[0057] First, the clustering problem is defined as finding a transport mapping T from sample measure α to class measure β:

[0058]

[0059] is the sample measure α, is the category measure β, where z i represents the characteristics of sample i, N represents the number of samples; q j represents the features of category j, and K represents the number of categories.

[0060] Since the purpose of this method is to perform clustering while preserving the original category probability distribution in the domain, the category conditional probability estimate p(y|x, q, θ) estimated by the network parameters can be defined as the cost function C of the transportation problem.

[0061] Secondly, according to the problem definition, the clustering problem in the source domain is converted into a Mongo problem:

[0062]

[0063] Where T #α=β represents the transportation strategy from measure α to measure β, represents the feature space, T(z) represents the transportation result of z under the transportation strategy T; c(z,T(z)) represents the transportation cost from z to T(z), and α(z) represents the measure of z.

[0064] Since the source domain samples are labeled samples, the solution to this problem can be simplified to the following form:

[0065]

[0066] Where N represents the number of samples in the source domain, Represents the source domain sample x i The category label of Represents the source domain sample x i Through neural network The extracted features.

[0067] Afterwards, for the clustering problem in the target domain, due to the lack of sample annotations, that is, is unknown, so it is necessary to obtain the category distribution of samples by solving the transportation problem *Since the Mongo problem is a non-convex optimization problem and is difficult to solve, its discrete assignment constraints are relaxed and converted into a Katrovich relaxation problem:

[0068]

[0069]

[0070] in represents the joint probability space of target domain samples and categories, represents the category, c(z t ,q) represents the sample z t The cost of transporting to category q, γ(z t ,q) represents the sample z under the current transportation strategy γ t Transport to size of category q.

[0071] U(a,b) represents the uniform distribution of [a,b]

[0072] In order to prevent the collapse of the solution and simplify the calculation, a regular term constraint is added to the above problem, and the problem is converted into the following form:

[0073]

[0074]

[0075] ∈ is the penalty coefficient, usually 0.05, γ i,j It represents the size of the measurement transported from sample i to category j, that is, sample x i Assigned to category y j probability.

[0076] Finally, the category assignment γ of the transportation problem can be obtained by solving the above Katrovich problem with regularization term through the Sinkhorn-Knopp algorithm. * , and used to update the clustering network:

[0077]

[0078] Taking image classification as an example, there is an active domain image dataset X S , target domain image dataset X T .

[0079] The parameter introduced is θ R A neural network and a set of cluster centers

[0080] Source domain image x is input to network θ R Then we get the feature z, using the above formula

[0081]

[0082] Update θ R and

[0083] Target domain image x is input into network θ R Then we get the feature z, using the above formula

[0084]

[0085] Get the category class assignment strategy¶ * , and using the above formula

[0086]

[0087] Update θ R and

[0088] After training is completed, input the test set image X, network θ R Output feature Z and compare Z with Calculate the class conditional distribution probability:

[0089]

[0090] Take the category q with the largest category conditional distribution probability max As the classification result.

[0091] The following is a system embodiment corresponding to the above method embodiment. This embodiment can be implemented in conjunction with the above embodiment. The relevant technical details mentioned in the above embodiment are still valid in this embodiment. In order to reduce repetition, they are not repeated here. Accordingly, the relevant technical details mentioned in this embodiment can also be applied in the above embodiment.

[0092] The present invention also proposes an image classification system based on online domain adaptive deep learning, which includes:

[0093] Module 1 is used to extract features of source domain image data and target domain image data respectively through a feature extraction network to obtain source domain features and target domain features;

[0094] Module 2 is used to obtain the category cluster center and explicitly express the conditional distribution probability of the category within the domain based on the distance between the source domain feature and the target domain feature and the cluster center;

[0095] Module 3 is used to update the feature extraction network and the category clustering center by minimizing the relative entropy distance between the source domain data label and the conditional distribution probability of the category in the domain;

[0096] Module 4 is used to solve the transportation problem and obtain the category assignment of the target domain samples through the following formula:

[0097]

[0098] Where γ * Assign to this category, where represents the joint probability space of target domain samples and categories, q represents the category, is the cluster center of all categories, c(z t ,q) represents the sample z t The cost of transporting to category q, γ(z t ,q) represents the target domain sample z under the current transportation strategy γ t The size of transport to category q, α is the sample measure, and β is the category measure β;

[0099] Module 5 is used to update the feature extraction network and cluster center again by minimizing the relative entropy distance between the category assignment and the conditional distribution probability of the category in the domain;

[0100] Module 6 is used to input the image to be classified into the feature extraction network that has been updated again, obtain the image features of the image to be classified, calculate the distance between the image features and all category centers in the category clustering center that has been updated again, represent the probability that the image to be classified belongs to each category, and take the category with the largest probability as the image classification result of the image to be classified.

[0101] The image classification system based on joint clustering domain adaptation, wherein the module 1 includes extracting features of source domain and target domain image data using the feature extraction network:

[0102]

[0103] in To input image data, Denote the parameter as θ R The neural network, Represents the output features.

[0104] The picture classification system based on joint clustering domain adaptation, wherein the module 2 includes:

[0105]

[0106] in represents the feature cluster center of category j pictures, p(y j |x i ,q) represents the image sample x i Belongs to category y j The conditional distribution probability p of the category within the domain.

[0107] In the image classification system based on joint clustering domain adaptation, the relative entropy distance between the source domain data label and the conditional distribution probability of the category in the domain is minimized in module 3 as follows:

[0108]

[0109] Where N represents the number of samples in the source domain, Represents the sample x in the source domain i The category label of Represents the source domain sample x i Through neural network The extracted features.

[0110] In the image classification system based on joint clustering domain adaptation, the relative entropy distance between the category assignment of target domain data and the conditional distribution probability of the category within the domain that is minimized in module 5 is specifically:

[0111]

[0112] where γ i,j represents the size of the transport from i to j, Represents the target domain sample x j The category assignment of Represents the target domain sample x i Through neural network The extracted features.

Claims

1. An image classification method based on online domain adaptive deep learning, It is characterized in that include: Step 1: Extract the features of the source domain image data and the target domain image data through a feature extraction network to obtain source domain features and target domain features; Step 2: Obtain the category cluster center, and explicitly express the conditional distribution probability of the category within the domain based on the distance from the source domain feature and the target domain feature to the cluster center; Step 3: Update the feature extraction network and category clustering center by minimizing the relative entropy distance between the source domain data label and the conditional distribution probability of the category in the domain; Step 4: Solve the transportation problem by the following formula to obtain the category distribution of the target domain samples: Where γ * Assign to this category, where represents the joint probability space of target domain samples and categories, q represents the category, is the cluster center of all categories, c(z t ,q) represents the sample z t The cost of transporting to category q, γ(z t ,q) represents the target domain sample z under the current transportation strategy γ t The size of transport to category q, α is the sample measure, and β is the category measure β; Step 5: Update the feature extraction network and cluster center again by minimizing the relative entropy distance between the category assignment and the conditional distribution probability of the category in the domain; Step 6: Input the image to be classified into the updated feature extraction network to obtain the image features of the image to be classified, calculate the distance between the image features and all category centers in the updated category clustering center, and express the probability that the image to be classified belongs to each category. Take the category with the largest probability as the image classification result of the image to be classified.

2. The image classification method based on online domain adaptive deep learning as claimed in claim 1, It is characterized in that The step 1 includes extracting features of the source domain and target domain image data using the feature extraction network: in To input image data, Denote the parameter as θ R The neural network, Represents the output features.

3. The image classification method based on online domain adaptive deep learning as claimed in claim 2, It is characterized in that This step 2 includes: Where q represents The characteristics of the cluster centers in represents the cluster center feature of category j pictures, p(y j |x i ,q) represents the image sample x i Belongs to category y j The conditional distribution probability p of the category within the domain.

4. The image classification method based on online domain adaptive deep learning as claimed in claim 3, It is characterized in that In step 3, the relative entropy distance between the source domain data label and the conditional distribution probability of the domain category is minimized as follows: Where N represents the number of samples in the source domain, Represents the sample x in the source domain i The category label of Represents the source domain sample x i Through neural network The extracted features.

5. The image classification method based on online domain adaptive deep learning as claimed in claim 4, It is characterized in that The relative entropy distance between the category assignment of the target domain data and the conditional distribution probability of the category in the domain that is minimized in step 5 is specifically: where γ i,j represents the size of the transport from i to j, Represents the source domain sample x j The category assignment of Represents the source domain sample x i Through neural network The extracted features.

6. An image classification system based on online domain adaptive deep learning, It is characterized in that include: Module 1 is used to extract features of source domain image data and target domain image data respectively through a feature extraction network to obtain source domain features and target domain features; Module 2 is used to obtain the category cluster center and explicitly express the conditional distribution probability of the category within the domain based on the distance between the source domain feature and the target domain feature and the cluster center; Module 3 is used to update the feature extraction network and the category clustering center by minimizing the relative entropy distance between the source domain data label and the conditional distribution probability of the category in the domain; Module 4 is used to solve the transportation problem and obtain the category assignment of the target domain samples through the following formula: Where γ * Assign to this category, where represents the joint probability space of target domain samples and categories, q represents the category, is the cluster center of all categories, c(z t ,q) represents the sample z t The cost of transporting to category q, γ(z t ,q) represents the target domain sample z under the current transportation strategy γ t The size of transport to category q, α is the sample measure, and β is the category measure β; Module 5 is used to update the feature extraction network and cluster center again by minimizing the relative entropy distance between the category assignment and the conditional distribution probability of the category in the domain; Module 6 is used to input the image to be classified into the feature extraction network that has been updated again, obtain the image features of the image to be classified, calculate the distance between the image features and all category centers in the category clustering center that has been updated again, represent the probability that the image to be classified belongs to each category, and take the category with the largest probability as the image classification result of the image to be classified.

7. The image classification system based on online domain adaptive deep learning as claimed in claim 6, It is characterized in that The module 1 includes extracting features of source domain and target domain image data using the feature extraction network: in To input image data, Denote the parameter as θ R The neural network, Represents the output features.

8. The image classification system based on online domain adaptive deep learning as claimed in claim 7, It is characterized in that This module 2 includes: in represents the feature cluster center of category j pictures, p(y j |x i ,q) represents the image sample x i Belongs to category y j The conditional distribution probability p of the category within the domain.

9. The image classification system based on online domain adaptive deep learning as claimed in claim 8, It is characterized in that In module 3, the relative entropy distance between the source domain data label and the conditional distribution probability of the category in the domain is minimized as follows: Where N represents the number of samples in the source domain, Represents the sample x in the source domain i The category label of Represents the source domain sample x i Through neural network The extracted features.

10. The image classification system based on online domain adaptive deep learning as claimed in claim 9, It is characterized in that In module 5, the relative entropy distance between the category assignment of the target domain data and the conditional distribution probability of the category within the domain is minimized as follows; where γ i,j represents the size of the transport from i to j, Represents the source domain sample x j The category assignment of Represents the source domain sample x i Through neural network The extracted features.