A long-tail distribution object identification scheme fusing unsupervised domain adaptation technology
By combining unsupervised domain adaptation techniques with transfer learning and contrastive learning, the problem of recognition accuracy caused by differences in data distribution and long-tail distribution in real-world scenarios is solved, achieving more efficient object recognition results.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- HUNAN UNIV
- Filing Date
- 2023-01-09
- Publication Date
- 2026-05-29
AI Technical Summary
Existing image recognition algorithms struggle to effectively identify objects uploaded by users in real-world scenarios due to performance degradation and cost issues caused by differences in the distribution of training data and real data, long-tailed distribution, and reliance on massive labeled datasets.
We employ unsupervised domain adaptation techniques combined with transfer learning, contrastive learning, and data-enhanced consistency. Through prototype-level contrastive learning and consistency loss based on smooth labels, we correct feature space bias and improve the model's recognition accuracy under long-tailed distributions.
It improves the accuracy and robustness of the photo recognition module in shopping and science popularization applications in real-world scenarios, reduces reliance on massive amounts of tag data, and lowers manpower and time costs.
Smart Images
Figure CN116051894B_ABST
Abstract
Description
Technical Field
[0001] This invention mainly relates to the field of object recognition by photo in shopping / science popularization applications, specifically a long-tailed distribution object recognition scheme that integrates unsupervised domain adaptation technology. Background Technology
[0002] Existing image-based object recognition algorithms typically train a deep learning model on a clean, balanced, massive dataset for object identification. Based on this large amount of labeled data, the model exhibits good classification performance. However, in real-world classification tasks, user-uploaded images may have flaws, such as insufficient lighting, angle discrepancies, or partial occlusion. Furthermore, real-world objects exhibit a long-tailed distribution, unlike the balanced dataset used for model training. Finally, the significant human and time costs of labeling the training set cannot be ignored.
[0003] Therefore, when dealing with object recognition tasks in this real-world scenario, the existing image recognition algorithms on the market have the following problems: (1) There is a distribution difference between the training dataset and the real data. The recognition algorithm does not specifically handle this difference (feature distribution difference, label distribution difference), which leads to a sharp decline in classification performance. (2) The biased feature space caused by the imbalanced dataset is not corrected, which leads to classifier bias and incorrect object recognition results. (3) The model accuracy depends on the huge cost caused by the massive labeled dataset. How to directly use unlabeled samples is a challenge. Summary of the Invention
[0004] Image recognition algorithms for shopping / science popularization apps require high accuracy and robustness. To address the challenges of object recognition in real-world scenarios, we propose a long-tailed distribution object recognition scheme based on unsupervised domain adaptation. This scheme combines transfer learning (unsupervised domain adaptation), contrastive learning, and data augmentation consistency to provide a high-performance and efficient solution to address the problems mentioned in the background.
[0005] To achieve the above objectives, this invention provides a long-tailed distributed object recognition scheme based on unsupervised domain adaptation technology, including an object recognition main system. The object recognition main system includes a domain adaptation system, an object database, and a model center. The domain adaptation system includes prototype-level contrastive learning and consistency loss based on smooth labels. The object database includes an object image acquisition device and a database. The model center includes an object pre-trained model and a long-tail recognition model. This design enables the photo-based object recognition module in shopping / science popularization applications to more accurately identify objects in natural life scenes.
[0006] The object image acquisition device can acquire the corresponding data of the object to be identified.
[0007] Classification algorithm models typically use cross-entropy loss to update feature extractor parameters. However, in the case of long-tailed data, the traditional cross-entropy loss cannot correct the skewed feature space. We combine this with unsupervised domain adaptation techniques to design a prototype-level contrastive learning in the feature space to balance the feature space.
[0008] The prototype-level comparative learning includes:
[0009] The classifier F of the long-tail recognition model supports the classification of K categories. Long-tail data leads to a biased feature space. To correct this bias, we designed the following contrastive loss:
[0010]
[0011] Among them, D S D T These are existing source domain data from the database and newly collected unlabeled target domain data to be identified, respectively. CE It is cross-entropy loss. Let y be a K-dimensional probability vector, representing the probability that a sample feature vector in the source / target domain is predicted to be each class in the feature space. S One-hot encoding of source domain sample labels, The soft pseudo-labels for the target domain samples. The calculation method is as follows:
[0012]
[0013] in, This represents the probability that the sample belongs to the k-th class, z. T c is the feature vector of the target domain sample. k Let be the cluster center of the k-th category. That is, we use the distance between the feature vector and each cluster center to measure the probability that a sample belongs to each category. The calculation method is as follows:
[0014]
[0015] in, yes The k-th dimension value. N t It is the number of all samples in the target domain. Based on all target domain samples in the k-th dimension The result of normalization is then normalized across K categories. That is, we use the probability prediction values of the feature vectors of the target domain sample to calculate the soft pseudo-label of the sample.
[0016] When there are differences in feature distribution and label distribution between training data and real data, and the data in the domain exhibits a long-tail distribution, the confidence of soft pseudo-labels drops sharply, thus hindering the effective training of the model. To address this, we designed a consistency loss based on smooth labels, which uses strong and weak data of the same sample to enhance the predictive consistency of instances and promote the training of the classification model.
[0017] Furthermore, the consistency loss based on smooth labels includes:
[0018] L aug =L CE (p(y|t(x t )),q) ④
[0019] Where, x t The target domain sample is t, which is the strong augmentation technique RandAugment, and L is L. CE It is the cross-entropy loss, where p is the value of sample x. t The predicted probability value of a strongly augmented instance, where q is the probability of sample x. t The soft pseudo-label value of weakly augmented instances. Since the source domain data, i.e., clean, available laboratory data, already has labels, this consistency loss applies to unlabeled samples collected from real-world scenarios, i.e., target domain samples.
[0020] Step 1:
[0021] a. Use t to sample the target domain x t Perform strong enhancement to obtain instance t(x) t );
[0022] b. Use common image enhancement techniques, such as scaling, cropping, and flipping, to obtain weakly enhanced instances t of the target domain samples. w (x t );
[0023] c. Obtain a weakly enhanced instance t w (x t The predicted probability value p) w If this consistency loss is used in the feature space, then p w Calculate using formula ②; if this consistency loss is used on the classifier, then p w The output value of the classifier;
[0024] d. Using formula ③ with p w Calculate the soft pseudo-label value q for the weakly enhanced instance;
[0025] e. Obtain the strongly enhanced instance t(x) tThe predicted probability value p is calculated using formula ② if the consistency loss is used in the feature space; if the consistency loss is used on the classifier, p is the classifier output value.
[0026] f. Use formula ④ to calculate the consistency loss based on smooth labels.
[0027] Compared with the prior art, the beneficial effects of the present invention are as follows:
[0028] This invention designs a prototype-level contrastive learning technique based on unsupervised domain adaptation, which brings samples closer to their own cluster and away from other clusters, thereby correcting the biased feature space caused by long-tailed data and promoting unbiased training of the classifier. It also designs a smooth label consistency loss technique based on unsupervised domain adaptation to solve the problem of a sharp drop in classification accuracy and the model's inability to adapt due to differences in the distribution of training and real-world data. The combination of prototype-level contrastive learning and smooth label consistency loss effectively improves the accuracy and robustness of the algorithm model, enabling the photo recognition module of shopping / science popularization applications to accurately identify objects in real-life scenarios. Attached Figure Description
[0029] Figure 1 This is a schematic diagram of the structure of the present invention;
[0030] Figure 2 This is a schematic diagram of the long-tail recognition model training of the present invention;
[0031] Figure 3 This is a schematic diagram illustrating the prototype-level comparative learning of the present invention;
[0032] Figure 4 This is a schematic diagram of the consistency loss based on smooth labels according to the present invention;
[0033] Figure 1 In the Chinese system: A. Main system for object recognition; 1. Object database; 2. Model center; 3. Domain adaptation system; 11. Object image acquisition device; 12. Database; 21. Object pre-trained model; 22. Long-tail recognition model; 31. Prototype-level contrastive learning; 32. Consistency loss based on smooth labels. Detailed Implementation
[0034] To facilitate understanding of the present invention, a more comprehensive description of the present invention will be given below with reference to the accompanying drawings, which illustrate several embodiments of the present invention. However, the present invention can be implemented in different forms and is not limited to the embodiments described in the text. Rather, these embodiments are provided to make the disclosure of the present invention more thorough and complete.
[0035] It should be noted that when an element is referred to as being "fixed to" another element, it can be directly on the other element or there may be an intervening element. When an element is referred to as being "connected to" another element, it can be directly connected to the other element or there may be an intervening element. The terms "vertical," "horizontal," "left," "right," and similar expressions used in this document are for illustrative purposes only.
[0036] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly associated with those skilled in the art to which this invention pertains. The terminology used herein in the specification of this invention is for the purpose of describing particular embodiments and is not intended to limit the invention. The term "and / or" as used herein includes any and all combinations of one or more of the associated listed items.
[0037] Please see Figure 1-4 This invention provides a long-tailed object recognition scheme based on unsupervised domain adaptation technology, including an object recognition main system A. The object recognition main system A includes a domain adaptation system 3, an object database 1, and a model center 2. The domain adaptation system 3 includes a consistency loss based on smooth labels 32 and a prototype-level contrastive learning 31. The object database 1 includes an object image acquisition device 11 and a database 12. The model center 2 includes an object pre-trained model 21 and a long-tail recognition model 22. This design can realize a domain-adaptive classification model that is more suitable for object recognition tasks in real-world scenarios.
[0038] Please see Figure 3 The prototype-level contrastive learning 31 includes:
[0039] The classifier F of the long-tail recognition model supports the classification of K categories. Long-tail data leads to a biased feature space. To correct this bias, we designed the following contrastive loss:
[0040]
[0041] Among them, D S D T These are existing source domain data from the database and newly collected unlabeled target domain data to be identified, respectively. CE It is cross-entropy loss. Let y be a K-dimensional probability vector, representing the probability that a sample feature vector in the source / target domain is predicted to be each class in the feature space. S One-hot encoding of source domain sample labels, The soft pseudo-labels for the target domain samples. The calculation method is as follows:
[0042]
[0043] in, This represents the probability that the sample belongs to the k-th class, z. T c is the feature vector of the target domain sample. k Let be the cluster center of the k-th category. That is, we use the distance between the feature vector and each cluster center to measure the probability that a sample belongs to each category. The calculation method is as follows:
[0044]
[0045] in, yes The k-th dimension value. N t It is the number of all samples in the target domain. Based on all target domain samples in the k-th dimension The result of normalization is then normalized across K categories. That is, we use the probability prediction values of the feature vectors of the target domain sample to calculate the soft pseudo-label of the sample.
[0046] This design can address the situation where the classification model's accuracy drops sharply when the source domain data of database 12 and the collected real-world objects (target domain data) exhibit a long-tailed distribution.
[0047] Please see Figure 4 The consistency loss 32 based on smooth labels includes:
[0048] L aug =L CE (p(y|t(x t )),q) ④
[0049] Where, x t The target domain sample is t, which is the strong augmentation technique RandAugment, and L is L. CE It is the cross-entropy loss, where p is the value of sample x. t The predicted probability value of a strongly augmented instance, where q is the probability of sample x. t The soft pseudo-label value of weakly augmented instances. Since the source domain data, i.e., clean, available laboratory data, already has labels, this consistency loss applies to unlabeled samples collected from real-world scenarios, i.e., target domain samples.
[0050] Step 1:
[0051] a. Use t to sample the target domain x t Perform strong enhancement to obtain instance t(x) t );
[0052] b. Use common image enhancement techniques, such as scaling, cropping, and flipping, to obtain weakly enhanced instances t of the target domain samples. w(x t );
[0053] c. Obtain a weakly enhanced instance t w (x t The predicted probability value p) w If this consistency loss is used in the feature space, then p w Calculate using formula ②; if this consistency loss is used on the classifier, then p w The output value of the classifier;
[0054] d. Using formula ③ with p w Calculate the soft pseudo-label value q for the weakly enhanced instance;
[0055] e. Obtain the strongly enhanced instance t(x) t The predicted probability value p is calculated using formula ② if the consistency loss is used in the feature space; if the consistency loss is used on the classifier, p is the classifier output value.
[0056] f. Use formula ④ to calculate the consistency loss based on smooth labels.
[0057] This design can mitigate the problem of decreased confidence in pseudo-labels of target domain data during classification model training caused by differences in feature distribution and label distribution between training data and real-world data.
[0058] The pseudocode for the algorithm that generates the long-tail recognition model 22 is as follows:
[0059] Input: Source domain data D S Target domain data D T The pre-trained classification model 21 is represented as H = C(F(x)), where C is the classifier and F is the feature extractor.
[0060] Output: Long-tail recognition model 22
[0061] 1. Using D s Traditional k-means initialization of cluster centers for each category
[0062] / / MAX_ITER represents the maximum number of iterations, where I represents the I-th iteration and E represents the Epoch.
[0063] 2 repetitions
[0064] 3|for I←1,MAX_ITER do
[0065] 4||From D S D T Sampling batch B S B T
[0066] 5||Based on B S B T Update the F parameter using contrast loss ① and consistency loss ④.
[0067] 6||Based on B T ④ Update the H parameters of the entire model using consistency loss.
[0068] 7|end for
[0069] 8| Recalculate the cluster centers for each category using k-means, and recalculate for... Fu
[0070] 9|value
[0071] 10|E=E+1
[0072] 11 until convergence;
[0073] The specific operation method of this invention is as follows:
[0074] When using a long-tailed object recognition scheme based on unsupervised domain adaptation technology, during the model training phase, the system collects a certain amount of images of objects to be recognized as target domain data through the object image acquisition device 11. The database 12 contains cleaned and labeled source domain data. A classification model trained on open-source big data (such as ImageNet) can be used as the object pre-training model 21, or a customized object pre-training model 21 can be obtained by training a portion of the data in the database 12. Through prototype-level contrastive learning 31 and consistency loss based on smooth labels 32, combined with the data of related objects in the database 12 and the data of the object image acquisition device 11, unsupervised domain adaptation training is performed on the object pre-training model 21. Further, after iteratively updating the parameters, the corresponding long-tailed recognition model 22 is generated and deployed.
[0075] During the model usage phase, the object image acquisition device 11 acquires the object to be identified and uses the long-tail recognition model 22 to obtain the recognition result.
[0076] The present invention has been described by way of example in conjunction with the accompanying drawings. Obviously, the specific implementation of the present invention is not limited to the above-described manner. Any non-substantial improvement made by adopting the inventive concept and technical solution of the present invention, or the direct application of the inventive concept and technical solution of the present invention to other occasions without modification, shall be within the protection scope of the present invention.
Claims
1. A long-tailed distributed object recognition scheme integrating unsupervised domain adaptation technology, comprising an object recognition main system (A), characterized in that: The object recognition main system (A) includes a domain adaptation system (3), an object database (1), and a model center (2). The domain adaptation system (3) includes prototype-level contrastive learning (31) and consistency loss based on smooth labels (32). The object database (1) includes an object image acquisition device (11) and a database (12). The model center (2) includes an object pre-trained model (21) and a long-tail recognition model (22). The prototype-level contrastive learning (31) specifically includes the following process: Step 1: Constructing the feature space and prototype mapping: Using the classifier F of the long-tail recognition model (22) to support the classification of k categories, the source domain data DS and the unlabeled target domain data DT are mapped to the feature space; in the feature space, a cluster center is set for each category as the prototype of that category. ; Step 2: Calculate distance-based classification probability: Obtain the feature vector of the target domain sample. Through calculation With prototypes of various categories The Euclidean distance between the samples is used to measure the probability that a sample belongs to each category, and a k-dimensional probability vector is calculated. The probability of the k-th category The calculation method is as follows: ① in It's a hyperparameter; Step 3: Generate soft pseudo-labels: Based on the normalization of the predicted probabilities of all target domain samples in the k-th dimension, calculate the soft pseudo-labels for the target domain samples. The calculation method is as follows: ② in, yes The k-th dimension value, It is the number of all samples in the target domain. Based on all target domain samples in the k-th dimension The result of normalization is then normalized on the k categories; Step 4: Correcting the biased feature space through contrastive optimization: Constructing the contrastive loss function The feature space bias caused by long-tailed data is corrected by minimizing the sum of the cross-entropy loss of the source domain samples and the cross-entropy loss of the target domain samples with soft pseudo-labels. The formula is as follows: ③ in It is cross-entropy loss. , Let be a k-dimensional probability vector, representing the probability that a sample feature vector in the source / target domain is predicted to be each class in the feature space. This is the one-hot encoding of the source domain sample labels.
2. The long-tailed object recognition scheme integrating unsupervised domain adaptation technology according to claim 1, characterized in that: The consistency loss (32) based on smooth labels includes: ④ Where, x t The target domain sample is t, which is the strong augmentation technique RandAugment, and L is L. CE It is the cross-entropy loss, where p is the value of sample x. t The predicted probability value of a strongly augmented instance, where q is the probability of sample x. t The soft pseudo-label value of a weakly enhanced instance. Step 1: a. Use t to sample the target domain x t Perform strong enhancement to obtain instance t(x) t ) ; b. Use common image enhancement techniques, such as scaling, cropping, and flipping, to obtain weakly enhanced instances t of the target domain samples. w (x t ); c. Obtain a weakly enhanced instance t w (x t If the predicted probability value pw is used in the feature space, then p w Calculate using formula ②; if this consistency loss is used on the classifier, then p w The output value of the classifier; d. Using formula ③ with p w Calculate the soft pseudo-label value q for the weakly enhanced instance; e. Obtain the strongly enhanced instance t(x) t The predicted probability value p is calculated using formula ② if the consistency loss is used in the feature space; if the consistency loss is used on the classifier, p is the classifier output value. f. Use formula ④ to calculate the consistency loss based on smooth labels.