Methods, apparatus, equipment and readable storage media for calibration of long-tailed noise tags

Abstract

This application relates to a method, apparatus, device, and readable storage medium for correcting long-tailed noise labels. The method includes: obtaining the top K first confidence scores (from largest to smallest) in the text confidence vector of a target image in a dataset, and the top K second confidence scores (from largest to smallest) in the image confidence vector; determining a first accumulated confidence score based on the confidence scores in the text confidence vector excluding the first confidence scores, and determining a second accumulated confidence score based on the confidence scores in the image confidence vector excluding the second confidence scores; determining the cumulative frequency of the target category label based on the first accumulated confidence score, the second accumulated confidence score, the confidence scores of the target category label in the text confidence vector and image confidence vector, and the observation confidence score; and determining the corrected label corresponding to the observation label of the target image based on the cumulative frequency of each candidate category label, and determining the label distribution of the corrected label. This method can improve the correction effect of long-tailed noise labels.

Description

Technical Field

[0001] This application relates to the field of computer technology, and in particular to a method, apparatus, computer device, computer-readable storage medium, and computer program product for correcting long-tailed noise tags. Background Technology

[0002] Training deep learning models heavily relies on large-scale datasets. These datasets not only need precise labeling but also require a balanced distribution of labeled categories to ensure robust generalization capabilities. The rise of basic models has further amplified the demand for high-quality labeled data; insufficient labeling quality can introduce unexpected prior biases during training. In real-world scenarios, precise labeling typically requires extremely high human resources, and data collection itself can lead to distribution imbalances due to varying collection difficulties across different categories. Rare or underrepresented categories (tail classes) are particularly difficult to collect, resulting in extreme data skew and datasets simultaneously facing labeling errors (noisy labels) and extreme class imbalance (long-tail distribution). Using such datasets for model training severely weakens the model's generalization ability and significantly impairs its ability to identify tail classes.

[0003] To address the problem of both noisy labels and long-tailed distribution, traditional techniques first detect noisy samples and then apply long-tailed learning techniques to achieve uniform category distribution. However, this results in poor correction of the labels for high-noise tail classes. Summary of the Invention

[0004] Therefore, it is necessary to provide a correction method, apparatus, computer equipment, computer-readable storage medium, and computer program product for long-tailed noise tags that can improve the correction effect of tail-type tags, in order to address the above-mentioned technical problems.

[0005] In a first aspect, this application provides a method for correcting long-tailed noise tags, comprising:

[0006] Obtain the top K first confidence scores from the text confidence vector corresponding to the target image in the dataset, ranked from largest to smallest, and the top K second confidence scores from the image confidence vector corresponding to the target image, ranked from largest to smallest. The text confidence vector is obtained based on the similarity between the image features of the target image and each candidate category label in the dataset. The image confidence vector is obtained based on the similarity between the image features of the target image and each image classification weight. K is a positive integer.

[0007] Based on the confidence scores in the text confidence vector other than the first confidence score, a first cumulative confidence score for the target category label is determined, and based on the confidence scores in the image confidence vector other than the second confidence score, a second cumulative confidence score for the target category label is determined; the target category label can be any candidate category label.

[0008] Based on the first accumulated confidence, the second accumulated confidence, the confidence of the target category label in the text confidence vector, the confidence of the target category label in the image confidence vector, and the observed confidence of the target category label, the cumulative frequency of the target category label is determined.

[0009] Based on the cumulative frequency of each candidate category label, the correction label corresponding to the observation label of the target image is determined, and the label distribution of the correction label is determined.

[0010] In one embodiment, K is positively correlated with the number of candidate category labels.

[0011] In one embodiment, the method for determining the text confidence vector corresponding to the target image includes:

[0012] The candidate category labels are encoded using a text encoder to obtain the category label features for each candidate category label;

[0013] The target image is encoded using an image encoder to obtain its image features;

[0014] Based on the similarity between the image features of the target image and the category label features of each candidate category label, the text confidence vector corresponding to the target image is determined.

[0015] In one embodiment, the method for determining the image confidence vector corresponding to the target image includes:

[0016] The target image is encoded using an image encoder to obtain its image features;

[0017] The initial classifier is trained based on the image features of the target image until the classification confidence meets the preset conditions, thus obtaining the target classifier.

[0018] Based on the similarity between the image features of the target image and the class weights of the target classifier, the image confidence vector corresponding to the target image is determined.

[0019] In one embodiment, determining the correction label corresponding to the observation label of the target image based on the cumulative frequency of each candidate category label includes:

[0020] The candidate category label corresponding to the largest cumulative frequency among the cumulative frequencies of the above candidate category labels is determined as the correction label corresponding to the observation label of the above target image.

[0021] In one embodiment, the method further includes:

[0022] Based on the label distribution of the above correction labels, the prior probability of the candidate category label corresponding to the above correction labels is determined.

[0023] The label generation loss is determined based on the prior probabilities mentioned above, and the label generation model is trained based on the label generation loss mentioned above.

[0024] Secondly, this application also provides a correction device for long-tailed noise tags, comprising:

[0025] The confidence score acquisition module is used to acquire the top K first confidence scores in descending order of the text confidence score vector corresponding to the target image in the dataset, and the top K second confidence scores in descending order of the image confidence score vector corresponding to the target image; the text confidence score vector is obtained based on the similarity between the image features of the target image and each candidate category label in the dataset; the image confidence score vector is obtained based on the similarity between the image features of the target image and each image classification weight; K is a positive integer.

[0026] The confidence determination module is used to determine a first accumulated confidence of the target category label based on the confidence in the text confidence vector other than the first confidence, and to determine a second accumulated confidence of the target category label based on the confidence in the image confidence vector other than the second confidence; the target category label is any candidate category label;

[0027] The frequency determination module is used to determine the cumulative frequency distribution of the target category label based on the first accumulated confidence, the second accumulated confidence, the confidence of the target category label in the text confidence vector, the confidence of the target category label in the image confidence vector, and the observed confidence of the target category label.

[0028] The label correction module is used to determine the correction label corresponding to the observation label of the target image based on the cumulative frequency distribution of each candidate category label, and to determine the label distribution of the correction label.

[0029] Thirdly, this application also provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps of the long-tailed noise label correction method provided in the first aspect.

[0030] Fourthly, this application also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the long-tailed noise label correction method provided in the first aspect.

[0031] Fifthly, this application also provides a computer program product, including a computer program that, when executed by a processor, implements the steps of the long-tailed noise label correction method provided in the first aspect.

[0032] The aforementioned method, apparatus, computer equipment, computer-readable storage medium, and computer program product for correcting long-tailed noise labels obtain the top K first confidence scores (from largest to smallest) and the top K second confidence scores (from largest to smallest) of the text confidence scores corresponding to the target image in the dataset. Based on the confidence scores in the text confidence vector excluding the first confidence scores, a first accumulated confidence score for the target category label is determined. Furthermore, based on the confidence scores in the image confidence vector excluding the second confidence scores, a second accumulated confidence score is determined. Finally, based on the first accumulated confidence score, the second accumulated confidence score, and the target category label in the text confidence vector... The confidence level in the confidence vector, the confidence level in the image confidence vector, and the observed confidence level of the target category label are used to determine the cumulative frequency of the target category label. Based on the cumulative frequency of each candidate category label, the corrected label of the observed label of the target image is determined, and the label distribution of the corrected label is determined. Since both the first and second cumulative confidence levels are related to K, that is, they will be different with different K, the cumulative frequency of different candidate category labels will change with the change of K. This can achieve different degrees of correction for different categories of labels, thereby improving the correction effect of long-tailed noise labels. Attached Figure Description

[0033] To more clearly illustrate the technical solutions in the embodiments of this application or related technologies, the drawings used in the description of the embodiments of this application or related technologies will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0034] Figure 1 This is an application environment diagram of the long-tailed noise label correction method in one embodiment;

[0035] Figure 2 This is a flowchart illustrating a method for correcting long-tailed noise tags in one embodiment.

[0036] Figure 3 This is a flowchart illustrating a method for correcting long-tailed noise tags in another embodiment;

[0037] Figure 4This is a structural block diagram of a long-tailed noise tag correction device in one embodiment;

[0038] Figure 5 This is an internal structural diagram of a computer device in one embodiment. Detailed Implementation

[0039] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0040] It should be noted that the terms "first," "second," etc., used in this application can be used to describe various elements, but these elements are not limited by these terms. These terms are only used to distinguish the first element from the second element. The terms "comprising" and "having," and any variations thereof, used in this application, are intended to cover non-exclusive inclusion. The term "multiple" used in this application refers to two or more. The term "and / or" used in this application refers to one of the embodiments, or any combination of multiple embodiments.

[0041] The long-tailed noise tag correction method provided in this application embodiment can be applied to, for example, Figure 1In the application environment shown, terminal 102 communicates with server 104 via a network. A data storage system can store the data that server 104 needs to process. The data storage system can be integrated onto server 104, or it can be located in the cloud or on another network server. Terminal 102 can acquire the text confidence vector and image confidence vector corresponding to the target image, and send them to server 104. Server 104 can acquire the top K first confidence values ​​(from largest to smallest) in the text confidence vector and the top K second confidence values ​​(from largest to smallest) in the image confidence vector corresponding to the target image in the dataset. Based on the confidence values ​​in the text confidence vector excluding the first confidence values, server 104 determines the first accumulated confidence value of the target category label and the second accumulated confidence value of the target category label based on the confidence values ​​in the image confidence vector excluding the second confidence values. The target category label can be any candidate category label. Based on the first accumulated confidence value, the second accumulated confidence value, the confidence value of the target category label in the text confidence vector, the confidence value of the target category label in the image confidence vector, and the observed confidence value of the target category label, server 104 determines the cumulative frequency of the target category label. Based on the cumulative frequency of each candidate category label, server 104 determines the correction label corresponding to the observed label of the target image and determines the label distribution of the correction label. The terminal 102 can be, but is not limited to, various personal computers, laptops, smartphones, tablets, drones, low-altitude aircraft, IoT devices, and portable wearable devices. IoT devices can include smart speakers, smart TVs, smart air conditioners, smart in-vehicle devices, and projection devices. Portable wearable devices can include smartwatches, smart bracelets, and head-mounted displays. Head-mounted displays can be virtual reality (VR) devices, augmented reality (AR) devices, and smart glasses. The server 104 can be a standalone physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing cloud computing services. It should be noted that the long-tail noise label correction method provided in this application embodiment is applicable not only to application scenarios involving server and terminal interaction but also to application scenarios with a single server or a single terminal.

[0042] In one exemplary embodiment, such as Figure 2 As shown, a method for correcting long-tailed noise labels is provided, which can be applied to... Figure 1 Taking the server in the example, the explanation includes the following steps 202 to 208. Wherein:

[0043] Step 202: Obtain the top K first confidence scores in the text confidence vector corresponding to the target image in the dataset, from largest to smallest, and the top K second confidence scores in the image confidence vector corresponding to the target image, from largest to smallest. The text confidence vector is obtained based on the similarity between the image features of the target image and each candidate category label in the dataset; the image confidence vector is obtained based on the similarity between the image features of the target image and each image classification weight, where K is a positive integer.

[0044] The dataset is the data sample used for model training. It includes images and corresponding labels for each image. Understandably, a dataset typically contains a large number of images and corresponding labels for each image. These labels are also called the observation labels for that image. However, the labels for images in the dataset may be incorrectly labeled; these incorrectly labeled labels can be called noise labels. Furthermore, there may be uneven distribution of label categories, with some categories having more categories than others. This uneven distribution is called a long-tailed distribution, and the fewer categories in the long-tailed distribution are called the tail classes. The target image can be any image from the dataset.

[0045] Candidate category labels refer to the label categories included in the dataset. For example, in a dataset used to train an animal recognition model, the label categories could be cat, dog, and tiger. That is, the animal recognition model trained on this dataset would only recognize cats, dogs, and tigers. Therefore, the candidate category labels would include the label categories corresponding to cats, dogs, and tigers. A text confidence vector can include one or more text confidences, each representing the similarity between the image features of the target image and the corresponding candidate category label in the dataset. For example, the vector formed by the similarity between the image features of the target image and each candidate category label in the dataset can be used as the text confidence vector. An image confidence vector can include one or more image confidences, each representing the similarity between the image features of the target image and the corresponding image classification weight. For example, the vector formed by the similarity between the image features of the target image and each image classification weight can be used as the image confidence vector. Here, image classification weights refer to the category weights used to classify image features. Image classification weights can be determined by an image feature classifier. The number of image feature categories in the dataset is the same as the number of image classification weights.

[0046] In practical applications, text confidence scores in the text confidence vector are usually arranged in the order of candidate class labels, while image confidence scores in the image confidence vector are usually arranged in the order of image classification weights, not necessarily in order of confidence magnitude. From the text confidence vector, K text confidence scores are selected from the highest to the lowest, as the first confidence score. Similarly, from the image confidence vector, K image confidence scores are selected from the highest to the lowest, as the second confidence score. In other words, the first confidence score is the first K text confidence scores in the text confidence vector, and the second confidence score is the first K image confidence scores in the image confidence vector, in descending order. It's easy to understand that the value of K can be less than or equal to the number of candidate class labels.

[0047] Step 204: Determine the first accumulated confidence of the target category label based on the confidence in the text confidence vector excluding the first confidence, and determine the second accumulated confidence of the target category label based on the confidence in the image confidence vector excluding the second confidence; wherein, the target category label is any candidate category label.

[0048] The value of K varies depending on the number of candidate category labels. For example, K is larger for candidate category labels with a larger number of labels, and smaller for candidate category labels with a smaller number of labels. Therefore, the number of confidence values ​​(excluding the first confidence value) in the text confidence vector or the number of confidence values ​​(excluding the second confidence value) in the image confidence vector also differs depending on the number of candidate category labels with different numbers of labels.

[0049] In an optional embodiment, the absolute value of the difference between the sum of confidence scores (excluding the first confidence score) in the text confidence vector and 1 can be used as the first accumulated confidence score for the target category label, and the absolute value of the difference between the sum of confidence scores (excluding the second confidence score) in the image confidence vector and 1 can be used as the second accumulated confidence score for the target category label. For example, the first accumulated confidence score... And the second accumulated confidence level The calculation formula is shown in formula (1) below.

[0050] Formula (1)

[0051] in, This represents the j-th text confidence score in the text confidence vector, ranked from largest to smallest. This represents the confidence score of the j-th image in the image confidence vector, ranked from largest to smallest. C represents the number of candidate class labels, and Kc represents the K corresponding to the candidate class label c. For example, if the K corresponding to the candidate class label c is 3, then j starts taking values ​​from 4. If K is 2 for candidate category label c, then j starts from 3 and takes corresponding values. By analogy, we can obtain the first and second cumulative confidence scores for each candidate category label.

[0052] Step 206: Based on the first accumulated confidence, the second accumulated confidence, the confidence of the target category label in the text confidence vector, the confidence of the target category label in the image confidence vector, and the observed confidence of the target category label, determine the cumulative frequency of the target category label.

[0053] The observation confidence score characterizes the similarity between the target category label and the observed label. The observed label is the annotation label corresponding to the target image. Since the target category label and the observed label are either the same or different, the observation confidence score is typically 0 or 1. If the target category label and the observed label are the same, the observation confidence score for the target category label is 1; otherwise, if the target category label and the observed label are different, the observation confidence score for the target category label is 0. It is easy to understand that for each target image's observed label, only one candidate category label has an observation confidence score of 1, while the observation confidence scores for all other candidate category labels are 0. The cumulative frequency characterizes the cumulative confidence score of the number of target category labels. The cumulative frequency may be greater than 1.

[0054] In an optional embodiment, if the target category label is any candidate category label corresponding to the first confidence level or the second confidence level, and the target category label is the same as the observed label of the target image, then the accumulated frequency of the target category label is determined using the first accumulated confidence level, the second accumulated confidence level, the confidence level of the target category label in the text confidence vector, the confidence level of the target category label in the image confidence vector, and the observed confidence level of the target category label. If the target category label is different from both the candidate category labels corresponding to the first and second confidence levels, and the target category label is different from the observed label, then the accumulated frequency obtained in the previous round is used as the accumulated frequency of the current round. Accumulated frequency of the target category label The calculation formula can be shown in the following formula (2).

[0055] Formula (2)

[0056] Where e represents the accumulation round; This indicates the cumulative frequency of the target category labels obtained in the previous round of the current round; = It is the Kronecker delta function. If the target category label is the same as the observation label of the target image, the value is 1; if the target category label is different from the observation label of the target image, the value is 0. Indicates the first confidence level. Indicates the second confidence level; This represents the confidence level of the target category label in the text confidence vector; This represents the confidence level of the target category label in the image confidence vector. This represents the observed confidence level of the target category label. Similarly, the cumulative frequency of each candidate category label in each round can be obtained. This indicates an indicator function. The value is 1 if the target category label is the same as the observation label of the target image, and 0 if the target category label is different from the observation label of the target image.

[0057] If the observed label is the same as the candidate category label corresponding to either the first or second confidence level, the observed label is considered clean (i.e., low noise). In this case, the contribution of candidate category labels corresponding to lower confidence levels (i.e., label categories that are unlikely to be correct) can be suppressed, and the remaining confidence level is accumulated as frequency information of the observed category. Conversely, if the observed label is different from both the first and second confidence levels, the observed label is considered potential noise, and the corresponding accumulated confidence level will not be accumulated to update the accumulated frequency.

[0058] Step 208: Based on the cumulative frequency of each candidate category label, determine the correction label corresponding to the observation label of the target image, and determine the label distribution of the correction label.

[0059] In this context, the calibration label refers to the label of the target image used to correct the observation label. The calibration label is considered a more accurate label for the target image than the observation label. After determining the calibration label, the observation label of the target image can be replaced with the calibration label; that is, the determined calibration label is used as the new observation label for the target image.

[0060] For example, the candidate category label corresponding to the highest accumulated frequency among the accumulated frequencies of each candidate category label can be used as the correction label. The observation label of the target image is replaced by the correction label, and the label distribution of the correction label is determined. Here, the label distribution is used to characterize the number of labels.

[0061] For example, the distribution of correction labels can be determined statistically. That is, the distribution of correction labels can be obtained by counting the frequency of occurrence of correction labels in the dataset. For example, suppose the correction label is... Correcting the label distribution It can be obtained through the following formula (3).

[0062] Formula (3)

[0063] in, Indicates if the calibration label If the label is the same as the candidate category label c, the result is 1; if the correction label is... If the result is different from the candidate category label c, the result is 0. Each correction label is compared sequentially with the candidate category label c to obtain the label distribution of the correction labels corresponding to the candidate category label c.

[0064] In the aforementioned method for correcting long-tailed noise labels, the first accumulated confidence of the target category label is determined by obtaining the top K first confidence scores (from largest to smallest) in the text confidence scores of the target image in the dataset, and the top K second confidence scores (from largest to smallest) in the image confidence scores. The first accumulated confidence score of the target category label is determined based on the confidence scores in the text confidence vector excluding the first confidence scores, and the second accumulated confidence score is determined based on the confidence scores in the image confidence vector excluding the second confidence scores. Finally, the method is based on the first accumulated confidence score, the second accumulated confidence score, the confidence score of the target category label in the text confidence vector, and the confidence score in the image confidence vector. The confidence level in the confidence vector and the observed confidence level of the target category label are used to determine the cumulative frequency of the target category label. Based on the cumulative frequency of each candidate category label, the corrected label of the observed label of the target image is determined, and the label distribution of the corrected label is determined. Since both the first and second accumulated confidence levels are related to K, that is, they will be different with different K, the cumulative frequency of different candidate category labels will change with the change of K. This enables different degrees of correction for labels of different categories, thereby improving the correction effect of long-tailed noise labels.

[0065] In some embodiments, K is positively correlated with the number of candidate category labels.

[0066] In determining the cumulative frequency of the target category label, it is necessary to determine the first and second cumulative confidence levels based on the k largest first confidence levels in the text confidence vector and the k largest second confidence levels in the image confidence vector, respectively. Then, based on the first and second cumulative confidence levels, the confidence level of the target category label in the text confidence vector, the confidence level of the target category label in the image confidence vector, and the observed confidence level of the target category label, the cumulative frequency of the target category label is determined. This process is repeated to determine the cumulative frequency of each candidate category label. The number of each candidate category label varies, and therefore the corresponding K also varies.

[0067] K is positively correlated with the number of candidate category labels. In other words, the more candidate category labels there are, the larger the corresponding K value, and the fewer candidate category labels there are, the smaller the corresponding K value.

[0068] In this embodiment, by setting K to be positively correlated with the number of candidate category labels, it is possible to adaptively determine K based on the number of candidate category labels. A larger number of head category labels corresponds to a larger K, and a smaller number of tail category labels corresponds to a smaller K. Setting different K based on the differences in the number of candidate category labels can reduce the confirmation deviation of tail category labels and ensure the consistency of correction between different category labels.

[0069] In some embodiments, the method for determining the text confidence vector corresponding to the target image includes:

[0070] The candidate category labels are encoded using a text encoder to obtain the category label features of each candidate category label; the target image is encoded using an image encoder to obtain the image features of the target image; and the text confidence vector corresponding to the target image is determined based on the similarity between the image features of the target image and the category label features of each candidate category label.

[0071] Candidate category labels can be input into a text encoder in text form. The text encoder encodes each candidate category label sequentially to obtain the category label features of the corresponding candidate category label. Similarity can be represented by Euclidean distance, cosine similarity, Manhattan distance, or Mahalanobis distance.

[0072] For example, candidate category labels can be encoded using a text encoder of Vision-Language Models (VLMs) to obtain category label features for each candidate category label. Similarly, the target image can be encoded using an image encoder of a Vision-Language Model to obtain image features for the target image. Then, the cosine similarity between the image features of the target image and the category label features of each candidate category label is calculated. These cosine similarities are then arranged in a preset order to obtain a text confidence vector. The Vision-Language Model can be, for example, CLIP, ALIGN, or Flamingo, and the specific model chosen depends on the application scenario.

[0073] In one example, the image features of the target image x Category label features of candidate category labels Cosine similarity between As shown in formula (4) below.

[0074] Formula (4)

[0075] Where s represents the scale parameter.

[0076] Image features of target image x Category label features of candidate category label c Cosine similarity between Normalization is performed to obtain the text confidence scores corresponding to the candidate category labels. As shown in formula (5) below, the text confidence vector is obtained by dividing the text confidence scores corresponding to each candidate category label into the order of the candidate category labels. .

[0077] Formula (5)

[0078] In this embodiment, the text confidence vector of the target image is determined by the similarity between the image features obtained by encoding the target image using an image encoder and the category label features obtained by encoding the candidate category labels using a text encoder. This accurate determination of the text confidence vector of the target image lays a solid foundation for improving the correction effect of long-tailed noise labels. Semantic priors based on the alignment of the image with each category label can be obtained without adjusting the model parameters.

[0079] In some embodiments, the method for determining the image confidence vector corresponding to the target image includes:

[0080] The target image is encoded by an image encoder to obtain its image features; an initial classifier is trained based on the image features of the target image until the classification confidence meets the preset conditions to obtain the target classifier; the image confidence vector corresponding to the target image is determined based on the similarity between the image features of the target image and the class weights of the target classifier.

[0081] The initial classifier is used to classify image features. It can be a linear classifier, meaning it divides image features into different categories using a linear decision boundary. Examples of initial classifiers include Logistic Regression, Linear Support Vector Machine (SVM), Perceptron, and Naive Bayes. The initial classifier can be chosen based on the specific application scenario. Classification confidence characterizes the accuracy of the classification. Generally, higher classification accuracy corresponds to higher classification confidence. Class weights are the weights assigned to each image feature category by the classifier. Higher class weights indicate a higher probability that the classifier will classify the image feature into its corresponding category.

[0082] For example, the image features of the target image can be input into an initial classifier to obtain a predicted classification result. The classification confidence is determined based on the difference between the predicted and standard classification results. The initial classifier is then trained based on this confidence until the classification confidence reaches a confidence threshold, resulting in the target classifier. This allows the acquisition of the class weights of the target classifier. The confidence threshold can be set according to the specific application scenario.

[0083] In one example, the similarity between the image features of the target image and the class weights of the target classifier can be characterized by the cosine similarity between the image features of the target image and the class weights of the target classifier. Image features of target image x Class weights of the target classifier Cosine similarity between The calculation formula is shown in formula (6) below.

[0084] Formula (6)

[0085] Image features of target image x Class weights of the target classifier The cosine similarity between the two images is normalized to obtain the image confidence score corresponding to the class weight. As shown in formula (7) below, the image confidence vector is obtained by sorting the image confidence scores corresponding to each category weight according to the category weight order. It is easy to understand that the order of the candidate category labels is consistent with the order of the category weights.

[0086] Formula (7)

[0087] For example, the parameters of CLIP's image encoder can be fine-tuned using AdaptFormer (a lightweight and efficient fine-tuning method for adapting to vision-language models), thereby enhancing adaptability to downstream tasks while maintaining efficiency. It should be noted that the fine-tuning of the image encoder is used to adapt to the classification of image features from different datasets.

[0088] In this embodiment, the image confidence vector corresponding to the target image is determined by the similarity between the image features of the target image and the weights of each category of the target classifier. This enables accurate determination of the image confidence vector, laying a solid foundation for improving the correction effect of long-tail noise labels.

[0089] In some embodiments, the correction label corresponding to the observation label of the target image is determined based on the cumulative frequency of each candidate category label, including:

[0090] The candidate category label corresponding to the largest cumulative frequency among the cumulative frequencies of each candidate category label is determined as the correction label corresponding to the observation label of the target image.

[0091] After determining the text confidence vector and image confidence vector based on the target image, the cumulative frequency of each candidate category label is determined based on the top K first confidence scores in the text confidence vector (from largest to smallest) and the top K second confidence scores in the image confidence vector (from largest to smallest). Among the cumulative frequencies of each candidate category label, the candidate category label corresponding to the highest cumulative frequency is used as the correction label corresponding to the observation label of the target image. It is easy to understand that the above process can be iterated multiple times, with each iteration yielding a correction label for the target image.

[0092] For example, correction label It can be obtained through the following formula (8).

[0093] Formula (8)

[0094] In this embodiment, by using the candidate category label corresponding to the maximum cumulative frequency among the cumulative frequencies of each candidate category label as the correction label corresponding to the observation label of the target image, the accuracy of the correction label can be improved, thereby improving the correction effect of the long-tailed noise label.

[0095] In some embodiments, the above method further includes:

[0096] Based on the label distribution of the corrected label, the prior probability of the candidate category label corresponding to the corrected label is determined; the label generation loss is determined based on the prior probability, and the label generation model is trained based on the label generation loss.

[0097] The label generation model is the model used to generate labels. The generation loss is the training loss of the label generation model.

[0098] For example, the label generation loss is shown in the following formula (9).

[0099] Formula (9)

[0100] in, Source The prior probability of the calibration label is calculated. The prior probability of the calibration label is used to characterize the accuracy of the calibration label.

[0101] In this embodiment, the prior probability of the corresponding candidate category label is redetermined based on the label distribution of the corrected label, the label generation loss is determined based on the prior probability, and the label generation model is trained based on the label generation loss. Since the distribution of each candidate category label is corrected, the accuracy is higher and the distribution is more uniform, so that the model pays more attention to the learning of tail class labels during the training process, and the trained label generation model can generate more accurate tail class labels.

[0102] In an exemplary embodiment, the correction process for long-tailed noise labels can be implemented using a model. The model performing label correction can be called a Category-Aware Collaborative Expert (CACE), which includes text experts (TE), image experts (IE), and observation label experts (BE). Label correction is completed through a category-aware expert consensus mechanism.

[0103] Text Expert (TE): A CLIP-based text encoder encodes each candidate category label to obtain category label features. The target image is encoded using an image encoder to obtain its image features. Based on the cosine similarity between these image features and the category label features of each candidate category label, a text confidence vector is determined. The model can calculate the similarity between image features and candidate category labels, and the model parameters do not require additional fine-tuning.

[0104] Image Expert (IE): A CLIP-based image encoder encodes an input image x to obtain a visual representation f (i.e., image features) adapted to the task scenario. To enhance adaptability to downstream tasks while maintaining efficiency, an AdaptFormer is used for parameter fine-tuning. This fine-tuning is only used for the final classification. The fine-tuned image encoder is reused for label correction during the expert consensus stage without introducing additional training objectives. Cosine similarity between image features and class weights is calculated, and image confidence vectors are determined based on these cosine similarities. .

[0105] Observation Label Experts (BE): Observation label experts, also known as foundation experts, rely directly on the original (potentially noisy) label annotations. Although they are unreliable, these labels still contain valuable supervision, especially for head classes containing a large number of accurate labels. Observation Labels This is represented as a one-hot confidence vector (i.e., the observation confidence vector). Only of them The entry is set to 1. That is, in the observation confidence vector of the target image, the observation confidence is 1 when the observation label is the same as the candidate category label, and 0 otherwise. There is only one position in the observation confidence vector with an observation confidence of 1, and the observation confidence of the other positions is 0.

[0106] Category-aware expert consensus: To evaluate the reliability of observed labels, the top-k confidence scores are identified from both TE and IE, resulting in... ,in , This represents the confidence level from the relevant experts. To improve robustness to noisy labels and long-tailed category distributions, a category-aware top-K consensus mechanism is introduced. K can be adaptively determined based on the number of candidate category labels. Specifically, candidate label categories with higher sample counts (head classes) are assigned a larger K, while candidate category labels with lower sample counts (tail classes) are assigned a smaller K, allowing the model to allocate confidence levels more appropriately across categories. Formally, the category-specific K is defined as proportional to the number of candidate category labels, i.e. ,in This represents the sample count for candidate category label c at round e. In the initial stage (i.e., e=0), Calculated based on the number of observed labels. For e > 0, Update using the calibration labels from the previous training round.

[0107] Category frequency accumulation: Let This represents the cumulative frequency vector of the target image x. For each candidate class label... F can be updated to:

[0108]

[0109] Noise label correction and long-tail correction:

[0110] During the dynamic accumulation process, the dynamic correction label of sample x is obtained from the accumulation frequency distribution of each round and used for model supervision. Let... The correction label for round e: Based on the corrected labels, the label distribution of category c can be recalculated. .

[0111] like Figure 3The label correction process for a single input image (target image) is illustrated below. Candidate category labels from the dataset are input into the model's text encoder for encoding, resulting in category label features for each candidate category label. The target image is input into the image encoder for encoding, resulting in image features for the target image. The similarity between the target image's image features and the category label features of each candidate category label is calculated, determining the text confidence vector as {0.4 0.2 0.2 0.1 0.1}. Based on the similarity between the target image's image features and the classification weights of each image, the image confidence vector is determined as {0.05 0.4 0.3 0.2 0.05}. Assuming K is 3, the top-K confidence scores from the text confidence vector are selected as {0.4 0.2 0.2} (first confidence score), and the top-K confidence scores from the image confidence vector are selected as {0.4 0.3 0.2} (second confidence score). Based on the first confidence level, a first cumulative frequency is determined. If the candidate category label does not belong to the top-K confidence level, the frequency accumulated by the text expert is 0. Based on the second confidence level, a second cumulative frequency is determined. If the candidate category label does not belong to the top-K confidence level, the frequency accumulated by the image expert is also 0. Based on the first cumulative frequency, the second cumulative frequency, and the observation confidence level, the cumulative frequencies of each candidate category label are determined as {1.8 0.2 0.5 0.2 0}. This process continues, allowing the cumulative frequency of each candidate type to be determined in each training round. This enables the determination of the corrected label for the observed label of the target image in each training round, and the determination of the distribution of the corrected label until the distribution of the corrected label meets preset conditions. For example, if the distribution of the corrected label does not belong to a long-tailed distribution and the noise rate (the proportion of noise labels) is lower than a preset threshold, then label correction is stopped, and the corrected label is obtained.

[0112] In the above embodiments, low-noise samples are identified using a top-K voting strategy for each category without introducing additional parameters. A larger K value is used for the head class and a smaller K value for the tail class, reducing confirmation bias in the tail class. Low-confidence categories are filtered out to ensure consistency between categories. This allows the model to generate a more reliable label distribution in a lightweight manner, significantly reducing both the overall noise rate and the head class noise rate, while also correcting noisy labels in the tail class, thus improving the correction effect of long-tail noisy labels. Based on the corrected labels, a label distribution closer to the real situation can be statistically analyzed as corrected prior information. Combined with various long-tail learning methods, this allows for better learning of the characteristics of tail class labels, resulting in better tail class identification performance.

[0113] It should be understood that although the steps in the flowcharts of the embodiments described above are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the embodiments described above may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the steps or stages in other steps. It is understood that the steps in different embodiments can be freely combined as needed, and all non-contradictory solutions formed by such combinations are within the scope of protection of this application.

[0114] Based on the same inventive concept, this application also provides a long-tailed noise tag correction device for implementing the above-described long-tailed noise tag correction method. The solution provided by this device is similar to the solution described in the above method; therefore, the specific limitations of one or more long-tailed noise tag correction device embodiments provided below can be found in the limitations of the long-tailed noise tag correction method described above, and will not be repeated here.

[0115] In one exemplary embodiment, such as Figure 4 As shown, a long-tailed noise tag correction device 400 is provided, including: a confidence acquisition module 402, a confidence determination module 404, a frequency determination module 406, and a tag correction module 408, wherein:

[0116] The confidence acquisition module 402 is used to acquire the top K first confidence scores in descending order from the text confidence vector corresponding to the target image in the dataset, and the top K second confidence scores in descending order from the image confidence vector corresponding to the target image; the text confidence vector is obtained based on the similarity between the image features of the target image and each candidate category label in the dataset; the image confidence vector is obtained based on the similarity between the image features of the target image and each image classification weight; K is a positive integer;

[0117] The confidence determination module 404 is used to determine the first accumulated confidence of the target category label based on the confidence in the text confidence vector excluding the first confidence, and to determine the second accumulated confidence of the target category label based on the confidence in the image confidence vector excluding the second confidence; the target category label is any candidate category label;

[0118] The frequency determination module 406 is used to determine the cumulative frequency distribution of the target category label based on the first accumulated confidence, the second accumulated confidence, the confidence of the target category label in the text confidence vector, the confidence of the target category label in the image confidence vector, and the observed confidence of the target category label.

[0119] The label correction module 408 is used to determine the correction label corresponding to the observation label of the target image based on the cumulative frequency distribution of each candidate category label, and to determine the label distribution of the correction label.

[0120] In some embodiments, the above apparatus further includes a text confidence determination module, configured to encode candidate category labels using a text encoder to obtain category label features for each candidate category label; encode the target image using an image encoder to obtain image features for the target image; and determine the text confidence vector corresponding to the target image based on the similarity between the image features of the target image and the category label features of each candidate category label.

[0121] In some embodiments, the above-described apparatus further includes an image confidence determination module, configured to encode the target image using an image encoder to obtain image features of the target image; train an initial classifier based on the image features of the target image until the classification confidence meets a preset condition to obtain a target classifier; and determine the image confidence vector corresponding to the target image based on the similarity between the image features of the target image and the class weights of the target classifier.

[0122] In some embodiments, the label correction module 408 is further configured to determine the candidate category label corresponding to the largest cumulative frequency among the cumulative frequencies of each candidate category label as the correction label corresponding to the observation label of the target image.

[0123] In some embodiments, the above apparatus further includes a label generation module, configured to determine the prior probability of the candidate category label corresponding to the corrected label based on the label distribution of the corrected label; determine the label generation loss based on the prior probability; and train a label generation model based on the label generation loss.

[0124] Each module in the aforementioned long-tailed noise tag correction device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device, or stored in the memory of a computer device as software, so that the processor can call and execute the corresponding operations of each module.

[0125] In one exemplary embodiment, a computer device is provided, which may be a server, and its internal structure diagram may be as follows: Figure 5As shown, this computer device includes a processor, memory, input / output interfaces (I / O), and a communication interface. The processor, memory, and I / O interfaces are connected via a system bus, and the communication interface is also connected to the system bus via the I / O interfaces. The processor provides computational and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system, computer programs, and a database. The internal memory provides the environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The database stores correction-related data for long-tailed noise tags. The I / O interfaces are used for exchanging information between the processor and external devices. The communication interface is used for communication with external terminals via a network connection. When the computer program is executed by the processor, it implements a correction method for long-tailed noise tags.

[0126] Those skilled in the art will understand that Figure 5 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0127] In one exemplary embodiment, a computer device is provided, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps in the above-described method embodiments.

[0128] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps in the above-mentioned method embodiments are implemented.

[0129] In one embodiment, a computer program product is provided, including a computer program that, when executed by a processor, implements the steps in the above method embodiments.

[0130] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of the relevant data must comply with relevant regulations.

[0131] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. Any references to memory, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile memory and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM). The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, artificial intelligence (AI) processors, etc., and are not limited to these.

[0132] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this application.

[0133] The above-described embodiments merely represent several implementation methods of the present application. While the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the present application. It should be noted that a person of ordinary skill in the art may make various modifications and improvements without departing from the spirit of the present application, and these modifications and improvements fall within the scope of protection of the present application. Therefore, the scope of protection of the present application shall be determined by the appended claims.

Claims

1. A method for correcting long-tailed noise tags, characterized in that, The method includes: Obtain the top K first confidence scores (in descending order) from the text confidence vector corresponding to the target image in the dataset, and the top K second confidence scores (in descending order) from the image confidence vector corresponding to the target image; the text confidence vector is obtained based on the similarity between the image features of the target image and each candidate category label in the dataset; the image confidence vector is obtained based on the similarity between the image features of the target image and each image classification weight; K is a positive integer; Based on the confidence scores in the text confidence vector excluding the first confidence score, a first accumulated confidence score for the target category label is determined, and based on the confidence scores in the image confidence vector excluding the second confidence score, a second accumulated confidence score for the target category label is determined; the target category label can be any candidate category label. Based on the first accumulated confidence score, the second accumulated confidence score, the confidence score of the target category label in the text confidence vector, the confidence score of the target category label in the image confidence vector, and the observed confidence score of the target category label, the cumulative frequency of the target category label is determined; Based on the cumulative frequency of each candidate category label, the correction label corresponding to the observation label of the target image is determined, and the label distribution of the correction label is determined.

2. The method according to claim 1, characterized in that, The number of candidate category labels is positively correlated with the number of K.

3. The method according to claim 1 or 2, characterized in that, The method for determining the text confidence vector corresponding to the target image includes: The candidate category labels are encoded using a text encoder to obtain the category label features for each candidate category label; The target image is encoded using an image encoder to obtain its image features; Based on the similarity between the image features of the target image and the category label features of each candidate category label, the text confidence vector corresponding to the target image is determined.

4. The method according to claim 1 or 2, characterized in that, The method for determining the image confidence vector corresponding to the target image includes: The target image is encoded using an image encoder to obtain its image features; The initial classifier is trained based on the image features of the target image until the classification confidence meets the preset conditions, thus obtaining the target classifier; Based on the similarity between the image features of the target image and the class weights of the target classifier, the image confidence vector corresponding to the target image is determined.

5. The method according to claim 1, characterized in that, The step of determining the correction label corresponding to the observation label of the target image based on the cumulative frequency of each candidate category label includes: The candidate category label corresponding to the largest cumulative frequency among the cumulative frequencies of each candidate category label is determined as the correction label corresponding to the observation label of the target image.

6. The method according to claim 1, characterized in that, The method further includes: Based on the label distribution of the correction label, determine the prior probability of the candidate category label corresponding to the correction label; The label generation loss is determined based on the prior probability, and the label generation model is trained based on the label generation loss.

7. A calibration device for a long-tailed noise tag, characterized in that, The device includes: The confidence score acquisition module is used to acquire the top K first confidence scores in descending order of the text confidence score vector corresponding to the target image in the dataset, and the top K second confidence scores in descending order of the image confidence score vector corresponding to the target image; the text confidence score vector is obtained based on the similarity between the image features of the target image and each candidate category label in the dataset; the image confidence score vector is obtained based on the similarity between the image features of the target image and each image classification weight; K is a positive integer; The confidence determination module is used to determine a first cumulative confidence of the target category label based on the confidence in the text confidence vector excluding the first confidence, and to determine a second cumulative confidence of the target category label based on the confidence in the image confidence vector excluding the second confidence; the target category label is any candidate category label; The frequency determination module is used to determine the cumulative frequency distribution of the target category label based on the first accumulated confidence, the second accumulated confidence, the confidence of the target category label in the text confidence vector, the confidence of the target category label in the image confidence vector, and the observed confidence of the target category label; The label correction module is used to determine the correction label corresponding to the observation label of the target image based on the cumulative frequency distribution of each candidate category label, and to determine the label distribution of the correction label.

8. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 6.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 6.

10. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 6.

Images