Long-tail noise tag correction method, device and equipment and readable storage medium

Abstract

The invention relates to a long-tail noise tag correction method, device and equipment and a readable storage medium. The method comprises the following steps: acquiring first K first confidence coefficients from large to small in a text confidence coefficient vector of a target image in a data set and first K second confidence coefficients from large to small in an image confidence coefficient vector; determining a first accumulation confidence coefficient according to confidence coefficients except the first confidence coefficient in the text confidence coefficient vector, and determining a second accumulation confidence coefficient according to confidence coefficients except the second confidence coefficient in the image confidence coefficient vector; based on the first accumulation confidence coefficient, the second accumulation confidence coefficient, the confidence coefficient of the target category label in the text confidence coefficient vector and the image confidence coefficient vector, and the observation confidence coefficient, determining the accumulation frequency of the target category label; and based on the cumulative frequency of each candidate category label, determining a correction label corresponding to the observation label of the target image, and determining label distribution of the correction label. By adopting the method, the correction effect of the tail class label can be improved.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a method, apparatus, computer device, computer-readable storage medium, and computer program product for correcting long-tail noise labels. Background Art

[0002] Training deep learning models relies heavily on large-scale datasets. Training datasets must not only be accurately labeled but also maintain a balanced distribution of labeled categories to ensure robust generalization. The rise of foundational models has further intensified the demand for high-quality labeled data. Inadequate labeling can introduce unintended prior biases during training. In real-world scenarios, accurate labeling often requires significant labor costs, and data collection itself can lead to distribution imbalances due to varying difficulty levels across different categories. Rare or underrepresented categories (tail categories) are particularly difficult to collect, leading to extreme data skew and resulting in datasets that are both labeled incorrectly (noisy labels) and extremely imbalanced (long-tail distribution). Using such datasets for model training can severely weaken the model's generalization and severely impair its ability to identify tail categories.

[0003] To address the problem of the coexistence of noisy labels and long-tail distribution, traditional technology first detects noisy samples and then applies long-tail learning technology to achieve uniform distribution of categories. This results in poor correction effect on high-noise tail class labels. Summary of the Invention

[0004] Based on this, it is necessary to provide a correction method, device, computer equipment, computer-readable storage medium and computer program product for long-tail noise labels that can improve the correction effect of tail class labels in order to address the above technical problems.

[0005] In a first aspect, the present application provides a method for correcting long-tail noise labels, comprising:

[0006] Obtain the top K first confidences in descending order from the text confidence vector corresponding to the target image in the dataset, and the top K second confidences in descending order from the image confidence vector corresponding to the target image; the text confidence vector is obtained based on the similarity between the image features of the target image and each candidate category label in the dataset; the image confidence vector is obtained based on the similarity between the image features of the target image and each image classification weight; K is a positive integer;

[0007] Determining a first accumulated confidence of a target category label based on the confidences other than the first confidence in the text confidence vector, and determining a second accumulated confidence of the target category label based on the confidences other than the second confidence in the image confidence vector; the target category label is any candidate category label;

[0008] Determining a cumulative frequency of the target category label based on the first accumulated confidence, the second accumulated confidence, the confidence of the target category label in the text confidence vector, the confidence of the target category label in the image confidence vector, and the observed confidence of the target category label;

[0009] Based on the cumulative frequency of each candidate category label, the correction label corresponding to the observation label of the target image is determined, and the label distribution of the correction label is determined.

[0010] In one embodiment, the above K is positively correlated with the number of candidate category labels.

[0011] In one embodiment, the method for determining the text confidence vector corresponding to the target image includes:

[0012] Encode the candidate category labels through the text encoder to obtain the category label features of each candidate category label;

[0013] Encoding the target image through an image encoder to obtain image features of the target image;

[0014] Based on the similarity between the image features of the target image and the category label features of each candidate category label, a text confidence vector corresponding to the target image is determined.

[0015] In one embodiment, a method for determining the image confidence vector corresponding to the target image includes:

[0016] Encoding the target image through an image encoder to obtain image features of the target image;

[0017] The initial classifier is trained based on the image features of the target image until the classification confidence meets the preset conditions, thereby obtaining the target classifier;

[0018] Based on the similarity between the image features of the target image and the category weights of the target classifier, an image confidence vector corresponding to the target image is determined.

[0019] In one embodiment, determining the correction label corresponding to the observed label of the target image based on the cumulative frequency of each candidate category label includes:

[0020] The candidate category label corresponding to the maximum cumulative frequency among the cumulative frequencies of the candidate category labels is determined as the correction label corresponding to the observation label of the target image.

[0021] In one embodiment, the method further includes:

[0022] Based on the label distribution of the above-mentioned correction labels, determining the prior probability of the candidate category labels corresponding to the above-mentioned correction labels;

[0023] The label generation loss is determined based on the prior probability, and the label generation model is trained based on the label generation loss.

[0024] In a second aspect, the present application also provides a device for correcting long-tail noise labels, comprising:

[0025] A confidence acquisition module is configured to acquire the top K first confidences, from large to small, in the text confidence vector corresponding to the target image in the data set, and the top K second confidences, from large to small, in the image confidence vector corresponding to the target image; the text confidence vector is obtained based on the similarity between the image features of the target image and each candidate category label in the data set; the image confidence vector is obtained based on the similarity between the image features of the target image and each image classification weight; and K is a positive integer;

[0026] a confidence determination module, configured to determine a first accumulated confidence of a target category label based on confidences other than the first confidence in the text confidence vector, and to determine a second accumulated confidence of the target category label based on confidences other than the second confidence in the image confidence vector; the target category label being any candidate category label;

[0027] a frequency determination module, configured to determine a cumulative frequency distribution of the target category label based on the first accumulated confidence, the second accumulated confidence, the confidence of the target category label in the text confidence vector, the confidence of the target category label in the image confidence vector, and the observed confidence of the target category label;

[0028] The label correction module is used to determine the correction label corresponding to the observation label of the above-mentioned target image based on the cumulative frequency distribution of each candidate category label, and determine the label distribution of the above-mentioned correction label.

[0029] In a third aspect, the present application further provides a computer device comprising a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the steps of the method for correcting long-tail noise labels provided in the first aspect are implemented.

[0030] In a fourth aspect, the present application further provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the method for correcting long-tail noise labels provided in the first aspect.

[0031] In a fifth aspect, the present application further provides a computer program product, comprising a computer program, which, when executed by a processor, implements the steps of the method for correcting long-tail noise labels provided in the first aspect.

[0032] The correction method, device, computer equipment, computer-readable storage medium and computer program product of the long-tail noise label are as follows: by obtaining the first K first confidences from large to small in the text confidence corresponding to the target image in the data set and the first K second confidences from large to small in the image confidence, determining the first accumulated confidence of the target category label according to the confidences other than the first confidence in the text confidence vector, and determining the second accumulated confidence according to the confidences other than the second confidence in the image confidence vector, and based on the first accumulated confidence, the second accumulated confidence and the target category label in the text position. The confidence in the confidence vector and the confidence in the image confidence vector, the observation confidence of the target category label, determine the cumulative frequency of the target category label, and determine the correction label of the observation label of the target image according to the cumulative frequency of each candidate category label, and determine the label distribution of the correction label. Since the first cumulative confidence and the second cumulative confidence are both related to K, that is, they will be different with different K, the cumulative frequency of different candidate category labels will change with the change of K, which can achieve different degrees of correction for different categories of labels, thereby improving the correction effect of long-tail noise labels. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] In order to more clearly illustrate the technical solutions in the embodiments of the present application or related technologies, the following briefly introduces the drawings required for use in the embodiments of the present application or related technical descriptions. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying any creative work.

[0034] Figure 1 FIG2 is an application environment diagram of a method for correcting long-tail noise labels in one embodiment;

[0035] Figure 2 1 is a flow chart of a method for correcting long-tail noise labels in one embodiment;

[0036] Figure 3 2 is a flow chart of a method for correcting long-tail noise labels in another embodiment;

[0037] Figure 41 is a structural block diagram of a device for correcting long-tail noise labels in one embodiment;

[0038] Figure 5 FIG. 1 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION

[0039] In order to make the purpose, technical solutions and advantages of this application more clear, the following further describes this application in detail with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain this application and are not intended to limit this application.

[0040] It should be noted that the terms "first", "second", etc. used in this application may be used to describe various elements, but these elements are not limited by these terms. These terms are only used to distinguish the first element from the second element. The terms "including" and "having" used in this application and any variations thereof are intended to cover non-exclusive inclusions. The term "plurality" used in this application refers to two or more. The term "and / or" used in this application refers to one of the solutions or any combination of multiple solutions.

[0041] The correction method for long-tail noise labels provided in the embodiment of the present application can be applied to Figure 1In the application environment shown, the terminal 102 communicates with the server 104 via a network. The data storage system can store data that the server 104 needs to process. The data storage system can be integrated on the server 104 or placed on the cloud or other network servers. The terminal 102 can obtain the text confidence vector and the image confidence vector corresponding to the target image, and send the text confidence vector and the image confidence vector corresponding to the target image to the server 104. The server 104 can obtain the top K first confidences from large to small in the text confidence vector corresponding to the target image in the data set, and the top K second confidences from large to small in the image confidence vector, and determine the first accumulated confidence of the target category label based on the confidences other than the first confidence in the text confidence vector, and determine the second accumulated confidence of the target category label based on the confidences other than the second confidence in the image confidence vector; the target category label is any candidate category label; based on the first accumulated confidence, the second accumulated confidence, the confidence of the target category label in the text confidence vector, the confidence of the target category label in the image confidence vector and the observed confidence of the target category label, determine the cumulative frequency of the target category label; based on the cumulative frequencies of each candidate category label, determine the correction label corresponding to the observed label of the target image, and determine the label distribution of the correction label. Terminal 102 may be, but is not limited to, various personal computers, laptops, smartphones, tablets, drones, low-altitude aircraft, IoT devices, and portable wearable devices. IoT devices may include smart speakers, smart TVs, smart air conditioners, smart car devices, projectors, and the like. Portable wearable devices may include smart watches, smart bracelets, head-mounted devices, and the like. Head-mounted devices may include virtual reality (VR) devices, augmented reality (AR) devices, smart glasses, and the like. Server 104 may be an independent physical server, a server cluster or distributed system consisting of multiple physical servers, or a cloud server providing cloud computing services. It should be noted that the long-tail noise label correction method provided in the embodiments of the present application is applicable not only to application scenarios where a server and terminal interact, but also to application scenarios where a single server or a single terminal is used.

[0042] In an exemplary embodiment, Figure 2 As shown in the figure, a correction method for long-tail noise labels is provided. Figure 1 The server in the example is used to illustrate the process, including the following steps 202 to 208.

[0043] Step 202: Obtain the top K first confidences from large to small in the text confidence vector corresponding to the target image in the data set, and the top K second confidences from large to small in the image confidence vector corresponding to the target image; wherein the text confidence vector is obtained based on the similarity between the image features of the target image and each candidate category label in the data set; the image confidence vector is obtained based on the similarity between the image features of the target image and each image classification weight, and K is a positive integer.

[0044] Among them, the dataset is the data sample used for model training. The dataset includes images and labels for the corresponding images. It is easy to understand that the dataset usually includes a large number of images and the corresponding labeled labels for each image. The labeled labels corresponding to the images are also called the observed labels of the images. However, the labels corresponding to the images in the dataset may be incorrectly labeled, and the incorrectly labeled labels can be called noise labels. At the same time, there may also be uneven label categories, some label categories are more, and some label categories are less. This phenomenon of uneven distribution of label categories is called long-tail distribution, and the label categories with a smaller number in the long-tail distribution are called tail categories. The target image can be any image in the dataset.

[0045] Candidate category labels refer to the label categories included in a dataset. For example, a dataset used to train an animal recognition model may include the label categories cat, dog, and tiger. This means that the animal recognition model trained based on the dataset is only used to identify cats, dogs, and tigers. Therefore, the candidate category labels include the label categories corresponding to cats, dogs, and tigers. A text confidence vector may include one or more text confidences, each of which represents the similarity between the image features of a target image and the corresponding candidate category labels in the dataset. For example, a vector consisting of the similarities between the image features of a target image and each candidate category label in the dataset can be used as a text confidence vector. An image confidence vector may include one or more image confidences, each of which represents the similarity between the image features of a target image and the corresponding image classification weights. For example, a vector consisting of the similarities between the image features of a target image and each image classification weight can be used as an image confidence vector. Image classification weights refer to the category weights used to classify image features. Image classification weights can be determined by an image feature classifier. The number of image feature categories for images included in a dataset is the same as the number of image classification weights.

[0046] In actual application scenarios, the text confidences in the text confidence vector are usually arranged in the order of candidate category labels, and the image confidences in the image confidence vector are usually arranged in the order of image classification weights, that is, they are usually not arranged in the order of confidence size. From the various text confidences in the text confidence vector, K text confidences are selected from the largest text confidence to the smallest as the first confidence. And from the various image confidences in the image confidence vector, K image confidences are selected from the largest image confidence to the smallest as the second confidence. In other words, the first confidence is the first K text confidences in the text confidence vector from the largest to the smallest, and the second confidence is the first K image confidences in the image confidence vector from the largest to the smallest. It is easy to understand that the value of K can be less than or equal to the number of candidate category labels.

[0047] Step 204: Determine a first accumulated confidence of the target category label based on the confidences other than the first confidence in the text confidence vector, and determine a second accumulated confidence of the target category label based on the confidences other than the second confidence in the image confidence vector; wherein the target category label is any candidate category label.

[0048] Among them, different candidate category labels correspond to different K. For example, for candidate category labels with a large number of labels, the corresponding K is larger, and for candidate category labels with a small number of labels, the corresponding K is smaller. Therefore, for candidate category labels with different numbers of labels, the number of confidence levels other than the first confidence level in the corresponding text confidence vector or the number of confidence levels other than the second confidence level in the corresponding image confidence vector will also be different.

[0049] In an optional embodiment, the absolute value of the difference between the sum of the confidences in the text confidence vector except the first confidence and 1 can be used as the first accumulated confidence of the target category label, and the absolute value of the difference between the sum of the confidences in the image confidence vector except the second confidence and 1 can be used as the second accumulated confidence of the target category label. and the second cumulative confidence The calculation formula is shown in the following formula (1).

[0050] Formula (1)

[0051] in, Represents the jth text confidence in the text confidence vector from large to small, Indicates the jth image confidence in the image confidence vector from large to small. C represents the number of candidate class labels, and Kc represents the K corresponding to the corresponding candidate class label c. For example, if the K corresponding to the candidate class label c is 3, then j starts from 4. , if the K corresponding to the candidate category label c is 2, then j starts from 3 and takes the corresponding value By analogy, the first accumulated confidence and the second accumulated confidence corresponding to each candidate category label can be obtained.

[0052] Step 206 : Determine the cumulative frequency of the target category label based on the first accumulated confidence, the second accumulated confidence, the confidence of the target category label in the text confidence vector, the confidence of the target category label in the image confidence vector, and the observed confidence of the target category label.

[0053] Among them, the observation confidence is used to characterize the similarity between the target category label and the observation label. The observation label is the annotation label corresponding to the target image. Since the target category label and the observation label are either the same or different, the observation confidence is usually 0 or 1. If the target category label is the same as the observation label, the observation confidence of the target category label is 1, otherwise, if the target category label is different from the observation label, the observation confidence of the target category label is 0. It is easy to understand that for the observation label of each target image, only one candidate category label has an observation confidence of 1, and the observation confidence of other candidate category labels is 0. The cumulative frequency is used to characterize the cumulative confidence of the number of target category labels. The cumulative frequency may be greater than 1.

[0054] In an optional embodiment, if the target category label is any candidate category label corresponding to the first confidence or the second confidence, and the target category label is the same as the observed label of the target image, then the cumulative frequency of the target category label is determined by the first accumulated confidence, the second accumulated confidence, the confidence of the target category label in the text confidence vector, the confidence of the target category label in the image confidence vector, and the observed confidence of the target category label. If the target category label is different from the candidate category label corresponding to the first confidence and the second confidence, and the target category label is different from the observed label, then the accumulated frequency obtained in the previous round is used as the accumulated frequency of the current round. Cumulative frequency of target category label The calculation formula can be shown as the following formula (2).

[0055] Formula (2)

[0056] Among them, e represents the round of accumulation; Indicates the cumulative frequency of target category labels obtained in the previous round of the current round; = is the Kronecker delta function, which takes a value of 1 if the target class label is the same as the observed label of the target image, and a value of 0 if the target class label is different from the observed label of the target image; represents the first confidence level, represents the second confidence level; Represents the confidence of the target category label in the text confidence vector; represents the confidence of the target category label in the image confidence vector, Represents the observed confidence of the target category label. Similarly, the cumulative frequency of each candidate category label in each round can be obtained. Represents the indicator function, which takes the value 1 if the target category label is the same as the observed label of the target image, and takes the value 0 if the target category label is different from the observed label of the target image.

[0057] If the observed label is the same as the candidate class label corresponding to either the first or second confidence level, the observed label is considered clean (i.e., low noise). In this case, the contribution of the candidate class labels corresponding to the lower confidence levels (i.e., less likely to be correct labels) can be suppressed, and the remaining confidence levels are accumulated as the frequency information of the observed class. Conversely, if the observed label is different from both the first and second confidence levels, the observed label is considered potential noise, and the corresponding accumulated confidence levels are not accumulated to update the accumulated frequency.

[0058] Step 208 : Based on the cumulative frequencies of the candidate category labels, the correction labels corresponding to the observed labels of the target image are determined, and the label distribution of the correction labels is determined.

[0059] The correction label is the label of the target image used to correct the observed label. The correction label is considered a more accurate label for the target image than the observed label. After the correction label is determined, the observed label of the target image can be replaced by the correction label. That is, the determined correction label is used as the new observed label for the target image.

[0060] For example, the candidate category label corresponding to the maximum cumulative frequency among the cumulative frequencies of the candidate category labels can be used as a correction label, the observed label of the target image is replaced by the correction label, and the label distribution of the correction label is determined. The label distribution is used to represent the number of labels.

[0061] For example, the label distribution of the correction label can be determined based on statistics. That is, the distribution of the correction label can be obtained by counting the number of times the correction label appears in the data set. For example, assuming the correction label is , label distribution of the corrected labels It can be obtained by the following formula (3).

[0062] Formula (3)

[0063] in, Indicates that if the calibration label The result is 1 if the corrected label is the same as the candidate category label c. If it is different from the candidate class label c, the result is 0. Compare each correction label with the candidate class label c in turn to obtain the label distribution of the correction label corresponding to the corresponding candidate class label c.

[0064] In the correction method for the long-tail noise label, the first K first confidences from large to small in the text confidence corresponding to the target image in the data set and the first K second confidences from large to small in the image confidence are obtained, and the first accumulated confidence of the target category label is determined according to the confidence in the text confidence vector except the first confidence, and the second accumulated confidence is determined according to the confidence in the image confidence vector except the second confidence, based on the first accumulated confidence, the second accumulated confidence, the confidence of the target category label in the text confidence vector and the confidence in the image. The confidence in the image confidence vector and the observation confidence of the target category label are used to determine the cumulative frequency of the target category label. According to the cumulative frequency of each candidate category label, the correction label of the observation label of the target image is determined, and the label distribution of the correction label is determined. Since the first cumulative confidence and the second cumulative confidence are both related to K, that is, they will be different with different K, the cumulative frequency of different candidate category labels will change with the change of K, which can achieve different degrees of correction for different categories of labels, thereby improving the correction effect on long-tail noise labels.

[0065] In some embodiments, K is positively correlated with the number of candidate class labels.

[0066] In the process of determining the cumulative frequency of the target category label, it is necessary to determine the first cumulative confidence and the second cumulative confidence based on the first k first confidences from large to small in the text confidence vector and the first k second confidences from large to small in the image confidence vector, and further determine the cumulative frequency of the target category label based on the first cumulative confidence, the second cumulative confidence, the confidence of the target category label in the text confidence vector, the confidence of the target category label in the image confidence vector and the observed confidence of the target category label. By analogy, the cumulative frequency of each candidate category label can be determined. The number of each candidate category label is different, and the corresponding K is also different.

[0067] K is positively correlated with the number of candidate class labels. In other words, the more candidate class labels there are, the larger the corresponding K value is, and the fewer the number of candidate class labels is, the smaller the corresponding K value is.

[0068] In this embodiment, by setting K to be positively correlated with the number of candidate category labels, K can be adaptively determined based on the number of candidate category labels. A larger number of head category labels corresponds to a larger K, and a smaller number of tail category labels corresponds to a smaller K. Setting different Ks based on the difference in the number of candidate category labels can reduce the confirmation bias of the tail category labels and ensure the correction consistency between different category labels.

[0069] In some embodiments, a method for determining a text confidence vector corresponding to a target image includes:

[0070] The candidate category labels are encoded by a text encoder to obtain the category label features of each candidate category label; the target image is encoded by an image encoder to obtain the image features of the target image; based on the similarity between the image features of the target image and the category label features of each candidate category label, the text confidence vector corresponding to the target image is determined.

[0071] The candidate category labels can be input into the text encoder in the form of text. The text encoder then encodes each candidate category label in turn to obtain the category label features of the corresponding candidate category label. Similarity can be represented by Euclidean distance, cosine similarity, Manhattan distance, or Mahalanobis distance.

[0072] For example, the candidate category labels can be encoded using a text encoder of a vision-language model (VLM) to obtain the category label features of each candidate category label. Furthermore, the target image can be encoded using an image encoder of the vision-language model to obtain the image features of the target image. The cosine similarity between the image features of the target image and the category label features of each candidate category label is then calculated. The cosine similarities between the image features of the target image and the category label features of each candidate category label are then arranged in a preset order to obtain a text confidence vector. The vision-language model can be, for example, CLIP, ALIGN, or Flamingo, and the specific model can be selected based on the actual application scenario.

[0073] In one example, the image features of the target image x Class label features with candidate class labels The cosine similarity between As shown in the following formula (4).

[0074] Formula (4)

[0075] Where s represents the scale parameter.

[0076] The image features of the target image x and the category label features of the candidate category label c The cosine similarity between Perform normalization to obtain the text confidence corresponding to the candidate category label , as shown in the following formula (5). The text confidence vector corresponding to each candidate category label is obtained according to the order of the candidate category label. .

[0077] Formula (5)

[0078] In this embodiment, the text confidence vector of the target image is determined by comparing the similarity between the image features obtained by encoding the target image using the image encoder and the category label features obtained by encoding the candidate category labels using the text encoder. This can accurately determine the text confidence vector of the target image, laying a solid foundation for subsequently improving the correction effect of long-tail noise labels. Without adjusting the model parameters, a semantic prior based on the alignment of the image and each category label can be obtained.

[0079] In some embodiments, a method for determining an image confidence vector corresponding to a target image includes:

[0080] The target image is encoded by an image encoder to obtain the image features of the target image; the initial classifier is trained based on the image features of the target image until the classification confidence meets the preset conditions to obtain the target classifier; based on the similarity between the image features of the target image and the category weights of the target classifier, the image confidence vector corresponding to the target image is determined.

[0081] The initial classifier is a classifier that classifies image features. It can be a linear classifier, which divides image features into different categories using a linear decision boundary. Examples of initial classifiers include logistic regression, linear support vector machine (SVM), perceptron, or naive Bayes. The initial classifier can be selected based on the actual application scenario. Classification confidence is used to indicate classification accuracy. Generally, higher classification accuracy indicates higher classification confidence. Classification weight refers to the weight assigned by the classifier to each image feature category. A higher class weight indicates a higher probability that the classifier will classify an image feature into the corresponding image feature category.

[0082] For example, the image features of the target image can be input into an initial classifier to obtain a predicted classification result. The classification confidence is determined based on the difference between the predicted classification result and the standard classification result. The initial classifier is trained based on the classification confidence until the classification confidence reaches a confidence threshold, thereby obtaining a target classifier. The class weight of the target classifier can thus be obtained. The confidence threshold can be set according to the actual application scenario.

[0083] In one example, the similarity between the image features of the target image and the class weights of the target classifier can be represented by the cosine similarity between the image features of the target image and the class weights of the target classifier. and the class weights of the target classifier The cosine similarity between The calculation formula is shown in the following formula (6).

[0084] Formula (6)

[0085] The image features of the target image x and the class weights of the target classifier The cosine similarity between them is normalized to obtain the image confidence corresponding to the corresponding category weight As shown in the following formula (7), the image confidence vector corresponding to each category weight is obtained in the order of category weight. It is easy to understand that the order of candidate class labels is consistent with the order of class weights.

[0086] Formula (7)

[0087] For example, AdaptFormer (a lightweight and efficient fine-tuning method for adaptive vision-language models) can be used to fine-tune the parameters of CLIP's image encoder, achieving enhanced adaptability to downstream tasks while maintaining efficiency. It should be noted that fine-tuning the image encoder is used to adapt the classification of image features from different datasets.

[0088] In this embodiment, by determining the image confidence vector corresponding to the target image based on the similarity between the image features of the target image and the weights of each category of the target classifier, the image confidence vector can be accurately determined, laying a solid foundation for subsequently improving the correction effect of long-tail noise labels.

[0089] In some embodiments, determining a correction label corresponding to an observed label of a target image based on the cumulative frequency of each candidate class label includes:

[0090] The candidate category label corresponding to the maximum cumulative frequency among the cumulative frequencies of the candidate category labels is determined as the correction label corresponding to the observation label of the target image.

[0091] After determining the text confidence vector and image confidence vector based on the target image, the cumulative frequency of each candidate class label is determined based on the top K first confidences in the text confidence vector and the top K second confidences in the image confidence vector, ranked from largest to smallest. Among the cumulative frequencies of each candidate class label, the candidate class label with the largest cumulative frequency is used as the correction label corresponding to the observed label of the target image. As will be readily understood, the above process can be repeated multiple times, with each iteration yielding a correction label for the target image.

[0092] For example, the calibration label It can be obtained by the following formula (8).

[0093] Formula (8)

[0094] In this embodiment, by using the candidate category label corresponding to the maximum cumulative frequency among the cumulative frequencies of the candidate category labels as the correction label corresponding to the observation label of the target image, the accuracy of the correction label can be improved, thereby improving the correction effect of the long-tail noise label.

[0095] In some embodiments, the above method further comprises:

[0096] Based on the label distribution of the corrected label, the prior probability of the candidate category label corresponding to the corrected label is determined; the label generation loss is determined based on the prior probability, and the label generation model is trained based on the label generation loss.

[0097] The label generation model is the model used to generate labels, and the generation loss is the training loss of the label generation model.

[0098] For example, the label generation loss is shown in the following formula (9).

[0099] Formula (9)

[0100] in, Originated from The calculated prior probability of the corrected label is used to characterize the accuracy of the corrected label.

[0101] In this embodiment, the prior probability of the corresponding candidate category label is re-determined based on the label distribution of the corrected label, the label generation loss is determined based on the prior probability, and the label generation model is trained based on the label generation loss. Since the distribution of each candidate category label is corrected, the accuracy is higher and the distribution is more uniform, so that the model pays more attention to the learning of the tail class label during the training process, so that the trained label generation model can generate more accurate tail class labels.

[0102] In one exemplary embodiment, the process of correcting long-tail noisy labels can be implemented using a model. The model used for label correction can be called Category-Aware Collaborative Expert (CACE). CACE includes text experts (TE), image experts (IE), and observed label experts (BE). Label correction is achieved through a category-aware expert consensus mechanism.

[0103] Text Expert (TE): A CLIP-based text encoder encodes each candidate category label to obtain category label features , encode the target image through the image encoder to obtain the image features of the target image, and determine the text confidence vector based on the cosine similarity between the image features and the category label features of each candidate category label. The model can calculate the similarity between image features and candidate category labels without additional fine-tuning of model parameters.

[0104] Image Expert (IE): The CLIP-based image encoder can encode the input image x to obtain a visual representation f (i.e., image features) adapted to the task scenario. In order to maintain efficiency while enhancing adaptability to downstream tasks, AdaptFormer is used for parameter fine-tuning. This fine-tuning is only used for the final classification. The fine-tuned image encoder is reused for label correction in the expert consensus phase without introducing additional training targets. The cosine similarity between the image features and the class weights is calculated, and the image confidence vector is determined based on each cosine similarity. .

[0105] Observed Label Expert (BE): Observed label experts, also known as base experts, directly rely on the original (possibly noisy) label annotations. Although they are unreliable, these labels still contain valuable supervision, especially for head classes that contain a large number of accurate labels. Represented as a one-hot confidence vector (i.e., observation confidence vector) , of which only The entry is set to 1. That is, in the observation confidence vector of the target image, when the observation label is the same as the candidate class label, the observation confidence is 1, otherwise, the observation confidence is 0. In the observation confidence vector, only one position has an observation confidence of 1, and the observation confidences of the other positions are 0.

[0106] Category-aware expert consensus: To evaluate the reliability of observed labels, the top-k confidence scores are identified from TE and IE respectively, and the obtained ,in , Represents the confidence from the corresponding expert. In order to improve the robustness to noisy labels and long-tail category distributions, a category-aware top-K consensus mechanism is introduced. K can be adaptively determined based on the number of each candidate category label. Specifically, candidate label categories with higher sample counts (head categories) are assigned larger K, while candidate category labels with fewer sample counts (tail categories) are assigned smaller K, allowing the model to more appropriately distribute confidence within the category range. Formally, the category-specific K is defined as proportional to the number of candidate category labels, that is, ,in Indicates the sample count of candidate class label c in the e-th round. In the initial stage (i.e. e=0), Calculated based on the number of observed labels. For e>0, Updated using the corrected labels from the previous training epoch.

[0107] Category frequency accumulation: set Represents the cumulative frequency vector of the target image x. For each candidate class label , F can be updated as:

[0108]

[0109] Noise label correction and long tail correction:

[0110] During the dynamic accumulation process, the dynamic correction label of sample x is obtained from the accumulated frequency distribution of each round and used for model supervision. is the correction label of the e-th round: , based on the correction label, the label distribution of category c can be recalculated .

[0111] like Figure 3As shown in the figure, the label correction process for a single input image (target image) is described. The candidate category labels in the dataset are input into the model's text encoder for encoding, obtaining the category label features for each candidate category label. The target image is input into the image encoder for encoding, obtaining the target image's image features. The similarity between the target image's image features and the category label features of each candidate category label is calculated, and the text confidence vector is determined to be {0.4 0.2 0.2 0.1 0.1}. Based on the similarity between the target image's image features and each image classification weight, the image confidence vector is determined to be {0.05 0.4 0.3 0.2 0.05}. Assuming K is 3, the top-K confidences are selected from the text confidence vector to be {0.4 0.2 0.2} (first confidence level), and the top-K confidences are selected from the image confidence vector to be {0.4 0.3 0.2} (second confidence level). Based on the first confidence level, a corresponding first cumulative frequency is determined. If a candidate class label does not belong to the top-K confidence levels, the frequency accumulated by the text expert is 0. Based on the second confidence level, a corresponding second cumulative frequency is determined. If a candidate class label does not belong to the top-K confidence levels, the frequency accumulated by the image expert is also 0. Based on the first cumulative frequency, the second cumulative frequency, and the observed confidence level, the cumulative frequencies of each candidate class label are determined to be {1.8 0.2 0.5 0.2 0}, respectively. Similarly, the cumulative frequency of each candidate class can be determined in each training round, thereby determining the corrected label of the observed label of the target image in each training round. The distribution of the corrected labels is determined until the distribution of the corrected labels meets preset conditions. For example, if the distribution of the corrected labels does not belong to a long-tail distribution and the noise rate (the proportion of noise labels) is below a preset threshold, label correction is stopped, resulting in the corrected labels.

[0112] In the above embodiment, by identifying low-noise samples according to the top-K voting strategy of the category without introducing additional parameters, the head category uses a larger K value and the tail category uses a smaller K value, reducing the confirmation bias on the tail category, and the low-confidence category is screened and filtered to ensure consistency between categories, so that the model can generate a more reliable label distribution in a lightweight manner, which not only greatly reduces the overall noise rate and the head category noise rate, but also can achieve the repair of the noise label of the tail category, thereby improving the correction effect of the long-tail noise label. Based on the corrected label, a label distribution that is closer to the actual situation can be statistically used as the corrected prior information. Combined with a variety of long-tail learning methods, the characteristics of the tail category label can be better learned to better identify the performance of the tail category.

[0113] It should be understood that, although the various steps in the flowcharts involved in the various embodiments described above are displayed in sequence according to the instructions of the arrows, these steps are not necessarily performed in sequence in the order indicated by the arrows. Unless clearly stated herein, the execution of these steps is not strictly limited in order, and these steps can be performed in other orders. Moreover, at least a portion of the steps in the flowcharts involved in the various embodiments described above may include multiple steps or multiple stages, and these steps or stages are not necessarily performed at the same time, but can be performed at different times, and the execution order of these steps or stages is not necessarily performed in sequence, but can be performed in turn or alternately with at least a portion of the steps or stages in other steps or other steps. It is understandable that the various steps in different embodiments can be freely combined as needed, and the various non-contradictory schemes formed by the combination all fall within the scope of protection of this application.

[0114] Based on the same inventive concept, embodiments of the present application also provide a device for correcting long-tail noise labels for implementing the aforementioned method for correcting long-tail noise labels. The solution provided by this device is similar to the solution described in the aforementioned method. Therefore, the specific limitations of the embodiments of the device for correcting one or more long-tail noise labels provided below can be found in the limitations of the method for correcting long-tail noise labels above and will not be repeated here.

[0115] In an exemplary embodiment, Figure 4 As shown, a long-tail noise label correction device 400 is provided, comprising: a confidence acquisition module 402, a confidence determination module 404, a frequency determination module 406 and a label correction module 408, wherein:

[0116] Confidence acquisition module 402 is used to obtain the top K first confidences in descending order from the text confidence vector corresponding to the target image in the data set, and the top K second confidences in descending order from the image confidence vector corresponding to the target image; the text confidence vector is obtained based on the similarity between the image features of the target image and each candidate category label in the data set; the image confidence vector is obtained based on the similarity between the image features of the target image and each image classification weight; K is a positive integer;

[0117] a confidence determination module 404 for determining a first accumulated confidence of a target category label based on confidences other than the first confidence in the text confidence vector, and determining a second accumulated confidence of the target category label based on confidences other than the second confidence in the image confidence vector; the target category label being any candidate category label;

[0118] a frequency determination module 406 for determining a cumulative frequency distribution of the target category label based on the first accumulated confidence, the second accumulated confidence, the confidence of the target category label in the text confidence vector, the confidence of the target category label in the image confidence vector, and the observed confidence of the target category label;

[0119] The label correction module 408 is configured to determine the correction labels corresponding to the observed labels of the target image based on the cumulative frequency distribution of each candidate category label, and to determine the label distribution of the correction labels.

[0120] In some embodiments, the above-mentioned device also includes a text confidence determination module, which is used to encode the candidate category labels through a text encoder to obtain the category label features of each candidate category label; encode the target image through an image encoder to obtain the image features of the target image; and determine the text confidence vector corresponding to the target image based on the similarity between the image features of the target image and the category label features of each candidate category label.

[0121] In some embodiments, the above-mentioned device also includes an image confidence determination module, which is used to encode the target image through an image encoder to obtain image features of the target image; train the initial classifier based on the image features of the target image until the classification confidence meets preset conditions to obtain a target classifier; and determine the image confidence vector corresponding to the target image based on the similarity between the image features of the target image and the category weights of the target classifier.

[0122] In some embodiments, the label correction module 408 is further configured to determine the candidate category label corresponding to the maximum cumulative frequency among the cumulative frequencies of the candidate category labels as the correction label corresponding to the observed label of the target image.

[0123] In some embodiments, the above-mentioned device also includes a label generation module, which is used to determine the prior probability of the candidate category label corresponding to the correction label based on the label distribution of the correction label; determine the label generation loss based on the prior probability, and train the label generation model based on the label generation loss.

[0124] Each module in the long-tail noise label correction device described above can be implemented in whole or in part through software, hardware, or a combination thereof. Each module can be embedded in or independent of a processor in a computer device in hardware form, or can be stored in a memory in the computer device in software form, so that the processor can call and execute the corresponding operations of each module.

[0125] In an exemplary embodiment, a computer device is provided. The computer device may be a server, and its internal structure diagram may be as shown in FIG. Figure 5As shown. The computer device includes a processor, a memory, an input / output interface (Input / Output, abbreviated as I / O) and a communication interface. The processor, memory and input / output interface are connected through a system bus, and the communication interface is connected to the system bus through the input / output interface. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and computer program in the non-volatile storage medium. The database of the computer device is used to store correction-related data for long-tail noise labels. The input / output interface of the computer device is used to exchange information between the processor and an external device. The communication interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, a method for correcting long-tail noise labels is implemented.

[0126] Those skilled in the art will understand that Figure 5 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than shown in the figure, or combine certain components, or have a different component arrangement.

[0127] In an exemplary embodiment, a computer device is provided, including a memory and a processor. The memory stores a computer program, and the processor implements the steps in the above method embodiments when executing the computer program.

[0128] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps in the above-mentioned method embodiments are implemented.

[0129] In one embodiment, a computer program product is provided, including a computer program, which implements the steps in the above method embodiments when executed by a processor.

[0130] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with relevant regulations.

[0131] Those skilled in the art will understand that all or part of the processes in the above-mentioned embodiments can be implemented by instructing the relevant hardware through a computer program. The computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. In particular, any reference to memory, database, or other media used in the embodiments provided in this application can include at least one of non-volatile memory and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The databases involved in the various embodiments provided herein may include at least one of a relational database and a non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the various embodiments provided herein may be, but are not limited to, general-purpose processors, central processing units (CPUs), graphics processing units (GPUs), digital signal processors (DSPs), programmable logic devices (PLDs), quantum computing-based data processing logic devices, artificial intelligence (AI) processors, and the like.

[0132] The technical features of the above embodiments can be combined arbitrarily. In order to make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this application.

[0133] The above-described embodiments merely represent several implementation methods of the present application. While the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the present application. It should be noted that a person of ordinary skill in the art may make various modifications and improvements without departing from the spirit of the present application, and these modifications and improvements fall within the scope of protection of the present application. Therefore, the scope of protection of the present application shall be determined by the appended claims.

Claims

1. A method for correcting long-tail noise labels, characterized in that: The method comprises: Obtain the top K first confidences in descending order in the text confidence vector corresponding to the target image in the data set, and the top K second confidences in descending order in the image confidence vector corresponding to the target image; the text confidence vector is obtained based on the similarity between the image features of the target image and each candidate category label in the data set; the image confidence vector is obtained based on the similarity between the image features of the target image and each image classification weight; K is a positive integer; Determining a first accumulated confidence of a target category label based on confidences other than the first confidence in the text confidence vector, and determining a second accumulated confidence of the target category label based on confidences other than the second confidence in the image confidence vector; the target category label is any candidate category label; Determining a cumulative frequency of the target category label based on the first accumulated confidence, the second accumulated confidence, the confidence of the target category label in the text confidence vector, the confidence of the target category label in the image confidence vector, and the observed confidence of the target category label; Based on the cumulative frequency of each candidate category label, a correction label corresponding to the observation label of the target image is determined, and a label distribution of the correction label is determined.

2. The method according to claim 1, characterized in that The K is positively correlated with the number of candidate class labels.

3. The method according to claim 1 or 2, characterized in that The method for determining the text confidence vector corresponding to the target image includes: Encode the candidate category labels through the text encoder to obtain the category label features of each candidate category label; Encoding the target image by an image encoder to obtain image features of the target image; Based on the similarity between the image feature of the target image and the category label feature of each candidate category label, a text confidence vector corresponding to the target image is determined.

4. The method according to claim 1 or 2, characterized in that The method for determining the image confidence vector corresponding to the target image includes: Encoding the target image by an image encoder to obtain image features of the target image; Training the initial classifier based on the image features of the target image until the classification confidence meets a preset condition, thereby obtaining a target classifier; An image confidence vector corresponding to the target image is determined based on the similarity between the image features of the target image and the category weights of the target classifier.

5. The method according to claim 1, wherein The determining, based on the cumulative frequency of each candidate category label, a correction label corresponding to the observation label of the target image includes: The candidate category label corresponding to the maximum cumulative frequency among the cumulative frequencies of the candidate category labels is determined as the correction label corresponding to the observation label of the target image.

6. The method according to claim 1, characterized in that The method further comprises: Determining, based on the label distribution of the correction label, a priori probability of the candidate category label corresponding to the correction label; A label generation loss is determined based on the prior probability, and a label generation model is trained based on the label generation loss.

7. A correction device for long-tail noise labels, characterized in that: The device comprises: A confidence acquisition module is configured to acquire the top K first confidences, from large to small, in the text confidence vector corresponding to the target image in the data set, and the top K second confidences, from large to small, in the image confidence vector corresponding to the target image; the text confidence vector is obtained based on the similarity between the image features of the target image and each candidate category label in the data set; the image confidence vector is obtained based on the similarity between the image features of the target image and each image classification weight; K is a positive integer; a confidence determination module, configured to determine a first accumulated confidence of a target category label based on confidences other than the first confidence in the text confidence vector, and to determine a second accumulated confidence of the target category label based on confidences other than the second confidence in the image confidence vector; the target category label being any candidate category label; a frequency determination module, configured to determine a cumulative frequency distribution of the target category label based on the first accumulated confidence, the second accumulated confidence, the confidence of the target category label in the text confidence vector, the confidence of the target category label in the image confidence vector, and the observed confidence of the target category label; The label correction module is used to determine the correction label corresponding to the observation label of the target image based on the cumulative frequency distribution of each candidate category label, and determine the label distribution of the correction label.

8. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 6 are implemented.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

10. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

Images