A method and system for constructing and applying a robust hint tuning model

Abstract

The application provides a kind of robust prompt tuning model construction, application method and system, comprising: the consensus feature of the training set of image classification scene is calculated, global probability matrix of training set is obtained according to consensus feature combined with mixed prompt pool;Based on global probability matrix, using entropy regular optimal transport algorithm, get calibrated pseudo label, adopt pseudo label and divide training set into clean subset and noise subset;Based on clean subset and the noise subset, using overall loss function, iteratively optimize mixed prompt pool, complete the construction of robust prompt tuning model;The pure class semantic reference prompt in the mixed prompt pool of the application, and the regularization term corresponding to the to-be-optimized sparse correction term that absorbs the internal alignment noise introduced in the training process, can break the vicious cycle of "prompt drift-unreliable division", improve the robustness of prompt tuning under noise condition, significantly improve the classification accuracy under noisy label.

Description

Technical Field

[0001] This invention belongs to the field of artificial intelligence deep learning technology, specifically relating to a robust suggestion tuning model construction, application method and system. Background Technology

[0002] CLIP (Contrastive Language-Image Pre-training) is a visual language pre-training model that has achieved significant breakthroughs in computer vision in recent years. Its core idea is to achieve powerful zero-shot recognition capabilities by aligning images and text in a shared embedding space. CLIP consists of two frozen encoders: an image encoder and a text encoder. It classifies images by calculating the cosine similarity between image and text features. Softmax is a normalized exponential function used to transform the classification scores (logits) into a probability distribution. To adapt the pre-trained CLIP model to specific downstream tasks, researchers have proposed PromptTuning—keeping the pre-trained model parameters frozen and learning only a small number of learnable text tokens—offering advantages such as high parameter efficiency and low computational cost.

[0003] However, the effectiveness of cue tuning is highly dependent on the quality of the training data. In real-world applications, training data is often inevitably contaminated by label noise. When training data contains label noise, cue tuning methods face a severe challenge: the flexibility of learnable cueing can capture subtle semantic differences when the labels are clean, but it becomes a double-edged sword when the labels are noisy—the cueing may not only learn the correct category semantics, but may also "remember" incorrect labels, thus overfitting to noisy labels and losing the ability to learn true category features.

[0004] For robust learning under noisy label conditions, existing methods mainly fall into two categories: one is robust loss function design, such as MAE (Mean Absolute Error) loss and GCE (Generalized Cross Entropy) loss, which limits the impact of mislabeled samples by designing special loss functions, but only addressing the loss level and cannot solve the sample selection problem; the other is sample selection methods, such as NLPrompt (a cue-based noisy label learning method), which utilizes the memory effect of deep networks to divide samples into clean subsets and noisy subsets through the cue-image consistency index, and then applies different training strategies to different subsets. Among them, CE (Cross Entropy) is a commonly used classification loss function.

[0005] The aforementioned existing methods generally suffer from a critical but overlooked problem—intrinsic alignment noise. In the frozen CLIP space, an inherent modal gap exists between the image encoder and the text encoder. During cue tuning, the cue must not only learn category semantics but also compensate for this systematic cross-modal mismatch. Therefore, the learned cue actually contains two parts of information: category semantic information and alignment bias information. When external label noise is present, this bias is further amplified, causing the cue to gradually deviate from the true category semantics. Existing methods (such as NLPrompt) rely on cue-image consistency to determine whether samples are clean, but these methods assume that the cue itself is a reliable semantic anchor. This assumption ignores the fact that the cue will drift during the learning process due to intrinsic alignment noise. When the cue itself drifts, the consistency signal becomes unreliable, leading to incorrect sample classification and forming a vicious cycle of "cue drift → unreliable classification → exacerbated drift," severely limiting the robustness of cue tuning under noisy conditions and resulting in low classification accuracy under noisy labels. Summary of the Invention

[0006] To overcome the shortcomings of the prior art, this invention proposes a method for constructing a robust hint tuning model, comprising: Calculate the consensus features of the training set for the image classification scene, and obtain the global probability matrix of the training set based on the consensus features and a pre-constructed hybrid cue pool. Based on the global probability matrix, the entropy regularized optimal transmission algorithm is used to obtain calibrated pseudo-labels, and the training set is divided into a clean subset and a noisy subset using the pseudo-labels. Based on the clean subset and the noisy subset, the hybrid cue pool is iteratively optimized using the overall loss function to complete the construction of the robust cue tuning model; The training set includes sample images and original labels; the original labels include correct labels and incorrect labels; the hybrid cue pool includes category semantic reference cue for the category corresponding to the sample images; the overall loss function includes a cross-entropy loss term, a mean absolute error loss term, and a regularization term corresponding to the sparse correction term to be optimized for absorbing inherent alignment noise.

[0007] Preferably, the consensus features of the training set for the image classification scene are used to obtain the global probability matrix of the training set based on the consensus features and a pre-constructed hybrid cue pool, including: Based on the training set of the image classification scenario, multiple augmented views are generated for each sample image, and the consensus features of each sample image are calculated. Based on the consensus features of each sample image and the semantic reference hints of the category of each sample image in the hybrid hint pool, calculate the prediction probability of each sample image with each category; Construct a global probability matrix based on the predicted probabilities; The image classification scenarios include one or more of the following: animal recognition scenarios, flower classification scenarios, texture classification scenarios, action and behavior classification scenarios, video behavior classification scenarios, and food image classification scenarios.

[0008] Preferably, the multiple augmented views generated for each sample image in the training set based on the image classification scenario, and the consensus features of each sample image are calculated, include: For each sample image in the training set for an image classification scenario, multiple augmented views are generated; The image encoder, which is frozen in the visual language pre-trained model, is used to extract the image features corresponding to each augmented view. The mode estimation method is used to aggregate the image features of all augmented views corresponding to each sample image into consensus features; The visual language pre-trained model includes a pre-trained image encoder and a text encoder; the enhanced view includes one or more of the following: a randomly cropped view, a horizontally flipped view, and a color-transformed view.

[0009] Preferably, the hybrid notification pool is constructed as follows: For each category of the sample images in the training set, a large language model is used to generate multiple short sentences describing visual attributes using a unified description template; the visual attributes include visual cues that distinguish the category from other categories. The frozen text encoder in the visual language pre-trained model is used to encode multiple short sentences, generate multiple text features, and average the multiple text features to obtain a fixed semantic prototype for each category. After concatenating the category lexical units of each category with the pre-obtained shared lexical units, the text encoder of the visual language pre-trained model is used to obtain the prompt features to be optimized. Based on the fixed semantic prototype, the prompt features to be optimized, and the hybrid weights to be optimized, semantic reference prompts for each category are generated. Aggregate all category semantic reference hints to generate a hybrid hint pool; The shared lexical units are derived from pre-set continuous context lexical units to be optimized that are shared across all categories.

[0010] Preferably, after aggregating semantic reference cues from all categories and generating a hybrid cue pool, the method further includes: Using a sparse overparameterized form, based on two variables to be optimized, a sparse correction term to be optimized is set for each category semantic reference cue in the hybrid cue pool to absorb inherent alignment noise; Based on the sparse correction terms to be optimized, generate the sparse correction term matrix to be optimized; The sparse correction term matrix to be optimized is regularized to generate regularization terms corresponding to the sparse correction terms to be optimized, which are used to construct the overall loss function. The intrinsic alignment noise is a systematic cross-modal feature representation bias that exists between the frozen text encoder and image encoder in the visual language pre-trained model and is independent of category semantics. This systematic cross-modal feature representation bias can cause the prompts to deviate from the true category semantics during the prompt optimization process based on the wrong labels, resulting in prompt drift.

[0011] Preferably, the step of obtaining calibrated pseudo-labels based on the global probability matrix using the entropy regularized optimal transmission algorithm, and using the pseudo-labels to divide the training set into a clean subset and a noisy subset, includes: Based on the global probability matrix, the sample image and the category are coupled through a global allocation plan using the entropy regularized optimal transfer algorithm under marginal constraints to generate a soft allocation plan. After normalizing the soft allocation plan by row, the class posterior of each sample image is obtained, and pseudo-labels for each sample image are generated based on the class posterior. When the pseudo-label and the original label are consistent, the sample image corresponding to the pseudo-label and the original label are regarded as clean samples; otherwise, the sample image corresponding to the pseudo-label and the original label are regarded as noise samples. Summarize all clean samples to generate a clean subset, and summarize all noisy samples to generate a noisy subset.

[0012] Preferably, the step of differentially training the clean subset and the noisy subset to complete the construction of the robust cue tuning model includes: The clean subset is trained under supervision using the cross-entropy loss function to obtain the cross-entropy loss term; The mean absolute error loss function is used to constrain the noise subset to obtain the mean absolute error loss term; The overall loss function is used to calculate the overall loss by weighted summation of the regularization terms corresponding to the cross-entropy loss term, the mean absolute error loss term, and the sparse correction term to be optimized. Based on the overall loss, the sparse correction term to be optimized and the category semantic reference cues in the hybrid cue pool are updated. Based on the updated category semantic reference cues and the sparse correction term to be optimized, iterative training is performed until a preset number of training rounds are reached. The final category semantic reference cues are retained, and the sparse correction term to be optimized is discarded, thus completing the construction of the robust cue tuning model.

[0013] Based on the same inventive concept, this invention also provides a method for applying a robust suggestion tuning model, including: Based on the image classification scenario, the probability of the image in each category is obtained by optimizing the model through robust prompts; The category to which the image belongs is determined based on the probability of each category.

[0014] The robust hint tuning model is constructed using any of the robust hint tuning model construction methods described above.

[0015] Based on the same inventive concept, the present invention also provides a robust suggestion tuning model construction system, including: a global probability matrix calculation module, a training set partitioning module, and a robust suggestion tuning model construction module; The global probability matrix calculation module is used to calculate the consensus features of the training set for the image classification scene, and obtain the global probability matrix of the training set based on the consensus features and a pre-built hybrid cue pool. The training set partitioning module is used to obtain calibrated pseudo-labels based on the global probability matrix using the entropy regularized optimal transmission algorithm, and to partition the training set into a clean subset and a noisy subset using the pseudo-labels. The robust suggestion tuning model construction module is used to iteratively optimize the hybrid suggestion pool based on the clean subset and the noisy subset using the overall loss function, thereby completing the construction of the robust suggestion tuning model. The training set includes sample images and original labels; the original labels include correct labels and incorrect labels; the hybrid cue pool includes category semantic reference cue for the category corresponding to the sample images; the overall loss function includes a cross-entropy loss term, a mean absolute error loss term, and a regularization term corresponding to the sparse correction term to be optimized for absorbing inherent alignment noise.

[0016] Preferably, the global probability matrix calculation module includes: The consensus feature calculation submodule is used to calculate the consensus features of each sample image generated from multiple augmented views in the training set for image classification scenarios. The prediction probability calculation submodule is used to calculate the prediction probability of each sample image with each category based on the consensus features of each sample image and the semantic reference hints of the category of each sample image in the mixed hint pool. The global probability matrix construction submodule is used to construct a global probability matrix based on the predicted probabilities. The image classification scenarios include one or more of the following: animal recognition scenarios, flower classification scenarios, texture classification scenarios, action and behavior classification scenarios, video behavior classification scenarios, and food image classification scenarios.

[0017] Preferably, the consensus feature calculation submodule is specifically used for: For each sample image in the training set for an image classification scenario, multiple augmented views are generated; The image encoder, which is frozen in the visual language pre-trained model, is used to extract the image features corresponding to each augmented view. The mode estimation method is used to aggregate the image features of all augmented views corresponding to each sample image into consensus features; The visual language pre-trained model includes a pre-trained image encoder and a text encoder; the enhanced view includes one or more of the following: a randomly cropped view, a horizontally flipped view, and a color-transformed view.

[0018] Preferably, the system further includes: a hybrid prompt pool construction module, used for: For each category of the sample images in the training set, a large language model is used to generate multiple short sentences describing visual attributes using a unified description template; the visual attributes include visual cues that distinguish the category from other categories. The frozen text encoder in the visual language pre-trained model is used to encode multiple short sentences, generate multiple text features, and average the multiple text features to obtain a fixed semantic prototype for each category. After concatenating the category lexical units of each category with the pre-obtained shared lexical units, the text encoder of the visual language pre-trained model is used to obtain the prompt features to be optimized. Based on the fixed semantic prototype, the prompt features to be optimized, and the hybrid weights to be optimized, semantic reference prompts for each category are generated. Aggregate all category semantic reference hints to generate a hybrid hint pool; The shared lexical units are derived from pre-set continuous context lexical units to be optimized that are shared across all categories.

[0019] Preferably, the system further includes: a regularization term setting module corresponding to the sparse correction term to be optimized, used for: Using a sparse overparameterized form, based on two variables to be optimized, a sparse correction term to be optimized is set for each category semantic reference cue in the hybrid cue pool to absorb inherent alignment noise; Based on the sparse correction terms to be optimized, generate the sparse correction term matrix to be optimized; The sparse correction term matrix to be optimized is regularized to generate regularization terms corresponding to the sparse correction terms to be optimized, which are used to construct the overall loss function. The intrinsic alignment noise is a systematic cross-modal feature representation bias that exists between the frozen text encoder and image encoder in the visual language pre-trained model and is independent of category semantics. This systematic cross-modal feature representation bias can cause the prompts to deviate from the true category semantics during the prompt optimization process based on the wrong labels, resulting in prompt drift.

[0020] Preferably, the training set partitioning module is specifically used for: Based on the global probability matrix, the sample image and the category are coupled through a global allocation plan using the entropy regularized optimal transfer algorithm under marginal constraints to generate a soft allocation plan. After normalizing the soft allocation plan by row, the class posterior of each sample image is obtained, and pseudo-labels for each sample image are generated based on the class posterior. When the pseudo-label and the original label are consistent, the sample image corresponding to the pseudo-label and the original label are regarded as clean samples; otherwise, the sample image corresponding to the pseudo-label and the original label are regarded as noise samples. Summarize all clean samples to generate a clean subset, and summarize all noisy samples to generate a noisy subset.

[0021] Preferably, the robust hint tuning model construction module is specifically used for: The clean subset is trained under supervision using the cross-entropy loss function to obtain the cross-entropy loss term; The mean absolute error loss function is used to constrain the noise subset to obtain the mean absolute error loss term; The overall loss function is used to calculate the overall loss by weighted summation of the regularization terms corresponding to the cross-entropy loss term, the mean absolute error loss term, and the sparse correction term to be optimized. Based on the overall loss, the sparse correction term to be optimized and the category semantic reference cues in the hybrid cue pool are updated. Based on the updated category semantic reference cues and the sparse correction term to be optimized, iterative training is performed until a preset number of training rounds are reached. The final category semantic reference cues are retained, and the sparse correction term to be optimized is discarded, thus completing the construction of the robust cue tuning model.

[0022] Based on the same inventive concept, the present invention also provides an application system for a robust prompting optimization model, including: an image category probability calculation module and an image category determination module; The image category probability calculation module is used to obtain the probability of the image in each category based on the image classification scenario and through robust prompting to optimize the model. The image category determination module is used to determine the category to which the image belongs based on the probability of each category.

[0023] The robust hint tuning model is constructed using any of the robust hint tuning model construction methods described above.

[0024] Based on the same inventive concept, the present invention also provides an electronic device, comprising: at least one processor and a memory; wherein the memory and the processor are connected via a bus; The memory is used to store one or more programs; When the one or more programs are executed by the at least one processor, a robust hint tuning model construction method as described above is implemented.

[0025] Based on the same inventive concept, the present invention also provides a readable storage medium having a computer program stored thereon and an executable program stored thereon, wherein when the executable program is executed, a method for constructing a robust hint tuning model as described above is implemented.

[0026] Compared with the closest existing technology, the present invention has the following beneficial effects: This invention provides a method and system for constructing and applying a robust suggestion tuning model, comprising: calculating consensus features of a training set for an image classification scenario; obtaining a global probability matrix of the training set based on the consensus features and a pre-constructed hybrid suggestion pool; obtaining calibrated pseudo-labels based on the global probability matrix using an entropy-normalized optimal transfer algorithm; dividing the training set into a clean subset and a noisy subset using the pseudo-labels; iteratively optimizing the hybrid suggestion pool using an overall loss function based on the clean subset and the noisy subset to complete the construction of the robust suggestion tuning model; wherein, the training set includes sample images and original labels; the original labels include correct labels and incorrect labels; the hybrid suggestion pool includes category terms corresponding to the categories of the sample images. The overall loss function includes a cross-entropy loss term, a mean absolute error loss term, and a regularization term corresponding to the sparse correction term to be optimized to absorb inherent alignment noise. The hybrid cue pool of this invention serves as a pure category semantic reference cue, used to measure the consistency between the cue and the image during the training set partitioning stage. Simultaneously, during training, a regularization term corresponding to the sparse correction term to be optimized to absorb inherent alignment noise is introduced, reducing the contamination of the category semantic reference cue by inherent alignment noise. The clear division of labor between the two ensures that the cue does not shift. Thus, the trained hybrid cue pool is suitable for both sample partitioning and final inference, breaking the vicious cycle of "cue drift - unreliable partitioning," improving the robustness of cue tuning under noisy conditions, and significantly improving the classification accuracy under noisy labels. Attached Figure Description

[0027] Figure 1A flowchart illustrating a method for constructing a robust hint tuning model provided by this invention; Figure 2 This is a schematic diagram of the overall framework for robust few-sample cue optimization under label noise in animal identification scenarios provided by the present invention. Figure 3 A flowchart illustrating the application method of a robust hint tuning model provided by this invention; Figure 4 A schematic diagram of a robust hint tuning model construction system provided by the present invention; Figure 5 A schematic diagram of an application system for a robust hinting tuning model provided by the present invention; Figure 6 This is a schematic diagram of a device structure provided by the present invention. Detailed Implementation

[0028] The specific embodiments of the present invention will be further described in detail below with reference to the accompanying drawings.

[0029] Example 1: This invention provides a method for constructing a robust hint tuning model, as follows: Figure 1 As shown, it includes: Calculate the consensus features of the training set for the image classification scene, and obtain the global probability matrix of the training set based on the consensus features and a pre-constructed hybrid cue pool. Based on the global probability matrix, the entropy regularized optimal transmission algorithm is used to obtain calibrated pseudo-labels, and the training set is divided into a clean subset and a noisy subset using the pseudo-labels. Based on the clean subset and the noisy subset, the hybrid cue pool is iteratively optimized using the overall loss function to complete the construction of the robust cue tuning model; The training set includes sample images and original labels; the original labels include correct labels and incorrect labels; the hybrid cue pool includes category semantic reference cue for the category corresponding to the sample images; the overall loss function includes a cross-entropy loss term, a mean absolute error loss term, and a regularization term corresponding to the sparse correction term to be optimized for absorbing inherent alignment noise.

[0030] Achieving robust cue tuning under noisy label conditions, particularly addressing cue drift and improving the reliability of sample segmentation, is a pressing technical challenge in the application of visual language models. This invention offers a solution to this problem. To overcome the shortcomings of existing technologies, this invention proposes a robust cue tuning model based on a pre-trained visual language model. Through a multi-layer noise correction mechanism, it enhances the learning ability of the robust cue tuning model under noisy training label conditions, enabling the model to learn true category features from noisy data and thus achieve higher classification accuracy on clean test sets.

[0031] Before presenting the technical solution of this invention, a problem model is performed as follows: Hint tuning, by learning only a small number of text tokens while keeping the two encoders in the visual language pre-trained model frozen, provides a parameter-efficient adaptation method for visual language pre-trained models such as CLIP. However, the application of this type of method in real-world scenarios is significantly limited by the label noise in the mislabeled labels of the training data. This step formally defines the noisy few-sample image classification problem to be solved and clarifies the working method of the basic model. The specific method is as follows: (1) Problem definition: Let the training set be...

[0032] in, For the training set, For the i-th sample image, Let be the original label corresponding to the i-th sample image; the original label includes correct labels and noisy incorrect labels. The total number of samples in the training set; The training set labels are noisy, while the test set labels used to test the robust cue-tuned model are clean. The ultimate goal is to learn a robust cue-tuned model from noisy training data that can accurately generalize to the clean test set.

[0033] (2) Label noise modeling: The label contamination process is characterized by a conditional noise model.

[0034] set up This is the noise transition matrix, where each element represents the probability that the true label is incorrectly labeled as another category. Let be the total number of categories; the noise transition matrix satisfies the class-conditional noise transition model, the expression of which is as follows: ,

[0035] in, When the true label y is the k-th category, the original label Let be the conditional probability of the j-th class. For the original tag, For the j-th category, For real labels, Let be the probability that a sample image with the true label of class k is labeled as class j. For the kth category; This represents the total number of categories; Class-conditional noise transfer models can simultaneously cover both symmetric noise (equal error rates across classes) and asymmetric noise (higher error rates between adjacent classes). The above formula represents the class-conditional noise transfer model; (3) Basic Model Definition: This method is based on the pre-trained CLIP model. The CLIP model contains an image encoder. and text encoder Both remain frozen during training. For each sample image Image features normalized by L2 are extracted using an image encoder. ;in, Let d be a d-dimensional real vector space, where d is the feature dimension, for each category L2-normalized text features are generated using prompt templates. Given image features and text feature pairs CLIP then uses the scaled cosine similarity between image features and text features, and performs softmax to predict the category, obtaining the corresponding category prediction probability:

[0036] in, For the i-th sample image Predict the probability for the k-th category. For temperature parameters, and ; Image features transpose, For the text features of the k-th category, The text features of the j-th category; (4) Parameter learning objective: Throughout the tuning process, the image encoder and text encoder are kept frozen, and only the category cue is learned. The parameters. The ultimate goal is to utilize only noisy supervision. Under the premise of training, we obtain hints that enable accurate generalization on a clean test set.

[0037] In one implementation, the consensus features of the training set for the image classification scene are calculated, and a global probability matrix of the training set is obtained based on the consensus features and a pre-constructed hybrid cue pool, including: Based on the training set of the image classification scenario, multiple augmented views are generated for each sample image, and the consensus features of each sample image are calculated. For example, consider the training set mentioned above. For each sample image First, calculate its stable consensus characteristics. .

[0038] Based on the consensus features of each sample image and the semantic reference hints of the category of each sample image in the hybrid hint pool, calculate the prediction probability of each sample image with each category; Example, based on a hybrid cue pool and consensus features Calculate the predicted probability of each sample and each category on the entire training set.

[0039] Construct a global probability matrix based on the predicted probabilities; For example, the global probability matrix is ​​as follows:

[0040] in, This is the global probability matrix; The total number of samples in the training set; The total number of categories; the global probability matrix summarizes cue-image consistency in normalized form and serves as an affinity signal for global sample partitioning.

[0041] The image classification scenarios include one or more of the following: animal recognition scenarios, flower classification scenarios, texture classification scenarios, action and behavior classification scenarios, video behavior classification scenarios, and food image classification scenarios.

[0042] In one implementation, the multiple augmented views generated for each sample image in the training set based on the image classification scenario, and the consensus features of each sample image are calculated, include: For each sample image in the training set for an image classification scenario, multiple augmented views are generated; For example, to ensure truly reliable sample selection, the visual evidence itself needs to be stable. In the current process, sample selection relies on cue-image similarity, which is directly influenced by the features of each training sample image. In low-sample scenarios, the CLIP image features often fluctuate significantly after different enhancements to the same sample image. If only a single view is relied upon, the cue-image consistency signal may change with the enhancement method, introducing unnecessary noise into the global sample partitioning. The goal of MVV (Multi-View Visual Consensus) is to reduce this bias by aggregating information from multiple enhanced views of the same image to obtain a single consensus direction on a unit sphere; specifically, given a sample image... First, generate multiple enhanced views. ;in, For sample images The An enhanced view; To increase the total number of views.

[0043] The image encoder, which is frozen in the visual language pre-trained model, is used to extract the image features corresponding to each augmented view. For example, L2-normalized image features are extracted using a frozen CLIP image encoder. ;in, For sample images The The image features of each view; the goal is to obtain a consensus feature. This allows the mean to represent the direction most supported by evidence among these views. Simply averaging is often insufficiently robust: when a few enhanced views produce outliers, the mean can be pulled away from the dominant direction. In such cases, even if most views are consistent with each other, a simple average may fall into low-density regions on the sphere. This is clearly undesirable for sample selection, as it aligns with the semantic reference cues of the category. The similarity will reflect more the amplified noise than the image content itself.

[0044] The mode estimation method is used to aggregate the image features of all augmented views corresponding to each sample image into consensus features; For example, will Defined as the mode of the kernel density estimate on a unit sphere, the mode estimation method of the von Mises-Fisher distribution (a probability distribution defined on a unit sphere, often used to model the directional distribution of normalized feature vectors) is used to aggregate the features of all views into consensus features. This ensures that the consensus direction represents the direction supported by the most views. Specifically, the optimization objective is updated through iterative mean shift:

[0045] in, For the i-th consensus feature, The candidate consensus direction is on the unit sphere. The total number of augmented views, where q is the augmented view number. For candidate consensus direction transpose, The image features corresponding to the q-th augmented view of the i-th sample image. For concentration parameters, and ;parameter Selectivity of the control estimator: when When the value is smaller, the target is closer to the mean aggregation; when When the value is large, the focus is more on the most consistent view and outliers are suppressed more strongly. The above formula tends to find a direction with a large cosine similarity to most views, thus remaining robust to a small number of deviations from the view. The iterative mean shift update process is as follows: starting from the direction of the normalized mean, at the... In this iteration, soft weights are first assigned to each view based on the current estimate and its similarity to each view: , in, The soft weights are the values ​​corresponding to the q-th augmented view of the i-th sample image in the t-th iteration. For concentration parameters, Estimating the current consensus direction transpose, For the image features of the q-th view, The image features corresponding to the r-th enhanced view; Soft weights enhance views aligned with the current orientation while reducing those deviating from it; then these weights are used to... Perform a weighted summation and renormalize the result to unit length. Repeat the iteration until convergence. Stop the iteration when the change between two consecutive estimates is less than a preset threshold; output the consensus feature. , used for sample partitioning calculation.

[0046] Because the image encoder remains frozen, MVV does not introduce any additional trainable parameters on the visual side. MVV and OPrompt (Over-parameterized Prompt) are complementary sparse over-parameterized correction modules: OPrompt stabilizes the semantic reference on the text side, and MVV stabilizes the visual evidence on the image side. In subsequent processes, consensus features are used to compute the cue-image consistency required for sample selection. Replacing single-view image features makes similarity scores less sensitive to augmentation methods and reduces the variance of the optimal transmission cost matrix. In subsequent processes, MVV is used whenever stable image features are required. Therefore, global sample partitioning is driven more by stable and consistent visual evidence than by random view noise. Multiple augmented views are generated for each image, and features are aggregated using the mode estimation method of the von Mises-Fisher distribution (a probability distribution defined on a unit sphere, often used to model the directional distribution of normalized feature vectors). This automatically suppresses interference from outlier views, resulting in a more robust visual representation.

[0047] The visual language pre-trained model includes a pre-trained image encoder and a text encoder. The augmented views include one or more of the following: randomly cropped views, horizontally flipped views, and color-transformed views. This step reduces image feature variance by aggregating features from multiple augmented views of the same sample image, thereby improving the stability of few-sample selection and making subsequent sample partitioning based on cue-sample image consistency more reliable. In few-sample scenarios, the CLIP image features of the same sample image often fluctuate significantly after different augmentations. If only a single view is relied upon, the cue-sample image consistency signal may change with the augmentation method, introducing unnecessary noise into global sample partitioning. The goal of MVV is to reduce this variance by aggregating information from multiple augmented views of the same sample image to obtain a single consensus direction on a unit sphere. This complements OPrompt: OPrompt stabilizes semantic references on the text side, while MVV stabilizes visual evidence on the image side; both are indispensable. This invention enhances the anti-interference capability of image features: This method uses a multi-view visual consensus (MVV) mechanism to aggregate features from multiple augmented views into a single consensus feature, effectively reducing feature variance and improving the stability of sample partitioning.

[0048] In one implementation, the hybrid prompt pool is constructed as follows: For each category of the sample images in the training set, a large language model is used to generate multiple short sentences describing visual attributes using a unified description template; the visual attributes include visual cues that distinguish the category from other categories. For example, for each category The large language model is used to generate several short sentences describing the visual attributes of the category, using a uniform template such as "A photo of a {class} with / which {attribute}", which means "a photo of an image of a certain class containing a certain visual attribute". The selected attribute emphasizes the visual cues that can distinguish this category from other categories. Using multiple descriptions can reduce sensitivity to individual expressions, thereby forming a more stable category-level prior.

[0049] The frozen text encoder in the visual language pre-trained model is used to encode multiple short sentences, generate multiple text features, and average the multiple text features to obtain a fixed semantic prototype for each category. For example, each sentence is encoded using a frozen CLIP text encoder, and the resulting text features are averaged to obtain a fixed semantic prototype for that category. Each sentence is processed by a frozen text encoder. Encode, then average the obtained text features to obtain a fixed semantic prototype for each category, denoted as . Using multiple descriptions can reduce sensitivity to individual expressions, thus forming a more stable category-level prior.

[0050] After concatenating the category lexical units of each category with the pre-obtained shared lexical units, the text encoder of the visual language pre-trained model is used to obtain the prompt features to be optimized. For example, add a suggestion branch to be optimized, following the standard suggestion tuning paradigm. Specifically, introduce a set of continuous context terms to be optimized that are shared across all categories: token

[0051] Where token is a word element. To share lexical units, As the first shared word unit, For the first Shared word units, The total number of shared lexical units; For each category After concatenating shared lexical units and category lexical units, the results are fed into the frozen text encoder, yielding the following prompt features to be optimized:

[0052] in, For category The corresponding features to be optimized For the frozen text encoder in the visual language pre-trained model, To share lexical units, For the category word of the k-th category; The feature to be optimized enables the text representation to adapt to the target dataset without altering the pre-trained encoder itself.

[0053] Based on the fixed semantic prototype, the prompt features to be optimized, and the hybrid weights to be optimized, semantic reference prompts for each category are generated. For example, in order to combine the fixed semantic prototype of the category Features that need optimization Set the blend weights to be optimized for each category. In implementation, From a scalar to be optimized After the sigmoid function We obtain the following formula: and

[0054] in, For category The corresponding mixed weights to be optimized For category The corresponding scalar to be optimized It is the sigmoid function; The final category semantic reference hint is defined as a weighted combination of the two, as follows:

[0055] In the formula, For category Corresponding category semantic reference hints, For category The corresponding mixed weights to be optimized For category The corresponding fixed semantic prototype, For category The corresponding features to be optimized; The reason for using the mixed weights to be optimized is that the reliability of attribute descriptions varies across different categories and datasets.

[0056] Aggregate all category semantic reference hints to generate a hybrid hint pool; Example, collection It serves as a semantic reference hint throughout the entire subsequent process for sample partitioning.

[0057] The shared lexical units are derived from pre-set, continuously contextual lexical units shared across all categories that are to be optimized. In the presence of label noise, a fully optimized prompt has sufficient flexibility to overfit to incorrect labels, thus compromising its semantic reliability. In contrast, a fixed text description provides a more stable semantic prior and is less prone to drifting with noisy labels, thus serving as a more robust reference signal.

[0058] In one implementation, after aggregating semantic reference hints of all categories and generating a hybrid hint pool, the method further includes: Using a sparse overparameterized form, based on two variables to be optimized, a sparse correction term to be optimized is set for each category semantic reference cue in the hybrid cue pool to absorb inherent alignment noise; For example, it is given This invention aims to maintain this as a category semantic reference used during sample selection. The challenge lies in the fact that, within the frozen CLIP space, cue tuning must not only learn category semantics but also compensate for the systematic cross-modal mismatch between image and text features. This mismatch is not unidirectional but consists of two parts: one is the inherent systematic residual in the frozen representation space, and the other is a context-dependent relevance that varies with samples and cannot be characterized by a single-directional shift. This invention argues that these mismatch effects repeatedly occur during optimization, continuously pulling on the cue and gradually accumulating into semantic drift. This invention refers to this persistent and structured bias as intrinsic alignment noise. If this mismatch is absorbed, the reference cue will deviate from its original category semantics, thus compromising the stability of the sample selection signal; this mechanism can be derived from the update form of cross-entropy on text features. For a given set of normalized image features and original labels... The loss calculation formula for the sample is as follows:

[0059] in, For images with features v and original labels The cross-entropy loss corresponding to the sample, Image feature v is predicted as the original label. The probability of the corresponding category; Will Considered The text prototype that needs optimization is then... The gradient is calculated as follows:

[0060] In the formula, For the loss function l(v, Text features for the k-th category gradient, For temperature parameters, Let v be the probability that an image feature v is predicted to be the k-th category. For indicator functions, For the original tag, For the kth category, Image features; The above equation shows that the cue update is essentially a weighted sum of image features.

[0061] To explicitly show the direction at the category level, define:

[0062] In the formula, Let be the prior probability that the original label belongs to the k-th category. For the original label It equals the probability of the k-th category. For the original tag, For the kth category; definition:

[0063] In the formula, This represents the average image features corresponding to the sample image whose original label is the k-th category. For mathematical expectation; That is, the original label is a category. The average image features corresponding to the samples. Taking the expectation of the above equation, we get:

[0064] In the formula, loss function Text features for the k-th category Gradient expectation, For the loss function l(v, Text features for the k-th category gradient, For temperature parameters, Let v be the probability that an image feature v is predicted to be in the k-th category. Let be the prior probability that the original label belongs to the k-th category. The average image features corresponding to the sample whose original label is the k-th category; In the above formula, a term can be separated from... For directly related terms, it anchors the update direction to the mean direction of the original class. While the remaining terms depend on the current prediction, they do not eliminate the influence of... This anchor term becomes a dependency. During repeated optimization processes, this anchor term can continuously induce the prompt to drift.

[0065] Tag noise will From a single-category mean to a mixed quantity. Let:

[0066] In the formula, Let be the average image features corresponding to the sample image whose true label is the j-th category. For mathematical expectation, For image features, For real labels, For the j-th category; For the real category The mean characteristic of the mean is given by:

[0067] In the formula, Let be the posterior weights for the sample image whose original label is the k-th class, indicating that it truly belongs to the j-th class. For the original label For the k-th category, the true label Let be the conditional probability of the j-th category; For real labels, Original tag; Let the posterior weights be those jointly induced by the noise process and the class prior, then we have:

[0068] In the formula, The mixed average image features corresponding to the sample image with the original label of the k-th category are... The average image features corresponding to the sample image whose true label is the j-th category; This indicates that when the label is contaminated, the anchoring direction driving the update in the above formula no longer corresponds to a single category.

[0069] Even more seriously, within the frozen CLIP space, It is not a purely semantic quantity either. Let's break it down into:

[0070] in, The average image features corresponding to the sample image whose true label is the j-th category; The category-level residual corresponding to the j-th category; The residual This describes a systematic mismatch between image and text features that cannot be completely eliminated through cue tuning when the encoder is kept frozen. Substituting this decomposition into... From the mixed expression, we can obtain:

[0071] In the formula, The mixed average image features corresponding to the sample image with the original label of the k-th category are... For the j-th category, the ideal semantic prototype is... The category-level residual corresponding to the j-th category; The above equation specifically reveals this failure mode: under label noise conditions, the anchoring direction driving cue updates simultaneously mixes semantic components and residual components from multiple categories. If the same cue is used both as a classification prototype and as a semantic reference during sample selection, it will be pushed toward this mixed direction. This entangles category semantics with inherent mismatches and amplifies cross-category interference, making category semantic reference cue increasingly unsuitable for selection.

[0072] This analysis also explains why drift can still occur even when the labels are clean. When, the weights satisfy ,therefore In the above formula, For the original tag, For real labels, Let be the posterior weights for the sample image whose original label is the k-th class, indicating that it truly belongs to the j-th class. For indicator functions, For the j-th category, For the k-th category, the above expression degenerates into: In the formula, The average image features of the k-th category, This represents the ideal semantic prototype corresponding to the k-th category. This represents the category-level residual corresponding to the k-th category; it means that improving alignment will still drive the hints to absorb a portion of the residual. In other words, in the frozen space, better alignment and pure semantics are not entirely consistent optimization goals; and label noise further amplifies this tension, making the drift stronger and more unstable. These observations inspire a "separation principle": to break the vicious cycle caused by drift, the semantic reference used for sample selection must be explicitly separated from the correction term used to compensate for mismatches during classification. OPrompt is designed according to this principle, absorbing recurring mismatches by introducing an explicit correction term while maintaining the purity of the reference cue itself. The relationship between the text representation during the training phase and the final category cue is given below. For each category, the hybrid cue pool generates a category semantic reference cue. And introduce the correction term to be optimized during the training phase. To absorb systematic cross-modal mismatch, so that The category semantics are cleaner. After training, the final category semantic reference hints used for sample partitioning and inference are... ;

[0073] in, For text features, Provide category semantic reference hints, These are the correction items to be optimized. Within the framework of this invention, As a semantic representation of the category, it is used to measure cue-image consistency in the sample selection stage and participates in classification as a category cue in the final inference stage. This separation is used only during the training phase to absorb systematic biases and is not used directly as a final inference cue. This separation is crucial because it prevents the selector from directly relying on a cue that is continuously pulled towards semantically irrelevant directions during optimization. The next challenge is how to parameterize and constrain this cue. An ideal correction term should have sufficient expressive power to absorb recurring residual directions, but should not have the freedom to rewrite category semantics. Therefore, a sparse overparameterized form is adopted, using two free variables to represent the correction term:

[0074] in, The correction item to be optimized. and All are variables to be optimized for category k. ; When this form is initialized near zero, it exhibits a favorable optimization bias: most coordinates remain small unless certain dimensions are repeatedly reinforced during training. This characteristic perfectly aligns with the objectives of this invention— Only capture directions that consistently appear across multiple updates, while suppressing directions that only appear sporadically in a few samples; for each class k, a correction term to be optimized is introduced during the training phase. This is used to absorb systematic cross-modal mismatches in the frozen visual-linguistic space. Correspondingly, the text features during the training phase can be written as: This follows the "separation principle": It consistently serves as the primary semantic representation of the category, measuring cue-image consistency during the sample selection phase, and participating in classification as a category cue during the final inference phase; while Introduced only during the training phase to absorb systematic biases and prevent such biases from being directly written into the cue pool representation itself. This split prevents the sample selector from directly relying on a cue that is continuously pulled towards non-semantic directions during optimization. A sparse overparameterized form is used for... Perform parameterization; this form, when initialized near zero, keeps most coordinates small, preserving only the directions that persist throughout multiple updates.

[0075] Based on the sparse correction terms to be optimized, generate the sparse correction term matrix to be optimized; For example, the sparse correction term matrix to be optimized is as follows:

[0076] In the formula, The sparse correction term matrix to be optimized This is the first sparsity correction term to be optimized. For the first One sparse correction term to be optimized.

[0077] and ; The sparse correction term matrix to be optimized is regularized to generate regularization terms corresponding to the sparse correction terms to be optimized, which are used to construct the overall loss function. For example, apply structural constraints, i.e., regularization, to the sparse correction term matrix to be optimized:

[0078] In the formula, The regularization term corresponding to the sparse correction term to be optimized is... The weight coefficients for the orthogonal regularization term are... The sparse correction term matrix to be optimized The transpose of the sparse correction term matrix S to be optimized. Let I be the Frobenius norm of the sparse correction term matrix S to be optimized, and let I be the identity matrix. The weight coefficients for the L1 sparse regularization term; In the formula As an orthogonal regularization term, it encourages different categories to use different correction directions to mitigate cross-category leakage caused by label noise; The L1 sparse regularization term prevents the correction term from spreading to too many coordinates, reducing the possibility of tracking weak or unstable correlations.

[0079] During training, the relevant text representations are normalized to ensure scale stability in cosine similarity calculations. The sparse correction branch serves to absorb and freeze systematic cross-modal mismatches in the visual-linguistic space during training, preventing such biases from directly contaminating the semantic representation of the subject formed by the mixed cue pool. After training, the final retained category cue remains... In actual training, the cue pool parameters and the OPrompt parameters are jointly optimized. After adding a correction term, it can be used during the training phase to absorb and freeze systematic cross-modal mismatches in the visual-linguistic space, mitigating the interference of such biases on the semantics of the main cue. After training, the category cue generated by the hybrid cue pool is still ultimately used for sample segmentation and inference. OPrompt primarily serves to absorb bias during the training phase.

[0080] The inherent alignment noise refers to a systematic cross-modal feature representation bias, independent of category semantics, existing between the frozen text encoder and image encoder in the visual-language pre-trained model. This systematic cross-modal feature representation bias, during the prompt optimization process based on incorrect labels, can cause prompts to deviate from the true category semantics, resulting in prompt drift. Existing methods rely on prompt-image consistency for sample partitioning, but the prompts themselves can drift due to inherent alignment noise, leading to unreliable consistency signals and a vicious cycle of "prompt drift → unreliable partitioning → exacerbated drift." This invention constructs a hybrid prompt pool as a pure semantic prompt representation, used to measure the consistency between prompts and images during the sample selection stage and participating in classification decisions as category prompts during the final inference stage. Simultaneously, an OPrompt sparse correction branch is introduced during training to absorb the systematic cross-modal mismatch in the frozen visual-language space, reducing the contamination of the main prompt semantics by this type of bias. The two components have clearly defined roles: the hybrid cue pool is responsible for providing a stable and clean category semantic representation, while the OPrompt sparse correction branch is responsible for absorbing systematic biases during the training phase, thus ensuring that the final cue pool representation is suitable for both sample partitioning and final inference. In this invention, within the frozen CLIP space, cue tuning not only learns category semantics but also compensates for the systematic cross-modal mismatch between image and text features. This mismatch repeatedly occurs during optimization, continuously affecting the cue and gradually accumulating into semantic drift. Absorbing this mismatch causes the reference cue to deviate from its intended category semantics, thus compromising the stability of the sample selection signal. These observations inspired a "separation principle": to break the vicious cycle caused by drift, the semantic reference used for sample selection must be explicitly separated from the correction term used to compensate for mismatch during classification. OPrompt is designed according to this principle, absorbing recurring mismatches by introducing an explicit correction term while maintaining the purity of the reference cue itself. This step introduces a parameterized but sparse component to absorb systematic feature mismatches, thereby protecting the semantic integrity of the main cue pool. This method, by introducing the "separation principle," explicitly separates semantic learning from bias absorption, ensuring the reliability of the cue-image consistency signal upon which sample partitioning depends, thus breaking this vicious cycle. However, its limitation is that even with a relatively complete description, it may lack fine-grained cues and dataset-specific information unique to the target dataset. This step provides a stable semantic reference decoupled from transient training fluctuations for subsequent steps. Combining fixed semantic priors with components to be optimized: the former provides stable and noise-resistant anchors, preventing cues from drifting completely with training fluctuations under noisy conditions; the latter preserves the adaptability required for the model to achieve high performance under clean supervision.

[0081] In one implementation, based on the global probability matrix, an entropy-regularized optimal transmission algorithm is used to obtain calibrated pseudo-labels, and the training set is divided into a clean subset and a noisy subset using the pseudo-labels, including: Based on the global probability matrix, the sample image and the category are coupled through a global allocation plan using the entropy regularized optimal transfer algorithm under marginal constraints to generate a soft allocation plan. For example, in a few-sample setting, even a small number of mislabeled samples can dominate supervision within a certain category. A straightforward approach is to threshold the local confidence score, but this method is unstable: the confidence scales for different categories and cues are not consistent, and a uniform threshold often over-selects samples from categories with sharper cues, leaving other categories under-covered. What is needed is a globally calibrated partitioning signal, rather than a per-sample thresholding rule easily affected by local biases. Using entropy-regularized optimal transfer, samples and categories are coupled through a global allocation plan. Unlike independent thresholding, OT introduces competition between samples globally through marginal constraints, thereby obtaining calibrated allocation scores.

[0082] A soft allocation plan is obtained by solving an entropy-canonical optimal transport problem. Since maximizing affinity is desired, [the following is omitted as the original text is incomplete and cannot be translated]. Treat it as a cost matrix and solve for it:

[0083]

[0084] In the formula, For soft allocation plan, Let P be the cost matrix, and P represent the global allocation plan. This represents the transpose of the global allocation plan. Assign the element of the k-th class to the i-th sample image in the global assignment plan P. As constraints, Let K be a column vector of all 1s. Let N be a column vector of all 1s. This indicates that the sample-side marginal vector 'a' is obtained by summing the global allocation plan matrix P according to the category dimension. This indicates that the category-side marginal vector b is obtained by summing the global allocation plan matrix P along the sample dimensions; both are marginal constraints. ε is the coefficient of the entropy regularization term. and Using a uniform prior: , This forces the allocation to remain balanced globally, reducing the risk of collapse to a minority class even if there is a bias in local confidence levels.

[0085] Marginal constraints and This introduces a globally assigned prior. In practice, a uniform prior is used for both samples and classes.

[0086]

[0087] In the formula, This is the sample-side marginal prior corresponding to the i-th sample image. For the category-side marginal prior corresponding to the k-th category, The total number of samples in the training set; This represents the total number of categories; This forces the allocation to remain globally balanced, reducing the risk of collapse to minority classes even if local confidence levels are biased. The resulting allocation plan provides a globally calibrated signal. After normalizing the soft allocation plan by row, the class posterior of each sample image is obtained, and pseudo-labels for each sample image are generated based on the class posterior. For example, to By row normalization, we obtain the class posterior for each sample:

[0088] In the formula, Let be the posterior probability of the i-th sample image belonging to the k-th class. For soft allocation plan The i-th sample image is assigned to an element of the k-th category. Soft allocation plan The element of the j-th class is assigned to the i-th sample image. Subsequently defined:

[0089] In the formula, For the i-th pseudo-label, Let be the posterior probability of the i-th sample image belonging to the k-th category;

[0090] In the formula, For the i-th confidence level; Confidence level through marginal constraints It offers better comparability across different categories.

[0091] When the pseudo-label and the original label are consistent, the sample image corresponding to the pseudo-label and the original label are regarded as clean samples; otherwise, the sample image corresponding to the pseudo-label and the original label are regarded as noise samples. For example, refine the sample partitioning by combining the original labels: only when the pseudo-label matches the original label ( Only if the sample meets the following criteria will it be considered a clean sample; otherwise, it will be considered a noisy sample.

[0092] This method aggregates all clean samples to generate a clean subset and all noisy samples to generate a noisy subset. Traditional methods perform threshold judgments independently for each sample, resulting in inconsistent confidence scales across different categories and biased classification results. This method employs entropy-regularized optimal transport for global optimization, ensuring balance in category allocation through marginal constraints, making confidence scores comparable across different categories. Specifically, this invention constructs a global probability matrix and uses entropy-regularized optimal transport to solve for a global allocation plan, ensuring category-level allocation balance through uniform marginal constraints. Specifically, through entropy-regularized optimal transport (OT), samples and categories are coupled through a global allocation plan to obtain calibrated pseudo-labels, thereby dividing the samples into clean subsets. and noise subset This invention achieves globally balanced sample partitioning; the cue pool and OPrompt stabilize the text side of the selector, while MVV compresses multiple enhanced views into a single consensus feature. The visual side is stabilized. With these two components, cue-image consistency becomes a meaningful signal. Next, this signal is transformed into reliable sample partitions on the training set; in a few-shot setting, even a small number of mislabeled images can dominate supervision within a certain category. Meanwhile, in the cue-based consistency selection framework, subsequent partitioning decisions often rely on the logits generated by the current cue, creating a significant feedback effect. Therefore, a globally calibrated partitioning signal is needed, rather than a sample-by-sample thresholding rule easily affected by local biases; a direct approach is to set a threshold for local confidence scores, such as calculating the similarity between an image feature and its original label corresponding to the cue. However, this method is unstable in the current setting. The confidence scales of different categories and different cues are not consistent, and few-shot training further amplifies this difference. A uniform threshold often over-selects samples from categories with sharper cues, leaving other categories under-covered; therefore, an entropy-regularized optimal transfer method is used to couple samples and categories through a global allocation plan. Unlike independent thresholds, OT introduces competition between samples globally through marginal constraints, thereby obtaining calibrated allocation scores.

[0093] In one implementation, the step of differentially training the clean subset and the noisy subset to construct a robust cue tuning model includes: The clean subset is trained under supervision using the cross-entropy loss function to obtain the cross-entropy loss term; For example, for samples in a clean subset, supervised training is performed using cross-entropy loss, where the clean subset is obtained by partitioning the samples. This step allows for full utilization of samples with relatively reliable original labels, improving the model's class discrimination ability and completing parameter updates based on clean samples; for The samples in the dataset are considered to have sufficiently reliable original labels, which can be used as standard supervision signals. The cross-entropy loss term calculated based on the clean subset is then obtained.

[0094] In the formula, For cross-entropy loss term, For a clean subset, For the i-th sample image, The label is the one corresponding to the i-th sample image. The model will use the i-th sample image Predict as label The probability of the corresponding category; Cross-entropy exist Time is unbounded, and it is about The derivative is This allows mislabeled samples to have a significant impact. In contrast, MAE's loss is bounded and... The derivative of is a constant. As the gradient continues to propagate back through the softmax logit, the gradient of MAE is further multiplied by . Therefore when When it is very small, its impact will also be smaller.

[0095] The mean absolute error loss function is used to constrain the noise subset to obtain the mean absolute error loss term; For example, for samples in a noisy subset, training is constrained using the mean absolute error loss in probabilistic form, where the noisy subset is derived from the sample partitioning. This step reduces the impact of mislabeled data on model updates and minimizes the misleading effect of noisy samples on the training process, thereby improving the model's stability under noisy supervision. The original labels of the samples in the dataset may be incorrect. Therefore, a more conservative loss function is adopted to limit the impact of samples that do not match the current learned representation. The mean absolute error loss term, calculated based on the noisy subset, is obtained using the mean absolute error in probabilistic form.

[0096] In the formula, Mean absolute error loss term For a subset of noise, For the i-th sample image, The label is the one corresponding to the i-th sample image. The model will use the i-th sample image Predict as label The probability of the corresponding category; The overall loss function is used to calculate the overall loss by weighted summation of the regularization terms corresponding to the cross-entropy loss term, the mean absolute error loss term, and the sparse correction term to be optimized. For example, the cross-entropy loss corresponding to the clean subset, the mean absolute error loss corresponding to the noisy subset, and the regularization term corresponding to the sparse correction branch are weighted and summed to form the overall loss function. By optimizing this overall loss function, the cue pool parameters and the sparse correction branch parameters are jointly updated, thereby completing the current round of training and obtaining the model parameters for the next round of multi-view consensus feature calculation and sample re-partitioning. The overall loss function coordinates the roles of clean subset supervision, noisy subset suppression, and correction branch constraints to ensure that the training process utilizes reliable label information while limiting the negative impact of incorrect labels. After this step, the updated model state for the current round of sample partitioning is obtained. This model state is used for subsequent iterative optimization until the preset number of training rounds is reached or the convergence condition is met.

[0097] The overall loss function is as follows:

[0098] in, For the overall loss function, For cross-entropy loss term, The weighting coefficients for the mean absolute error loss term. This is the mean absolute error loss term. For the regularization term corresponding to the sparse correction term to be optimized; The loss term is used to balance the contribution of the noisy subset. The first term is used to guide the model to learn the correct class discrimination relationship by utilizing reliable label information from the clean subset. The second term is used to reduce the impact of erroneous labels in the noisy subset on model parameter updates. The third term is used to constrain the parameter distribution of the sparse correction branch, so that it maintains sparsity and inter-class diversity while absorbing systematic cross-modal mismatch. In each round of training, the corresponding loss terms are first calculated based on the clean subset and noisy subset obtained in step 5, and then the losses are weighted and summed to obtain the overall loss used in the current round of training. Subsequently, while keeping the CLIP image encoder and text encoder frozen, only the parameters to be optimized and the parameters of the sparse correction branch in the mixed cue pool are updated. After the parameter update is completed, the class cue representation and correction term after the current round of training are obtained, and they are used for the next round of multi-view consensus feature calculation, global sample partitioning, and differential training.

[0099] Based on the overall loss, the sparse correction term to be optimized and the category semantic reference cues in the hybrid cue pool are updated. Based on the updated category semantic reference cues and the sparse correction term to be optimized, iterative training is performed until a preset number of training rounds are reached. The final category semantic reference cues are retained, and the sparse correction term to be optimized is discarded, thus completing the construction of the robust cue tuning model.

[0100] For example, by iteratively executing the above steps, the model parameters are gradually optimized and the sample partitioning is updated, achieving a virtuous cycle between cue drift and reliable sample selection. After reaching the preset number of training rounds, the robust cue-tuned model is output. The output model uses category cues generated by the hybrid cue pool for classification during the inference phase. In each iteration, the continuous context parameters to be optimized and the sparse correction branch parameters in the hybrid cue pool are updated, and the MVV consensus features are recalculated, samples are re-partitioned, and differential training is performed based on the updated cue.

[0101] Taking animal recognition scenarios as an example, such as Figure 2 This diagram illustrates the organized structure of the animal recognition scenario throughout the learning process. First, a hybrid cue pool is constructed to provide a stable semantic reference decoupled from transient training fluctuations for subsequent steps. Second, an OPrompt module is introduced, an overparameterized but sparsity component used to absorb systematic feature mismatches, thus preserving the semantic integrity of the main cue pool. Simultaneously, Multi-View Visual Consensus (MVV) is used to aggregate features from multiple augmented views of the same image into a single consensus feature to stabilize the visual representation. After obtaining the corrected cue representation and stable image features, the samples are globally divided into a clean subset and a noisy subset using optimal transfer (OT). Finally, cross-entropy (CE) and mean absolute error (MAE) are used to train these two subsets, respectively. This invention builds upon a pre-trained CLIP model, keeping the CLIP's image encoder and text encoder frozen throughout the tuning process, learning only cue-related parameters. In the figure, , and All represent image features extracted by the image encoder; the image uses a unified English prompt template "A photo of a {class} with xxx", meaning "a photo of a certain type of image containing a certain visual attribute"; parametric modeling. in, The correction item to be optimized. and All are variables to be optimized for category k. α represents the weight coefficient corresponding to each prompt in the prompt word pool, used to weight and combine multiple prompts to generate category text features; MeanShift represents the mean shift algorithm, used to aggregate multi-view image features of the same image to obtain a stable visual representation; Dclean represents the clean sample subset obtained from the partitioning; Dnoisy represents the noisy sample subset obtained from the partitioning. CE Loss is the cross-entropy loss value; MAE Loss is the mean absolute error loss value; the beneficial effects of this invention are as follows: (1) Significantly improves classification accuracy under noisy labels Under label noise conditions, especially in high-noise scenarios, the classification accuracy of the method in this invention significantly outperforms existing methods. Experimental results show that, with a few-sample setting of only 16 training samples per category and 75% symmetric noise as an example, the classification accuracy of the method in this invention on the OxfordPets pet dataset is improved from 70.77% to 75.13%, an improvement of 4.36 percentage points; and the classification accuracy on the UCF101 action / video behavior classification dataset is improved from 60.53% to 66.14%, an improvement of 5.61 percentage points. Under 75% asymmetric noise conditions, the classification accuracy of the method in this invention on the OxfordPets pet dataset, Flowers102 flower classification, DTD texture classification, and UCF101 action / video behavior classification dataset is improved by 8.60, 3.95, 4.08, and 1.82 percentage points respectively compared to NLPrompt, indicating that the invention still has strong classification robustness under high noise conditions. Furthermore, in the real-world noisy dataset Food101N, the classification accuracy of the method in this invention reaches 77.39%, higher than NLPrompt's 76.46%. This demonstrates that the invention is not dependent on specific image categories and is applicable to general image classification scenarios. It also shows that the invention is not only suitable for synthetic noise scenarios but also exhibits stable performance under real-world data acquisition noise conditions. This is because the invention effectively alleviates the cue drift problem and improves the quality of sample partitioning through the synergistic effect of a hybrid cue pool, sparse correction branch, multi-view visual consensus, and a global sample partitioning mechanism, thereby ensuring the training stability and final classification performance of the model under high-noise conditions. Existing methods typically use the consistency between the learned cue and the image directly for sample partitioning. However, when the cue itself drifts due to inherent alignment noise, this consistency signal loses its reliability, leading to incorrect samples being classified into clean subsets. This invention improves the reliability of sample partitioning by explicitly separating the category semantic representation from the training-stage bias absorption term and combining multi-view visual consensus with an optimal transmission global partitioning mechanism. Experimental results show that, under 75% asymmetric noise conditions on the DTD dataset, the method of this invention improves the clean sample classification accuracy from 37.70% to 56.84%, while increasing the number of samples judged as clean and actually correct from 118 to 133, and ultimately improving the test accuracy from 28.37% to 32.45%. These results demonstrate that this invention can effectively reduce erroneous selection caused by cue drift and further translate more reliable sample classification into better final classification performance. Implementation: A stable semantic reference anchor is provided through a hybrid cue pool, combined with the OPrompt sparse correction module to compensate for cross-modal mismatch, and multi-view visual consensus (MVV) is used to stabilize image feature representations. This allows the model to accurately identify clean samples and learn effectively even under noisy conditions, ultimately achieving a significant improvement in classification accuracy.If the same loss function is used to train all samples, mislabeled samples will negatively impact the model, causing it to learn incorrect patterns. Leveraging the memory effect of deep networks, the network typically learns from clean samples first and then gradually fits to noisy samples. Therefore, after dividing the samples into clean and noisy subsets, standard cross-entropy loss is used on the clean subset to fully utilize reliable supervision, while a more conservative MAE loss is used on the noisy subset to limit the impact of mislabeled samples. This step optimizes the clean and noisy subsets obtained in step five using cross-entropy and mean absolute error respectively, thereby mitigating the feedback loop between cue drift and unreliable selection.

[0102] Example 2: Based on the same inventive concept, this invention also provides an application method for a robust suggestion tuning model, such as... Figure 3 As shown, it includes: Based on the image classification scenario, the probability of the image in each category is obtained by optimizing the model through robust prompts; The category to which the image belongs is determined based on the probability of each category.

[0103] The robust hint tuning model is constructed using any of the robust hint tuning model construction methods described above.

[0104] Example 3: Based on the same inventive concept, this invention also provides a robust suggestion tuning model construction system, such as... Figure 4 As shown, it includes: a global probability matrix calculation module, a training set partitioning module, and a robust hint-based model optimization construction module; The global probability matrix calculation module is used to calculate the consensus features of the training set for the image classification scene, and obtain the global probability matrix of the training set based on the consensus features and a pre-built hybrid cue pool. The training set partitioning module is used to obtain calibrated pseudo-labels based on the global probability matrix using the entropy regularized optimal transmission algorithm, and to partition the training set into a clean subset and a noisy subset using the pseudo-labels. The robust suggestion tuning model construction module is used to iteratively optimize the hybrid suggestion pool based on the clean subset and the noisy subset using the overall loss function, thereby completing the construction of the robust suggestion tuning model. The training set includes sample images and original labels; the original labels include correct labels and incorrect labels; the hybrid cue pool includes category semantic reference cue for the category corresponding to the sample images; the overall loss function includes a cross-entropy loss term, a mean absolute error loss term, and a regularization term corresponding to the sparse correction term to be optimized for absorbing inherent alignment noise.

[0105] Preferably, the global probability matrix calculation module includes: The consensus feature calculation submodule is used to calculate the consensus features of each sample image generated from multiple augmented views in the training set for image classification scenarios. The prediction probability calculation submodule is used to calculate the prediction probability of each sample image with each category based on the consensus features of each sample image and the semantic reference hints of the category of each sample image in the mixed hint pool. The global probability matrix construction submodule is used to construct a global probability matrix based on the predicted probabilities. The image classification scenarios include one or more of the following: animal recognition scenarios, flower classification scenarios, texture classification scenarios, action behavior classification scenarios, and video behavior classification scenarios.

[0106] Preferably, the consensus feature calculation submodule is specifically used for: For each sample image in the training set for an image classification scenario, multiple augmented views are generated; The image encoder, which is frozen in the visual language pre-trained model, is used to extract the image features corresponding to each augmented view. The mode estimation method is used to aggregate the image features of all augmented views corresponding to each sample image into consensus features; The visual language pre-trained model includes a pre-trained image encoder and a text encoder; the enhanced view includes one or more of the following: a randomly cropped view, a horizontally flipped view, and a color-transformed view.

[0107] Preferably, the system further includes: a hybrid prompt pool construction module, used for: For each category of the sample images in the training set, a large language model is used to generate multiple short sentences describing visual attributes using a unified description template; the visual attributes include visual cues that distinguish the category from other categories. The frozen text encoder in the visual language pre-trained model is used to encode multiple short sentences, generate multiple text features, and average the multiple text features to obtain a fixed semantic prototype for each category. After concatenating the category lexical units of each category with the pre-obtained shared lexical units, the text encoder of the visual language pre-trained model is used to obtain the prompt features to be optimized. Based on the fixed semantic prototype, the prompt features to be optimized, and the hybrid weights to be optimized, semantic reference prompts for each category are generated. Aggregate all category semantic reference hints to generate a hybrid hint pool; The shared lexical units are derived from pre-set continuous context lexical units to be optimized that are shared across all categories.

[0108] Preferably, the system further includes: a regularization term setting module corresponding to the sparse correction term to be optimized, used for: Using a sparse overparameterized form, based on two variables to be optimized, a sparse correction term to be optimized is set for each category semantic reference cue in the hybrid cue pool to absorb inherent alignment noise; Based on the sparse correction terms to be optimized, generate the sparse correction term matrix to be optimized; The sparse correction term matrix to be optimized is regularized to generate regularization terms corresponding to the sparse correction terms to be optimized, which are used to construct the overall loss function. The intrinsic alignment noise is a systematic cross-modal feature representation bias that exists between the frozen text encoder and image encoder in the visual language pre-trained model and is independent of category semantics. This systematic cross-modal feature representation bias can cause the prompts to deviate from the true category semantics during the prompt optimization process based on the wrong labels, resulting in prompt drift.

[0109] Preferably, the training set partitioning module is specifically used for: Based on the global probability matrix, the sample image and the category are coupled through a global allocation plan using the entropy regularized optimal transfer algorithm under marginal constraints to generate a soft allocation plan. After normalizing the soft allocation plan by row, the class posterior of each sample image is obtained, and pseudo-labels for each sample image are generated based on the class posterior. When the pseudo-label and the original label are consistent, the sample image corresponding to the pseudo-label and the original label are regarded as clean samples; otherwise, the sample image corresponding to the pseudo-label and the original label are regarded as noise samples. Summarize all clean samples to generate a clean subset, and summarize all noisy samples to generate a noisy subset.

[0110] Preferably, the robust hint tuning model construction module is specifically used for: The clean subset is trained under supervision using the cross-entropy loss function to obtain the cross-entropy loss term; The mean absolute error loss function is used to constrain the noise subset to obtain the mean absolute error loss term; The overall loss function is used to calculate the overall loss by weighted summation of the regularization terms corresponding to the cross-entropy loss term, the mean absolute error loss term, and the sparse correction term to be optimized. Based on the overall loss, the sparse correction term to be optimized and the category semantic reference cues in the hybrid cue pool are updated. Based on the updated category semantic reference cues and the sparse correction term to be optimized, iterative training is performed until a preset number of training rounds are reached. The final category semantic reference cues are retained, and the sparse correction term to be optimized is discarded, thus completing the construction of the robust cue tuning model.

[0111] Example 4: Based on the same inventive concept, this invention also provides an application system for a robust suggestion tuning model, such as... Figure 5 As shown, it includes: an image category probability calculation module and an image category determination module; The image category probability calculation module is used to obtain the probability of the image in each category based on the image classification scenario and through robust prompting to optimize the model. The image category determination module is used to determine the category to which the image belongs based on the probability of each category.

[0112] The robust hint tuning model is constructed using any of the robust hint tuning model construction methods described above.

[0113] Example 5 like Figure 6 As shown, the present invention also provides an electronic device, which may be a computer device, a microcontroller device, a smart mobile device, etc. The electronic device in this embodiment may include a processor, a memory, a transceiver component, etc. The memory, processor, and transceiver component are connected via a bus; the memory can be used to store executable programs, and an exemplary executable program may include instructions; the processor is used to execute the instructions stored in the memory. The memory can also be used to store data, which can be accessed and / or modified when instructions are executed.

[0114] The processor may be a Central Processing Unit (CPU), or it may be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. It is the computing and control core of the terminal, and it is suitable for implementing one or more instructions. Specifically, it is suitable for loading and executing one or more instructions in the storage medium to implement the corresponding method flow or corresponding function, so as to realize the steps of the robust hint tuning model construction method in the above embodiments.

[0115] Example 6 Based on the same inventive concept, this invention also provides a readable storage medium, specifically an electronic device readable storage medium (Memory). This readable storage medium is a memory device within an electronic device used to store programs and data. It is understood that the storage medium here can include both built-in storage media within the electronic device and extended storage media supported by the electronic device. The storage medium provides storage space, which stores the terminal's operating system. Furthermore, this storage space also stores one or more instructions suitable for loading and execution by a processor. These instructions can be one or more executable programs (including program code). It should be noted that the storage medium here can be high-speed RAM or non-volatile memory, such as at least one disk storage device. Loading and executing one or more instructions stored in the storage medium by the processor can implement the steps of the robust hint tuning model construction method in the above embodiments.

[0116] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0117] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0118] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0119] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0120] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and not to limit its scope of protection. Although the present invention has been described in detail with reference to the above embodiments, those skilled in the art should understand that after reading the present invention, they can still make various changes, modifications or equivalent substitutions to the specific implementation methods of the application, but these changes, modifications or equivalent substitutions are all within the scope of protection of the claims pending approval.

Claims

1. A method for constructing a robust suggestion tuning model, characterized in that, include: Calculate the consensus features of the training set for the image classification scene, and obtain the global probability matrix of the training set based on the consensus features and a pre-constructed hybrid cue pool. Based on the global probability matrix, the entropy regularized optimal transmission algorithm is used to obtain calibrated pseudo-labels, and the training set is divided into a clean subset and a noisy subset using the pseudo-labels. Based on the clean subset and the noisy subset, the hybrid cue pool is iteratively optimized using the overall loss function to complete the construction of the robust cue tuning model; The training set includes sample images and original labels; the original labels include correct labels and incorrect labels; the hybrid cue pool includes category semantic reference cue for the category corresponding to the sample images; the overall loss function includes a cross-entropy loss term, a mean absolute error loss term, and a regularization term corresponding to the sparse correction term to be optimized for absorbing inherent alignment noise.

2. The method as described in claim 1, characterized in that, The consensus features of the training set for the image classification scenario are used to obtain the global probability matrix of the training set based on the consensus features and a pre-constructed hybrid cue pool, including: Based on the training set of the image classification scenario, multiple augmented views are generated for each sample image, and the consensus features of each sample image are calculated. Based on the consensus features of each sample image and the semantic reference hints of the category of each sample image in the hybrid hint pool, calculate the prediction probability of each sample image with each category; Construct a global probability matrix based on the predicted probabilities; The image classification scenarios include one or more of the following: animal recognition scenarios, flower classification scenarios, texture classification scenarios, action and behavior classification scenarios, video behavior classification scenarios, and food image classification scenarios.

3. The method as described in claim 2, characterized in that, For each sample image generated in the training set based on the image classification scenario, multiple augmented views are calculated, and consensus features for each sample image are computed, including: For each sample image in the training set for an image classification scenario, multiple augmented views are generated; The image encoder, which is frozen in the visual language pre-trained model, is used to extract the image features corresponding to each augmented view. The mode estimation method is used to aggregate the image features of all augmented views corresponding to each sample image into consensus features; The visual language pre-trained model includes a pre-trained image encoder and a text encoder; the enhanced view includes one or more of the following: a randomly cropped view, a horizontally flipped view, and a color-transformed view.

4. The method as described in claim 1 or 2, characterized in that, The hybrid cue pool is constructed as follows: For each category of the sample images in the training set, a large language model is used to generate multiple short sentences describing visual attributes using a unified description template; the visual attributes include visual cues that distinguish the category from other categories. The frozen text encoder in the visual language pre-trained model is used to encode multiple short sentences, generate multiple text features, and average the multiple text features to obtain a fixed semantic prototype for each category. After concatenating the category lexical units of each category with the pre-obtained shared lexical units, the text encoder of the visual language pre-trained model is used to obtain the prompt features to be optimized. Based on the fixed semantic prototype, the prompt features to be optimized, and the hybrid weights to be optimized, semantic reference prompts for each category are generated. Aggregate all category semantic reference hints to generate a hybrid hint pool; The shared lexical units are derived from pre-set continuous context lexical units to be optimized that are shared across all categories.

5. The method as described in claim 4, characterized in that, After aggregating semantic reference cues from all categories and generating a hybrid cue pool, the process also includes: Using a sparse overparameterized form, based on two variables to be optimized, a sparse correction term to be optimized is set for each category semantic reference cue in the hybrid cue pool to absorb inherent alignment noise; Based on the sparse correction terms to be optimized, generate the sparse correction term matrix to be optimized; The sparse correction term matrix to be optimized is regularized to generate regularization terms corresponding to the sparse correction terms to be optimized, which are used to construct the overall loss function. The intrinsic alignment noise is a systematic cross-modal feature representation bias that exists between the frozen text encoder and image encoder in the visual language pre-trained model and is independent of category semantics. This systematic cross-modal feature representation bias can cause the prompts to deviate from the true category semantics during the prompt optimization process based on the wrong labels, resulting in prompt drift.

6. The method as described in claim 1, characterized in that, Based on the global probability matrix, the entropy-regularized optimal transmission algorithm is used to obtain calibrated pseudo-labels. These pseudo-labels are then used to divide the training set into a clean subset and a noisy subset, including: Based on the global probability matrix, the sample image and the category are coupled through a global allocation plan using the entropy regularized optimal transfer algorithm under marginal constraints to generate a soft allocation plan. After normalizing the soft allocation plan by row, the class posterior of each sample image is obtained, and pseudo-labels for each sample image are generated based on the class posterior. When the pseudo-label and the original label are consistent, the sample image corresponding to the pseudo-label and the original label are regarded as clean samples; otherwise, the sample image corresponding to the pseudo-label and the original label are regarded as noise samples. Summarize all clean samples to generate a clean subset, and summarize all noisy samples to generate a noisy subset.

7. The method as described in claim 1, characterized in that, The step of differentially training the clean subset and the noisy subset to construct a robust cue tuning model includes: The clean subset is trained under supervision using the cross-entropy loss function to obtain the cross-entropy loss term; The mean absolute error loss function is used to constrain the noise subset to obtain the mean absolute error loss term; The overall loss function is used to calculate the overall loss by weighted summation of the regularization terms corresponding to the cross-entropy loss term, the mean absolute error loss term, and the sparse correction term to be optimized. Based on the overall loss, the sparse correction term to be optimized and the category semantic reference cues in the hybrid cue pool are updated. Based on the updated category semantic reference cues and the sparse correction term to be optimized, iterative training is performed until a preset number of training rounds are reached. The final category semantic reference cues are retained, and the sparse correction term to be optimized is discarded, thus completing the construction of the robust cue tuning model.

8. A method for applying a robust suggestion tuning model, characterized in that, include: Based on the image classification scenario, the probability of the image in each category is obtained by optimizing the model through robust prompts; The category to which the image belongs is determined based on the probability of each category. The robust hint tuning model is constructed using a method for constructing a robust hint tuning model as described in any one of claims 1 to 7.

9. A system for constructing a robust suggestion tuning model, characterized in that, include: The module includes a global probability matrix calculation module, a training set partitioning module, and a robust hint-based model optimization construction module. The global probability matrix calculation module is used to calculate the consensus features of the training set for the image classification scene, and obtain the global probability matrix of the training set based on the consensus features and a pre-built hybrid cue pool. The training set partitioning module is used to obtain calibrated pseudo-labels based on the global probability matrix using the entropy regularized optimal transmission algorithm, and to partition the training set into a clean subset and a noisy subset using the pseudo-labels. The robust suggestion tuning model construction module is used to iteratively optimize the hybrid suggestion pool based on the clean subset and the noisy subset using the overall loss function, thereby completing the construction of the robust suggestion tuning model. The training set includes sample images and original labels; the original labels include correct labels and incorrect labels; the hybrid cue pool includes category semantic reference cue for the category corresponding to the sample images; the overall loss function includes a cross-entropy loss term, a mean absolute error loss term, and a regularization term corresponding to the sparse correction term to be optimized for absorbing inherent alignment noise.

10. An application system for a robust suggestion tuning model, characterized in that, include: Image category probability calculation module and image category determination module; The image category probability calculation module is used to obtain the probability of the image in each category based on the image classification scenario and through robust prompting to optimize the model. The image category determination module is used to determine the category to which the image belongs based on the probability of each category. The robust hint tuning model is constructed using a method for constructing a robust hint tuning model as described in any one of claims 1 to 7.

Images