Uncertainty-guided dynamic prompt fine tuning method, system and equipment and medium

Abstract

The invention discloses an uncertainty-guided dynamic prompt fine tuning method, system and device and a medium, and is applied to the technical field of computer vision, and the method comprises the steps: carrying out the visual feature extraction of an input image, and calculating an uncertainty scalar based on the extracted visual feature; inputting the uncertainty scalar into an uncertainty injection network to output a cue word disturbance vector; adding the obtained basic cue word and the scaled cue word disturbance vector to generate a dynamic cue word corresponding to the input image; encoding the dynamic cue words through a pre-trained text encoder to obtain text features; calculating the similarity between the visual features and the text features, and determining a classification result of the input image according to a similarity calculation result; according to the method, the generalization ability and decision interpretability of the model on distributed data are improved by constructing a lightweight cross-modal collaboration mechanism for driving dynamic generation of the text cue words by using the visual uncertainty scalar.

Description

Technical Field

[0001] This invention relates to the field of computer vision technology, and in particular to a method, system, device, and medium for dynamic prompting fine-tuning guided by uncertainty. Background Technology

[0002] With the emergence of large-scale vision-language pre-trained models, cue word-based fine-tuning techniques have become a key paradigm for efficiently adapting pre-trained models to downstream vision tasks. The core idea is to parameterize the originally human-designed, discrete text cue templates into continuous, learnable vectors. During fine-tuning, only these cue vectors are optimized, while the massive pre-trained model backbone is frozen, thus achieving task adaptation with extremely low parameter costs. However, existing cue word learning techniques still have significant limitations in terms of dynamic adaptability, computational efficiency, and cross-modal collaboration.

[0003] To overcome these shortcomings, this application proposes an uncertainty-guided dynamic cueing fine-tuning method, system, device, and medium. Summary of the Invention

[0004] The purpose of this application is to provide a dynamic cue fine-tuning method, system, device, and medium guided by uncertainty, aiming to solve the problems of large number of parameters and low computational efficiency caused by the reliance on high-dimensional feature generation in existing dynamic cue word learning methods, as well as insufficient generalization ability caused by the lack of cross-modal collaborative mechanisms.

[0005] To achieve the above objectives, this application provides the following technical solution: Firstly, this application provides a dynamic prompting fine-tuning method guided by uncertainty, the steps of which include: Visual features are extracted from the input image, and an uncertainty scalar is calculated based on the extracted visual features. The uncertainty scalar is input into the uncertainty injection network to output the cue word perturbation vector; The acquired basic prompt words are added to the scaled prompt word perturbation vector to generate dynamic prompt words corresponding to the input image; The dynamic prompt words are encoded using a pre-trained text encoder to obtain text features; Calculate the similarity between the visual features and the text features, and determine the classification result of the input image based on the similarity calculation result.

[0006] Secondly, this application provides a dynamic prompting and fine-tuning system guided by uncertainty, specifically including: The feature extraction and uncertainty estimation module is used to extract visual features from the input image and calculate the uncertainty scalar based on the extracted visual features. The uncertainty injection network module is a trainable parameter module used to input the uncertainty scalar into the uncertainty injection network to output a cue word perturbation vector; The dynamic prompt word generation module is used to add the acquired basic prompt words to the scaled prompt word perturbation vector to generate dynamic prompt words corresponding to the input image; The text encoding module is a pre-trained model with frozen parameters, used to encode the dynamic prompt words through a pre-trained text encoder to obtain text features; The similarity calculation and classification module is used to calculate the similarity between the visual features and the text features, and to determine the classification result of the input image based on the similarity calculation result.

[0007] Thirdly, this application provides a computer device, the computer device including a processor and a memory coupled to the processor, wherein the memory stores program instructions for implementing a dynamic prompting fine-tuning method guided by uncertainty; the processor is used to execute the program instructions stored in the memory to implement a dynamic prompting fine-tuning method guided by uncertainty.

[0008] Fourthly, this application provides a computer-readable storage medium storing processor-executable program instructions for executing a dynamic prompting fine-tuning method with uncertainty guidance.

[0009] This application provides a dynamic prompting fine-tuning method, system, device, and medium guided by uncertainty, which has the following beneficial effects: Compared with existing technologies, this application achieves cross-modal collaborative optimization by synchronously driving visual and textual dual-path adaptation through the same uncertainty signal. Specifically, it uses uncertainty to drive a lightweight Uncertainty Injection Network (UIN) to generate text-side cue word perturbation vectors, enabling text semantics to dynamically expand with the increase of visual uncertainty. This directly solves the problem of misalignment of visual-text alignment in the AFF scheme under high uncertainty, thus achieving better generalization performance on out-of-distribution data. At the same time, since it uses scalar rather than high-dimensional features as conditions, the number of UIN parameters is extremely low (33k), which makes this application lightweight and efficient, far superior to similar dynamic methods, while maintaining high accuracy in the source domain. Attached Figure Description

[0010] Figure 1 This is a flowchart illustrating an uncertainty-guided dynamic prompting fine-tuning method according to Embodiment 1 of this application; Figure 2 This is an architectural block diagram of an uncertainty-guided dynamic prompting fine-tuning method according to Embodiment 1 of this application; Figure 3This is a schematic diagram of the structure of an uncertainty-guided dynamic prompting fine-tuning system according to Embodiment 2 of this application; Figure 4 This is a schematic diagram of the computer device structure according to Embodiment 3 of this application; Figure 5 This is a schematic diagram of the storage medium structure of Embodiment 4 of this application. Detailed Implementation

[0011] It should be understood that the specific embodiments described herein are merely illustrative of this application and are not intended to limit this application.

[0012] The following analysis, based on relevant technologies, examines existing solutions.

[0013] Existing technologies mainly revolve around two types of solutions: static prompt word learning and dynamic prompt word generation. However, both have inherent technical flaws, which are analyzed in detail below: Static cue word learning schemes, such as CoOp (Co-Context Optimization), learn cue vectors that remain fixed after training. This approach ignores the significant differences in visual content complexity, sharpness, and recognition difficulty among different input image samples. For blurry, occluded images, or images whose distribution differs greatly from the training data, the fixed textual semantic representation cannot effectively match dynamically changing visual features. This leads to a significant performance degradation of the model in out-of-distribution (OOD) scenarios, such as sketches, artwork, or images containing adversarial perturbations. The root cause lies in the lack of a perception and response mechanism for input uncertainty.

[0014] To overcome the shortcomings of static cue words, dynamic solutions, such as Conditional Context Optimization (CoCoOp), have been proposed. These solutions typically introduce an additional meta-network (Meta-Net), using the complete high-dimensional image features extracted in the previous stage as conditional input to dynamically generate bias vectors for adjusting the base cue words. While achieving dynamic cue word processing, this approach introduces a heavy computational burden: its meta-network has a massive number of parameters (e.g., up to 2.1M), severely contradicting the "lightweight" principle of cue word learning and leading to increased inference latency. More importantly, the large number of parameters easily leads to overfitting in small-sample training scenarios, impairing the model's generalization robustness, thus offsetting the benefits of dynamic processing against the risk of overfitting.

[0015] Further improvements, such as Adaptive Feature Fusion (AFF), have begun to address the uncertainty of visual features and dynamically fuse features from different sources accordingly. However, the dynamic adjustment mechanism of such schemes is unidirectional and incomplete: it only operates on the visual feature side, adjusting the fusion weights of visual features based on visual uncertainty, while the text-side prompts remain static. This results in the text semantic space failing to adjust collaboratively when visual features undergo adaptive changes due to high uncertainty, causing misalignment in the alignment relationship between modalities and limiting the upper limit of performance improvement. Furthermore, existing dynamic generation processes are often a "black box" lacking interpretability, making it difficult to trace and explain the decision-making basis for the generation of specific prompts, reducing the model's credibility in critical applications.

[0016] Therefore, existing technologies struggle to achieve a dynamic prompt word generation mechanism capable of sensing input uncertainty and performing deep collaborative adaptation between visual and textual aspects while maintaining low parameter costs and high computational efficiency. This has become a key technical bottleneck hindering the efficient and reliable application of visual-language models in complex and open scenarios.

[0017] This application addresses the contradictions in dynamic adaptability, computational efficiency, and cross-modal collaboration among existing visual-language prompting learning methods. Through a lightweight, uncertainty-driven cross-modal collaboration mechanism, it achieves a significant improvement in generalization ability for out-of-distribution (OOD) data with extremely low computational overhead.

[0018] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of the embodiments. Based on the embodiments of this application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of this application.

[0019] Example 1 Please see Figure 1 This is a flowchart illustrating an uncertainty-guided dynamic prompting fine-tuning method according to Embodiment 1 of this application; the steps include: S1: Extract visual features from the input image and calculate the uncertainty scalar based on the extracted visual features.

[0020] In this embodiment, robust visual feature representations are extracted from the input image, and a scalar signal that quantifies the model's confidence in predicting the image content is generated simultaneously. This process is mainly implemented through a visual processing pathway that includes a dual-branch visual encoder and an adaptive feature fusion module.

[0021] First, the input image is fed in parallel to two pre-trained visual encoders whose parameters are frozen throughout the entire method execution. These two encoders are preferably models with different architectures or complementary training objectives: the first visual encoder uses the Vision Transformer (ViT), and the second visual encoder uses the visual encoder from the CLIP model (CLIP-ViT). They encode the input image separately and independently output high-dimensional feature vectors, denoted as the first visual feature vector. With the second visual feature vector The dual-branch visual encoder aims to comprehensively utilize different pre-trained models to extract visual representations, providing a richer feature base for subsequent fusion.

[0022] The obtained first visual feature vector With the second visual feature vector Both inputs are fed into an Adaptive Feature Fusion (AFF) module whose parameters are also frozen. This module dynamically determines the fusion weights based on the information inherent in the input features themselves. In one specific implementation, a lightweight gating network or weighted network is used to calculate the weight coefficients of the two input feature vectors separately, followed by a weighted summation, ultimately outputting a single visual feature that fuses information from both inputs. This fused feature exhibits stronger robustness and representational ability compared to any single feature.

[0023] In obtaining fused visual features Then, uncertainty quantification is performed based on this feature. Specifically, a lightweight auxiliary classifier, such as a simple multilayer perceptron (MLP), is applied to... The auxiliary classifier outputs a probability distribution across all candidate categories. The information entropy of this probability distribution is calculated to obtain an uncertainty scalar ranging from [0,1]. .That The larger the value, the more uncertain the model's prediction of the current input is; for example, the image is blurry, there is occlusion, or the feature lies between multiple categories. The smaller the value, the more certain the prediction. Uncertainty scalar. This serves as a key control signal for dynamically adjusting the guiding text prompts in subsequent steps.

[0024] Optionally, the input image is input into a frozen ViT encoder to obtain a first visual feature vector; The input image is input into the frozen CLIP-ViT encoder to obtain the second visual feature vector; The first visual feature vector and the second visual feature vector are input into the parameter-frozen adaptive feature fusion module to obtain visual features; The visual features are classified and predicted, and the information entropy of the probability distribution of the prediction results is calculated. The information entropy is used as the uncertainty scalar.

[0025] This embodiment extracts visual feature vectors using a frozen dual encoder, obtains fused visual features via a frozen adaptive fusion module, and finally generates an uncertainty scalar by calculating the information entropy of the feature prediction distribution. This process achieves an effective measure of the difficulty of input image recognition without introducing a large number of trainable parameters, and the visual features... This will be used for subsequent similarity calculations, while the uncertainty scalar Inject uncertainty into the network.

[0026] S2: Input the uncertainty scalar into the uncertainty injection network to output the cue word perturbation vector.

[0027] In this embodiment, the uncertainty scalar generated in step S1 is transformed into a high-dimensional vector that can directly affect the semantic space of the text, thereby providing a precise operational quantity for dynamically adjusting text prompts. This is specifically achieved through a trainable uncertainty injection network.

[0028] The uncertainty injection network is structurally a lightweight multilayer perceptron, employing a two-layer fully connected structure. The first fully connected layer receives the input uncertainty scalar. The first layer maps the intermediate features to an intermediate feature space, such as 64 dimensions, and then processes them through a non-linear activation function, such as ReLU, to introduce the necessary expressive power. The second fully connected layer then maps these intermediate features to the final output dimension. This output dimension is designed to match the text embedding dimension used in the system, such as 512 dimensions, ensuring that the generated vectors can be directly computed with the text prompts. Its mathematical expression is: , in, These are the weights of the two network layers, These are the bias parameters for the two network layers, respectively. This is the output prompt word perturbation vector.

[0029] The core design feature of this Uncertainty Injection Network (UIN) lies in its extreme parameter efficiency. Compared to existing dynamic generative networks that rely on high-dimensional image features as conditions and have millions of parameters, the UIN in this application uses a single scalar... For input, the total number of parameters can be strictly controlled at a very low level, approximately 33k. This is suitable when the input image is simple and straightforward. When the value approaches 0, the network output The amplitude also approaches zero, having a negligible effect on adjusting text prompts; when the input image is difficult to recognize, As the amplitude increases, the network will output a value whose magnitude and direction have both been learned. This perturbation vector aims to guide the semantics of the base cue words in a more generalized and adaptable direction to uncertain scenarios.

[0030] Optionally, the uncertainty injection network is a multilayer perceptron, into which the uncertainty scalar is input and the prompt word perturbation vector is output.

[0031] This embodiment uses a minimally sized trainable multilayer perceptron network to transform uncertain scalars into high-dimensional semantic perturbation vectors, thereby achieving a core mechanism for efficient and accurate cross-modal dynamic collaboration.

[0032] S3: Add the acquired basic prompt words to the scaled prompt word perturbation vector to generate dynamic prompt words corresponding to the input image.

[0033] In this embodiment, the prompt word perturbation vector generated in step S2 is integrated into the dynamic prompt generator to generate prompt words that dynamically adapt to the input content. This process eliminates the redundant design of generating different perturbations independently for each prompt word position.

[0034] Specifically, we define a learnable basic cue word tensor, denoted as . ;in For the length of the prompt word (e.g., 16), each It is dimensional vectors (e.g., ), which corresponds to the continuous representation of the text embedding space. It can be viewed as a set of initial, task-related, learnable contexts.

[0035] The prompt word perturbation vector injected into the network output by uncertainty Its dimensions are also To enable the model to adaptively control the perturbation strength, a learnable scalar scaling factor is introduced. The perturbation vector With this scaling factor Multiplying these results in a scaled perturbation vector for the prompt words. This operation allows the model to learn and adjust the degree of uncertainty's influence on the final prompt words based on the overall task requirements.

[0036] Through formula Dynamic prompts are generated by specifically by combining the scaled prompt perturbation vector with the base prompt tensor. This is obtained by adding each vector in the vector array element by element. The formula for this process is: For all... ,have .

[0037] Generating dynamic prompts In the process, it is in all The cue words are completely shared across all positions. This means that regardless of the cue word length... The source of the perturbation used to adjust the semantics is always the same vector determined by the uncertainty of the current image. This avoids the need for generating a large number of parameters for each location's perturbation independently, achieving dynamic prompts while strictly maintaining the model's lightweight characteristics.

[0038] Optionally, learnable basic prompt words are obtained, which contain M vectors, where M is an integer greater than 1; the prompt word perturbation vector is multiplied by a scaling factor to obtain a scaled perturbation vector; each vector in the basic prompt words is added to the scaled perturbation vector to generate the dynamic prompt word. The prompt word perturbation vector is shared across all prompt word positions corresponding to the M vectors when generating the dynamic prompt word.

[0039] This embodiment generates dynamic prompts closely related to the input image content by superimposing the prompt perturbation vectors with adjustable strength onto a set of learnable base prompts. With extremely low parameter overhead, it achieves an adaptive response of text-based semantic representation to visual uncertainties.

[0040] S4: The dynamic prompt words are encoded using a pre-trained text encoder to obtain text features.

[0041] In this embodiment, the dynamic prompt words generated in step S3 are combined with discrete category label text, and converted into feature representations in semantic space by a pre-trained text encoder, providing a benchmark for subsequent visual-text alignment matching.

[0042] For each category to be classified in the dataset (e.g., the ImageNet dataset contains categories such as "Golden Retriever", "Persian Cat", and "Car"), first obtain its category name text, denoted as . Obtain the current input image from step S3. The corresponding dynamic prompt words The dynamic prompts are concatenated with the acquired category label text to obtain a text sequence that can be processed by the text encoder. For example: Golden Retriever. This construction method is semantically equivalent to the cue templates used by vision-language pre-trained models such as CLIP, but here the templates are composed of continuous, learnable vectors that change dynamically with the input.

[0043] The text sequences constructed for each category are then input into a pre-trained text encoder with its parameters frozen. This text encoder is a CLIP text encoder paired with the CLIP visual encoder from step S1, such as a Transformer-based model. The text encoder encodes the input text sequences and ultimately outputs a text feature vector that represents the semantics of the entire sequence.

[0044] The text encoder itself is frozen, meaning its powerful linguistic representation capabilities are fully preserved without introducing additional trainable parameters. For the same category, the context of the input text sequence differs under different image inputs, resulting in dynamically changing encoded text features. This achieves indirect adaptation of textual semantic representation to visual input content, i.e., modulating the frozen encoder's encoding results for the same category name through dynamic prefix context.

[0045] Optionally, the dynamic prompt words are concatenated with the acquired category label text to obtain a text sequence; the text sequence is then input into a frozen CLIP text encoder to obtain text features.

[0046] This embodiment combines dynamic cue words related to the input image as prefixes with category labels and inputs them into a frozen pre-trained text encoder to obtain a text feature representation related to the context of the input image. Visual uncertainty information is transmitted to the text feature space through cue words, laying the foundation for subsequent accurate alignment with visual features.

[0047] S5: Calculate the similarity between the visual features and the text features, and determine the classification result of the input image based on the similarity calculation result.

[0048] In this embodiment, the classification result is obtained by calculating the similarity between visual features and text features of all candidate categories in the shared semantic space; this process utilizes the dynamically adapted bimodal features generated in the previous steps to achieve more robust matching.

[0049] Specifically, the input image is obtained from step S1. Fusion visual features This vector is the visual representation obtained after extraction and fusion via dual encoders. Text features for all candidate categories are obtained from step S4, and visual features are calculated for each category. The similarity score is calculated between the vector and the text features of each category. The similarity function is cosine similarity, which effectively measures the directional proximity of two vectors. Its value range is [-1, 1], with larger values ​​indicating higher alignment. After calculation, a similarity score vector is obtained.

[0050] The similarity score vector is used as the logits (logistic values) of the classifier. The class with the highest similarity score is selected, and the class with the highest confidence is the final classification result.

[0051] Since the input text features are dynamically modulated by uncertainties associated with the current visual input, the similarity comparison between visual and text features is performed in an adaptively calibrated semantic space. When the input image has high uncertainty, dynamic prompts guide the text features to adjust towards a more generalized semantic direction, making the matching process more tolerant of visual feature distribution drift, ultimately improving the model's classification robustness and accuracy in complex scenes. This embodiment completes the entire process from image input to category output by calculating the similarity between visual and text features and making decisions based on this similarity.

[0052] Please see Figure 2 This is an architectural block diagram of an uncertainty-guided dynamic prompting fine-tuning method according to Embodiment 1 of this application. Specifically, it includes: simultaneously feeding the input image to two parallel, parameter-frozen visual encoders (i.e., the ViT encoder and the CLIP-ViT encoder) for feature extraction. The extracted visual features are input to a parameter-frozen adaptive feature fusion module (AFF), which outputs a robust fused visual feature and calculates an uncertainty scalar reflecting the current image recognition difficulty based on the visual feature.

[0053] An uncertainty scalar is input into a trainable Uncertainty Injection Network (UIN), which generates a semantic perturbation vector. This perturbation vector is added to a set of learnable base cue word parameters in a trainable dynamic cue generator to generate dynamic cue words adapted to the content of the current input image. The generated dynamic cue words are then input into a parameter-frozen CLIP text encoder to obtain text feature representations. Finally, the visual features are compared with the text features of all candidate categories, and their similarity is calculated to output the classification result and the corresponding confidence score.

[0054] During the model training phase, parameters are updated only for designated trainable parts, including the Uncertainty Injection Network (UIN), base cue word parameters, and scaling factors in the dynamic cue generator. The parameters of the ViT encoder, CLIP-ViT encoder, Adaptive Feature Fusion (AFF) module, and CLIP text encoder remain frozen. This strategy aims to efficiently utilize pre-trained knowledge and prevent overfitting. The model's training objective is to optimize classification accuracy, and the loss function used is based on standard cross-entropy loss. To further enhance training stability and generalization ability, an L2 norm regularization constraint term on the perturbation vector (δ) can be introduced into the loss function.

[0055] Example 2 Please see Figure 3 This is a schematic diagram of the structure of an uncertainty-guided dynamic prompting fine-tuning system according to Embodiment 2 of this application; the specific content includes: The feature extraction and uncertainty estimation module 100 is used to extract visual features from the input image and calculate the uncertainty scalar based on the extracted visual features; Uncertainty injection network module 200 is a trainable parameter module used to input the uncertainty scalar into the uncertainty injection network to output a cue word perturbation vector; The dynamic prompt word generation module 300 is used to add the acquired basic prompt words to the scaled prompt word perturbation vector to generate dynamic prompt words corresponding to the input image; The text encoding module 400 is a pre-trained model with frozen parameters, used to encode the dynamic prompt words through a pre-trained text encoder to obtain text features; The similarity calculation and classification module 500 is used to calculate the similarity between the visual features and the text features, and to determine the classification result of the input image based on the similarity calculation result.

[0056] In this embodiment, the system achieves robust classification of input images through the collaboration of multiple modules. Specifically, it includes the following functional modules: The feature extraction and uncertainty estimation module 100 extracts visual feature vectors in parallel using a pre-trained visual encoder with two parameters frozen, based on the acquired input image. Subsequently, an adaptive feature fusion module with frozen parameters fuses these features, outputting a robust fused visual feature. This module also includes an auxiliary computation unit that calculates an uncertainty scalar based on the fused visual features, used to quantify the recognition difficulty of the current input.

[0057] The uncertainty injection network module 200 acquires the uncertainty scalar generated by the feature extraction and uncertainty estimation module 100. The uncertainty injection network module 200 is composed of a lightweight multilayer perceptron and is used to map the input uncertainty scalar into a high-dimensional cue word perturbation vector, thereby establishing a bridge from visual uncertainty to the text semantic space.

[0058] The dynamic prompt generation module 300 is a trainable component that maintains a set of learnable basic prompt parameters. Uncertainty is injected into the prompt perturbation vector generated by the network module 200, and this vector is scaled using a learnable scaling factor. Finally, the scaled perturbation vector is added to the basic prompt parameters to generate a dynamic prompt sequence that dynamically adapts to the content of the current input image.

[0059] The text encoding module 400 consists of a pre-trained text encoder (CLIP text encoder) whose parameters are always frozen. It is used to encode the dynamic prompt word sequence output by the dynamic prompt word generation module 300, combined with the label text of each category, into a set of corresponding text feature vectors.

[0060] The similarity calculation and classification module 500 calculates the similarity (cosine similarity) between visual and text features, and outputs the final classification label and optional confidence score based on the similarity comparison results.

[0061] During system training, only the parameters in the uncertainty injection network module and the dynamic prompt word generation module are updated, while the parameters of the other modules remain frozen, thereby achieving efficient and stable model adaptation.

[0062] Example 3 Please see Figure 4 This is a schematic diagram of the computer device structure according to Embodiment 3 of this application. The computer device 50 includes a processor 51 and a memory 52 coupled to the processor 51.

[0063] The memory 52 stores program instructions for implementing the aforementioned dynamic prompting fine-tuning method for uncertainty-guided programming.

[0064] The processor 51 is used to execute program instructions stored in the memory 52 to implement a dynamic cueing fine-tuning with an undefined guidance.

[0065] The processor 51 can also be referred to as a CPU (Central Processing Unit).

[0066] Processor 51 may be an integrated circuit chip with signal processing capabilities. Processor 51 may also be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), an off-the-shelf programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. A general-purpose processor may be a microprocessor or any conventional processor.

[0067] Example 4 Please see Figure 5 This is a schematic diagram of the storage medium in Embodiment 4 of this application. The storage medium in this embodiment stores a program file 61 capable of implementing all the above methods. This program file 61 can be stored in the storage medium in the form of a software product, including several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) or processor to execute all or part of the steps of the methods of various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks, or devices such as computers, servers, mobile phones, and tablets.

[0068] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, apparatus, article, or method that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, apparatus, article, or method. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, apparatus, article, or method that includes that element.

[0069] The above description is only a preferred embodiment of this application and does not limit the patent scope of this application. Any equivalent structural or procedural changes made based on the content of this application's specification and drawings, or direct or indirect applications in other related technical fields, are similarly included within the patent protection scope of this application.

[0070] Although embodiments of this application have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and variations can be made to these embodiments without departing from the principles and spirit of this application, the scope of which is defined by the appended claims and their equivalents.

[0071] Of course, the present invention may have many other embodiments. Based on this embodiment, other embodiments obtained by those skilled in the art without any creative effort are all within the scope of protection of the present invention.

Claims

1. A dynamic prompting fine-tuning method guided by uncertainty, characterized in that, include: Visual features are extracted from the input image, and an uncertainty scalar is calculated based on the extracted visual features. The uncertainty scalar is input into the uncertainty injection network to output the cue word perturbation vector; The acquired basic prompt words are added to the scaled prompt word perturbation vector to generate dynamic prompt words corresponding to the input image; The dynamic prompt words are encoded using a pre-trained text encoder to obtain text features; Calculate the similarity between the visual features and the text features, and determine the classification result of the input image based on the similarity calculation result.

2. The uncertainty-guided dynamic prompting fine-tuning method according to claim 1, characterized in that, The steps of extracting visual features from the input image and calculating the uncertainty scalar based on the extracted visual features specifically include: The input image is input into the frozen ViT encoder to obtain the first visual feature vector; The input image is input into the frozen CLIP-ViT encoder to obtain the second visual feature vector; The first visual feature vector and the second visual feature vector are input into the parameter-frozen adaptive feature fusion module to obtain visual features; The visual features are classified and predicted, and the information entropy of the probability distribution of the prediction results is calculated. The information entropy is used as the uncertainty scalar.

3. The uncertainty-guided dynamic prompting fine-tuning method according to claim 2, characterized in that, The uncertainty injection network is a multilayer perceptron. The uncertainty scalar is input into the multilayer perceptron, and the prompt word perturbation vector is output.

4. The uncertainty-guided dynamic prompting fine-tuning method according to claim 3, characterized in that, The step of adding the acquired basic prompt words to the scaled prompt word perturbation vector to generate dynamic prompt words corresponding to the input image specifically includes: Obtain learnable basic cue words, which contain M vectors, where M is an integer greater than 1; Multiply the prompt word perturbation vector by the scaling factor to obtain the scaled perturbation vector; Each vector in the basic prompt words is added to the scaled perturbation vector to generate the dynamic prompt words; The prompt word perturbation vector is shared across all prompt word positions corresponding to the M vectors when generating the dynamic prompt word.

5. The uncertainty-guided dynamic prompting fine-tuning method according to claim 4, characterized in that, The step of encoding the dynamic prompt words using a pre-trained text encoder to obtain text features specifically includes: The dynamic prompt words are concatenated with the acquired category label text to obtain a text sequence; The text sequence is input into a frozen CLIP text encoder to obtain text features.

6. The uncertainty-guided dynamic prompting fine-tuning method according to claim 5, characterized in that, The step of calculating the similarity between the visual features and the text features, and determining the classification result of the input image based on the similarity calculation result, specifically includes: Calculate the cosine similarity between the visual features and the text features, convert the highest similarity score into a classification confidence score, and obtain the classification result of the input image.

7. The uncertainty-guided dynamic prompting fine-tuning method according to claim 6, characterized in that, The method further includes: The parameters of the uncertainty injection network, the basic cue words, and the scaling factor are updated using a loss function. The parameters of the ViT encoder, the CLIP-ViT encoder, the adaptive feature fusion module, and the text encoder are not updated during the training phase.

8. A dynamic prompting and fine-tuning system guided by uncertainty, characterized in that, For performing the dynamic cueing fine-tuning method for uncertainty guidance according to any one of claims 1 to 7, the dynamic cueing fine-tuning system for uncertainty guidance comprises: The feature extraction and uncertainty estimation module is used to extract visual features from the input image and calculate the uncertainty scalar based on the extracted visual features. The uncertainty injection network module is a trainable parameter module used to input the uncertainty scalar into the uncertainty injection network to output a cue word perturbation vector; The dynamic prompt word generation module is used to add the acquired basic prompt words to the scaled prompt word perturbation vector to generate dynamic prompt words corresponding to the input image; The text encoding module is a pre-trained model with frozen parameters, used to encode the dynamic prompt words through a pre-trained text encoder to obtain text features; The similarity calculation and classification module is used to calculate the similarity between the visual features and the text features, and to determine the classification result of the input image based on the similarity calculation result.

9. A computer device, characterized in that, The computer device includes a processor and a memory coupled to the processor, wherein the memory stores program instructions for implementing the uncertainty-guided dynamic prompting fine-tuning method according to any one of claims 1-7; the processor is used to execute the program instructions stored in the memory to implement uncertainty-guided dynamic prompting fine-tuning.

10. A computer-readable storage medium, characterized in that, The device stores processor-executable program instructions for performing the uncertainty-guided dynamic prompting fine-tuning method according to any one of claims 1-7.

Images