Visual-linguistic model prompting method and system based on coupled prompt field

Abstract

The application provides a visual-language model prompting method and system based on a coupling prompt field, and belongs to the fields of artificial intelligence and computer vision. The method comprises the following steps: mapping a base class task and a new class task to a shared feature space, defining a coupling prompt field, and enabling the base class and the new class task to form mutual constraints in the shared feature space; performing scale constraints on the coupling prompt field through projection layer norm alignment and coding layer layer-by-layer norm alignment; integrating the projection layer norm alignment loss and the coding layer layer-by-layer norm alignment loss, fusing the alignment loss and a task loss of a visual-language model, forming a model overall training target, and performing training; performing reasoning on the base class task and the new class task based on the trained model, and outputting a classification prediction result. The application solves the problems of existing end-to-end, decoupling prompt learning, base class-new class isolated optimization, easy falling into local optimum, norm drift and entanglement collapse, and improves the cross-task generalization and robustness of the model.

Description

Technical Field

[0001] This invention belongs to the field of artificial intelligence and computer vision technology, and particularly relates to a visual-language model prompting method and system based on coupled cue fields. Background Technology

[0002] The statements in this section are merely background information related to the present invention and do not necessarily constitute prior art.

[0003] Visual-language pre-trained models have become the core foundational models in the field of multimodal intelligence due to their cross-modal semantic alignment capabilities and strong generalization characteristics. Cue learning is a key technology for efficiently transferring such pre-trained models to downstream tasks. By introducing learnable cue tokens into the visual and text encoders, the model can be adapted to specific tasks without significant fine-tuning of the model's backbone weights. It balances transfer efficiency and model generalization and has become the mainstream technical path for the practical application of visual-language models.

[0004] Existing cue learning methods for visual-language models mainly fall into two paradigms: end-to-end cue learning and decoupled cue learning. While both paradigms have achieved certain results in different scenarios, they both suffer from insurmountable technical flaws. The core problem lies in the fact that the optimization process is confined to the independent feature space of a specific task, failing to establish an effective connection between the base class and the new task class. This leads to the model easily getting trapped in local optima and failing to converge to the global optimum. Simultaneously, existing methods neglect the embedding norm drift problem during cue tuning. Non-uniform norm changes severely disrupt the inherent feature uniformity and tolerance trade-off in visual-language models, impairing the model's cross-task transfer performance. Furthermore, due to the inherent characteristics of the self-attention mechanism, randomly initialized cue tokens are highly orthogonal to other tokens in the model in the high-dimensional feature space, causing the model to exhibit an "entanglement collapse" phenomenon. This results in the model paying almost no attention to cue tokens, blocking effective information exchange between cue and global representation.

[0005] To address the aforementioned issues, some existing technologies have attempted to alleviate the performance trade-off between base classes and new classes through self-regularization, external knowledge distillation, and uncertainty modeling, or to improve feature adaptability through multi-layered hint injection. However, these methods are all local improvements for specific problems and do not construct a global coupling relationship between base class and new class tasks. They also do not address the core issues of norm drift and "entanglement collapse." They either require the introduction of additional external knowledge or complex decoupling reasoning mechanisms, increasing the computational overhead and application complexity of the model, or they still cannot break through the technical limitations of local optima, resulting in limited improvement in cross-task generalization performance. Summary of the Invention

[0006] To overcome the shortcomings of the prior art, this invention provides a visual-language model prompting method and system based on coupled cue fields. By constructing a coupled cue field, the base class and new class tasks are incorporated into a mutually constrained shared feature space. A dynamic norm alignment strategy ensures coupling within the field and global consistency, thereby achieving global joint optimization of the base class and new class tasks. This effectively solves the local optimum dilemma and "entanglement collapse" problem in existing methods, and improves the generalization performance and robustness of the model.

[0007] To achieve the above objectives, one or more embodiments of the present invention provide the following technical solutions: The first aspect of this invention provides a visual-language model-based cueing method based on coupled cueing fields; Visual-language model-based cueing methods based on coupled cue fields include: The base class task and the new class task of the visual-language model are mapped to a shared feature space, and a coupling cue field is defined so that the base class and the new class tasks form mutual constraints in the shared feature space. The scale constraint of the coupled cue field is applied by norm alignment of the projection layer and norm alignment of the coding layer, so that the cue feature norm is dynamically aligned to the native scale of the visual-language model. The alignment loss is obtained by integrating the projection layer norm alignment loss and the coding layer layer-by-layer norm alignment loss. The alignment loss is then fused with the task loss of the visual-language model to form the overall training objective of the model. The model is trained based on the overall training objective. The trained visual-language model is used to infer base class tasks and new class tasks, and the classification prediction results are output based on the coupled cue field.

[0008] As a further technical solution, the base class task and the new class task of the visual-language model are mapped to a shared feature space, and a coupling cue field is defined so that the base class and the new class tasks form mutual constraints in the shared feature space, including: Using the category branch embeddings in the decoupled cue learning framework as stable anchors in the feature space, a cue field is constructed based on the difference between the cue branch embeddings and the stable anchors: Based on the constructed hint field, a unified reasoning strategy is adopted for base class and new class tasks. By transforming mutually independent decoupled reasoning into coupling constraints, a coupled hint field is defined.

[0009] As a further technical solution, the unified reasoning strategy is as follows:

[0010] in, Let be the predicted probability of the k-th category; Let be the predictor classifier for the k-th category; Let x be the anchor point representing x; The coupling hint field for x.

[0011] As a further technical solution, the loss function for projection layer norm alignment is:

[0012] in, The loss function is for alignment of the projection layer norm; This is a cue representation of the unnormalized projection layer; Gradient cutoff; This represents the unnormalized category embedding representation of the projection layer.

[0013] As a further technical solution, the layer-by-layer norm alignment of the encoding layer starts from the cue activation layer J of the visual encoder and covers the last layer L of the encoder, with the loss function being:

[0014] in, The loss function is used for layer-by-layer norm alignment of the coding layer; The mean of the lexical representations of the coding layer; This represents the category embedding representation of the encoding layer.

[0015] As a further technical solution, the formula for calculating the alignment loss is:

[0016] in, For alignment loss; The hyperparameter for alignment loss of the projection layer; This is the hyperparameter for the alignment loss of the coding layer.

[0017] A second aspect of the present invention provides a visual-language model prompting system based on coupled cue fields.

[0018] A visual-language model-based cueing system based on coupled cue fields includes: The coupling cue field construction module is configured to: map the base class task and the new class task of the visual-language model to a shared feature space, and define a coupling cue field so that the base class and the new class tasks form mutual constraints in the shared feature space; The two-dimensional norm alignment module is configured to: constrain the scale of the coupled cue field by means of projection layer norm alignment and coding layer layer-by-layer norm alignment, and dynamically align the cue feature norm to the native scale of the visual-language model; The loss fusion module is configured to: integrate the projection layer norm alignment loss and the coding layer layer-by-layer norm alignment loss to obtain the alignment loss, and fuse the alignment loss with the task loss of the visual-language model to form the overall training objective of the model; The global inference module is configured to: train the model based on the overall training objective, infer the base class and new class tasks based on the trained vision-language model, and output the classification prediction results based on the coupled cue field.

[0019] A third aspect of the present invention provides a computer-readable storage medium having a program stored thereon that, when executed by a processor, implements the steps of the visual-language model prompting method based on coupled cue fields as described in the first aspect of the present invention.

[0020] A fourth aspect of the present invention provides an electronic device, including a memory, a processor, and a program stored in the memory and executable on the processor, wherein the processor executes the program to implement the steps of the visual-language model prompting method based on coupled cue fields as described in the first aspect of the present invention.

[0021] The fifth aspect of the present invention provides a computer program product, including a computer program / instruction that, when executed by a processor, implements the steps in the visual-language model prompting method based on coupled cue fields described in the first aspect of the present invention.

[0022] The above one or more technical solutions have the following beneficial effects: (1) This invention breaks through the limitations of isolated task optimization under the existing end-to-end, decoupled paradigm by constructing a shared feature space for base class and new class tasks and forming mutual constraints. This effectively avoids the model getting trapped in local optima, realizes global collaborative optimization of base class and new class tasks, and allows the performance gains of base class tasks to be efficiently transferred to new class tasks, greatly improving the model's cross-task generalization ability. By constraining the consistency of the norm ratio distribution between base class and new class, it avoids unexpected distortions in the coupled cue field, maintains the inherent feature uniformity and tolerance trade-off relationship of the visual-language model, and improves the robustness of the coupled field to stochastic gradient noise, ensuring the representation stability of the model in cross-distribution scenarios.

[0023] (2) This invention balances the scale of cue tokens and category tokens by applying norm constraints layer by layer to the encoding layer, restoring the coupling relationship and effective information exchange between cue tokens and global representation, solving the technical pain point of cue tokens being ignored by the model, and enhancing the expressive power of cue token features. At the same time, a task-agnostic unified inference rule is designed, which eliminates the need to pre-identify base class / new class task identifiers during the testing phase, gets rid of the dependence of the decoupling paradigm on task identifiers, and perfectly adapts to real-world application scenarios such as open worlds where there is no prior task information, greatly reducing the deployment complexity and inference cost of the model.

[0024] Advantages of additional aspects of the invention will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of the invention. Attached Figure Description

[0025] The accompanying drawings, which form part of this invention, are used to provide a further understanding of the invention. The illustrative embodiments of the invention and their descriptions are used to explain the invention and do not constitute an improper limitation of the invention.

[0026] Figure 1 This is a flowchart of the method in the first embodiment.

[0027] Figure 2 The embedding results of the method of the present invention in the first embodiment with other decoupling methods on three different image classification datasets are shown.

[0028] Figure 3 This is a system structure diagram of the second embodiment. Detailed Implementation

[0029] It should be noted that the following detailed descriptions are exemplary and intended to provide further illustration of the invention. Unless otherwise specified, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains.

[0030] It should be noted that the terminology used herein is for the purpose of describing particular implementations only and is not intended to limit the exemplary implementations of the present invention.

[0031] Where there is no conflict, the embodiments and features in the embodiments of the present invention can be combined with each other.

[0032] The overall approach proposed in this invention addresses the core problem in visual-language model cue learning: isolated optimization of base and new class tasks, which easily leads to local optima. It proposes constructing a coupled cue field. The two types of tasks are mapped to a shared feature space to form mutual constraints. The coupled cue field is defined using visually anchored category embeddings as stable anchor points. Dynamic norm alignment in two dimensions (projection and encoding layers) solves the field distortion problem caused by norm drift, while also mitigating attentional "entanglement collapse." A unified training objective is constructed by fusing alignment loss and task loss, and task-independent global inference rules are designed. This eliminates the need for pre-identifying task identifiers, achieving global collaborative optimization between base and new classes, significantly improving the model's generalization ability and robustness across tasks and scenarios.

[0033] Example 1 This embodiment discloses a visual-language model-based prompting method based on coupled cue fields; like Figure 1 As shown, the visual-language model-based cueing method based on coupled cue fields includes: Step S1: Map the base class task and the new class task of the visual-language model to a shared feature space, and define a coupling cue field so that the base class and the new class tasks form mutual constraints in the shared feature space.

[0034] Step S11: Based on the core structure of the decoupled cue learning framework, extract the category branch embeddings and use them as stable anchor points in the shared feature space, providing a basic reference for constructing the coupled cue field. The specific implementation process is as follows: The CLIP model is selected as the basic vision-language model. Its visual encoder contains an L-layer Transformer structure for the input image. The final layer of category token embeddings is obtained through layer-by-layer processing by a visual encoder. The input is then processed by a visual projection head to perform dimensional mapping, resulting in the category branch projection output. .

[0035] right implement Normalization operation, i.e. A stable anchor point is generated. This anchor point preserves the original semantic features of the CLIP model through a visual anchoring mechanism, avoids representation drift caused by cue-based optimization, ensures the stability of its position in the feature space, and provides a unified reference benchmark for the coupling of base class and new class tasks.

[0036] Step S12: Based on the differential features between stable anchor points and cue branch embeddings, a coupled cue field is constructed to achieve mutual constraints between base class and new class tasks. Learnable cues t are injected into the specified activation layers J to the last layer L of the visual encoder. Mean pooling is performed on each layer of cue tokens to obtain layer-by-layer aggregated features. Finally, the pooling result of the last layer is input into the cue projection head to obtain the cue branch projection output. ,through After normalization, the hint branch embedding is obtained .

[0037] Based on stable anchor points Embedding with hint branches Differences, constructing a cue field:

[0038] in, The weighting coefficients are used to adjust the correction strength of the cue adaptation terms to the anchor points. This cue field is essentially a task-shared transformation in the feature space. Through the cue correction rules learned from the base class task, it provides adaptive correction terms for the new class task, forming a mutual constraint between the two classes of tasks.

[0039] Step S13: In order to break the limitation of independent reasoning between base class and new class tasks in the decoupling framework, a unified reasoning strategy is designed based on the constructed hint field, transforming decoupling reasoning into coupling constraints, and completing the final definition of the coupled hint field.

[0040] The unified reasoning strategy is specifically as follows:

[0041] in, Let be the predicted probability of the k-th category; Let be the predictor classifier for the k-th category; Let x be the anchor point representing x; The coupling hint field for x.

[0042] In the decoupled cue learning framework, base class task inference needs to integrate features from category branches and cue branches:

[0043] in, It serves as a representation of the base class.

[0044] The new task class relies solely on category branch features, and the reasoning rules for the two task classes are independent of each other. This embodiment unifies the fusion reasoning of the base class and the single-branch reasoning of the new class into a coupled form of anchor point + cue field through a unified reasoning strategy. Specifically, in the base class task, the cue field... Equivalent to anchor point The adaptive correction is consistent with the fusion logic of the decoupling framework but in a unified form; in new types of tasks, the cue field... It inherits the feature correction rules learned from the base class task and realizes the transfer of base class knowledge to the new class through the superposition of anchor points and cue fields.

[0045] Step S2 involves applying scale constraints to the coupled cue field through projection layer norm alignment and coding layer-by-layer norm alignment, dynamically aligning the cue feature norms to the native scale of the visual-language model.

[0046] Step S21: Based on the completion of shared feature space mapping, stable anchor point determination and basic cue field construction, further through norm ratio distribution analysis, norm level influence modeling, and perturbation stability optimization, combined with the norm alignment mechanism, the precise definition of the coupled cue field is completed, achieving triple protection of global consistency, stability and robustness of the coupled cue field.

[0047] First, in the image-text shared feature space of the vision-language model, the feature norm ratio of the base class and the new class task is defined. By analyzing the relationship between the norm ratio and the coupled cue field, the coupled cue field is expressed as a function of the norm ratio, clarifying the core influence of the norm ratio distribution consistency on the coupled cue field.

[0048] Output by category branch projection Prompt for branch projection output Based on this, the norm ratio per sample in the shared feature space is defined as:

[0049] in, For L2 norm calculation operations, Project the category branch output; To prompt for branch projection output; Used to quantify the relative scale of clique branch features and categorical branch features.

[0050] For base class task datasets New type of task dataset Calculate the norm ratio for each sample in the dataset. Calculate the expected norm ratio of the two datasets:

[0051] Based on the fundamental definition of coupled cue fields and the norm ratio, the coupled cue field is reconstructed into a functional form with respect to the norm ratio:

[0052] This form directly reflects the decisive role of the norm ratio r(z) in the coupled cueing field, and it is clear that the consistency of the norm ratio distribution is the core prerequisite for maintaining the global scale uniformity of the coupled cueing field and avoiding field distortion.

[0053] Step S22, consider any loss term If it is only embedded through normalization Related to the hint branch. (Note) Backpropagation yields:

[0054] in, It is the identity matrix; This is the gradient of the loss function with respect to the normalized representation.

[0055] Therefore, the gradient magnitude and Inversely scaled. Assume the stochastic gradient can be decomposed into... ,in Therefore, the update noise on the hint branch is amplified as follows:

[0056] in, The gradient representing z; This is gradient noise; This is a representation of unnormalized data.

[0057] Therefore, when Less than At that time, through The factor amplifies the effective update scale on the cue branch, increasing its sensitivity to stochastic gradient noise.

[0058] Step S23, consider the random perturbation on the cue branch, ,in Simulate estimation noise caused by finite data and stochastic optimization. Utilize The first-order approximation at that point, The following limits are obtained:

[0059] in, This is a non-normalized cue representation; To indicate the mean of the representation; This is a normalized cue representation; The mean of the normalized hint representation; This is a transpose.

[0060] Anchoring Maintaining relative stability, therefore control This is key to improving the robustness of coupled cue field noise.

[0061] Step S3: Integrate the projection layer norm alignment loss and the coding layer layer-by-layer norm alignment loss to obtain the alignment loss. Then, fuse the alignment loss with the task loss of the vision-language model to form the overall training objective of the model.

[0062] The projection layer norm alignment loss is the core loss term for ensuring the consistency of the norm of the final projection outputs of the constraint cue branch and the category branch. The loss function for projection layer norm alignment is:

[0063] in, The loss function is for alignment of the projection layer norm; This is a non-normalized cue representation; Gradient cutoff.

[0064] Given that the original output Logits of the attention mechanism depends on the dot product, i.e. In high-dimensional cases, randomly initialized cue tokens are highly likely to be orthogonal to each other in direction. Therefore, in the early stages of training, cos θ approaches 0, resulting in weak and easily imbalanced Logits between cue tokens and image patches. During training, the norm of cue tokens is often much larger than that of class tokens, which further amplifies the Logits difference when unaligned. This causes the Softmax gating to quickly saturate to a one-sided state and is difficult to recover from. Ultimately, the attention weights assigned to cue tokens by a large number of global tokens are almost zero, leading to "entanglement collapse," meaning that cue tokens cannot effectively exchange semantic information with the global representation.

[0065] Therefore, in this embodiment, the layer-by-layer norm alignment loss of the encoding layer is used to constrain the norm consistency between the cue tokens and the category tokens in each layer of the encoder, thereby alleviating entanglement collapse. The layer-by-layer norm alignment of the encoding layer starts from the cue activation layer J of the visual encoder and covers the last layer L of the encoder. Its loss function is:

[0066] in, The loss function is used for layer-by-layer norm alignment of the coding layer; The mean of the lexical representations of the coding layer; This represents the category embedding representation of the encoding layer.

[0067] Finally, by adjusting the relative strength of the alignment loss between the projection layer and the coding layer using weighting coefficients, a unified alignment loss is obtained. The formula for calculating the alignment loss is as follows:

[0068] in, For alignment loss; The hyperparameter for alignment loss of the projection layer; This is the hyperparameter for the alignment loss of the coding layer.

[0069] By fusing the alignment loss with the task loss of the visual-language model, the final overall training objective of the model is formed, thereby achieving synergistic optimization of coupled cue field constraints and downstream task performance.

[0070] Step S4: Train the model based on the overall training objective, perform inference on the base class task and the new class task based on the trained vision-language model, and output the classification prediction result based on the coupled cue field.

[0071] During the training phase, CLIP ViT-B / 16 was used as the base model. The dataset was first split (70% base class / 30% new class) and parameters were initialized. Pre-trained backbone weights were frozen, and only learnable cue tokens and projection head parameters were optimized. Hyperparameters were configured with the SGD optimizer (learning rate 0.005), batch size 32, and 5 training epochs, along with early stopping and gradient pruning strategies to ensure stability. During training, forward propagation was used to calculate the two-dimensional alignment loss (projection layer L1 loss + encoding layer layer-by-layer L1 loss, both with weight coefficients of 0.5) and the cross-entropy task loss. These were summed to obtain the overall training objective. Backpropagation was used to update parameters, and the harmonic mean (HM) between the base class and new class was monitored for convergence of the loss. Ultimately, the alignment loss converged to below 0.08, and the base class accuracy stabilized above 89%.

[0072] The inference phase employs a unified set of rules, eliminating the need to distinguish between base class / new class task identifiers. After preprocessing the image to be inferred, it is input into the trained model. The class branch anchors and cue branch embeddings are extracted, a coupled cue field is generated and fused into the final features, and the predicted probability is calculated using Softmax with a temperature parameter (τ=0.07). The optimal class result is then output. Inference supports batch processing acceleration, with a single sample processing time of <10ms. Validation on 15 datasets shows that the new class generalization performance is improved by 0.74% compared to the decoupled method, and robustness to few-shot and domain generalization scenarios is significantly enhanced. Its tolerance to input perturbations is superior to the baseline model.

[0073] The entire process achieves efficient transfer learning that requires no external strategy and is adapted to the open world by using "two-dimensional norm constraints during training + unified coupled field fusion during inference", which balances performance, efficiency and robustness.

[0074] Furthermore, representative methods from both end-to-end and decoupled paradigms were selected for comparison. In the end-to-end setting, the benchmarks included MaPLe, PromptSRC, and HicroPL; for the decoupled strategy, the two most competitive models, Guo and Gu, were chosen. To ensure a fair comparison, the specific inference mechanisms of these decoupled methods were removed, and a unified inference rule was used for evaluation. Experiments showed that, without introducing decoupled inference, the method of this invention effectively improved the accuracy of the new task while maintaining the performance of the base class. Figure 2 As shown, the novel class embedding features extracted by the method of the present invention have stronger discriminative power.

[0075] Example 2 This embodiment discloses a visual-language model-based prompting system based on coupled cue fields; like Figure 3As shown, a visual-language model-based cueing system based on coupled cueing fields includes: The coupling cue field construction module is configured to: map the base class task and the new class task of the visual-language model to a shared feature space, and define a coupling cue field so that the base class and the new class tasks form mutual constraints in the shared feature space; The two-dimensional norm alignment module is configured to: constrain the scale of the coupled cue field by means of projection layer norm alignment and coding layer layer-by-layer norm alignment, and dynamically align the cue feature norm to the native scale of the visual-language model; The loss fusion module is configured to: integrate the projection layer norm alignment loss and the coding layer layer-by-layer norm alignment loss to obtain the alignment loss, and fuse the alignment loss with the task loss of the visual-language model to form the overall training objective of the model; The global inference module is configured to: train the model based on the overall training objective, infer the base class and new class tasks based on the trained vision-language model, and output the classification prediction results based on the coupled cue field.

[0076] Example 3 The purpose of this embodiment is to provide a computer-readable storage medium.

[0077] A computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the visual-language model prompting method based on coupled cue fields as described in Embodiment 1.

[0078] Example 4 The purpose of this embodiment is to provide an electronic device.

[0079] An electronic device includes a memory, a processor, and a program stored in the memory and executable on the processor. When the processor executes the program, it implements the steps in the visual-language model prompting method based on coupled cue fields as described in Embodiment 1.

[0080] Example 5 Embodiment 5 of the present invention provides a computer program product, including a computer program / instruction, which, when executed by a processor, implements the steps in the visual-language model prompting method based on coupled cue fields as described in Embodiment 1.

[0081] The steps and methods involved in the apparatuses of Embodiments 2, 3, 4, and 5 above correspond to those in Embodiment 1. For specific implementation details, please refer to the relevant description section of Embodiment 1. The term "computer-readable storage medium" should be understood as a single medium or multiple media including one or more instruction sets; it should also be understood as including any medium capable of storing, encoding, or carrying an instruction set for execution by a processor and enabling the processor to perform any of the methods in this invention.

[0082] Those skilled in the art will understand that the modules or steps of the present invention described above can be implemented using general-purpose computer devices. Optionally, they can be implemented using computer-executable program code, thereby allowing them to be stored in a storage device for execution by a computer device, or they can be fabricated as separate integrated circuit modules, or multiple modules or steps can be fabricated as a single integrated circuit module. The present invention is not limited to any particular combination of hardware and software.

[0083] While the specific embodiments of the present invention have been described above in conjunction with the accompanying drawings, this is not intended to limit the scope of protection of the present invention. Those skilled in the art should understand that various modifications or variations that can be made by those skilled in the art without creative effort based on the technical solutions of the present invention are still within the scope of protection of the present invention.

Claims

1. A visual-language model-based cueing method based on coupled cue fields, characterized in that, include: The base class task and the new class task of the visual-language model are mapped to a shared feature space, and a coupling cue field is defined so that the base class and the new class tasks form mutual constraints in the shared feature space. The scale constraint of the coupled cue field is applied by norm alignment of the projection layer and norm alignment of the coding layer, so that the cue feature norm is dynamically aligned to the native scale of the visual-language model. The alignment loss is obtained by integrating the projection layer norm alignment loss and the coding layer layer-by-layer norm alignment loss. The alignment loss is then fused with the task loss of the visual-language model to form the overall training objective of the model. The model is trained based on the overall training objective. The trained visual-language model is used to infer base class tasks and new class tasks, and the classification prediction results are output based on the coupled cue field.

2. The visual-language model prompting method based on coupled cue fields as described in claim 1, characterized in that, The base class task and the new class task of the visual-language model are mapped to a shared feature space, and a coupling cue field is defined so that the base class and the new class tasks form mutual constraints in the shared feature space, including: Using the category branch embeddings in the decoupled cue learning framework as stable anchors in the feature space, a cue field is constructed based on the difference between the cue branch embeddings and the stable anchors: Based on the constructed hint field, a unified reasoning strategy is adopted for base class and new class tasks. By transforming mutually independent decoupled reasoning into coupling constraints, a coupled hint field is defined.

3. The visual-language model prompting method based on coupled cue fields as described in claim 2, characterized in that, The unified reasoning strategy is as follows: in, Let be the predicted probability of the k-th category; Let be the predictor classifier for the k-th category; Let x be the anchor point representing x; The coupling hint field for x.

4. The visual-language model prompting method based on coupled cue fields as described in claim 1, characterized in that, The loss function for the projection layer norm alignment is: in, The loss function is the one used for alignment with the projection layer norm. This is a cue representation of the unnormalized projection layer; Gradient cutoff; This represents the unnormalized category embedding representation of the projection layer.

5. The visual-language model prompting method based on coupled cue fields as described in claim 1, characterized in that, The layer-by-layer norm alignment of the encoding layer starts from the cue activation layer J of the visual encoder and covers the last layer L of the encoder. Its loss function is: in, The loss function is used for norm alignment of the coding layers. The mean of the lexical representations of the coding layer; This represents the category embedding representation of the encoding layer.

6. The visual-language model prompting method based on coupled cue fields as described in claim 1, characterized in that, The formula for calculating the alignment loss is: in, For alignment loss; The loss function is the one used for alignment with the projection layer norm. The loss function is used for norm alignment of the coding layers. The hyperparameter for alignment loss of the projection layer; This is the hyperparameter for the alignment loss of the coding layer.

7. A visual-language model cueing system based on coupled cue fields, characterized in that, include: The coupling cue field construction module is configured to: map the base class task and the new class task of the visual-language model to a shared feature space, and define a coupling cue field so that the base class and the new class tasks form mutual constraints in the shared feature space; The two-dimensional norm alignment module is configured to: constrain the scale of the coupled cue field by means of projection layer norm alignment and coding layer layer-by-layer norm alignment, and dynamically align the cue feature norm to the native scale of the visual-language model; The loss fusion module is configured to: integrate the projection layer norm alignment loss and the coding layer layer-by-layer norm alignment loss to obtain the alignment loss, and fuse the alignment loss with the task loss of the visual-language model to form the overall training objective of the model; The global inference module is configured to: train the model based on the overall training objective, infer the base class and new class tasks based on the trained vision-language model, and output the classification prediction results based on the coupled cue field.

8. A computer-readable storage medium having a program stored thereon, characterized in that, When the program is executed by the processor, it implements the steps in the visual-language model prompting method based on coupled cue fields as described in any one of claims 1-6.

9. An electronic device comprising a memory, a processor, and a program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the steps in the visual-language model prompting method based on coupled cue fields as described in any one of claims 1-6.

10. A computer program product comprising a computer program / instructions, characterized in that, When the computer program / instruction is executed by the processor, it implements the steps in the visual-language model prompting method based on coupled cue fields as described in any one of claims 1-6.

Images