Multi-label field adaptive method based on prompt driving

Abstract

The invention discloses a multi-label field adaptive method based on prompt driving, which belongs to the technical field of computer vision and comprises the following steps: generating multi-dimensional semantic text description containing visual attributes, semantic levels and category names of common scenes for each type; embedding the semantic text description into an optimizable category vector, and embedding the optimized category vector into a CLIP text prompt; extracting and projecting multi-layer style statistical characteristics of the image, and synchronously realizing image-text cross-modal alignment and source-target domain distribution alignment in a frame through a cross-domain style mapping network and various alignment losses; semantic priori knowledge of CLIP is combined with a label co-occurrence mode of a source domain, so that the model can sense and adapt to a possibly changed label dependency relationship in a target domain; performing semantic propagation on label embedding by using a graph convolutional network; the whole system is jointly optimized through a multi-task loss function, and end-to-end cross-domain multi-label classification is achieved.

Description

Technical Field

[0001] This invention belongs to the field of computer vision technology, specifically relating to a cue-driven multi-label domain adaptation method. Background Technology

[0002] Multi-label classification is a core task in computer vision, aiming to predict multiple category labels for a single image. It is widely used in fields such as object detection, facial attribute analysis, and medical image diagnosis. However, in practical applications, multi-label learning often faces the problem of feature distribution shift in cross-domain scenarios—image features in different domains differ significantly, causing models trained on source domain data to experience a sharp decline in performance on unlabeled target domains.

[0003] To address this issue, Multi-Label Domain Adaptation (MLDA) technology has emerged, aiming to transfer multi-label learning models from the source domain to the unlabeled target domain, bridging the cross-domain distribution gap. Existing MLDA methods primarily combine feature alignment techniques with multi-label learning strategies to learn domain-invariant features while preserving label relevance in the source domain.

[0004] However, existing technology has three major drawbacks:

[0005] 1. Fixed label relevance assumption: Existing methods assume that the label relevance of the source domain remains unchanged in the target domain. However, in reality, label relevance is a dynamic association determined by domain characteristics. The label relevance of the source domain cannot adapt to the target domain, resulting in limited migration performance.

[0006] 2. Insufficient semantic representation of categories: Existing methods mostly rely solely on the category name to describe the semantics of categories. This simple representation cannot fully encompass the semantic information needed for cross-domain knowledge dissemination. When the semantic granularity of the source domain and the target domain does not match, this simple semantic representation cannot establish an effective cross-domain association, causing the alignment process to lose precise semantic guidance.

[0007] 3. The learning processes are independent of each other: Existing methods treat feature distribution alignment and label relevance learning as independent tasks, without building a unified framework. This makes it impossible to achieve synergistic optimization of knowledge transfer and relevance learning, and makes it difficult to make full use of cross-domain semantic information. Summary of the Invention

[0008] To address this, this invention proposes a prompt-driven, multi-label domain adaptation method. It utilizes a large language model to generate multi-dimensional semantic content descriptions for each category name, thereby constructing rich and fine-grained semantic representations that fundamentally enhance the representation of semantic content. Simultaneously, the enhanced semantic representations are dynamically fused with source domain statistics to construct a label association graph, enabling the model to possess both source domain label dependency knowledge and learn rich enhanced semantic dependencies. Furthermore, to solve cross-domain feature alignment, a unified prompt-driven framework is used to simultaneously perform dual alignment of feature distribution and label relevance, significantly improving the model's generalization ability in unlabeled target domains.

[0009] This invention provides a prompt-driven multi-tag domain adaptation method. By organically combining semantic reconstruction, dynamic graph construction and prompt-driven dual alignment, it provides an innovative, efficient and high-performance solution for multi-tag domain adaptation, effectively overcoming long-standing technical difficulties in this field.

[0010] The technical solution of the present invention is as follows: The method includes the following steps:

[0011] S1. Generate multi-dimensional semantic text descriptions for each type using a large language model, including category names containing visual attributes, semantic hierarchy, and co-occurrence scenarios;

[0012] S2. Through a learnable semantic reconstruction network, rich semantic text descriptions are embedded into optimizable category vectors, and the optimized category vectors are then embedded into CLIP text prompts.

[0013] S3. Extract and project multi-layer style statistical features of the image, generate domain-specific cue tokens, and embed them together with learnable contextual cue tokens and optimized category vectors; through cross-domain style mapping networks and multiple alignment losses, image-text cross-modal alignment and source-target domain distribution alignment are simultaneously achieved within a framework.

[0014] S4. By combining the semantic prior knowledge of CLIP with the label co-occurrence pattern of the source domain, the general semantic prior matrix derived from the large-scale visual-language model is weighted and fused with the specific co-occurrence statistical matrix derived from the source domain data, so that the model can perceive and adapt to the label dependency relationship that may change in the target domain.

[0015] S5. Use graph convolutional networks to perform semantic propagation on label embeddings;

[0016] S6. Finally, by jointly optimizing the entire system through a multi-task loss function, end-to-end cross-domain multi-label classification is achieved.

[0017] Preferably, S1 includes: an input source domain. With the target domain Image to CLIP Image Encoder Extract visual features.

[0018] Preferably, S2 includes: classifying the category name The large language model (LLM) is used to represent the structured text description. The semantic reconstruction network learns and optimizes the category vector. For each category Category hints are indicated as ,in It is a shared context vector, a structured text description. and category hints Both are encoded using CLIP's text encoder. Projected into a shared space, minimizing the Euclidean distance between two embeddings within the shared space. Meanwhile, pairwise loss is introduced in the shared space. To perform regularization, the specific formula is:

[0019] (1),

[0020] (2),

[0021] Ultimately, the optimized class vectors will be successful. Embedded in text prompts.

[0022] Preferably, S3 includes: for each input sample Calculate and stitch together the image encoder Statistical measures of the layer to obtain its style vector Through shared style projectors Source domain style vectors and target domain style vector Mapping to domain-specific hints and Embed it in the text prompt middle:

[0023] (3),

[0024] in This represents a soft hint token, and CLS represents the optimized category vector, achieved by introducing a cross-domain style mapping network. This mapper aligns the target and source domain style features during the inference phase, enabling the generation of domain-specific cues for both the source and target domains using only target domain data during inference, and transforming the target domain style vector... Using a cross-domain style mapping network Mapped to source domain hint token Style projection is supervised by minimizing the difference between the projected token and the real source domain token:

[0025] (4).

[0026] Preferably, S3 includes: employing binary cross-entropy loss in the domain-aware cueing network. To force precise alignment between the source image and the corresponding text prompt:

[0027] (5),

[0028] in Indicates the first The source sample at the th The actual label of the class, and This represents the probability predicted by the text prompt.

[0029] Preferably, S3 includes: minimizing the distributional difference between the source domain and target domain cue embeddings, and constructing a cue-aligned semantic space; given cue constructed from the source domain and target domain, computing the embeddings. and And force alignment:

[0030] (6),

[0031] in It is the Kullback-Leibler divergence. and These represent text embeddings concatenated by the CLIP text encoder from the source and target domains, respectively.

[0032] Preferably, S3 includes: further applying low-entropy regularization to the prediction of the target domain to encourage the generation of confident and discriminative outputs, defined as:

[0033] (7),

[0034] in It refers to the first The entropy of the predicted probability distribution of a target sample.

[0035] Preferably, step S4 includes: obtaining a source domain label correlation matrix by calculating the frequency of label co-occurrence in the source data. Meanwhile, a semantic matrix was constructed using CLIP semantic prior knowledge. Embedding text from source and target domains Calculate the matrix ,

[0036] (8),

[0037] in It is cosine similarity, for each entry Before choosing The elements are set to 0, and the remaining elements are set to zero, resulting in a sparse matrix. ,

[0038] (9).

[0039] Preferably, by adjusting the sparse graph To alleviate the oversmoothing problem in graph representation, based on:

[0040] (10)

[0041] in It is a hyperparameter used to determine the weights assigned to a node and its neighboring nodes. Finally, the CLIP-guided label correspondence is obtained as follows:

[0042] (11),

[0043] in Controlling the smoothness of the distribution, It is an indicator function that represents such a semantic adjacency matrix as The final dynamic tag-related graph is represented as follows:

[0044] (12)

[0045] As a hyperparameter, it is used to balance the contributions of the two correlation matrices.

[0046] Preferably, step S5 includes: performing semantic propagation of label embeddings using a graph convolutional network, which includes: using a network with... A graph convolutional network (GCN) with multiple layers is used to progressively optimize the input features. ,in This represents a concatenated text embedding from CLIP, each Corresponding to the Class, number Layered GCN updates features in the following way:

[0047] (13)

[0048] in, It is a learnable weight matrix. As a nonlinear activation function, in the... After the layers, a residual connection is used to enhance the discriminability of the features. This connection combines the initial features and the optimized features, i.e.:

[0049] (14)

[0050] in This represents the final label embedding, which has been enhanced through cross-domain label dependencies. Then, for the image features obtained by the image encoder... Its corresponding number The predicted probability of a class is determined by measuring image features. With refined tag embedding Likelihood is calculated based on the similarity between them. The calculation is as follows:

[0051] (15)

[0052] in, It is the sigmoid function. Represents cosine similarity. It is a temperature parameter used to calibrate prediction confidence. For multi-label prediction, if the probability of prediction... Exceeding the predefined threshold Then the category Assigned to image The multi-class objective loss function is:

[0053] (16).

[0054] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0055] 1) It fundamentally solves the bottleneck problem of sparse and mismatched granularity in cross-domain category semantic representation. By introducing a large language model to generate and reconstruct structured semantic descriptions, the category representations obtained by the model contain rich visual, hierarchical, and contextual information. This significantly improves the ability to represent and distinguish abstract and coarse-grained categories, ensuring that accurate cross-domain semantic associations can still be established even when the semantic annotation granularity of the source and target domains is inconsistent, providing a precise semantic representation for subsequent alignment.

[0056] 2) It enables dynamic exploration of label relevance, breaking the limitations of the static association assumption. The proposed dynamic label association graph construction method cleverly combines CLIP semantic prior knowledge with co-occurrence statistics reflecting the characteristics of the source domain. This allows the model to effectively avoid imposing inherent, potentially inapplicable, relevance biases from the source domain onto the target domain, greatly enhancing the model's adaptability and robustness in diverse and open scenarios.

[0057] 3) Through a unified prompt-driven framework, a collaborative dual alignment of feature distribution and label association is achieved, resulting in excellent optimization efficiency and effectiveness. This invention innovatively uses domain-aware prompt learning as the core hub, tightly coupling multiple sub-tasks such as image feature extraction, text semantic enhancement, cross-domain distribution alignment, and label relationship modeling within an end-to-end optimization framework. Feature alignment provides a domain-invariant feature foundation for relevance learning, while dynamic relevance cognition, in turn, guides more accurate feature alignment. This collaborative mechanism overcomes the shortcomings of traditional methods that isolate tasks and fragment optimization objectives, achieving a transfer effect of 1+1>2 and simplifying the training process.

[0058] 4) It possesses excellent practicality and scalability. Based on a pre-trained CLIP model, this invention eliminates the need for complex adversarial networks or additional domain discriminators, resulting in a more stable training process. Its modular design facilitates integration with more advanced base models or language models. This technology can be widely applied in areas such as automated annotation of remote sensing images, cross-device medical image-assisted diagnosis, cross-platform understanding of social media content, and environmental perception in autonomous driving systems under different climatic conditions, demonstrating significant industrial application value. Attached Figure Description

[0059] Figure 1 A schematic diagram of a prompt-driven multi-label domain adaptation method. Detailed Implementation

[0060] The present invention will be further illustrated below with reference to the accompanying drawings and specific embodiments. It should be understood that the following specific embodiments are for illustrative purposes only and are not intended to limit the scope of the invention.

[0061] Example 1

[0062] This invention relates to a prompt-driven multi-label domain adaptive system, aiming to address three major problems in existing technologies for cross-domain multi-label classification: insufficient semantic representation, fixed label relevance, and fragmented learning processes. The core of this invention lies in constructing a unified prompt-driven dual-alignment framework based on semantic enhancement. The system includes:

[0063] Category Semantic Augmentation Network: Class names typically provide limited information, especially for coarse-grained, abstract categories or those related to co-occurrence scenarios. This system proposes using a large language model to reshape class name information into multi-dimensional category semantic information that includes visual information, semantic hierarchy information, and category co-occurrence relationship information. This multi-dimensional category semantic information is then used to reconstruct class names, and sparse category name embeddings are transformed into rich semantic representations through learnable category vectors.

[0064] Dynamic Label Relevance Graph Construction: Previous MLDA methods typically construct label relevance graphs based on label co-occurrence patterns in the source domain. However, the co-occurrence patterns in the target domain may differ significantly from those in the source domain, so relying solely on label relevance in the source domain may lead to poor performance in the target domain. Therefore, this system proposes a dynamic label relevance graph by combining the semantic prior knowledge of CLIP with the label co-occurrence patterns in the source domain.

[0065] Domain-Aware Prompt Network: This system proposes a domain-aware prompt network that bridges domain gaps and preserves label relevance within a unified framework. Through domain-specific tokens and learnable context, it achieves dual alignment of feature distribution and label relevance within a unified prompt-driven framework.

[0066] Example 2

[0067] In a typical embodiment, the system performs the following steps:

[0068] Input source domain With the target domain Image to CLIP Image Encoder Extract visual features;

[0069] Category Name The large language model (LLM) is used to represent the structured text description. The semantic reconstruction network learns and optimizes the category vector. For each category Category hints can be represented as ,in It is a shared context vector. A structured text description. and category hints Both are encoded using CLIP's text encoder. Projected into a shared space, minimizing the Euclidean distance between two embeddings within the shared space. To enhance the distinguishability between categories, a pairwise loss is introduced into the shared space. To perform regularization, the specific formula is:

[0070] (1),

[0071] (2),

[0072] Ultimately, the optimized class vectors will be successful. Embedded in text prompts;

[0073] For each input sample This invention calculates and stitches an image encoder. Statistical measures of the layer to obtain its style vector Through shared style projectors Source domain style vectors and target domain style vector Mapping to domain-specific hints and Embed it in the text prompt middle:

[0074] (3),

[0075] in The symbol represents a soft hint token, and CLS represents the optimized category vector. Since the source domain data is not visible during the inference phase, this system introduces a cross-domain style mapping network. This mapper aligns target and source domain style features during the inference phase, enabling the generation of domain-specific cues for both the source and target domains using only target domain data during inference. Specifically, it aligns the target domain style vectors... Using a cross-domain style mapping network Mapped to source domain hint token Style projection is supervised by minimizing the difference between the projected token and the real source token:

[0076] (4),

[0077] In the domain-aware cueing network, a domain-independent cue token is learned through three complementary loss functions, thereby maintaining consistent semantic representation across domains. Specifically, a binary cross-entropy loss is employed. This forces precise alignment between the source image and the corresponding text prompt, ensuring that the model learns a stable cross-modal semantic mapping.

[0078] (5),

[0079] in Indicates the first The source sample at the th The actual label of the class, and This represents the probability predicted by the text prompt. Furthermore, this invention minimizes the distributional differences between the source and target domain prompt embeddings to enhance consistency between prompts from different domains and constructs a semantic space for prompt alignment. Given prompts constructed from the source and target domains, the embeddings are computed. and And force alignment:

[0080] (6),

[0081] in It is the Kullback-Leibler divergence. and These represent the text embeddings concatenated by the CLIP text encoder from the source and target domains, respectively. This invention also applies low-entropy regularization to the prediction of the target domain to encourage the generation of confident and discriminative outputs, defined as:

[0082] (7),

[0083] in It refers to the first The entropy of the predicted probability distribution of a target sample.

[0084] Since relying solely on source label relevance may lead to poor performance in the target domain, this system proposes constructing a dynamic label relevance graph. This aims to integrate semantic prior knowledge from CLIP into the label co-occurrence patterns of the source domain. To capture label co-occurrence patterns in the source domain, a source domain label relevance matrix is ​​obtained by calculating the frequency of label co-occurrence in the source data. Meanwhile, this system relies on CLIP's powerful semantic prior knowledge to construct a semantic matrix. Specifically, this invention embeds text from both the source and target domains. Calculate the matrix ;

[0085] (8),

[0086] in It's cosine similarity. For each entry... Before the present invention is selected The elements are set to 0, and the remaining elements are set to zero, resulting in a sparse matrix. ,

[0087] (9),

[0088] Furthermore, this invention adjusts the sparse graph To alleviate the oversmoothing problem in graph representation, based on:

[0089] (10)

[0090] in This is a hyperparameter used to determine the weights assigned to a node and its neighboring nodes. Finally, the label correspondence guided by CLIP can be derived as follows:

[0091] (11),

[0092] in Controlling the smoothness of the distribution, It is an indicator function. This invention represents such a semantic adjacency matrix as... The final dynamic tag correlation diagram of this invention is represented as follows:

[0093] (12)

[0094] here, As a hyperparameter, it is used to balance the contributions of the two correlation matrices.

[0095] Next, semantic propagation of the label embeddings is performed using a graph convolutional network. This invention uses a network with... A graph convolutional network (GCN) with multiple layers is used to progressively optimize the input features. , in This represents a concatenated text embedding from CLIP, each Corresponding to the Class. Number Layered GCN updates features in the following way:

[0096] (13)

[0097] in, It is a learnable weight matrix. As a nonlinear activation function. In the first... Following the layers, this invention enhances the discriminability of features through a residual connection that combines the initial and optimized features, namely:

[0098] (14)

[0099] in This represents the final label embedding, which has been enhanced through cross-domain label dependencies. Then, for the image features obtained after the image encoder... Its corresponding number The predicted probability of a class is determined by measuring image features. With refined tag embedding It is calculated based on the similarity between them. In other words, likelihood. It can be calculated as follows:

[0100] (15)

[0101] in, It is the sigmoid function. Represents cosine similarity. This is the temperature parameter used to calibrate prediction confidence. For multi-label prediction, if the probability of prediction... Exceeding the predefined threshold Then the category Assigned to image The multi-class objective loss function is:

[0102] (16)

[0103] Finally, by jointly optimizing the entire system through a multi-task loss function, end-to-end cross-domain multi-label classification is achieved.

[0104] It should be noted that the above content merely illustrates the technical concept of the present invention and should not be construed as limiting the scope of protection of the present invention. For those skilled in the art, various improvements and modifications can be made without departing from the principle of the present invention, and all such improvements and modifications fall within the scope of protection of the claims of the present invention.

Claims

1. A prompt-driven multi-label domain adaptation method, characterized in that, The method includes the following steps: S1. Generate multi-dimensional semantic text descriptions for each type using a large language model, including category names containing visual attributes, semantic hierarchy, and co-occurrence scenarios; S2. Through a learnable semantic reconstruction network, rich semantic text descriptions are embedded into optimizable category vectors, and the optimized category vectors are then embedded into CLIP text prompts. S3. Extract and project multi-layer style statistical features of the image, generate domain-specific cue tokens, and embed them together with learnable contextual cue tokens and optimized category vectors; through cross-domain style mapping networks and multiple alignment losses, image-text cross-modal alignment and source-target domain distribution alignment are simultaneously achieved within a framework. S4. By combining the semantic prior knowledge of CLIP with the label co-occurrence pattern of the source domain, the general semantic prior matrix derived from the large-scale visual-language model is weighted and fused with the specific co-occurrence statistical matrix derived from the source domain data, so that the model can perceive and adapt to the label dependency relationship that may change in the target domain. S5. Use graph convolutional networks to perform semantic propagation on label embeddings; S6. Finally, by jointly optimizing the entire system through a multi-task loss function, end-to-end cross-domain multi-label classification is achieved.

2. The method according to claim 1, characterized in that, S1 includes: input source domain With the target domain Image to CLIP Image Encoder Extract visual features.

3. The method according to claim 1, characterized in that, S2 includes: classifying the category name The large language model (LLM) is used to represent the structured text description. The semantic reconstruction network learns and optimizes the category vector. For each category Category hints are indicated as ,in It is a shared context vector, a structured text description. and category hints Both are encoded using CLIP's text encoder. Projected into a shared space, minimizing the Euclidean distance between two embeddings within the shared space. Meanwhile, pairwise loss is introduced in the shared space. To perform regularization, the specific formula is: （1）， （2）， Ultimately, the optimized class vectors will be successful. Embedded in text prompts.

4. The method according to claim 1, characterized in that, S3 includes: for each input sample Calculate and stitch together the image encoder Statistical measures of the layer to obtain its style vector Through shared style projectors Source domain style vectors and target domain style vector Mapping to domain-specific hints and Embed it in the text prompt middle: （3）， in This represents a soft hint token, and CLS represents the optimized category vector, achieved by introducing a cross-domain style mapping network. This mapper aligns target and source domain style features during the inference phase, enabling the generation of domain-specific cues for both the source and target domains using only target domain data during inference, and transforming the target domain style vector... Using a cross-domain style mapping network Mapped to source domain hint token Style projection is supervised by minimizing the difference between the projected token and the real source domain token: （4）。 5. The method according to claim 4, characterized in that, S3 includes: employing binary cross-entropy loss in the domain-aware cueing network. To force precise alignment between the source image and the corresponding text prompt: （5）， in Indicates the first The source sample at the th The actual label of the class, and This represents the probability predicted by the text prompt.

6. The method according to claim 5, characterized in that, S3 includes: minimizing the distributional difference between the source and target domain cue embeddings, and constructing a cue-aligned semantic space; given cue constructed from the source and target domains, computing the embeddings. and And force alignment: （6）， in It is the Kullback-Leibler divergence. and These represent text embeddings concatenated by the CLIP text encoder from the source and target domains, respectively.

7. The method according to claim 6, characterized in that, The S3 includes: further applying low-entropy regularization to the prediction of the target domain to encourage the generation of confident and discriminative outputs, defined as: （7）， in It refers to the first The entropy of the predicted probability distribution of a target sample.

8. The method according to claim 1, characterized in that, S4 includes: obtaining the source domain label correlation matrix by calculating the frequency of label co-occurrence in the source data. Meanwhile, a semantic matrix was constructed using CLIP semantic prior knowledge. Embedding text from source and target domains Calculate the matrix , （8）， in It is cosine similarity, for each entry Before choosing The elements are set to 0, and the remaining elements are set to zero, resulting in a sparse matrix. , （9）。 9. The method according to claim 8, characterized in that, By adjusting the sparse graph To alleviate the oversmoothing problem in graph representation, based on: （10）， in It is a hyperparameter used to determine the weights assigned to a node and its neighboring nodes. Finally, the CLIP-guided label correspondence is obtained as follows: （11）， in Controlling the smoothness of the distribution, It is an indicator function that represents such a semantic adjacency matrix as The final dynamic tag-related graph is represented as follows: （12）， As a hyperparameter, it is used to balance the contributions of the two correlation matrices.

10. The method according to claim 1, characterized in that, S5 includes: the step of semantic propagation of label embeddings using a graph convolutional network includes: using a graph convolutional network with... A graph convolutional network with multiple layers is used to progressively optimize the input features. ,in This represents a concatenated text embedding from CLIP, each Corresponding to the Class, number Layered GCN updates features in the following way: （13）， in, It is a learnable weight matrix. As a nonlinear activation function, in the... After the layers, a residual connection is used to enhance the discriminability of the features. This connection combines the initial features and the optimized features, i.e.: （14）， in This represents the final label embedding, which has been enhanced through cross-domain label dependencies. Then, for the image features obtained by the image encoder... Its corresponding number The predicted probability of a class is determined by measuring image features. With refined tag embedding Likelihood is calculated based on the similarity between them. The calculation is as follows: （15）， in, It is the sigmoid function. Represents cosine similarity. It is a temperature parameter used to calibrate prediction confidence. For multi-label prediction, if the probability of prediction... Exceeding the predefined threshold Then the category Assigned to image The multi-class objective loss function is: （16）。

Images