A cross-domain image recognition method and system based on natural language generalization processing

Abstract

The application provides a cross-domain image recognition method and system based on natural language generalization processing, which comprises the following steps: a class perception enhancement step: two template prompt images are received, different representations of field information prompts and different class information prompts in different fields are learned respectively to better capture the diversity of class representation, and a class perception enhanced image is output; and a field filtering step: for the class perception enhanced image, a group of field types with the highest relevance is selected for each class to construct a class prompt representation.

Description

Technical Field

[0001] This invention belongs to the field of source domain-independent domain generalization technology in artificial intelligence, and particularly relates to a cross-domain image recognition method and system based on natural language generalization processing. Background Technology

[0002] The statements in this section are merely background information related to the present invention and do not necessarily constitute prior art.

[0003] Currently, in the field of artificial intelligence, especially in cross-domain image recognition and natural language processing tasks, domain generalization technology is crucial for the widespread application of AI. However, in real-world environments, due to limitations such as data privacy, data security, and storage space, source domain data is often difficult to obtain directly; only a pre-trained source domain model based on the source domain data can be obtained. This challenge is particularly pronounced in sensitive data domains such as medicine and demographics. For example, a province has built a medical image recognition model based on local data to classify lung CT scan images to assist doctors in the early screening and diagnosis of diseases such as lung cancer, and hopes to transfer this model to other provinces. However, due to differences in the distribution of population data between regions (such as differences in gender, age, and measurement equipment), an unsupervised domain adaptation strategy is needed to adjust the model. But due to policy regulations and the difficulty of data collection, the original data is strictly protected and cannot be directly shared. Therefore, only a pre-trained model can be provided for transfer, ensuring data privacy while achieving knowledge transfer. This also forces researchers to explore source domain-independent domain generalization methods that do not require direct access to the source data.

[0004] Generalizing to unknown downstream tasks without exposing source domain data, relying solely on a source domain model, presents a significant challenge; this scenario is known as source-domain-independent generalization. The goal of source-domain-independent generalization is to enable the model to effectively adapt and generalize to a new target domain without access to source domain data. However, since neither source nor target domain data is readily available, robust source-domain-independent generalization is difficult to achieve using only a source domain model with limited performance. The model struggles to infer and generalize without sufficient information, increasing the task's complexity and challenge.

[0005] Therefore, traditional or deep learning-based domain generalization methods mainly consider methods based on multi-source domains and single-source domains. However, when considering more restrictive scenarios where source domain data is unavailable, the above methods cannot be used.

[0006] To address this challenge, Contrastive Language-Image Pre-Training (CLIP) models have attracted significant attention due to their powerful generalization potential. These models, through a general template prompt mechanism, can flexibly adapt to different tasks, demonstrating good adaptability to unknown domains.

[0007] Fine-tuning methods for visual language models based on contrastive language-image pre-trained models can be mainly divided into three categories:

[0008] Encoder-based fine-tuning: Fine-tuning the relevant parameters of the two major modules, Text Encoder and Image Encoder, in the CLIP model.

[0009] Few-shot fine-tuning: Fine-tuning the CLIP model using a small amount of labeled data.

[0010] Tip-based fine-tuning: Fine-tuning a multimodal model by modifying the representation of the input text (i.e., the prompt).

[0011] However, current research on fine-tuning the prompt still faces two major obstacles:

[0012] (1) Overemphasis on learning and representing domain information makes it difficult for categories to be learned effectively.

[0013] (2) The use of domain information is too extreme, or the generation of different styles for different photos is too detailed, or the equal treatment of all domain information is too coarse. Although the average pooling of multiple different templates can obtain a unified prompt, it is difficult to enable different categories of prompts to effectively obtain key information specific to certain domains, and it is difficult to achieve the source domain-independent domain generalization task.

[0014] In summary, existing technologies cannot effectively solve the above two problems to construct a richer and more unified prompt representation, thereby effectively improving classification tasks. Summary of the Invention

[0015] To overcome the shortcomings of the prior art, this invention provides a cross-domain image recognition method based on natural language generalization processing, and a Prompt fine-tuning method based on category enhancement and domain awareness, which better achieves source-domain-independent domain generalization tasks.

[0016] To achieve the above objectives, one or more embodiments of the present invention provide the following technical solutions:

[0017] Firstly, a cross-domain image recognition method based on natural language generalization processing is disclosed, including:

[0018] Class-aware enhancement step: Receive two template cue images, learn different representations of domain information cues and category information cues in different domains to better capture the diversity of category representations, and output a class-aware enhanced image;

[0019] Domain filtering step: For class-aware enhanced images, select the most relevant set of domain types for each category to construct a category cue representation.

[0020] As a further technical solution, two template prompt texts are received: "a{style}photo of a{class}" and "a{style}photo", where class represents the category of the target domain image, such as cat, dog, etc., and style represents the domain type, such as sketch, clip art, etc.

[0021] As a further technical solution, the process of learning different representations of domain information cues and category information cues in different domains is as follows:

[0022] Using a text encoder, D domain cue texts and D×C cue texts constructed based on a combined domain library are embedded into text features of the same dimension.

[0023] We use domain class and inter-domain loss functions to compare and learn the domain feature representation and the domain class feature representation.

[0024] As a further technical solution, the domain feature representation and the domain class feature representation are compared and learned using domain class and inter-domain loss functions. The specific steps are as follows:

[0025] The cosine similarity between the domain feature representation and the domain class feature representation is calculated using the domain class and inter-domain loss functions. The model is made to move away from each other by reducing their cosine similarity. The proportion of class features in the prompt representation is increased to make the model pay more attention to the feature representation of class information.

[0026] As a further technical solution, in the process of the domain feature representation and the domain class feature representation moving away from each other, an orthogonal loss function is used to orthogonally learn the domain feature representation in order to maximize the distance between different domains in the visual language space.

[0027] As a further technical solution, orthogonal loss functions are used to perform orthogonal learning on the domain feature representations to maximize the distance between different domains in the visual-language space, specifically:

[0028] In the process of reconstructing domain feature representations, orthogonal loss functions are used to enable different domain types to be mapped to different directions in the feature space, minimizing the correlation between domain feature representations and learning richer feature information.

[0029] As a further technical solution, it also includes: calculating the cosine similarity between the input and output of the cosine autoencoder by reconstructing the loss function, and keeping the two highly similar to maximize the preservation of the feature representation of the text encoder.

[0030] As a further technical solution, a set of domain types with the highest relevance is selected for each category to construct a category hint representation. The specific process is as follows:

[0031] The reconstruction prompts for each category output by the cosine autoencoder in different domains are used as the accuracy of each classification task when they are prompts.

[0032] Sort the accuracy rates from highest to lowest, and then select the top N cue symbols with the best classification performance.

[0033] The average of the N hint representations is used as the final unified hint representation for the domain.

[0034] Secondly, a cross-domain image recognition and natural language generalization processing system is disclosed, including:

[0035] The class-aware enhancement module is configured to receive two template cue images, learn different representations of domain information cues and category information cues in different domains respectively to better capture the diversity of category representations, and output a class-aware enhanced image.

[0036] The domain filtering module is configured to: for class-aware enhanced images, select the most relevant set of domain types for each category to construct a category cue representation.

[0037] The above one or more technical solutions have the following beneficial effects:

[0038] The technical solution of this invention enhances the diverse representation of prompts through class awareness. By adding attention to category information, it ensures effective learning of categories, thereby preventing redundant domain information from affecting the learning of category information.

[0039] The technical solution of this invention enhances the relevance between the prompt and the target domain by eliminating irrelevant domain information through domain filtering, thereby effectively improving the generalization performance of the model.

[0040] Advantages of additional aspects of the invention will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of the invention. Attached Figure Description

[0041] The accompanying drawings, which form part of this invention, are used to provide a further understanding of the invention. The illustrative embodiments of the invention and their descriptions are used to explain the invention and do not constitute an improper limitation of the invention.

[0042] Figure 1 This is a flowchart of the Prompt fine-tuning method for category enhancement and domain awareness in an embodiment of the present invention. Detailed Implementation

[0043] It should be noted that the following detailed descriptions are exemplary and intended to provide further illustration of the invention. Unless otherwise specified, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains.

[0044] It should be noted that the terminology used herein is for the purpose of describing particular implementations only and is not intended to limit the exemplary implementations of the present invention.

[0045] Where there is no conflict, the embodiments and features in the embodiments of the present invention can be combined with each other.

[0046] The overall concept proposed in this invention is as follows:

[0047] The cross-domain image recognition and natural language generalization processing method proposed in this invention adopts a category enhancement and domain-aware Prompt fine-tuning method, which aims to better guide the classification work of CLIP model by constructing an enhanced domain-unified prompt representation. It includes two key modules: class-aware enhancement and domain filtering.

[0048] Example 1

[0049] See appendix Figure 1 As shown, this embodiment discloses a cross-domain image recognition method based on natural language generalization processing, including:

[0050] To address the issue of insufficient information on the categories of interest, this embodiment proposes a class-aware enhancement step:

[0051] By learning different representations of domain information cues and category information cues in different domains, we can better capture the diversity of category representations;

[0052] To address the issue of weak relevance of domain information, this embodiment proposes a domain filtering step in its sub-solution:

[0053] For each category, select the group of domain types with the highest relevance to construct a category hint representation.

[0054] Step 1: Class-Aware Enhancement Steps:

[0055] See the appendix for details. Figure 1 As shown, the specific inputs are "a{style}photo of a{class}" and "a{style}photo", and the processing procedure is as follows:

[0056] First, the two inputs are encoded by the CLIP model's text encoder (this results in a more uniform representation of the encoded text).

[0057] The next step is to input the obtained encoding into a cosine autoencoder for re-encoding. In this process, the loss functions (1)-(4) proposed below are used for learning. The purpose is to compare and learn the two representations to gradually enhance the attention to the category in “a{style}photo of a{class}”.

[0058] The output is specifically: "a{style}photo of a{class}" and "a{style}photo" after being encoded twice.

[0059] In this implementation example, to achieve the goal of enhancing the focus on categories, a class-aware enhancement module is proposed to take two prompt inputs: "a{style}photo of a{class}" and "a{style}photo". The class refers to the category, which depends on the dataset being processed. For example, in animal image classification, it could be cat, dog, etc. The style represents the domain type, which here includes all types involved in the existing domain generalization dataset. This indicates that contrastive learning is performed simultaneously, aiming to make the model pay more attention to category information.

[0060] For the dataset being processed, the objective of this example is to train encoded representations of two types of cues, without involving specific images, but only all categories of the dataset to be classified.

[0061] Specifically, the CLIP model's text encoder first embeds D domain cue texts (i.e., "a{style}photo") and D×C cue texts (i.e., "a{style}photo of a{class}") constructed from the combined domain library into text features of the same dimension. and In the context of the text (i∈{1,2,...,C},j∈{1,2,...,D}), Z and T have the same dimension, d. Domain features are represented by Z, and class features are represented by T. Both belong to text features.

[0062] In this implementation example, the CLIP model's text encoder has obtained a unified representation after extensive training. Encoding with it first is beneficial for obtaining a unified text representation.

[0063] D represents the total number of domain types, which is the domains covered by the current domain generalization dataset. C represents the number of categories. D×C represents the total number of possible prompt texts for all different combinations of domain types and categories, i.e., the number of possible combinations of "a{style}photo of a{class}".

[0064] Z j This represents the text encoded by the text encoder for "a{style}photo", where j represents the encoding of the prompt text constructed using the j-th domain type, i.e., the style is replaced by the j-th domain type; T i j This represents the text encoded by the text encoder for "a{style}photo of a{class}", where j represents style replaced by the j-th domain type, and i represents class replaced by the i-th category. In other words, it represents the encoding of the prompt text constructed by the i-th category in the j-th domain.

[0065] To enhance the model's focus on categories, a method is proposed to use domain class and inter-domain loss functions to compare and learn domain feature representations and domain class feature representations.

[0066] Specifically, the domain feature representation Z is calculated using the domain class and the inter-domain loss function. j and domain class feature representation The model calculates the cosine similarity between two classes and reduces their cosine similarity to keep them apart. This aims to increase the proportion of class features in the prompt representation, making the model focus more on class information and thus enriching the domain class feature representation. The purpose of this representation. Domain class and inter-domain loss can be expressed as...

[0067]

[0068] in, Let represent the hint representation of the i-th type in the j-th domain, and D represent the number of domain types.

[0069] Meanwhile, considering the domain feature representation Z j and domain class feature representation As they move away from each other, the increased importance of shared semantic components may cause category feature representations from different domains to move closer together. This paper proposes using an orthogonal loss function to adjust the Z-axis. j Orthogonal learning is performed to maximize the distance between domains in the visual-language space. Specifically, in the Z... jDuring the reconstruction process, an orthogonal loss function is used to map different domain types to different directions in the feature space (as far apart as possible), aiming to minimize the correlation between domain feature representations in order to learn richer feature information. The orthogonal loss function can be expressed as:

[0070]

[0071] Among them, |Θ·Θ| 2 Z represents the dot product between two vectors. k This represents the text encoded by the text encoder for "a{style}photo", where k represents the encoding of the prompt text constructed by replacing style with the k-th domain type.

[0072] Furthermore, to avoid excessively large semantic changes before and after cosine autoencoder encoding, a cosine similarity between the input and output of the cosine autoencoder is calculated through a reconstruction loss function, aiming to maintain a high similarity between them and maximize the preservation of the feature representation of the CLIP model text encoder. For the domain feature representation Z... j In this regard, its reconstruction loss function can be expressed as:

[0073]

[0074] in, Z represents j The reconstructed output. For domain class feature representation. In this regard, its reconstruction loss function can be expressed as:

[0075]

[0076] in express The reconstructed output.

[0077] Step Two: Domain Filtering Steps

[0078] The input in this step is: "a{style}photo of a{class}" after secondary encoding, representing the processing procedure.

[0079] First, for each class category, its corresponding D "a{style}photo of a{class}" representations (each different part is style, i.e. domain type) are used as prompt inputs for the CLIP model, and the image classification task accuracy under this prompt is calculated.

[0080] The next step is to sort the obtained accuracy rates from largest to smallest, select the top N accuracy results, and obtain their corresponding prompts (i.e., a{style}photo of a{class}).

[0081] Finally, these N prompts are averaged and pooled to obtain the average representation output for this category: a strengthened domain unified prompt representation for each category, i.e., "a{S}photo of a{class}", where S represents the average type representation, which is auxiliary information.

[0082] The domain filtering module aims to perform fine-grained filtering and aggregation of domain feature representations on category prompt representations containing information from different domains, thereby eliminating the potential negative impact of irrelevant domain information on the current category classification task.

[0083] Specifically, the domain filtering module first calculates the reconstruction cue representation of each category output by the cosine autoencoder in different domains. The accuracy rates for each classification task during the prompt are used as the reference values. Furthermore, the domain filtering module sorts these accuracy rates from highest to lowest, selecting the top N best-performing prompts. This functionality is achieved through a filtering function, ensuring that only domain information that positively impacts the classification task is retained, while irrelevant or distracting information is effectively excluded. This function can be expressed as follows:

[0084]

[0085] in, This represents the accuracy of the current classification task when using the j-th domain hint representation, where N represents the number of preferred domain types. and Let represent the cue representations of class i in the 1st, 2nd, and Nth domains of the top N domain types that provide the best classification performance. Using the average of these N cue representations as the final unified domain cue representation, the optimal unified domain cue representation for class i can be defined as follows:

[0086]

[0087] in, This represents the hint for class i in the k-th domain among the top N domain types that provide the best classification results for the current classification. This represents the unified prompt representation for the i-th class. In this way, the sub-solution of this embodiment not only constructs a more accurate and effective prompt representation but also improves the performance of target domain classification tasks, thus better addressing the challenges of unknown domains.

[0088] Furthermore, since the two modules mentioned above focus on the diversity of category representations and the usefulness of domain information, respectively, they lack representations of domain class features. To consider the inherent correlation between categories, this embodiment further measures the representation differences of the same category in different domains and the differences between different categories in each domain. Specifically, this embodiment first calculates the cosine similarity between the representation of the same category in different domains and the average representation of that category in D domains, and uses this as the intra-class loss to reduce the representation differences of the same category in different domains. The intra-class loss function can be expressed as:

[0089]

[0090] in, Let represent the mean of the representation of class i across D domains. Simultaneously, to amplify the difference in cue representation between different categories, this sub-solution calculates the cosine similarity of the representations of different categories within the same domain, using this as the inter-class loss. The inter-class loss function can then be expressed as:

[0091]

[0092] Formulas (4), (7), and (8) are collectively defined as the domain loss function for learning:

[0093]

[0094] Therefore, the overall training loss can be expressed as

[0095]

[0096] Where λ1 represents the category attention learning parameter, and λ2 represents the orthogonal space mapping learning parameter.

[0097] In this step of the embodiment, the processing procedure is as follows:

[0098] First, for each class category, its corresponding D "a{style}photo of a{class}" representations (each different part is style, i.e. domain type) are used as prompt inputs for the CLIP model, and the image classification task accuracy under this prompt is calculated.

[0099] The next step is to sort the obtained accuracy rates from largest to smallest, select the top N accuracy results, and obtain their corresponding prompts (i.e., a{style}photo of a{class}).

[0100] Finally, these N prompts are averaged and pooled to obtain the average value representation for this category;

[0101] Based on the processing of this module, it can be shown that: for a specific category, ac() represents the task accuracy when using the "a{style}photo of a{class}" (the representation obtained by the previous secondary encoding) constructed under this category as the prompt input, N represents the number of useful "a{style}photo of a{class}" representations selected by this module, and the following three parameters represent the prompt representations corresponding to the top N largest accuracies retained by this module (according to the superscript, they are the first, second, and Nth largest accuracies respectively);

[0102] It should be noted that the accuracy here corresponds to the usefulness of the domain type information in the prompt representation. That is, if for a specific category, one of the D "a{style}photo of a{class}" representations achieves the highest accuracy when used as model input, then the style (i.e., domain type) corresponding to this representation is the most effective for the current classification task.

[0103] In this implementation example, the class-aware enhancement step reduces the similarity between the feature representations of the domain information cues (i.e., "a{style}photo") and the information cues of different domain categories (i.e., "a{style}photo of a{class}"), making their cosine distance greater. Thus, when representing category information cues from different domains, the feature representations of the categories become more prominent, allowing the model to pay more attention to the different possible representations of category information.

[0104] The domain filtering step aims to address the issue of weak relevance of domain information, ensuring that the role of important domain information is fully reflected. Specifically, the domain filtering module compares the training performance of all domain types and selects N domains that are highly relevant to the target category to exclude the influence of irrelevant domains on the current domain. The combination of these two steps enables the model to better handle inter-domain differences, reduces the negative impact of irrelevant information on performance, and improves the model's generalization ability in new domains.

[0105] Example 2

[0106] The purpose of this embodiment is to provide a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the steps of the above-described method.

[0107] Example 3

[0108] The purpose of this embodiment is to provide a computer-readable storage medium.

[0109] A computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, performs the steps of the above method.

[0110] Example 4

[0111] The purpose of this embodiment is to provide a cross-domain image recognition and natural language generalization processing system, including:

[0112] The class-aware enhancement module is configured to receive two template cue images, learn different representations of domain information cues and category information cues in different domains respectively to better capture the diversity of category representations, and output a class-aware enhanced image.

[0113] The domain filtering module is configured to: for class-aware enhanced images, select the most relevant set of domain types for each category to construct a category cue representation.

[0114] Example 5

[0115] The purpose of this embodiment is to provide a computer program product containing instructions that, when run on a computer, causes the computer to perform the methods and functions involved in any of the embodiments described above.

[0116] The steps and methods involved in the apparatus of the above embodiments correspond to those in Embodiment 1. For specific implementation details, please refer to the relevant description section of Embodiment 1. The term "computer-readable storage medium" should be understood as a single medium or multiple media including one or more instruction sets; it should also be understood as including any medium capable of storing, encoding, or carrying an instruction set for execution by a processor and enabling the processor to perform any of the methods in this invention.

[0117] Those skilled in the art will understand that the modules or steps of the present invention described above can be implemented using general-purpose computer devices. Optionally, they can be implemented using computer-executable program code, thereby allowing them to be stored in a storage device for execution by a computer device, or they can be fabricated as separate integrated circuit modules, or multiple modules or steps can be fabricated as a single integrated circuit module. The present invention is not limited to any particular combination of hardware and software.

[0118] While the specific embodiments of the present invention have been described above in conjunction with the accompanying drawings, this is not intended to limit the scope of protection of the present invention. Those skilled in the art should understand that various modifications or variations that can be made by those skilled in the art without creative effort based on the technical solutions of the present invention are still within the scope of protection of the present invention.

Claims

1. A cross-domain image recognition method based on natural language generalization processing, characterized in that, include: Class-aware enhancement step: Receive two template cue images, learn different representations of domain information cues and category information cues in different domains to better capture the diversity of category representations, and output a class-aware enhanced image; The process of learning different representations of domain-specific information cues and category-specific information cues in different domains is as follows: Using a text encoder, D domain cue texts and D×C cue texts constructed based on a combined domain library are embedded into text features of the same dimension. The domain feature representation and the domain class feature representation are compared and learned by using domain class and inter-domain loss functions; The domain feature representation and the domain class feature representation are compared and learned using domain class and inter-domain loss functions. The specific steps are as follows: The cosine similarity between the domain feature representation and the domain class feature representation is calculated by the domain class and inter-domain loss function. The cosine similarity between the two is reduced to make them move away from each other. The proportion of category features in the prompt representation is increased to make the model pay more attention to the feature representation of category information. In the process of the domain feature representation and the domain class feature representation moving away from each other: orthogonal loss function is used to orthogonally learn the domain feature representation in order to maximize the distance between different domains in the visual language space; Orthogonal learning of domain feature representations is performed using an orthogonal loss function to maximize the distance between different domains in the visual-language space, specifically: In the process of reconstructing domain feature representations, orthogonal loss functions are used to enable different domain types to be mapped to different directions in the feature space, minimizing the correlation between domain feature representations and learning richer feature information. Domain filtering step: For class-aware enhanced images, select the most relevant set of domain types for each category to construct a category cue representation.

2. The cross-domain image recognition method based on natural language generalization processing as described in claim 1, characterized in that it further... include: The cosine similarity between the input and output of the cosine autoencoder is calculated by reconstructing the loss function, and the similarity between the two is maintained, thereby maximizing the preservation of the feature representation of the text encoder.

3. The cross-domain image recognition method based on natural language generalization processing as described in claim 1, characterized in that, For each category, the most relevant set of domain types is selected to construct a category hint representation. The specific process is as follows: The reconstruction prompts for each category output by the cosine autoencoder in different domains are used as the accuracy of each classification task when they are prompts. Sort the accuracy rates from highest to lowest, and then select the top N cue symbols with the best classification performance. The average of the N hint representations is used as the final unified hint representation for the domain.

4. A cross-domain image recognition system employing a cross-domain image recognition method based on natural language generalization processing as described in any one of claims 1-3, characterized in that, include: The class-aware enhancement module is configured to receive two template cue images, learn different representations of domain information cues and category information cues in different domains respectively to better capture the diversity of category representations, and output a class-aware enhanced image. The domain filtering module is configured to: for class-aware enhanced images, select the most relevant set of domain types for each category to construct a category cue representation.

5. A computer device comprising a memory, a processor, and a computer program stored in the memory and running on the processor, characterized in that, When the processor executes the program, it implements the steps of the method described in any one of claims 1-3.

6. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it performs the steps of the method described in any one of claims 1-3 above.

Images