A domain generalization image category prediction method based on feature decoupling and class center matching

Abstract

The present invention discloses a method for predicting and processing image categories based on domain generalization of feature decoupling and class center matching, comprising: obtaining a domain generalization data set corresponding to an image to be processed; performing text decoupling on the text in the domain generalization data set; performing image decoupling on the image in the domain generalization data set; constructing a class center memory, using the class center memory to obtain a domain-related prediction probability, obtaining a domain-independent prediction probability by calculation, performing weighted summation on the domain-related prediction probability and the domain-independent prediction probability to obtain the prediction probability of the overall category of the image to be processed and outputting it. The present invention constructs a class center memory, obtains a domain-related prediction probability, combines the domain-related prediction probability with the domain-independent prediction probability, obtains the overall prediction probability, and greatly improves the prediction probability of the overall category of the image to be processed.

Description

Technical Field

[0001] The present invention belongs to the technical field of domain generalization prediction of image classification, and specifically relates to a domain generalization image category prediction processing method based on feature decoupling and class center matching. Background Art

[0002] Most machine learning methods are based on the assumption that the training and test data are independent and identically distributed. However, in real-world situations, this assumption does not always hold and there is often a distribution shift between the training and test data. Therefore, the domain generalization task is proposed, which aims to learn a generalized model that can perform well on unseen target domains using limited source domain data for training. DG (Domain Generalization) has always been an important topic in the field of machine learning and has attracted considerable attention.

[0003] The main challenge of domain generalization is to learn a model with good generalization ability that can extract domain-invariant representations from source domain datasets. Fortunately, recent studies have shown that large-scale pre-trained models can greatly enhance domain generalization capabilities. In particular, for pre-trained visual-language models (Visual Foundation Models, VFMs), they are trained with large-scale images and texts to obtain robust visual representations, which are inherently rich in semantic information from prior knowledge. Therefore, such VFMs are able to encode the semantic meaning of visual descriptions regardless of the image domain, which is in line with the goal of DG, which is to learn unified visual semantic representations across different domains.

[0004] Most current DG methods use multi-domain training datasets to train a model that generalizes to unseen domains. Intuitively, we can also use these datasets to fine-tune the capabilities of the base model. Subsequently, the idea of ​​using hints to fine-tune the base model was introduced, achieving better results on specific downstream tasks with very few training samples. However, directly applying hint fine-tuning to predict image classification in the context of domain generalization faces significant challenges. This is because existing hint fine-tuning methods adjust the base model to generate image-related domain-dependent features, while domain generalization requires the model to generate image-related domain-independent features that can work well on different unseen domains. Therefore, in order to effectively apply hint fine-tuning to domain generalization to predict image categories, it is key to be able to guide the base model to disentangle the connection between domain-independent features in all domains and domain-dependent features specific to certain domains. Summary of the invention

[0005] In order to solve the above problems existing in the prior art, the present invention provides an image category prediction processing method based on domain generalization of feature decoupling and class center matching. The technical problem to be solved by the present invention is achieved through the following technical solutions:

[0006] A method for image category prediction based on domain generalization of feature decoupling and class center matching, comprising:

[0007] For an image to be processed, a domain generalization dataset corresponding to the image to be processed is obtained; the domain generalization dataset includes: images of each preset category in different fields and labels corresponding to each image, and the labels include: category labels and domain labels;

[0008] The LLM model is used to generate a fine-grained text description corresponding to each category label and a fine-grained text description corresponding to each domain label respectively; the fine-grained text description corresponding to each category label is processed by the LLM model to obtain a domain-independent text description corresponding to each category label; the fine-grained text description corresponding to each domain label is processed by the LLM model to obtain a domain-related text description corresponding to each domain label; a preset text description is inserted into each domain-independent text description and each domain-related text description, and a domain-independent text feature set of all domain-independent text descriptions and a domain-related text feature set of all domain-related text descriptions are obtained through a text encoder;

[0009] Inserting a preset word vector into each image of the domain generalization dataset, and using the domain-independent text feature set and the domain-related text feature set to respectively train to obtain domain-independent image features corresponding to each image and domain-related image features corresponding to each image;

[0010] A class center memory is constructed using all domain-related image features; the class center memory and the domain-related text feature set are used to obtain domain-related prediction probabilities; the domain-independent text feature set is matched with all domain-independent image features to obtain domain-independent prediction probabilities; the domain-related prediction probabilities and the domain-independent prediction probabilities are weightedly summed to obtain the prediction probability of the overall category of the image to be processed and output.

[0011] In one embodiment of the present invention, the domain generalization dataset includes:

[0012] The data set D of the source domain S and the target domain data set D T ;in,

[0013] is the data set D of the source domain SFor each source domain, Among them, each data sample (x i ,y i ) is given by the input image x i And the input image x i The corresponding label y i Composition, N S is the total number of source domains, is the total number of all images in the current source domain; is the data set D of the target domain T For each target domain, Among them, each data sample (x i ′,y i ′) is composed of the input image x i ′ and input image x i ′ corresponds to the label y i 'Composition, N t is the total number of target domains, is the total number of all images in the current target domain.

[0014] In one embodiment of the present invention, the LLM model is used to generate a fine-grained text description corresponding to each category label and a fine-grained text description corresponding to each domain label, including:

[0015] The designed preset questions are used to guide GPT-3 in the LLM model to generate fine-grained text descriptions corresponding to each category label and each domain label.

[0016] In one embodiment of the present invention, the LLM model is used to process the fine-grained text description corresponding to each category label to obtain a field-independent text description corresponding to each category label; the LLM model is used to process the fine-grained text description corresponding to each field label to obtain a field-related text description corresponding to each field label, including:

[0017] The GPT-3 is used to summarize and analyze the shared and common attributes of the fine-grained text description corresponding to each category label to obtain a domain-independent text description corresponding to each category label;

[0018] The GPT-3 is used to process the fine-grained text description corresponding to each domain label to obtain the domain-related text description corresponding to each domain label.

[0019] In one embodiment of the present invention, a preset text description is inserted into each domain-independent text description and each domain-related text description, and a domain-independent text feature set of all domain-independent text descriptions and a domain-related text feature set of all domain-related text descriptions are obtained through a text encoder, including:

[0020] Using a text editor of the CLIP model, the preset text description is inserted into each field-independent text description to obtain a field-independent text feature set to be optimized;

[0021] The loss function L of the domain-independent text feature set to be optimized is obtained using the L2 distillation loss kg1 ;

[0022] According to the domain-independent text feature set to be optimized and all images of the domain generalization dataset, the comparative learning target L is obtained. ce1 ;

[0023] Using a text editor of the CLIP model, the preset text description is inserted into each field-related text description to obtain a field-related text feature set to be optimized;

[0024] The loss function L of the text feature set related to the field to be optimized is obtained by using L2 distillation loss kg2 ;

[0025] According to the text feature set related to the field to be optimized and all images of the field generalization dataset, the comparative learning target L is obtained. ce2 ;

[0026] According to the loss function L kg1 , the comparative learning objective L ce1 , the loss function L kg2 Compared with the above learning objectives L ce2 , and get the loss function L of the entire text text ;in,

[0027] L text =L ce1 +α1*L ce2 +β1*(L kg1 +L kg2 ), α1 and β1 are pre-set weight parameters;

[0028] Using the loss function L of the entire text text The domain-independent text feature set to be optimized and the domain-dependent text feature set to be optimized are processed separately to obtain the domain-independent text feature set and the domain-dependent text feature set.

[0029] In one embodiment of the present invention, a preset word vector is inserted into each image of the domain generalization dataset, and the domain-independent text feature set and the domain-related text feature set are used to respectively train to obtain domain-independent image features corresponding to each image and domain-related image features corresponding to each image, including:

[0030] Using deep VPT as a domain-independent visual encoder I and domain-dependent visual encoder f I Process each image and extract the image features that are independent of the field to be optimized corresponding to each image Image features related to the domain to be optimized corresponding to each image in, E I is the domain-independent visual encoder f I The preset word vector in E S is the domain-dependent visual encoder f S The preset word vectors in ;

[0031] According to all the domain-independent image features to be optimized and the set of domain-independent text features, the comparative learning target L′ is obtained ce1 ;

[0032] Perform domain confusion regularization on all the domain-independent image features to be optimized and the domain-related text feature set to obtain the loss function L mix ;

[0033] Use L2 distillation loss to obtain all the domain-independent image features to be optimized The loss function L kg ;

[0034] According to all the domain-related image features to be optimized and the domain-related text feature set, the comparative learning target L′ is obtained ce2 ;

[0035] According to the comparative learning objective L′ ce1 , the comparative learning objective L′ ce2 , the loss function L kg And the loss function L mix , we get the loss function L of the entire image img ;in,

[0036] L img =L′ ce1 +α2*L′ ce2 +β2*L kg +β3*L mix, α2, β2 and β3 are pre-set weight parameters;

[0037] Using the loss function L of the entire image img All the domain-independent image features to be optimized and all the domain-dependent image features to be optimized are processed to obtain the domain-independent image features corresponding to each image and the domain-dependent image features corresponding to each image.

[0038] In one embodiment of the present invention, all domain-related image features are used to construct a class-centric memory, including:

[0039] According to the data set D of the source domain S , the total number of source domains and the number of category labels under the source domain, and obtain the class center tensor C, in,

[0040] C m，k is the center tensor of the kth category in the mth source domain in the class center tensor C,

[0041] d is the feature dimension, M is the total number of category labels in the mth source domain, and N C is the total number of category labels in the data set of the source domain;

[0042] The class center tensor C is represented as a two-dimensional matrix and cached as the class center memory.

[0043] In one embodiment of the present invention, obtaining a domain-related prediction probability using the class center memory and the domain-related text feature set includes:

[0044] Select the target domain data set D T The image in is used as the test image x i ′, through the domain-dependent visual encoder f S Extract the test image x i The test image feature f in ′ S (x i ′,E S );

[0045] The correlation degree Φ(x) between the test image feature and the class center memory is obtained by the preset formula 1 i ′, C), Wherein, the preset formula 1 is Φ(xi′, C)=exp(-β(1-f S (xi′,E S )C T )); β is a hyperparameter for adjusting the sharpness of similarity output, C Tis the transpose of C, f S (x i ′,E S )C T is the cosine similarity between the test image feature and the class center tensor C;

[0046] The predicted probability P related to the field is obtained using the preset formula 2 i S ; Wherein, the preset formula 2 is P i S =Φ(x i ′,C)L C , L C is a collection of tags,

[0047] In one embodiment of the present invention, the domain-independent text feature set is matched with all domain-independent image features to obtain the domain-independent prediction probability, including:

[0048] Substitute the domain-independent text features and all the domain-independent image features into the preset formula 3 to obtain the domain-independent prediction probability P i I ; Wherein, the preset formula 3 is is the domain-independent image feature, is a set of text features that are independent of the field.

[0049] In one embodiment of the present invention, performing a weighted sum of the field-related prediction probability and the field-independent prediction probability to obtain the prediction probability of the overall category of the image to be processed and outputting the prediction probability includes:

[0050] Substitute the domain-related prediction probability and the domain-independent prediction probability into the preset formula 4 for weighted summation to obtain the prediction probability of the overall category of the image to be processed and output P i And output; wherein the preset formula 4 is P i =P i I +α3*P i S , α3 is the trade-off parameter between the prediction probability related to the field and the prediction probability independent of the field.

[0051] Beneficial effects of the present invention:

[0052] In the solution provided by the embodiment of the present invention, a novel text feature-guided visual cue tuning framework is proposed. First, a large language model LLM is used to process the label multiple times to obtain domain-independent text descriptions and domain-related text descriptions respectively; a text encoder is used to insert the learnable text description to obtain domain-independent text features and domain-related text features; the domain-independent text features and domain-related text features are then used to train to obtain domain-independent image features and domain-related image features, making the decoupling process easier; the embodiment of the present invention constructs a class-centric memory to obtain domain-related prediction probabilities, and the domain-related prediction probabilities are combined with the domain-independent prediction probabilities to obtain the prediction probability of the overall category of the image to be processed, thereby improving the accuracy of the prediction of the category of the image to be processed. BRIEF DESCRIPTION OF THE DRAWINGS

[0053] Figure 1 A schematic diagram of an image category prediction processing method based on domain generalization of feature decoupling and class center matching provided by an embodiment of the present invention;

[0054] Figure 2 A network framework diagram of an image category prediction processing method based on domain generalization of feature decoupling and class center matching provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0055] The following will be combined with the accompanying drawings of the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0056] Example 1

[0057] The embodiment of the present invention provides an image category prediction processing method based on domain generalization of feature decoupling and class center matching, such as Figure 1 As shown, it may include:

[0058] S1, for an image to be processed, obtaining a domain generalization dataset corresponding to the image to be processed; the domain generalization dataset includes: images of each preset category in different fields and labels corresponding to each image, and the labels include: category labels and domain labels;

[0059] S2, using the LLM model to generate a fine-grained text description corresponding to each category label and a fine-grained text description corresponding to each domain label; using the LLM model to process the fine-grained text description corresponding to each category label to obtain a domain-independent text description corresponding to each category label; using the LLM model to process the fine-grained text description corresponding to each domain label to obtain a domain-related text description corresponding to each domain label; inserting the preset text description into each domain-independent text description and each domain-related text description, and obtaining a domain-independent text feature set of all domain-independent text descriptions and a domain-related text feature set of all domain-related text descriptions through a text encoder;

[0060] S3, inserting the preset word vector into each image of the domain generalization dataset, and using the domain-independent text feature set and the domain-related text feature set to respectively train to obtain the domain-independent image features corresponding to each image and the domain-related image features corresponding to each image;

[0061] S4, using all domain-related image features to build a class-centric memory; using the class-centric memory and the domain-related text feature set to obtain the domain-related prediction probability; using the domain-independent text feature set and all domain-independent image feature matching to obtain the domain-independent prediction probability; performing weighted summation of the domain-related prediction probability and the domain-independent prediction probability to obtain the prediction probability of the overall category of the image to be processed and output it.

[0062] The embodiment of the present invention proposes a novel prompt fine-tuning framework, which uses LLM model-assisted text disentanglement and text-guided visual disentanglement model to predict image categories through domain generalization. Specifically, first, the corresponding domain generalization dataset is obtained for the image to be processed, and the domain-independent text feature set of all domain-independent text descriptions and the domain-related text feature set of all domain-related text descriptions are generated using the LLM model. Secondly, the learned domain-independent text feature set and domain-related text feature set are used to guide the learning of domain-independent image features corresponding to each image in the domain generalization dataset and domain-related image features corresponding to each image. A class center memory is proposed for class center matching learning, and the model finally outputs a combination of domain-independent prediction and class center matching learning. The purpose of the present invention is to find an effective fine-tuning method that can better extract prompts to disentangle the connection between invariant features and features specific to certain domains. The embodiment of the present invention makes full use of the controllable and flexible language prompt characteristics of VFM. The prompt of the present invention plays a vital role in guiding the disentanglement of image features. Intuitively speaking, text in VFM can be more easily disentangled because the text / language itself is rich in semantic information and easy to interpret. For example, given a comprehensive text description of the image category "sheep", the description can be divided into two parts, one is a domain-related text description (for example: the overall color and image of the sheep), and the other is a domain-independent description (for example: the shape and texture of the sheep). Using large language models such as GPT, these descriptions can be generated and disentangled.

[0063] For ease of understanding, each step of the embodiment of the present invention is described below.

[0064] For S1, for an image to be processed, a domain generalization dataset corresponding to the image to be processed is obtained. Different images to be processed obtain their respective corresponding domain generalization datasets. Among them, for any image to be processed, the domain generalization dataset corresponding to the image to be processed is obtained by using the existing technology, which will not be described in detail here.

[0065] Specifically, domain generalization datasets may include:

[0066] The data set D of the source domain S and the target domain data set D T ;in,

[0067] The data set D of the source domain S For each source domain, Among them, each data sample (x i ,y i ) is given by the input image x i And the input image xi The corresponding label y i Composition, N S is the total number of source domains, is the total number of all images in the current source domain; The target domain data set D T For each target domain, Among them, each data sample (x i ′,y i ′) is composed of the input image x i ′ and input image x i ′ corresponds to the label y i 'Composition, N t is the total number of target domains, is the total number of all images in the current target domain.

[0068] In specific experiments, the number of target domains is usually set to 1. The goal of domain-general learning is to perform well on the target domain when only source domain data is available during model training. The main challenge is to address the domain distribution shift between source and target domain data.

[0069] For S2, Figure 2 As shown in the text decoupling section (a), it can include:

[0070] S21, using the LLM model to generate a fine-grained text description corresponding to each category label and a fine-grained text description corresponding to each domain label;

[0071] S22, using the LLM model to process the fine-grained text description corresponding to each category label to obtain a domain-independent text description corresponding to each category label; using the LLM model to process the fine-grained text description corresponding to each domain label to obtain a domain-related text description corresponding to each domain label;

[0072] S23, inserting the preset text description into each domain-independent text description and each domain-related text description, and obtaining a domain-independent text feature set of all domain-independent text descriptions and a domain-related text feature set of all domain-related text descriptions through a text encoder.

[0073] Specifically, S21 uses the LLM model to generate a fine-grained text description corresponding to each category label and a fine-grained text description corresponding to each domain label, including:

[0074] The designed preset questions are used to guide GPT-3 in the LLM model to generate fine-grained text descriptions corresponding to each category label and each domain label.

[0075] Among them, GPT-3 (Generative Pre-trained Transformer 3) is a large language model developed by Google and released in May 2020. It has 175 billion parameters, making it the language model with the most parameters at the time. GPT-3 performs well on a variety of natural language processing tasks, including text generation, machine translation, question answering, and summarization.

[0076] The format of the designed preset questions may include: What is [category] like in [domain], How to identify [category] in [domain], Image title of [category] in [domain] and Please provide some information about the [domain] and other prompting questions, such as: Detailed description of objects under [category].

[0077] Fine-grained text description refers to a detailed and accurate description of things or events. This description usually contains rich details and information, which can help readers or listeners form a clearer and more comprehensive understanding.

[0078] S22, using the LLM model to process the fine-grained text description corresponding to each category label to obtain a domain-independent text description corresponding to each category label; using the LLM model to process the fine-grained text description corresponding to each domain label to obtain a domain-related text description corresponding to each domain label, including:

[0079] GPT-3 is used to summarize and analyze the shared and common attributes of the fine-grained text descriptions corresponding to each category label, and a domain-independent text description corresponding to each category label is obtained;

[0080] GPT-3 is used to process the fine-grained text description corresponding to each domain label to obtain the domain-related text description corresponding to each domain label.

[0081] Among them, the field-independent text description corresponding to each category label is relatively general and broad so as to be applicable to a variety of different fields and contexts.

[0082] The domain-related text description corresponding to each domain label is relatively specific and detailed so as to be applicable to a specific domain and context.

[0083] Domain-independent and domain-dependent text descriptions can help users better understand and process information. They can serve as the basis for information organization and retrieval, and can also help users perform tasks such as natural language processing and machine learning.

[0084] After obtaining the domain-independent text description and the domain-related text description, the embodiment of the present invention uses the text encoder of CLIP (Contrastive Language-Image Pre-Training) to generate domain-independent embedding and domain-independent embedding for each category label and each domain label, respectively, to obtain the domain-independent text description corresponding to each category label and the domain-related text description corresponding to each domain label. These domain-independent text embeddings will serve as a guide for learning domain-independent visual cues.

[0085] S23, inserting the preset text description into each field-independent text description and each field-related text description, and obtaining a field-independent text feature set of all field-independent text descriptions and a field-related text feature set of all field-related text descriptions through a text encoder, including:

[0086] 1) Using the text editor of the CLIP model, insert the preset text description into each field-independent text description to obtain a field-independent text feature set to be optimized;

[0087] The preset text descriptions may include learnable text descriptions. Learnable text descriptions refer to text descriptions that a model can learn from one or more source domains and apply to new target domains. In other words, a model can learn general knowledge between different domains and use it to solve problems in new domains. For example, a model can learn how to diagnose diseases in the medical field, and then apply this knowledge to the financial field to predict stock prices. Or, a model can learn how to recommend products in the e-commerce field, and then apply this knowledge to the tourism field to recommend attractions.

[0088] 2) Use L2 distillation loss to obtain the loss function L of the field-independent text feature set to be optimized kg1 ;

[0089] Specifically, in the L2 distillation loss, N C is the number of category labels, is the L2 norm, is a set of domain-independent text features, A set of domain-independent text features to be optimized.

[0090] 3) According to the domain-independent text feature set to be optimized and all images of the domain generalization dataset, the comparative learning target L is obtained ce1 ;

[0091] Comparative Learning Objectives ce1 The specific calculation method is as follows:

[0092]

[0093]

[0094] represents the one-hot encoded label, N C is the dimension of the label, for each input image x i , the actual labels are assigned to [1,0,…,0].

[0095] 4) Using the text editor of the CLIP model, insert the preset text description into each field-related text description to obtain a field-related text feature set to be optimized;

[0096] 5) Use L2 distillation loss to obtain the loss function L of the field-related text feature set to be optimized kg2 ;

[0097] Similar to 2), the loss function L is obtained by kg1 The same method is used to obtain the loss function L of the text feature set related to the field to be optimized. kg2 .

[0098]

[0099] 6) According to the domain-related text feature set to be optimized and all images of the domain generalization dataset, the comparative learning target L is obtained ce2 ;

[0100] Similar to 3), we use the contrast learning objective L ce1 The same method is used to obtain the comparative learning target L ce2 .

[0101]

[0102]

[0103] in, is a set of domain-related text features, h clip A collection of text features related to the field to be optimized.

[0104] 7) According to the loss function L kg1 , contrast learning objectives L ce1 , loss function L kg2 and contrast learning objectives L ce2 , and get the loss function L of the entire text text ;in,

[0105] L text =L ce1+α1*L ce2 +β1*(L kg1 +L kg2 ), α1 and β1 are pre-set weight parameters;

[0106] 8) Using the overall text loss function L text The domain-independent text feature set to be optimized and the domain-dependent text feature set to be optimized are processed separately to obtain a domain-independent text feature set and a domain-dependent text feature set.

[0107] After completing S2 text disentanglement, the embodiment of the present invention keeps the text encoder and its domain-independent and domain-dependent text prompts unchanged. Then, under the guidance of the disentangled text embedding, image disentanglement is performed.

[0108] Specifically, for S3, the preset word vector is inserted into each image of the domain generalization dataset, and the domain-independent text feature set and the domain-related text feature set are used to train the domain-independent image features corresponding to each image and the domain-related image features corresponding to each image, respectively, such as Figure 2 As shown in the image decoupling part (b), it can include:

[0109] S31, using deep VPT as a domain-independent visual encoder f I and domain-dependent visual encoder f I Process each image and extract the image features that are independent of the field to be optimized corresponding to each image Image features related to the domain to be optimized corresponding to each image in, E I is the domain-independent visual encoder f I The preset word vector in E S is the domain-dependent visual encoder f S The preset word vectors in ;

[0110] Deep VPT (Deep Virtual Prototype), referred to as Deep VPT in domain generalization, is a method for achieving domain generalization. The basic idea of ​​Deep VPT is to learn a virtual prototype that can represent the common knowledge between the source domain and the target domain. Then, the model can learn domain generalization knowledge by interacting with this virtual prototype.

[0111] Specifically, the specific steps of deep VPT are as follows:

[0112] a) Learning Virtual Prototype: First, the model learns a virtual prototype from the source domain. This virtual prototype can be a vector, a matrix, or a more complex data structure.

[0113] b) Interacting with the virtual prototype: The model then interacts with this virtual prototype to learn domain generalization knowledge. This can be achieved in a variety of ways, for example, the model can be trained with the virtual prototype as input, or the model can be trained adversarially with the virtual prototype.

[0114] c) Fine-tuning on the target domain: Finally, the model is fine-tuned on the target domain. This helps the model apply what it has learned from the virtual prototype to the target domain.

[0115] The preset word vectors may include learnable word vectors. In domain generalization, learnable word vectors refer to word vectors that the model can learn from one or more source domains and apply to word vectors in a new target domain. In other words, the model can learn general word vectors between different domains and use them to solve problems in new domains.

[0116] For example, a model can learn how to convert medical terms into word vectors in the medical field, and then apply these word vectors to the financial field to convert financial terms into word vectors. Or, a model can learn how to convert product names into word vectors in the e-commerce field, and then apply these word vectors to the tourism field to convert attraction names into word vectors.

[0117] S32, based on all the domain-independent image features and domain-independent text feature sets to be optimized, obtain the comparative learning target L′ ce1 ;

[0118] Comparative Learning Objectives

[0119] S33, perform domain confusion regularization on all domain-independent image features and domain-related text feature sets to be optimized to obtain the loss function L mix ;

[0120]

[0121] Loss Function

[0122] In order to train domain confusion regularization, the embodiment of the present invention replaces the one-hot domain labels of all images from different domains in the above formula Set to [1 / N S ,1 / N S ,…,1 / N S ].

[0123] S34, use L2 distillation loss to obtain all domain-independent image features to be optimized The loss function L kg ;

[0124] Loss Function

[0125] By calculating the loss function L kg Enhancement The generalization performance of and The distance between.

[0126] S35, based on all the domain-related image features and domain-related text feature sets to be optimized, obtain the comparative learning target L′ ce2 ;

[0127] Comparative Learning Objectives

[0128] S36, according to the comparative learning target L′ ce1 , contrast learning objectives L′ ce2 , loss function L kg And the loss function L mix , we get the loss function L of the entire image img ;in,

[0129] L img =L′ ce1 +α2*L′ ce2 +β2*L kg +β3*L mix , α2, β2 and β3 are pre-set weight parameters;

[0130] S37, using the loss function L of the entire image img All domain-independent image features to be optimized and all domain-related image features to be optimized are processed to obtain domain-independent image features corresponding to each image and domain-related image features corresponding to each image.

[0131] For S4, Figure 2 As shown in the domain-related class center matching part (c), it can include:

[0132] S41, constructing a class-centric memory using all domain-related image features;

[0133] S42, using the class center memory and the domain-related text feature set to obtain the domain-related prediction probability;

[0134] S43, using the domain-independent text feature set and all the domain-independent image features to match to obtain the domain-independent prediction probability;

[0135] S44, performing weighted summation of the domain-related prediction probability and the domain-independent prediction probability to obtain the prediction probability of the overall category of the image to be processed and output it.

[0136] Specifically, for S41, using all domain-related image features to build a class-centric memory may include:

[0137] S41A, based on the data set D of the source domain S , the total number of source domains and the number of category labels under the source domain, and obtain the class center tensor C, in,

[0138] C m，k is the center tensor of the kth category of the mth source domain in the class center tensor C;

[0139] d is the feature dimension, M is the total number of category labels in the mth source domain, and N C is the total number of category labels in the data set of the source domain;

[0140] S41B, represents the class center tensor C as a two-dimensional matrix and caches it as a class center memory.

[0141] S41C, after constructing the class-centric memory, the embodiment of the present invention also converts their corresponding labels into one-hot encoding, because each class-centric memory corresponds to a category label.

[0142] For S42, using the class center memory and the domain-related text feature set to obtain the domain-related prediction probability may include:

[0143] S42A, select the target domain data set D T The image in is used as the test image x′ i , through the domain-dependent visual encoder f S Extract the test image x′ i The test image features f in S (x′ i , E S );

[0144] S42B, the correlation degree Φ(x i ′, C),

[0145] Wherein, the preset formula 1 is Φ(xi′, C)=exp(-β(1-f S (xi′,E S )C T)); β is a hyperparameter for adjusting the sharpness of similarity output, C T is the transpose of C, f S (x i ′,E S )C T is the cosine similarity between the test image features and the class center tensor C; both key and query features are L2 normalized. Domain-dependent predictions based on the cache model can be obtained through L C Obtained by a linear combination of weighted query-key similarities.

[0146] S42C, using the preset formula 2 to obtain the domain-related prediction probability P i S ;

[0147] Among them, the preset formula 2 is P i S =Φ(x i ′,C)L C , L C is a collection of tags,

[0148] The class center matching learning can be called a key-value cache model, where the key is the class center of each class. Their corresponding values ​​are label sets During inference, the test image can be used as a query image.

[0149] For S43, the domain-independent text feature set is matched with all domain-independent image features to obtain the domain-independent prediction probability, which may include:

[0150] Substitute the domain-independent text features and all domain-independent image features into the preset formula 3 to obtain the domain-independent prediction probability P i I ;

[0151] Among them, the preset formula 3 is is a domain-independent image feature, It is a set of domain-independent text features.

[0152] For S44, the domain-related prediction probability and the domain-independent prediction probability are weighted and summed to obtain the prediction probability of the overall category of the image to be processed and output, including:

[0153] Substitute the domain-related prediction probability and the domain-independent prediction probability into the preset formula 4 for weighted summation to obtain the prediction probability of the overall category of the image to be processed and output P i And output;

[0154] Among them, the preset formula 4 is P i=P i I +α3*P i S , α3 is the trade-off parameter between the domain-related prediction probability and the domain-independent prediction probability.

[0155] In order to make full use of domain-independent and domain-related information for domain-general learning, the above method is used to combine the domain-related prediction probability and the domain-independent prediction probability to obtain the prediction probability of the overall category of the image to be processed.

[0156] In addition, the embodiment of the present invention further regards the cached class center memory C as a learnable parameter and initializes it with the mean feature, and then fine-tunes C through several rounds of training through SGD (Stochastic Gradient Descent). Updating the cached class center memory can improve the estimation of the association degree, thereby more accurately calculating the cosine similarity between the test image features and the learnable class center memory. In contrast, L C It is a one-hot encoding representing the true annotation and should be kept frozen to keep a good memory of the category information during model training. Among them, SGD is an optimization algorithm. The basic idea of ​​SGD is to iteratively update the model parameters along the gradient direction to minimize the loss function. In domain generalization, SGD is often used to train the performance of the model on the source domain. Then, the model is fine-tuned on the target domain. This can help the model apply the knowledge learned from the source domain to the target domain.

[0157] In order to evaluate the method provided by the embodiment of the present invention, the evaluation results of the data set and the evaluation index are shown in Table 1 below.

[0158] Table 1

[0159]

[0160] The details are as follows:

[0161] Datasets: In the experiments, the embodiments of the present invention follow the experimental settings of DomainBed. DomainBed is a benchmark suite for evaluating domain generalization algorithms. It contains six challenging domain generalization tasks, covering multiple fields such as image classification, natural language processing, and reinforcement learning. The embodiments of the present invention use five commonly used domain-general learning task datasets for evaluation: PACS (4 domains, 9,991 samples, 7 categories), VLCS (4 domains, 10,729 samples, 5 categories), OfficeHome (4 domains, 15,588 samples, 65 categories), DomainNet (6 domains, 586,575 samples, 345 categories) and TerraIncognita (4 domains, 24,788 samples, 10 categories).

[0162] Specifically, PACS (Pascal VOC 2012 Attribute and Class Segmentation) is an image dataset for domain generalization and attribute recognition tasks; VLCS (Visual Learning with Class Similarity) is an image dataset for domain generalization and visual learning tasks; OfficeHome is an image dataset for domain generalization and visual learning tasks; DomainNet is a large-scale image dataset for domain generalization and visual learning tasks; TerraIncognita is a large-scale image dataset for domain generalization and visual learning tasks.

[0163] Evaluation Metrics: In domain generalization learning experiments, one domain is usually designated as the target domain, and the remaining domains are considered source domains. To ensure reliable results, the embodiment of the present invention calculates the average performance of multiple experiments.

[0164] In the solution provided by the embodiment of the present invention, a novel text feature-guided visual cue tuning framework is proposed. First, the label is processed multiple times using a large language model (LLM) to obtain domain-independent text descriptions and domain-related text descriptions respectively; the learnable text description is inserted using a text encoder to obtain domain-independent text features and domain-related text features; the domain-independent text features and domain-related text features are then trained to obtain domain-independent image features and domain-related image features, making the decoupling process easier; the embodiment of the present invention constructs a class-centric memory to obtain domain-related prediction probabilities, combines the domain-related prediction probabilities with the domain-independent prediction probabilities, and obtains the prediction probability of the overall category of the image to be processed, thereby improving the accuracy of the prediction of the category of the image to be processed.

[0165] Example 2

[0166] The embodiment of the present invention provides a road condition environment prediction processing method for unmanned driving, which may include the following steps:

[0167] L1, take the image taken by the unmanned driving system as the image to be processed, and obtain the domain generalization dataset corresponding to the processed image; the domain generalization dataset includes: images of each road condition environment under different weather conditions and the labels corresponding to each image, and the labels may include: road condition environment labels and weather condition labels.

[0168] Among them, the acquired road condition environment label is equivalent to the category label in Example 1, and the weather condition label is equivalent to the field label in Example 1.

[0169] L2, using the LLM model to generate fine-grained text descriptions corresponding to different road conditions and fine-grained text descriptions corresponding to different weather conditions for each road; using the LLM model to process the fine-grained text descriptions corresponding to each road condition to obtain a text description that is independent of the weather condition (domain-independent) corresponding to each road condition; using the LLM model to process the fine-grained text descriptions corresponding to each weather condition to obtain a text description that is related to each weather condition (domain-related); inserting the preset text description into each weather condition-independent text description and each weather condition-related text description, and obtaining a weather condition-independent text feature set of all weather condition-independent text descriptions and a weather condition-related text feature set of all weather condition-related text descriptions through a text encoder;

[0170] For L2, the text description independent of weather conditions and the text description related to weather conditions correspond to the text description independent of the field and the text description related to the field in Example 1, respectively. The preset text description is a preset learnable text description, and the preset learnable text description is used to train the text description independent of weather conditions and the text description related to weather conditions on the text side to achieve better text-side decoupling. The preset learnable text description is an array, which is a torch vector in the program.

[0171] L3, inserting the preset word vector into each image of the data set of different weather conditions and road conditions, using the text feature set irrelevant to weather conditions and the text feature set relevant to weather conditions to respectively train to obtain the image features irrelevant to weather conditions and the image features relevant to weather conditions corresponding to each image;

[0172] For L3, by inserting preset word vectors at the image end, we train the image features that are independent of the weather conditions and the image features that are related to the weather conditions for each image, so as to achieve better image-end decoupling; we use confusion loss to constrain the image features that are independent of the weather conditions, and use the loss function L kg Close the distance between the features generated by the model and the features generated by the original CLIP model, effectively improving the generalization performance of the model itself. The preset word vector is a preset learnable word vector. The learnable word vector in domain generalization refers to the word vector that can be learned and adapted in different domains or tasks. Traditional word vector models, such as Word2Vec and GloVe, are trained on specific domains or tasks, so they may not perform well in other domains or tasks. Learnable word vector models can be trained and adjusted on different domains or tasks to improve their generalization ability.

[0173] L4, using all weather condition related image features to build a class center memory; using the class center memory and the weather condition related text feature set to obtain the weather condition related prediction probability; using the weather condition independent text feature set and all weather condition independent image feature matching to obtain the weather condition independent prediction probability; performing weighted summation of the weather condition related prediction probability and the weather condition independent prediction probability to obtain the prediction probability of the overall road condition environment of the unmanned vehicle and output it.

[0174] For L4, the class center memory is constructed using the image features related to weather conditions to make full use of the information related to weather conditions, and the test image and the class center are matched to generate the prediction probability related to weather conditions. Finally, the prediction probability related to weather conditions and the prediction probability unrelated to weather conditions are weighted and summed to obtain the prediction probability of the overall road environment of the unmanned vehicle and output it. The road environment where the current unmanned vehicle is located can be predicted.

[0175] The specific contents of L1 to L4 can be understood in conjunction with the relevant contents of S1 to S4 in Example 1, and will not be repeated here.

[0176] In the solution provided by the embodiment of the present invention, the road environment label and the weather condition label are first processed multiple times using a large language model LLM to obtain a field-independent text description and a field-related text description respectively; the learnable text description is inserted using a text encoder to obtain a text feature set independent of weather conditions and a text feature set related to weather conditions; the text feature set independent of weather conditions and the text feature set related to weather conditions are then trained to obtain image features independent of weather conditions and image features related to weather conditions, making the decoupling process easier; the embodiment of the present invention constructs a class-centric memory to obtain a prediction probability related to weather conditions, combines the prediction probability related to weather conditions with the prediction probability independent of weather conditions, and obtains a prediction probability of the overall road environment of the unmanned vehicle, thereby improving the accuracy of the overall road environment prediction of the unmanned vehicle.

[0177] After receiving the predicted probability of the overall road condition environment, the control system of the unmanned vehicle can make corresponding judgments and output control information to change the vehicle's operating state, control the vehicle to perform corresponding driving actions for different scenarios, and realize the unmanned driving function.

[0178] Each embodiment in this specification is described in a related manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the system embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0179] The above description is only a preferred embodiment of the present invention and is not intended to limit the protection scope of the present invention. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention are included in the protection scope of the present invention.

Claims

1. A method for image category prediction based on domain generalization of feature decoupling and class center matching, characterized in that: include: For an image to be processed, a domain generalization dataset corresponding to the image to be processed is obtained; The domain generalization dataset includes: images of each preset category in different domains and labels corresponding to each image, wherein the labels include: category labels and domain labels; The LLM model is used to generate a fine-grained text description corresponding to each category label and a fine-grained text description corresponding to each domain label respectively; the fine-grained text description corresponding to each category label is processed by the LLM model to obtain a domain-independent text description corresponding to each category label; the fine-grained text description corresponding to each domain label is processed by the LLM model to obtain a domain-related text description corresponding to each domain label; a preset text description is inserted into each domain-independent text description and each domain-related text description, and a domain-independent text feature set of all domain-independent text descriptions and a domain-related text feature set of all domain-related text descriptions are obtained through a text encoder; Inserting a preset word vector into each image of the domain generalization dataset, and using the domain-independent text feature set and the domain-related text feature set to respectively train to obtain domain-independent image features corresponding to each image and domain-related image features corresponding to each image; Using all domain-related image features to construct a class center memory; using the class center memory and the domain-related text feature set to obtain domain-related prediction probabilities; using the domain-independent text feature set and all domain-independent image features to match to obtain domain-independent prediction probabilities; performing weighted summation of the domain-related prediction probabilities and the domain-independent prediction probabilities to obtain the prediction probability of the overall category of the image to be processed and output it; The domain generalization dataset includes: The data set D of the source domain S and the target domain data set D T ;in, is the data set D of the source domain S For each source domain, Among them, each data sample (x i ,y i ) is given by the input image x i And the input image x i The corresponding label y i Composition, N S is the total number of source domains, is the total number of all images in the current source domain; is the data set D of the target domain T For each target domain, Among them, each data sample (x i ′,y i ′) is composed of the input image x i ′ and input image x i ′ corresponds to the label y i 'Composition, N t is the total number of target domains, is the total number of all images in the current target domain; The method of using all domain-related image features to construct a class-centric memory includes: According to the data set D of the source domain S , the total number of source domains and the number of category labels under the source domain, and obtain the class center tensor C, in, C m，k is the center tensor of the kth category in the mth source domain in the class center tensor C, d is the feature dimension, M is the total number of category labels in the mth source domain, and N C is the total number of category labels in the data set of the source domain, E S is the domain-dependent visual encoder f S The preset word vectors in ; Representing the class center tensor C as a two-dimensional matrix and caching it as the class center memory; The method of obtaining a domain-related prediction probability by using the class center memory and the domain-related text feature set includes: Select the target domain data set D T The image in is used as the test image x i ′, through the domain-dependent visual encoder f S Extract the test image x i The test image feature f in ′ S (x i ′,E S ); The correlation degree Φ(x) between the test image feature and the class center memory is obtained by the preset formula 1 i ′, C), Wherein, the preset formula 1 is Φ(x i ′, C) = exp(-β(1-f S (x i ′,E S )C T )); β is a hyperparameter for adjusting the sharpness of similarity output, C T is the transpose of C, f S (x i ′,E S )C T is the cosine similarity between the test image feature and the class center tensor C; The predicted probability P related to the field is obtained using the preset formula 2 i S ; Wherein, the preset formula 2 is P i S =Φ(x i ′,C)L C , L C is a collection of tags, 2. The image category prediction processing method based on domain generalization of feature decoupling and class center matching according to claim 1 is characterized in that: The LLM model is used to generate a fine-grained text description corresponding to each category label and a fine-grained text description corresponding to each domain label, including: The designed preset questions are used to guide GPT-3 in the LLM model to generate fine-grained text descriptions corresponding to each category label and each domain label.

3. The image category prediction processing method based on domain generalization of feature decoupling and class center matching according to claim 2 is characterized in that: The LLM model is used to process the fine-grained text description corresponding to each category label to obtain a field-independent text description corresponding to each category label; The LLM model is used to process the fine-grained text description corresponding to each domain label to obtain the domain-related text description corresponding to each domain label, including: The GPT-3 is used to summarize and analyze the shared and common attributes of the fine-grained text description corresponding to each category label to obtain a domain-independent text description corresponding to each category label; The GPT-3 is used to process the fine-grained text description corresponding to each domain label to obtain the domain-related text description corresponding to each domain label.

4. The image category prediction processing method based on domain generalization of feature decoupling and class center matching according to claim 3 is characterized in that: The method inserts the preset text description into each field-independent text description and each field-related text description, and obtains a field-independent text feature set of all field-independent text descriptions and a field-related text feature set of all field-related text descriptions through a text encoder, including: Using a text editor of the CLIP model, the preset text description is inserted into each field-independent text description to obtain a field-independent text feature set to be optimized; The loss function L of the domain-independent text feature set to be optimized is obtained using the L2 distillation loss kg1 ; According to the domain-independent text feature set to be optimized and all images of the domain generalization dataset, the comparative learning target L is obtained. ce1 ; Using a text editor of the CLIP model, the preset text description is inserted into each field-related text description to obtain a field-related text feature set to be optimized; The loss function L of the text feature set related to the field to be optimized is obtained by using L2 distillation loss kg2 ; According to the text feature set related to the field to be optimized and all images of the field generalization dataset, the comparative learning target L is obtained. ce2 ; According to the loss function L kg1 , the comparative learning objective L ce1 , the loss function L kg2 Compared with the above learning objectives L ce2 , and get the loss function L of the entire text text ;in, L text =L ce1 +α1*L ce2 +β1*(L kg1 +L kg2 ), α1 and β1 are pre-set weight parameters; Using the loss function L of the entire text text The domain-independent text feature set to be optimized and the domain-dependent text feature set to be optimized are processed separately to obtain the domain-independent text feature set and the domain-dependent text feature set.

5. The image category prediction processing method based on domain generalization of feature decoupling and class center matching according to claim 4 is characterized in that: The inserting of the preset word vector into each image of the domain generalization dataset, and using the domain-independent text feature set and the domain-related text feature set to respectively train to obtain domain-independent image features corresponding to each image and domain-related image features corresponding to each image, include: Using deep VPT as a domain-independent visual encoder I and domain-dependent visual encoder f I Process each image and extract the image features that are independent of the field to be optimized corresponding to each image Image features related to the domain to be optimized corresponding to each image in, E I is the domain-independent visual encoder f I The preset word vector in E S is the domain-dependent visual encoder f S The preset word vectors in ; According to all the domain-independent image features to be optimized and the set of domain-independent text features, the comparative learning target L′ is obtained ce1 ; Perform domain confusion regularization on all the domain-independent image features to be optimized and the domain-related text feature set to obtain the loss function L mix ; Use L2 distillation loss to obtain all the domain-independent image features to be optimized The loss function L kg ; According to all the domain-related image features to be optimized and the domain-related text feature set, the comparative learning target L′ is obtained ce2 ; According to the comparative learning objective L′ ce1 , the comparative learning objective L′ ce2 , the loss function L kg And the loss function L mix , we get the loss function L of the entire image img ;in, L img =L′ ce1 +α2*L′ ce2 +β2*L kg +β3*L mix , α2, β2 and β3 are pre-set weight parameters; Using the loss function L of the entire image img All the domain-independent image features to be optimized and all the domain-dependent image features to be optimized are processed to obtain the domain-independent image features corresponding to each image and the domain-dependent image features corresponding to each image.

6. The image category prediction processing method based on domain generalization of feature decoupling and class center matching according to claim 5, characterized in that: The method of obtaining a domain-independent prediction probability by matching the domain-independent text feature set with all domain-independent image features includes: Substitute the domain-independent text features and all the domain-independent image features into the preset formula 3 to obtain the domain-independent prediction probability P i I ; Wherein, the preset formula 3 is is the domain-independent image feature, is a set of text features that are independent of the field.

7. The image category prediction processing method based on domain generalization of feature decoupling and class center matching according to claim 6 is characterized in that: The predicted probability related to the field and the predicted probability unrelated to the field are weightedly summed to obtain the predicted probability of the overall category of the image to be processed and output it, including: Substitute the domain-related prediction probability and the domain-independent prediction probability into the preset formula 4 for weighted summation to obtain the prediction probability of the overall category of the image to be processed and output P i And output; wherein the preset formula 4 is P i =P i I +α3*P i S , α3 is the trade-off parameter between the prediction probability related to the field and the prediction probability independent of the field.

Images