Domain generalization method, apparatus, electronic device, and storage medium

Abstract

The application relates to the field of computer vision and provides a domain generalization method, device, electronic equipment and storage medium, wherein the method comprises the following steps: acquiring a source domain sample image and a target domain image; generating domain-related text comprising domain information and image category information based on a text prompt template; extracting visual features of the source domain sample image based on an image encoder and extracting text features of the domain-related text based on a text encoder; applying the visual features to perform image classification on the source domain sample image based on a classifier and applying the text features to perform text classification on the domain-related text; and performing parameter iteration on the image encoder and the classifier based on an image classification loss and a text classification loss, wherein the image encoder and the classifier after the parameter iteration are used for classifying the target domain image. The domain generalization method, device, electronic equipment and storage medium provided by the application effectively improve the generalization of the model.

Description

Technical Field

[0001] This invention relates to the field of computer vision technology, and more particularly to a domain generalization method, apparatus, electronic device, and storage medium. Background Technology

[0002] Image recognition has always been a fundamental and crucial task in the field of computer vision, aiming to enable machines to correctly classify input images. Common deep learning-based image recognition models often experience a significant performance degradation when faced with domain bias, i.e., when the distribution of test data differs greatly from that of training data. Therefore, numerous domain generalization methods have been proposed, attempting to generalize models trained on multiple source domains to unknown target domains.

[0003] Domain generalization research addresses the challenge of learning a strong generalization model from several source domain datasets with different data distributions to achieve good prediction results in an unknown target domain. Since the target domain data is invisible during training, domain generalization presents a highly challenging yet practical application.

[0004] Related domain generalization methods mainly focus on learning generalization from source domain images. However, due to limitations in model size and training data scale, the performance improvement brought by existing methods is often limited. Summary of the Invention

[0005] This invention provides a domain generalization method, apparatus, electronic device, and storage medium to address the limitation of performance improvement in existing domain generalization methods.

[0006] This invention provides a domain generalization method, comprising:

[0007] Obtain source domain sample images and target domain images;

[0008] Based on the text prompt template, generate domain-related text that includes domain information and image category information;

[0009] Based on the image encoder, visual features of the source domain sample images are extracted, and based on the text encoder, text features of the domain-related text are extracted.

[0010] Based on the classifier, the visual features are applied to classify the source domain sample images, and the text features are applied to classify the domain-related text.

[0011] Based on image classification loss and text classification loss, the image encoder and the classifier are iterated with parameters. After parameter iteration, the image encoder and classifier are used to classify the target domain image.

[0012] According to the domain generalization method provided by the present invention, the generation of domain-related text, including domain information and image category information, based on a text prompt template includes:

[0013] Based on a pre-trained vocabulary replacement model, domain-related vocabulary is generated, and category-related vocabulary is generated based on the categories of the source domain sample images and the target domain images;

[0014] The domain-related words and category-related words are filled into the text prompt template to generate domain-related text that includes domain information and image category information.

[0015] According to the domain generalization method provided by the present invention, the step of determining the text prompt template includes:

[0016] Based on the semantic similarity between the visual features and the text features, the semantic alignment loss is determined;

[0017] Based on the similarity between the text features of each category, the text similarity loss is determined;

[0018] Based on the semantic alignment loss and the text similarity loss, the initial text prompt template is iterated to obtain the text prompt template.

[0019] According to the domain generalization method provided by the present invention, determining the semantic alignment loss based on the semantic similarity between the visual features and the text features includes:

[0020] The semantic alignment loss is determined based on the following formula.

[0021]

[0022] In the formula, <·> represents the calculation of cosine similarity; α ni =1 if and only if If it belongs to the i-th class, otherwise α ni =0; N s N represents the number of source domain sample images. c N represents the number of image categories. d Indicates the number of domain-related texts; Represents the visual features of the nth source domain sample image. The text features represent the domain-related text of the i-th image category.

[0023] According to the domain generalization method provided by the present invention, determining the text similarity loss based on the similarity between the text features of each category includes:

[0024] The text similarity loss is determined based on the following formula.

[0025]

[0026] In the formula, The text features representing the domain-related text of the i-th image category are: Let represent the text features of the domain-related text for the i-th image category, specifically the j′-th region.

[0027] According to the domain generalization method provided by the present invention, the step of classifying the source domain sample images by applying the visual features based on the classifier, and classifying the domain-related text by applying the text features, includes:

[0028] The visual features and the text features are normalized respectively to obtain normalized visual features and normalized text features;

[0029] The classification header parameters of each category of the classifier are normalized to obtain the parameter-normalized classifier;

[0030] Based on the classifier with normalized parameters, the normalized visual features are applied to classify the source domain sample images to obtain image classification results; and the normalized text features are applied to classify the domain-related text to obtain text classification results.

[0031] According to the domain generalization method provided by the present invention, the image classification loss is determined based on the image classification result and the label of the source domain sample image; the text classification loss is determined based on the text classification result and the image category information of the domain-related text.

[0032] The present invention also provides a domain generalization device, comprising:

[0033] The image acquisition unit is used to acquire source domain sample images and target domain images;

[0034] The text generation unit is used to generate domain-related text, including domain information and image category information, based on text prompt templates;

[0035] The feature extraction unit is used to extract visual features of the source domain sample image based on the image encoder, and to extract text features of the domain-related text based on the text encoder.

[0036] A classification unit is used to classify the source domain sample images based on the classifier, applying the visual features, and to classify the domain-related texts based on the text features.

[0037] The parameter iteration unit is used to perform parameter iteration on the image encoder and the classifier based on image classification loss and text classification loss. After the parameter iteration is completed, the image encoder and classifier are used to classify the target domain image.

[0038] The present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor, when executing the program, implements the domain generalization method as described above.

[0039] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the domain generalization method as described above.

[0040] The present invention also provides a computer program product, including a computer program that, when executed by a processor, implements the domain generalization method as described above.

[0041] The domain generalization method, apparatus, electronic device, and storage medium provided by this invention generate domain-related text, including domain information and image category information, based on text prompt templates. The domain-related text and source domain sample images are used together as training data to enhance the domain diversity of the training data, thereby effectively improving the generalization of the model in an easy-to-implement manner. Attached Figure Description

[0042] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0043] Figure 1 This is one of the flowcharts illustrating the domain generalization method provided by the present invention;

[0044] Figure 2 This is a schematic diagram of the normalized linear classifier provided by the present invention;

[0045] Figure 3 This is the second flowchart illustrating the domain generalization method provided by the present invention;

[0046] Figure 4 This is the third flowchart illustrating the domain generalization method provided by the present invention;

[0047] Figure 5 This is a schematic diagram of the field generalization device provided by the present invention;

[0048] Figure 6This is a schematic diagram of the structure of the electronic device provided by the present invention. Detailed Implementation

[0049] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0050] Related domain generalization methods mainly focus on learning generalization from source domain images. However, due to limitations in model size and training data scale, the domain diversity of training data in current domain generalization tasks is insufficient, and the performance improvement brought by existing methods is often limited.

[0051] Based on the above considerations, in order to improve the domain generalization performance of the model, the inventive concept of this invention is: based on a large-scale visual language pre-trained model, by introducing additional text information to enhance the domain diversity of the training data, thereby improving the generalization of the model.

[0052] Based on the above-mentioned inventive concept, the present invention provides a domain generalization method, apparatus, electronic device and storage medium, which are applied to image recognition scenarios in artificial intelligence technology to improve the generalization of image recognition models.

[0053] The technical solution of the present invention will now be described in detail with reference to the accompanying drawings. Figure 1 This is one of the flowcharts illustrating the domain generalization method provided by this invention. The execution entity for each step in this method can be a domain generalization device, which can be implemented through software and / or hardware. This device can be integrated into an electronic device, which can be a terminal device (such as a smartphone, personal computer, etc.), a server (such as a local server or cloud server, or a server cluster, etc.), a processor, or a chip, etc. Figure 1 As shown, the method may include the following steps:

[0054] Step 110: Obtain the source domain sample image and the target domain image.

[0055] Specifically, the source domain sample image is the image used as training data, and the target domain image is the image used for testing or actual operation scenarios. The source domain may contain one or more domains, and the target domain may also contain one or more domains; this embodiment of the invention does not specifically limit this.

[0056] Taking the commonly used PACS dataset containing 9991 images as an example, which includes four different domains P, A, C, and S, we can consider domains P, A, and C as the source domains acting on the training model, and the data corresponding to domains P, A, and C are the source domain sample images; we can consider domain S as the target domain acting on the testing model, and the data corresponding to domain S are the target domain images.

[0057] Step 120: Based on the text prompt template, generate domain-related text including domain information and image category information.

[0058] The text prompt template is used to generate domain-related text. The text prompt template may include domain terms and image category terms. Domain terms are used to represent domain information, and image category terms are used to represent image category information.

[0059] For example, a text prompt template can be displayed in the form of "[v1][domain][v2][category]", where domain is used to represent domain information, category is used to represent image category information, and v1 and v2 are the parameters of the text prompt template, respectively.

[0060] Considering that related technologies mainly focus on learning generalization from source domain images, resulting in insufficient domain diversity in training data and limited improvement in model generalization, this invention introduces additional textual information, namely, domain-related text, as training data to enhance the domain diversity of the training data and thus improve the model's generalization.

[0061] Furthermore, to enhance the domain diversity of the generated domain-related text, the text prompt template can be trained. During training, parameters of v1 and v2 are iterated. After training, the text prompt template can generate more diverse domain-related text.

[0062] Step 130: Based on the image encoder, extract the visual features of the source domain sample images, and based on the text encoder, extract the text features of the domain-related text.

[0063] Specifically, an image encoder is used to extract visual features from source domain sample images. Image encoders include, for example, ResNet or other pre-trained visual neural network models (CLIP Image Encoder) capable of extracting visual features.

[0064] Text encoders are used to extract text features from domain-relevant text. Text encoders include, for example, Transformer networks or other pre-trained language neural network models (CLIP TextEncoder) capable of extracting text features.

[0065] For example, by inputting a source domain sample image into an image encoder, the visual features z output by the image encoder are obtained. img The domain-related text is input into the text encoder to obtain the text features z output by the text encoder. txt .

[0066] It should be noted that before extracting the visual features of the source domain sample images, data augmentation can be performed on the source domain sample images. Data augmentation can include image cropping, color dithering, horizontal flipping, grayscale transformation, etc.

[0067] Step 140: Based on the classifier, apply visual features to classify the source domain sample images, and apply text features to classify the domain-related text.

[0068] Specifically, the classifier is used to classify images and text. The classifier may include, for example, a multi-layer or single-layer MLP (Multi-Layer Perceptron), or other networks capable of classification. The classifier can be a traditional linear classifier or a normalized linear classifier; this embodiment of the invention does not specifically limit its application.

[0069] Visual features are input into a classifier, which classifies source domain sample images to obtain the image classification result. Text features are input into a classifier, which classifies domain-related text to obtain the text classification result.

[0070] Step 150: Based on image classification loss and text classification loss, perform parameter iteration on the image encoder and classifier. After parameter iteration, the image encoder and classifier are used to classify the target domain image.

[0071] Image classification loss characterizes the difference between the classification result of a source domain sample image and its label, while text classification loss characterizes the difference between the classification result of domain-related text and its label. Image classification loss and text classification loss can be, for example, employ cross-entropy loss.

[0072] The image classification loss and text classification loss are added together to obtain the final model loss. Based on this, the model loss L is minimized. ce The image encoder and classifier are iterated through parameters. The image encoder and classifier after parameter iteration are used to classify target domain images, and have better generalization in target domain image classification.

[0073] For example, the model loss can be expressed as L ce =L img +L txt , where L img L represents the image classification loss. txt This represents the text classification loss.

[0074] The method provided in this invention generates domain-related text, including domain information and image category information, based on a text prompt template. The domain-related text and source domain sample images are used together as training data to enhance the domain diversity of the training data, thereby effectively improving the generalization of the model in an easy-to-implement manner.

[0075] Based on the above embodiments, based on the text prompt template, domain-related text including domain information and image category information is generated, that is, step 120 specifically includes:

[0076] Step 121: Generate domain-related words based on the pre-trained word replacement model, and generate category-related words based on the categories of the source domain sample images and the target domain images;

[0077] Step 122: Fill the text prompt template with domain-related words and category-related words to generate domain-related text that includes domain information and image category information.

[0078] Specifically, domain-related vocabulary can be words related to the image domain and can be generated by a pre-trained vocabulary replacement model. For example, if the initial sentence is "An image of a dog", this initial sentence is input into the pre-trained vocabulary replacement model, and "image" is set as the word to be replaced, and the number of generated words is set to 20, the vocabulary replacement model will output 20 domain-related words, such as photo, painting, picture, portrait, etc.

[0079] Category-related vocabulary refers to the image categories in a dataset, which can be derived based on the categories of source domain sample images and target domain images. For example, category-related vocabulary may include dog, elephant, horse, person, etc.

[0080] After obtaining domain-related and category-related vocabulary, the domain-related and category-related vocabulary can be populated into the text prompt template for domain-related text.

[0081] For example, the text prompt template is "[v1][domain][v2][category]", where "domain" represents domain information and can be filled with domain-related words generated in step 121; "category" represents image category information and can be filled with category-related words. Assuming there are 20 domain-related words and 7 category-related words, the number of generated domain-related texts will be 20 * 7 = 140.

[0082] The method provided in this invention automatically generates various domain-related words through a pre-trained vocabulary replacement model. The generated domain-related words have good diversity and contain rich domain information. Based on this, the generated domain-related text can be used to improve the performance of the model in unknown domains.

[0083] Based on the above embodiments, the steps for determining the text prompt template include:

[0084] Based on the semantic similarity between visual features and text features, determine the semantic alignment loss;

[0085] The text similarity loss is determined based on the similarity between text features of each category;

[0086] Based on semantic alignment loss and text similarity loss, the initial text prompt template is iterated to obtain the text prompt template.

[0087] Specifically, in order to further improve the domain diversity of the generated text, the initial text prompt template can be iterated based on semantic alignment loss and text similarity loss. The text prompt template obtained after parameter iteration can generate diverse texts.

[0088] Semantic alignment loss is determined based on the semantic similarity between visual and textual features. It calculates the semantic similarity between the visual features of a source domain sample image of any category and the textual features of related text belonging to that category. Higher semantic similarity results in lower semantic alignment loss, and vice versa. The semantic alignment loss ensures that category information in the textual features is not destroyed during training.

[0089] The similarity here can be obtained using common methods such as cosine similarity, Euclidean distance, and Pearson correlation coefficient.

[0090] In some embodiments, the semantic alignment loss can be determined based on the following formula:

[0091]

[0092] In the formula, <·> represents the calculation of cosine similarity; α ni =1 if and only if If it belongs to the i-th class, otherwise α ni =0; N s N represents the number of source domain sample images. c N represents the number of image categories. d Indicates the number of domain-related texts; Represents the visual features of the nth source domain sample image. The text features represent the domain-related text of the i-th image category.

[0093] Text similarity loss is determined based on the similarity between text features across different categories. For domain-related texts within each category, the similarity between the text features of each domain-related text is calculated. Higher similarity results in lower text similarity loss; conversely, lower similarity results in higher text similarity loss. Text similarity loss aims to maximize the difference between text features containing domain information. The similarity between text features across different categories can be calculated using common methods such as cosine similarity, Euclidean distance, and Pearson correlation coefficient.

[0094] In some embodiments, the text similarity loss is determined based on the similarity between text features of each category, including: determining the text similarity loss based on the following formula:

[0095]

[0096] In the formula, The text features representing the domain-related text of the i-th image category are: Let represent the text features of the domain-related text for the i-th image category, specifically the j′-th region.

[0097] In some embodiments, based on a classifier, visual features are applied to classify source domain sample images, and text features are applied to classify domain-related text. Specifically, step 140 includes:

[0098] Step 141: Normalize the visual features and text features respectively to obtain normalized visual features and normalized text features;

[0099] Step 142: Normalize the classification head parameters of each category of the classifier to obtain the parameter-normalized classifier;

[0100] Step 143: Based on the classifier with normalized parameters, apply normalized visual features to classify the source domain sample images to obtain image classification results; and apply normalized text features to classify the domain-related text to obtain text classification results.

[0101] Specifically, the embodiments of the present invention employ a normalized linear classifier, which is more suitable for visual neural network models based on contrastive learning pre-training than traditional linear classifiers.

[0102] The input visual and textual features are first normalized into unit vectors: This yields normalized visual features and normalized text features.

[0103] Figure 2 This is a schematic diagram of the normalized linear classifier provided by the present invention, as shown below. Figure 2As shown, a linear classifier can be represented by G, and g can be used to represent it. i Let represent the parameters of the i-th classifier head corresponding to the i-th category. Similar to the input features, the parameters of each classifier head are also normalized to a unit vector: Then a normalized classifier is constructed.

[0104] Then, the normalized visual features z img and normalized text features z txt Input the normalized linear classifier respectively To calculate the classification result, image classification result and text classification results

[0105] Based on the above embodiments, the image classification loss is determined based on the image classification result and the label of the source domain sample image; the text classification loss is determined based on the text classification result and the image category information of the domain-related text.

[0106] Based on the above embodiments, Figure 3 This is the second flowchart illustrating the domain generalization method provided by the present invention. Figure 4 This is the third flowchart of the domain generalization method provided by the present invention. As shown in the figure, the method includes:

[0107] Step S0: Obtain source domain sample images. Randomly sample 32 RGB color images from each of the P, A, and C domains in the dataset, resulting in a total of 96 images. These images are then used to construct a batch of three-channel input data. Data augmentation operations such as cropping, color dithering, horizontal flipping, and grayscale transformation are then applied to the input data. Finally, the data is scaled to a uniform size, resulting in a final data dimension of 96×3×224×224.

[0108] Step S1: Based on the text prompt template, generate domain-related text including domain information and image category information. Input the initial sentence "An image of a dog" into the pre-trained word replacement model, set "image" as the word to be replaced, and set the generation quantity to 20, resulting in 20 domain-related words. Fill the text prompt template with the domain-related words and 7 category words from the dataset to generate 20 × 7 = 140 domain-related texts.

[0109] Step S2: Based on the image encoder, extract the visual features of the source domain sample images. The enhanced data obtained in step S0 is then fed into a pre-trained visual neural network model to obtain 96 visual features z with a dimension of 512. img ;

[0110] Based on the text encoder, text features of the domain-related text are extracted, and the text obtained in step S1 is input into the pre-trained language neural network model to obtain 140 text features z with a dimension of 512. txt .

[0111] Step S3: Calculate the semantic alignment loss L based on visual features and text features. a To ensure that the category information in the text features is not destroyed during training; then, the text similarity loss L is calculated based on different text features. s To maximize the differences between text features containing information from different domains. For L a and L s The text prompt loss is obtained by weighted summation: L p =L a +λL s , where λ is a weighting parameter, which can be set to 0.3.

[0112] Step S4, update the text prompt template to minimize L p .

[0113] Step S5, extract image features z im and text features z txt Input the normalized linear classifier respectively To calculate the normalized classification results with 96 dimensions and 7 elements. as well as

[0114] Step S6: Calculate the cross-entropy loss for both the image and text based on the classification results from step S5.

[0115] Image classification loss

[0116] In the formula, N s This indicates the number of source domain sample images. This represents the category label of the nth source domain sample image. This represents the normalized classification result of the nth source domain sample image.

[0117] Text classification loss

[0118] In the formula, N c N represents the number of image categories. d Indicates the number of domain-related texts. The image category representing domain-related text is class i. This represents the normalized classification result of the domain-related text for the i-th image category.

[0119] The two losses are then added together to obtain the final cross-entropy loss L.ce =L img +L txt .

[0120] Step S7, update the normalized linear classifier and visual neural network model to minimize L ce .

[0121] Step S8: The updated visual neural network model and normalized classifier can better classify the data in domain S. Repeat this training process 3000 times until the model converges, resulting in the final trained model.

[0122] The method provided in this embodiment of the invention designs a domain-related vocabulary generation method, which automatically generates a certain number of words related to the image domain based on a pre-trained vocabulary replacement model, thereby expanding the diverse descriptions of different domains.

[0123] A text feature generation method based on cue learning was designed. The generated domain-related vocabulary was used to further generate domain-related text. The domain information in the text was mapped to the common feature space of text features and image features through a text encoder. During this process, text cue templates were trained to make the text features have richer domain information.

[0124] Finally, a specially designed normalized classifier is trained using features from the input image and the generated text. This classifier exhibits better generalization ability in unknown target domains, while the image encoder is updated based on the gradients backpropagated from the classifier. This method effectively utilizes the generated text information and achieves superior performance on domain generalization tasks in an easily implementable manner.

[0125] The domain generalization apparatus provided by the present invention is described below. The domain generalization apparatus described below can be referred to in correspondence with the domain generalization method described above.

[0126] Figure 5 This is a schematic diagram of the field generalization device provided by the present invention, as shown below. Figure 5 As shown, the domain generalization device includes:

[0127] Image acquisition unit 510 is used to acquire source domain sample images and target domain images;

[0128] The text generation unit 520 is used to generate domain-related text, including domain information and image category information, based on a text prompt template.

[0129] The feature extraction unit 530 is used to extract visual features of the source domain sample image based on the image encoder, and to extract text features of the domain-related text based on the text encoder.

[0130] The classification unit 540 is used to perform image classification on the source domain sample image based on the classifier and the visual features, and to perform text classification on the domain-related text based on the text features.

[0131] The parameter iteration unit 550 is used to perform parameter iteration on the image encoder and the classifier based on image classification loss and text classification loss. After the parameter iteration is completed, the image encoder and classifier are used to classify the target domain image.

[0132] The domain generalization apparatus provided in this embodiment of the invention generates domain-related text, including domain information and image category information, based on a text prompt template. The domain-related text and source domain sample images are used together as training data to enhance the domain diversity of the training data, thereby effectively improving the generalization of the model in an easy-to-implement manner.

[0133] Based on the above embodiments, the text generation unit 520 is specifically used for:

[0134] Based on a pre-trained vocabulary replacement model, domain-related vocabulary is generated, and category-related vocabulary is generated based on the categories of the source domain sample images and the target domain images;

[0135] The domain-related words and category-related words are filled into the text prompt template to generate domain-related text that includes domain information and image category information.

[0136] Based on the above embodiments, a template determining unit is further included, used for:

[0137] Based on the semantic similarity between the visual features and the text features, the semantic alignment loss is determined;

[0138] Based on the similarity between the text features of each category, the text similarity loss is determined;

[0139] Based on the semantic alignment loss and the text similarity loss, the initial text prompt template is iterated to obtain the text prompt template.

[0140] Based on the above embodiments, the template determining unit is specifically used for:

[0141] The semantic alignment loss is determined based on the following formula.

[0142]

[0143] In the formula, <·> represents the calculation of cosine similarity; α ni =1 if and only if If it belongs to the i-th class, otherwise α ni =0; N sN represents the number of source domain sample images. c N represents the number of image categories. d Indicates the number of domain-related texts; Represents the visual features of the nth source domain sample image. The text features represent the domain-related text of the i-th image category.

[0144] Based on the above embodiments, the template determining unit is specifically used for:

[0145] The text similarity loss is determined based on the following formula.

[0146]

[0147] In the formula, The text features representing the domain-related text of the i-th image category are: Let represent the text features of the domain-related text for the i-th image category, specifically the j′-th region.

[0148] Based on the above embodiments, the classification unit 540 is specifically used for:

[0149] The visual features and the text features are normalized respectively to obtain normalized visual features and normalized text features;

[0150] The classification header parameters of each category of the classifier are normalized to obtain the parameter-normalized classifier;

[0151] Based on the classifier with normalized parameters, the normalized visual features are applied to classify the source domain sample images to obtain image classification results; and the normalized text features are applied to classify the domain-related text to obtain text classification results.

[0152] Based on the above embodiments, the image classification loss is determined based on the image classification result and the label of the source domain sample image; the text classification loss is determined based on the text classification result and the image category information of the domain-related text.

[0153] Figure 6 An example is a schematic diagram of the physical structure of an electronic device, such as... Figure 6 As shown, the electronic device may include: a processor 610, a communications interface 620, a memory 630, and a communication bus 640, wherein the processor 610, the communications interface 620, and the memory 630 communicate with each other via the communication bus 640. The processor 610 can call logical instructions in the memory 630 to execute a domain generalization method, which includes:

[0154] Obtain source domain sample images and target domain images;

[0155] Based on the text prompt template, generate domain-related text that includes domain information and image category information;

[0156] Based on the image encoder, visual features of the source domain sample images are extracted, and based on the text encoder, text features of the domain-related text are extracted.

[0157] Based on the classifier, the visual features are applied to classify the source domain sample images, and the text features are applied to classify the domain-related text.

[0158] Based on image classification loss and text classification loss, the image encoder and the classifier are iterated with parameters. After parameter iteration, the image encoder and classifier are used to classify the target domain image.

[0159] Furthermore, the logical instructions in the aforementioned memory 630 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, essentially, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0160] On the other hand, the present invention also provides a computer program product, the computer program product comprising a computer program that can be stored on a non-transitory computer-readable storage medium, wherein when the computer program is executed by a processor, the computer is capable of executing the domain generalization method provided by the above methods, the method comprising:

[0161] Obtain source domain sample images and target domain images;

[0162] Based on the text prompt template, generate domain-related text that includes domain information and image category information;

[0163] Based on the image encoder, visual features of the source domain sample images are extracted, and based on the text encoder, text features of the domain-related text are extracted.

[0164] Based on the classifier, the visual features are applied to classify the source domain sample images, and the text features are applied to classify the domain-related text.

[0165] Based on image classification loss and text classification loss, the image encoder and the classifier are iterated with parameters. After parameter iteration, the image encoder and classifier are used to classify the target domain image.

[0166] In another aspect, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, is implemented to perform the domain generalization methods provided by the methods described above, the method comprising:

[0167] Obtain source domain sample images and target domain images;

[0168] Based on the text prompt template, generate domain-related text that includes domain information and image category information;

[0169] Based on the image encoder, visual features of the source domain sample images are extracted, and based on the text encoder, text features of the domain-related text are extracted.

[0170] Based on the classifier, the visual features are applied to classify the source domain sample images, and the text features are applied to classify the domain-related text.

[0171] Based on image classification loss and text classification loss, the image encoder and the classifier are iterated with parameters. After parameter iteration, the image encoder and classifier are used to classify the target domain image.

[0172] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0173] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0174] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A domain generalization method, characterized in that, include: Obtain source domain sample images and target domain images; Based on the text prompt template, generate domain-related text that includes domain information and image category information; Based on the image encoder, visual features of the source domain sample images are extracted, and based on the text encoder, text features of the domain-related text are extracted. Based on the classifier, the visual features are applied to classify the source domain sample images, and the text features are applied to classify the domain-related text. Based on image classification loss and text classification loss, the image encoder and the classifier are iterated with parameters. After parameter iteration, the image encoder and classifier are used to classify the target domain image. The steps for determining the text prompt template include: Based on the semantic similarity between the visual features and the text features, the semantic alignment loss is determined; Based on the similarity between the text features of each category, the text similarity loss is determined; Based on the semantic alignment loss and the text similarity loss, the initial text prompt template is iterated to obtain the text prompt template. The determination of semantic alignment loss based on the semantic similarity between the visual features and the text features includes: The semantic alignment loss is determined based on the following formula. In the formula, <·> represents the calculation of cosine similarity; α ni =1 if and only if If it belongs to the i-th class, otherwise α ni =0; N s N represents the number of source domain sample images. c N represents the number of image categories. d This indicates the number of domain-related texts; Represents the visual features of the nth source domain sample image. Text features representing the domain-related text of the i-th image category; The determination of text similarity loss based on the similarity between text features of each category includes: The text similarity loss is determined based on the following formula. In the formula, The text features representing the domain-related text of the i-th image category are: Let N represent the text features of the j′-th domain-related text for the i-th image category. c N represents the number of image categories. d Indicates the number of domain-related texts.

2. The domain generalization method according to claim 1, characterized in that, The process of generating domain-related text based on text prompt templates, including domain information and image category information, includes: Based on a pre-trained vocabulary replacement model, domain-related vocabulary is generated, and category-related vocabulary is generated based on the categories of the source domain sample images and the target domain images; The domain-related words and category-related words are filled into the text prompt template to generate domain-related text that includes domain information and image category information.

3. The domain generalization method according to claim 1 or 2, characterized in that, The step of classifying source domain sample images using the visual features based on a classifier and classifying domain-related text using the text features includes: The visual features and the text features are normalized respectively to obtain normalized visual features and normalized text features; The classification header parameters of each category of the classifier are normalized to obtain the parameter-normalized classifier; Based on the classifier with normalized parameters, the normalized visual features are applied to classify the source domain sample images to obtain image classification results; and the normalized text features are applied to classify the domain-related text to obtain text classification results.

4. The domain generalization method according to claim 3, characterized in that, The image classification loss is determined based on the image classification result and the label of the source domain sample image; the text classification loss is determined based on the text classification result and the image category information of the domain-related text.

5. A domain generalization apparatus applying the domain generalization method as described in any one of claims 1 to 4, characterized in that, include: The image acquisition unit is used to acquire source domain sample images and target domain images; The text generation unit is used to generate domain-related text, including domain information and image category information, based on text prompt templates; The feature extraction unit is used to extract visual features of the source domain sample image based on the image encoder, and to extract text features of the domain-related text based on the text encoder. A classification unit is used to classify the source domain sample images based on the classifier, applying the visual features, and to classify the domain-related texts based on the text features. The parameter iteration unit is used to perform parameter iteration on the image encoder and the classifier based on image classification loss and text classification loss. After the parameter iteration is completed, the image encoder and classifier are used to classify the target domain image.

6. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the domain generalization method as described in any one of claims 1 to 4.

7. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the domain generalization method as described in any one of claims 1 to 4.

Images