Domain generalization method and system based on CLIP model semantic information extraction and target alignment
By aligning the model features in the training and testing stages, the problem of image features is solved, the domain generalization performance is significantly improved, and good results are achieved in practice.
Patent Information
- Application Number
- CN202411202598.2
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-29
- Publication Date
- 2024-12-06
AI Technical Summary
In the prior art, image features are not aligned in the model training stage and testing stage, resulting in information loss, and most domain generalization methods only focus on the alignment of the target domain and ignore the overall model representation improvement of the source domain.
Using semantic information extraction and target alignment methods based on CLIP model, dynamic weighted training is performed through similarity loss, semantic loss and spatial consistency loss of student model and teacher model, optimize the feature encoder to obtain semantic information, and use it in the test stage. The memory library of the target domain statistical distribution updates the classifier to achieve target alignment.
The performance of the model in any target domain is significantly improved, valuable information in large visual language models is retained, and combined with other domain generalization methods, demonstrating the superiority and universality of this method.
Smart Images

Figure CN119091214A_ABST
Abstract
Description
Technical Field
[0001] The present invention belongs to the technical field of domain generalization, and in particular relates to a domain generalization method and system based on CLIP model semantic information extraction and target alignment. Background Art
[0002] Machine learning has been widely studied and has achieved excellent results in many fields. However, most machine learning models are trained based on the iid (independent and identically distributed) assumption, which means that the training data and test data are independent and identically distributed. This assumption leads to a significant drop in model performance when the model is directly applied to data with a different statistical distribution from the training data. Out-of-Distribution Detection (OOD) scenarios are very common in the real world. This difference in the distribution of training data (source domain) and test data (target domain) is called domain shift. In order to solve the problem of domain shift, many research works have considered domain generalization (DG).
[0003] The goal of domain generalization techniques is to train a model using one or more source domains so that it can also perform well when tested using an unseen target domain. Since it is impractical to collect the statistical distributions of all domains, DG involves using data manipulation or representation learning techniques to help the model learn domain-invariant features that are consistent across different domains. These features contain semantic information about the image that remains constant regardless of the statistical distribution of the domain. At the same time, domain-specific features such as background, style, and viewpoint, which convey information about the statistical distribution of the image, are ignored. However, although these methods aim to minimize the influence of domain-related information, they still have limitations in generalization performance as they have difficulty distinguishing between domain-invariant and domain-specific features. Thanks to the development of multimodal models, large visual-language models pre-trained on a wide range of datasets have been able to learn from a wide variety of data distributions, resulting in improved feature representations and enhanced generalization capabilities.
[0004] Contrast language-image pre-training (CLIP) overcomes the barrier between vision and language, using both image and text inputs, and extracting text features that are universal to the corresponding images, regardless of the distribution of the domain in which the images are located. However, the training data used for large models is different from that used for DG, resulting in feature misalignment during model training, and directly fine-tuning CLIP for downstream tasks may impair its generalization ability. At the same time, domain drift is also a serious problem, and fine-tuning to the target domain will lead to a large loss of pre-training information. In addition to training models with strong generalization ability, it is also necessary to align features with the target domain. Test-Time Adaptation (TTA) is a method that slightly adjusts the model to the target domain during the test phase. UniDG updates the model for the target domain during the test phase, ensuring that the adapted model is not too different from the pre-trained model and produces more discriminative features in the target domain.
[0005] However, most TTA methods only focus on the alignment with the target domain without considering the overall improvement of the model representation from the perspective of the source domain. Summary of the invention
[0006] In order to solve the problems existing in the prior art, the present invention provides a domain generalization method based on CLIP model semantic information extraction and target alignment, which is used to solve the technical problem of information loss caused by misalignment of image features in the model training stage and the test stage in the prior art.
[0007] In order to achieve the above object, in a first aspect, the present invention provides a domain generalization method based on CLIP model semantic information extraction and target alignment, comprising the following steps:
[0008] Obtain a detection image and a text description corresponding to the detection image, input the detection image into a preset student model, and input the detection image and the text description corresponding to the detection image into a teacher model, where the teacher model is based on a trained CLIP model;
[0009] Based on the student model, its feature encoder extracts features from the detection image to obtain feature representation of the image; based on the teacher model, the text encoder extracts text features from the text description to obtain text feature representation; and the corresponding classification predictions are obtained through the pre-trained CLIP classifiers of the student model and the teacher model respectively;
[0010] Based on the classification prediction of the two models, the classification prediction similarity loss is obtained by minimizing the similarity between the classification predictions;
[0011] Based on the feature representation of the image and the feature representation of the text, obtaining a semantic loss by minimizing a difference between the feature representation of the image and the feature representation of the text;
[0012] Utilizing the relative position of source domain text feature representation to optimize the semantic loss;
[0013] Based on the optimized semantic loss and the feature representation of the image and the text feature representation, modify the general text input of CLIP, derive the domain-specific text feature representation, minimize the distance between the domain-specific text feature representation and the feature representation of the image and the text feature representation, respectively, to obtain the spatial consistency loss;
[0014] Based on the prediction similarity loss, semantic loss and spatial consistency loss, dynamic weighting is used to obtain the final training loss, and training is performed to obtain a feature encoder with semantic information extraction capability in the student model;
[0015] In the test phase, the classifier is updated using the memory library that introduces the statistical distribution of the target domain to promote the alignment of the target domain and obtain the final classifier with target alignment;
[0016] Based on the feature encoder extracted from the semantic information and the target aligned classifier, a classification result of the detection image is obtained.
[0017] Furthermore, the student model is represented as F(·; θ), which includes an encoder f(·) and a classifier q(·) for feature extraction. The ultimate goal of the domain generalization problem is summarized as an empirical risk minimization problem, which is calculated as follows:
[0018]
[0019] Among them, x represents the data, and y represents the label corresponding to x.
[0020] Furthermore, KL divergence is used to measure the similarity between the predictions of the teacher and the student during training. The similarity loss between the predictions of the teacher model and the student model during the training phase is calculated as follows:
[0021]
[0022] Among them, F l (x;θ) and Represents the logits output value of the training process, l KL represents the KL divergence.
[0023] Furthermore, the semantic loss is used to ensure that the feature representation extracted by the student is consistent with the text feature representation extracted by the teacher's text encoder, and to minimize the difference between the feature representation extracted by the student model and the text feature representation extracted by the teacher model to train the student's encoder. The calculation formula is:
[0024]
[0025] Where e t (c) represents the text feature representation extracted by the text encoder of the teacher model CLIP, f(x) represents the feature representation extracted by the feature encoder of the student model, k a (·,·) indicates the distance between the two;
[0026] The spatial consistency loss is calculated as follows:
[0027]
[0028] Among them, eD i (c) represents the domain-specific text feature representation derived from the general text, and k1(·, ·) and k2(·, ·) represent the distance between the two.
[0029] Furthermore, the final training loss is obtained by dynamically weighting all the above losses in the training stage, and its calculation formula is:
[0030] l train =λ1l0+λ2l distill +λ3(l ad +l rd )
[0031] Among them, l0 is the empirical risk minimization loss, l distill is the prediction similarity loss during the training phase of the teacher and student models, l ad is the semantic loss mentioned above, l rd is the above-mentioned spatial consistency loss, and λ1, λ2, and λ3 are weighting coefficients.
[0032] Furthermore, in the testing phase, the classifier is updated using the memory bank that introduces the statistical distribution of the target domain to align the target domain. The final classifier with target alignment is obtained, including:
[0033] Use a linear layer as a classifier, select the K highest confidence samples for each class c, and create a differentiable loss function v c , the formula is:
[0034]
[0035] When using the trained encoder and linear classifier, the probability of samples belonging to category C is calculated and stored in the memory bank, and the probability of samples belonging to category C is iteratively updated. The formula is expressed as:
[0036]
[0037] Where ωc represents the weight associated with category c,
[0038] The sample probability of each class is used to update the weight of the classifier in the corresponding dimension, and the entropy on the target domain is used to guide the parameter update:
[0039]
[0040] In a second aspect, the present invention further provides a domain generalization system for semantic information extraction and target alignment based on a CLIP model, comprising a student model, a teacher model based on CLIP, a classification prediction module, a classification prediction similarity loss acquisition module, a semantic loss optimization module, a spatial consistency loss acquisition module, a feature encoder training module, a classifier training module, and a classification module;
[0041] The classification prediction module obtains the image features and classification prediction results of the detection image based on the student model; obtains the image features of the detection image and the text features of the corresponding text description and the final classification prediction results according to the teacher model, and the teacher model is based on the CLIP model; obtains the corresponding classification prediction through the pre-trained CLIP classifier of the student model and the teacher model;
[0042] The classification prediction similarity loss acquisition module obtains the classification prediction similarity loss by minimizing the similarity between the classification predictions based on the classification predictions of the two models;
[0043] The semantic loss acquisition module is used to acquire the semantic loss by minimizing the difference between the feature representation of the image and the feature representation of the text based on the feature representation of the image and the feature representation of the text;
[0044] The semantic loss optimization module optimizes the semantic loss by using the relative position of the source domain text feature representation;
[0045] The spatial consistency loss acquisition module modifies the general text input of CLIP based on the optimized semantic loss and the feature representation of the image and the text feature representation, derives the domain-specific text feature representation, minimizes the distance between the domain-specific text feature representation and the feature representation of the image and the text feature representation, and obtains the spatial consistency loss;
[0046] The feature encoder training module uses dynamic weighting to obtain the final training loss based on the predicted similarity loss, semantic loss and spatial consistency loss, and performs training to obtain a feature encoder with semantic information extraction capability in the student model;
[0047] In the test phase, the classifier training module uses the memory bank that introduces the statistical distribution of the target domain to update the classifier so that the target domain is aligned and the final classifier with target alignment is obtained;
[0048] The classification module obtains the classification result of the detection image based on the feature encoder extracted by the semantic information and the target aligned classifier.
[0049] In a third aspect, the present invention may also provide a computer device, comprising a processor and a memory, wherein the memory is used to store a computer executable program, the processor reads the computer executable program from the memory and executes it, and when the processor executes the computer executable program, the domain generalization method based on CLIP model semantic information extraction and target alignment of the present invention can be implemented.
[0050] At the same time, a computer-readable storage medium is provided, in which a computer program is stored. When the computer program is executed by a processor, the domain generalization method based on CLIP model semantic information extraction and target alignment described in the present invention can be implemented.
[0051] Compared with the prior art, the present invention has at least the following beneficial effects: the method of the present invention is different from the existing domain generalization method, which only focuses on the domain shift during training and the alignment with the target domain during testing. Instead, it starts from the source domain and proposes semantic information extraction and target alignment (SIETA), which optimizes the generalization of the model by aligning the features of the model during the training and testing stages, making the model more suitable for domain generalization tasks while retaining the valuable information obtained from the large visual language model. This method can significantly improve the performance in any target domain. In addition, the method proposed by the present invention can be combined with other domain generalization methods. These results show the superiority and versatility of the method, which is more suitable for promotion and application in practice. BRIEF DESCRIPTION OF THE DRAWINGS
[0052] Figure 1 A schematic diagram of the flow of the domain generalization method based on CLIP model semantic information extraction and target alignment provided by the present invention;
[0053] Figure 2 A schematic diagram of the structure of an embodiment of a domain generalization system based on CLIP model semantic information extraction and target alignment provided by the present invention. DETAILED DESCRIPTION
[0054] The preferred embodiments of the present invention are described in detail below in conjunction with the accompanying drawings, which constitute a part of this application and are used together with the embodiments of the present invention to illustrate the principles of the present invention, but are not used to limit the scope of the present invention.
[0055] The present invention proposes a domain generalization method based on semantic information extraction and target alignment (SIETA) of the CLIP model, which optimizes the generalization of the model by aligning the features of the model in the training and testing stages. In the training stage, a semantic information extraction (SIE) method is proposed, and the pre-trained CLIP model is used as a teacher model to guide the training of the student model. The prior knowledge of the pre-trained CLIP model is transferred to the model of the present invention using knowledge distillation. In addition, the image features extracted by the model of the present invention are ensured to be closely aligned with the text features extracted by CLIP by limiting the distance. The model is enabled to learn domain-invariant semantic information. In the testing stage, target alignment (TA) is proposed, and the encoder obtained from training is used to prevent catastrophic forgetting of previously learned information during fine-tuning. A memory bank is used to update the classifier, making the model more suitable for pre-generalization tasks while retaining valuable information obtained from a large visual language model. This method can significantly improve performance in any target domain.
[0056] The present invention provides a domain generalization method, system and computer-readable storage medium based on CLIP model semantic information extraction and target alignment, which are described below respectively.
[0057] Combination Figure 1 As shown, a specific embodiment of the present invention discloses a domain generalization method based on CLIP model semantic information extraction and target alignment, comprising the following steps:
[0058] Step S101, obtaining a detection image and a text description corresponding to the detection image, inputting the detection image into a preset student model, and simultaneously inputting the detection image and the text description corresponding to the detection image into a pre-trained CLIP as a teacher model;
[0059] Step S102: Based on the student model, its feature encoder extracts features from the detection image to obtain feature representation of the image; at the same time, the text encoder based on the teacher model detects the text description corresponding to the image and extracts text features to obtain text features; at the same time, the corresponding classification predictions are obtained through the pre-trained CLIP classifiers of the student model and the teacher model respectively;
[0060] Step S103, based on the classification predictions of the two models, obtaining the classification prediction similarity loss by minimizing the similarity between the classification predictions;
[0061] Step S104, obtaining semantic loss by minimizing the difference between the feature representation of the image and the feature representation of the text based on the feature representation of the image and the feature representation of the text;
[0062] Step S105: Based on the semantic loss, the present invention optimizes the loss by using the relative position of the source domain text feature representation;
[0063] Step S106, based on the semantic loss optimization and based on the image features and the corresponding text features, modify the general text input of CLIP to derive the domain-specific text feature representation, and then minimize the distance between the domain-specific text feature representation and the image features and the corresponding text feature representation, respectively, to obtain a spatial consistency loss;
[0064] Step S107: Based on the prediction similarity loss, semantic loss and spatial consistency loss, a final training loss is obtained by dynamic weighting, so as to train a feature encoder with semantic information extraction capability in the student model;
[0065] Step S108: In the testing phase, the classifier is updated using the memory library that introduces the statistical distribution of the target domain to promote the alignment of the target domain and obtain a final classifier with target alignment;
[0066] Step S109: obtaining a classification result of the detection image based on the feature encoder extracted from the semantic information and the target aligned classifier.
[0067] Compared with the prior art, the method of this embodiment first uses a student model to obtain image features and classification prediction results of a detection image, and a teacher model to obtain image features of a detection image and text features of a corresponding text description and a final classification prediction result; secondly, based on the two classification prediction results, a prediction similarity loss is obtained by minimizing the similarity between the predictions; based on the image features obtained by the student model and the corresponding text features obtained by the teacher model, a semantic loss is obtained by minimizing the difference between the image features and the corresponding text features; based on the semantic loss, the relative position of the source domain specific text feature representation is used to optimize the loss, that is, the semantic loss is optimized, the image features and the corresponding text features are modified. The universal text input of CLIP is modified to derive domain-specific text feature representations, and then the distances between the domain-specific text feature representations and the image features and the corresponding text feature representations are minimized to obtain the spatial consistency loss; based on the prediction similarity loss, semantic loss and spatial consistency loss, the final training loss is obtained by simple addition, and the feature encoder with semantic information extraction capability in the student model is trained in this way; then, in the testing phase, the classifier is updated using a memory library that introduces the statistical distribution of the target domain to promote target domain alignment and obtain the final classifier with target alignment; finally, the detection image is passed through the feature encoder with semantic information extraction capability and the target aligned classification predictor to obtain the classification result.
[0068] The method of this embodiment is different from the existing domain generalization method that only focuses on the domain shift during training and the alignment with the target domain during testing. Instead, it starts from the source domain and proposes semantic information extraction and target alignment (SIETA). The generalization of the model is optimized by aligning the features of the model during the training and testing stages, which makes the model more suitable for specific tasks while retaining the valuable information obtained from the large visual language model. This method can significantly improve the performance in any target domain.
[0069] In some embodiments, the detected image is scaled to a uniform size and then input into the backbone network. The input image needs to be converted into a fixed size first for subsequent processing by the model, for example, uniformly converted into a 224×224 RGB three-channel color image, and at the same time, the corresponding general text description is generated through the CLIP prompting project.
[0070] As a preferred embodiment, the backbone network of the preset student model is constructed based on the ResNet50 network model, and the teacher model CLIP adopts the pre-trained ViT-B / 16 as the backbone network;
[0071] In a specific embodiment, the student model is a feature encoder and a classification predictor structure; the teacher model includes two encoders, namely a text feature encoder and an image feature encoder, and also includes a classification predictor.
[0072] The student model is represented by F(·; θ), which consists of two parts: an encoder f(·) for feature extraction and a classifier q(·). The ultimate goal of the DG problem described in the present invention can be summarized as an empirical risk minimization (ERM) problem, which is calculated as follows:
[0073]
[0074] Among them, x represents the data, and y represents the label corresponding to x.
[0075] In a specific embodiment, the CLIP model is used as a teacher model to guide the training of the student model. The goal is to optimize the student model so that its output is closely consistent with the output of the teacher model, thereby transferring the prior knowledge obtained by the CLIP model during pre-training to the student model. The KL divergence is used to measure the similarity between the predictions of the teacher and the student during training as a way of knowledge distillation. The calculation formula is:
[0076]
[0077] Among them, F l (x;θ) and Represents the logits output value of the training process, l KL represents the KL divergence.
[0078] In a specific embodiment, in order to transfer semantic knowledge from the teacher to the student so that the student model can obtain general semantic information based only on image input, it is necessary to ensure that the feature representation extracted by the student is closely consistent with the text feature representation extracted by the teacher's text encoder. Therefore, a semantic loss is proposed to minimize the difference between the feature representation extracted by the student model and the text feature representation extracted by the teacher model to train the student encoder. The calculation formula is:
[0079]
[0080] Where e t (c) represents the text feature representation extracted by the text encoder of the teacher model CLIP, f(x) represents the feature representation extracted by the feature encoder of the student model, k a (·,·) represents the distance between the two.
[0081] In practice, it is a challenge to directly obtain the optimal solution that satisfies the semantic loss. Therefore, this paper uses domain-related information from the source domain, uses relative positions to optimize the model, and proposes a spatial consistency loss, which means that the feature representation of each source domain is To e t (c) The distance should be equal to the The distance to f(x) is calculated as:
[0082]
[0083] Among them, eD i (c) represents the domain-specific text feature representation derived from the general text, and k1(·, ·) and k2(·, ·) represent the distance between the two.
[0084] During the training process, considering all the above loss functions, the prior knowledge of the CLIP model is transferred to the training model suitable for DG. The final loss of the complete training is as follows, and its calculation formula is:
[0085] l train =λ1l0+λ2l distill +λ3(l ad +l rd )
[0086] Among them, change: l0The above empirical risk minimizes the loss, l distill is the prediction similarity loss during the training phase of the teacher and student models, l ad is the semantic loss mentioned above, l rd is the above-mentioned spatial consistency loss, and λ1, λ2, and λ3 are weighting coefficients.
[0087] In the test phase, a linear layer is used as a classifier to select the K highest confidence samples for each class c to create a differentiable loss function v c , the formula is expressed as:
[0088]
[0089] When using the trained encoder and linear classifier, the probability of samples belonging to category C can be calculated and stored in the memory bank, which is updated with iterations. The formula is expressed as:
[0090]
[0091] Where ωc represents the weight associated with category c.
[0092] The sample probability of each class is used to update the weight of the classifier on the corresponding dimension, and the entropy on the target domain is used to guide parameter updating.
[0093]
[0094] As a preferred embodiment, the classification result of the detection image is obtained according to the feature encoder and classification predictor trained by feature alignment in the entire training phase and test phase.
[0095] This embodiment also provides a domain generalization system based on CLIP model semantic information extraction and target alignment, including a student model, a CLIP-based teacher model, a classification prediction module, a classification prediction similarity loss acquisition module, a semantic loss optimization module, a spatial consistency loss acquisition module, a feature encoder training module, a classifier training module and a classification module;
[0096] The classification prediction module obtains the image features and classification prediction results of the detection image based on the student model; obtains the image features of the detection image and the text features of the corresponding text description and the final classification prediction results according to the teacher model, and the teacher model is based on CLIP: a pre-trained CLIP model; obtains the corresponding classification prediction through the student model and the teacher model;
[0097] The classification prediction similarity loss acquisition module obtains the classification prediction similarity loss by minimizing the similarity between the classification predictions based on the classification predictions of the two models;
[0098] The semantic loss acquisition module is used to acquire the semantic loss by minimizing the difference between the feature representation of the image and the feature representation of the text based on the feature representation of the image and the feature representation of the text;
[0099] The semantic loss optimization module optimizes the semantic loss by using the relative position of the source domain text feature representation;
[0100] The spatial consistency loss acquisition module modifies the general text input of CLIP based on the optimized semantic loss and the feature representation of the image and the text feature representation, derives the domain-specific text feature representation, minimizes the distance between the domain-specific text feature representation and the feature representation of the image and the text feature representation, and obtains the spatial consistency loss;
[0101] The feature encoder training module uses dynamic weighting to obtain the final training loss based on the predicted similarity loss, semantic loss and spatial consistency loss, and performs training to obtain a feature encoder with semantic information extraction capability in the student model;
[0102] In the test phase, the classifier training module uses the memory bank that introduces the statistical distribution of the target domain to update the classifier so that the target domain is aligned and the final classifier with target alignment is obtained;
[0103] The classification module obtains the classification result of the detection image based on the feature encoder extracted by the semantic information and the target aligned classifier.
[0104] like Figure 2As shown, first, the detection image is input into the student model to obtain the image features and classification prediction results of the detection image, and then the detection image and the corresponding text description are input into the teacher model to obtain the image features of the detection image and the text features of the corresponding text description and the final classification prediction results; secondly, based on the two classification prediction results, the prediction similarity loss is obtained by minimizing the similarity between the predictions; based on the image features obtained by the student model and the corresponding text features obtained by the teacher model, the semantic loss is obtained by minimizing the difference between the image features and the corresponding text features; based on the semantic loss, the relative position of the source domain specific text feature representation is used to optimize the loss, that is, the semantic loss is optimized, the image features and the corresponding Text features, modify the general text input of CLIP to derive domain-specific text feature representations, and then minimize the distances between the domain-specific text feature representations to the image features and the corresponding text feature representations to obtain the spatial consistency loss; based on the prediction similarity loss, semantic loss and spatial consistency loss, the final training loss is obtained by simple addition, and the feature encoder with semantic information extraction capability in the student model is trained in this way; then, in the test phase, the classifier is updated using a memory library that introduces the statistical distribution of the target domain to promote target domain alignment and obtain the final classifier with target alignment; finally, the detection image is passed through the feature encoder with semantic information extraction capability and the target aligned classification predictor to obtain the classification result.
[0105] This embodiment also provides a computer-readable storage medium, on which a domain generalization program based on CLIP model semantic information extraction and target alignment is stored. When the domain generalization program based on CLIP model semantic information extraction and target alignment is executed by a processor, the steps in the above embodiments can be implemented.
[0106] The present invention provides a domain generalization method based on CLIP model semantic information extraction and target alignment, which optimizes the generalization of the model by aligning the features of the model in the training and testing stages.
[0107] The present invention may also provide a computer device, comprising a processor and a memory, wherein the memory is used to store a computer executable program, and the processor reads the computer executable program from the memory and executes it. When the processor executes the computer executable program, the domain generalization method based on CLIP model semantic information extraction and target alignment described in the present invention can be implemented.
[0108] The present invention provides a computer-readable storage medium, in which a computer program is stored. When the computer program is executed by a processor, the domain generalization method based on CLIP model semantic information extraction and target alignment described in the present invention can be implemented.
[0109] The computer device may be a laptop computer, a desktop computer or a workstation.
[0110] The processor may be a central processing unit (CPU), a digital signal processor (DSP), an application specific integrated circuit (ASIC), or an off-the-shelf field programmable gate array (FPGA).
[0111] The memory of the present invention may be an internal storage unit of a laptop computer, a desktop computer or a workstation, such as a memory or a hard disk; or an external storage unit, such as a mobile hard disk or a flash memory card.
[0112] Computer-readable storage media may include computer storage media and communication media. Computer storage media include volatile and non-volatile, removable and non-removable media implemented by any method or technology for storing information such as computer-readable instructions, data structures, program modules or other data. Computer-readable storage media may include: read-only memory (ROM), random access memory (RAM), solid-state drive (SSD) or optical disk, etc. Among them, random access memory may include resistance random access memory (ReRAM) and dynamic random access memory (DRAM).
[0113] The method of the present invention is different from the existing domain generalization methods that only focus on the domain shift during training and the alignment with the target domain during testing. Instead, it starts from the source domain and proposes semantic information extraction and target alignment (SIETA). The generalization of the model is optimized by aligning the features of the model during the training and testing stages, which makes the model more suitable for specific tasks while retaining the valuable information obtained from the large visual language model. This method can significantly improve the performance in any target domain. In addition, the method proposed by us can be combined with other domain generalization methods. These results show the superiority and versatility of the method, which is more suitable for promotion and application in practice. The method of the present invention overcomes the inertial thinking of the prior art. It does not adopt the mainstream approach of only focusing on alignment with the target domain, but does not consider the overall improvement of the model representation from the perspective of the source domain. Instead, it innovatively achieves better domain generalization performance through semantic information extraction alignment in the training stage and target alignment in the testing stage, and achieves good results in practical application. Therefore, the method has the advantages of good versatility, high feasibility, and innovative and efficient.
[0114] The above description is only a preferred specific implementation manner of the present invention, but the protection scope of the present invention is not limited thereto. Any changes or substitutions that can be easily conceived by any technician familiar with the technical field within the technical scope disclosed by the present invention should be covered within the protection scope of the present invention.
Claims
1. A domain generalization method based on CLIP model semantic information extraction and target alignment, characterized in that: The following steps are involved: Obtain a detection image and a text description corresponding to the detection image, input the detection image into a preset student model, and input the detection image and the text description corresponding to the detection image into a teacher model, where the teacher model is based on a trained CLIP model; Based on the student model, its feature encoder extracts features from the detection image to obtain feature representation of the image; based on the teacher model, the text encoder extracts text features from the text description to obtain text feature representation; and the corresponding classification predictions are obtained through the pre-trained CLIP classifiers of the student model and the teacher model respectively; Based on the classification prediction of the two models, the classification prediction similarity loss is obtained by minimizing the similarity between the classification predictions; Based on the feature representation of the image and the feature representation of the text, obtaining a semantic loss by minimizing a difference between the feature representation of the image and the feature representation of the text; Utilizing the relative position of source domain text feature representation to optimize the semantic loss; Based on the optimized semantic loss and the feature representation of the image and the text feature representation, modify the general text input of CLIP, derive the domain-specific text feature representation, minimize the distance between the domain-specific text feature representation and the feature representation of the image and the text feature representation, respectively, to obtain the spatial consistency loss; Based on the prediction similarity loss, semantic loss and spatial consistency loss, dynamic weighting is used to obtain the final training loss, and training is performed to obtain a feature encoder with semantic information extraction capability in the student model; In the test phase, the classifier is updated using the memory library that introduces the statistical distribution of the target domain to promote the alignment of the target domain and obtain the final classifier with target alignment; Based on the feature encoder extracted from the semantic information and the target aligned classifier, a classification result of the detection image is obtained.
2. The method for semantic information extraction and target alignment based on CLIP model according to claim 1, characterized in that: The student model is represented as F(·; θ), which includes an encoder f(·) and a classifier q(·) for feature extraction. The ultimate goal of the domain generalization problem is summarized as an empirical risk minimization problem, which is calculated as: Among them, x represents the data, and y represents the label corresponding to x.
3. The method for semantic information extraction and target alignment based on CLIP model according to claim 1, characterized in that: Using KL divergence to measure the similarity between the teacher and student predictions during training, the similarity loss between the teacher model and the student model predictions during the training phase is calculated as follows: Among them, F l (x;θ) and Represents the logits output value of the training process, l KL represents the KL divergence.
4. The method for semantic information extraction and target alignment based on CLIP model according to claim 1, characterized in that: The semantic loss is used to ensure that the feature representation extracted by the student is consistent with the text feature representation extracted by the teacher's text encoder, and to minimize the difference between the feature representation extracted by the student model and the text feature representation extracted by the teacher model to train the student's encoder. The calculation formula is: Where e t (c) represents the text feature representation extracted by the text encoder of the teacher model CLIP, f(x) represents the feature representation extracted by the feature encoder of the student model, k a (·,·) represents the distance between the two.
5. The method for semantic information extraction and target alignment based on CLIP model according to claim 4, characterized in that: The spatial consistency loss is calculated as follows: Among them, eD i (c) represents the domain-specific text feature representation derived from the general text, and k1(·, ·) and k2(·, ·) represent the distance between the two.
6. The method for semantic information extraction and target alignment based on CLIP model according to claim 5, characterized in that: The final training loss is obtained by dynamically weighting all the above losses in the training phase, and its calculation formula is: l train =λ1l0+λ2l distill +λ3(l ad +l rd ) Among them, l0 is the empirical risk minimization loss, l distill is the prediction similarity loss during the training phase of the teacher and student models, l ad is the semantic loss mentioned above, l rd is the above-mentioned spatial consistency loss, and λ1, λ2, and λ3 are weighting coefficients.
7. The method for semantic information extraction and target alignment based on CLIP model according to claim 1, characterized in that: In the test phase, the classifier is updated using the memory library that introduces the statistical distribution of the target domain to align the target domain. The final classifier with target alignment includes: Use a linear layer as a classifier, select the K highest confidence samples for each class c, and create a differentiable loss function v c , the formula is: When using the trained encoder and linear classifier, the probability of samples belonging to category C is calculated and stored in the memory bank, and the probability of samples belonging to category C is iteratively updated. The formula is expressed as: Where ωc represents the weight associated with category c, The sample probability of each class is used to update the weight of the classifier in the corresponding dimension, and the entropy on the target domain is used to guide the parameter update:
8. A domain generalization system based on CLIP model semantic information extraction and target alignment, characterized in that: It includes a student model, a CLIP-based teacher model, a classification prediction module, a classification prediction similarity loss acquisition module, a semantic loss optimization module, a spatial consistency loss acquisition module, a feature encoder training module, a classifier training module, and a classification module; The classification prediction module obtains the image features and classification prediction results of the detection image based on the student model; obtains the image features of the detection image and the text features of the corresponding text description and the final classification prediction results according to the teacher model, and the teacher model is based on the CLIP model; obtains the corresponding classification prediction through the pre-trained CLIP classifier of the student model and the teacher model; The classification prediction similarity loss acquisition module obtains the classification prediction similarity loss by minimizing the similarity between the classification predictions based on the classification predictions of the two models; The semantic loss acquisition module is used to acquire the semantic loss by minimizing the difference between the feature representation of the image and the feature representation of the text based on the feature representation of the image and the feature representation of the text; The semantic loss optimization module optimizes the semantic loss by using the relative position of the source domain text feature representation; The spatial consistency loss acquisition module modifies the general text input of CLIP based on the optimized semantic loss and the feature representation of the image and the text feature representation, derives the domain-specific text feature representation, minimizes the distance between the domain-specific text feature representation and the feature representation of the image and the text feature representation, and obtains the spatial consistency loss; The feature encoder training module uses dynamic weighting to obtain the final training loss based on the predicted similarity loss, semantic loss and spatial consistency loss, and performs training to obtain a feature encoder with semantic information extraction capability in the student model; In the test phase, the classifier training module uses the memory bank that introduces the statistical distribution of the target domain to update the classifier so that the target domain is aligned and the final classifier with target alignment is obtained; The classification module obtains the classification result of the detection image based on the feature encoder extracted by the semantic information and the target aligned classifier.
9. A computer device, characterized in that: The invention comprises a processor and a memory, wherein the memory is used to store a computer executable program, the processor reads part or all of the computer executable program from the memory and executes it, and when the processor executes part or all of the computer executable program, the domain generalization method based on CLIP model semantic information extraction and target alignment as described in any one of claims 1 to 7 can be implemented.
10. A computer-readable storage medium, characterized in that: Used to store computer-readable programs or instructions, which, when executed by a processor, can implement the domain generalization method based on CLIP model semantic information extraction and target alignment as described in any one of claims 1 to 7.
Citation Information
Patent Citations
Unsupervised semantic segmentation model training method and related device
CN115063585A
Cross-domain small sample image classification method based on parameter efficient module and related device
CN116977736A
Model training method and apparatus
US20200356852A1
Cited By
Adaptive learning method and system for power transmission line inspection
CN119295744A
Domain generalization method and system based on multi-source hierarchical semantic fusion and cross-layer calibration
CN119357903A
Power system equipment fault early warning method and system
CN120234677A
Passive cross-domain adaptive image classification method based on multi-modal pre-training model
CN121214038A
A passive cross-domain adaptive image classification method based on a multi-modal pre-training model
CN121214038B
