A feature decoupling and recombination domain generalization method combined with low rank adaptation

Abstract

The application discloses a feature decoupling and recombination domain generalization method combined with low rank adaptation, and belongs to the fields of machine learning and deep learning. The method constructs a student-teacher distillation framework comprising a LoRA-DG feature decoupling module, a cross-modal semantic distillation mechanism and a convolution-attention fusion module CAFM, utilizes two parallel low rank adaptation LoRA structures to decouple class-related features and domain-related features, verifies the purity of the decoupled features through a feature recombination comparison strategy, migrates cross-modal semantic knowledge to a student image encoder through semantic distillation of image-text alignment with the CLIP visual-language pre-training model as a teacher, avoids catastrophic forgetting, adopts the CAFM module to fuse local convolution and global attention mechanisms, realizes collaborative modeling of local structures and global dependencies, and finally completes joint optimization of the model through a total loss function weighted by multiple losses. The method effectively improves the cross-domain generalization capability and robustness of the model to out-of-distribution data.

Description

Technical Field

[0001] This invention relates to the fields of machine learning and deep learning, and in particular to a feature decoupling and recombination domain generalization method that combines low-rank adaptation. Background Technology

[0002] Deep neural networks demonstrate powerful feature learning capabilities in computer vision tasks, but their performance relies on the assumption that the training and test data are independently and identically distributed (i.i.d.). When there is a distributional difference (domain shift) between the test data (target domain) and the training data (source domain), the model performance will significantly degrade, and this problem has become a key factor restricting the practical application of deep neural networks.

[0003] Domain generalization (DG), as an effective solution to address domain shift, aims to train a model using multiple known source domains and enable it to maintain good performance on a completely unknown target domain. Its core idea is to learn robust domain-independent features. Existing domain generalization methods often treat domain-related information as noise and blindly remove it. However, domain features and class features have complex complementary relationships; blindly removing them leads to the loss of discriminative information, resulting in a decrease in the model's generalization ability. Furthermore, existing feature decoupling methods only achieve a formal separation of class-domain features and lack a mechanism to verify the purity of decoupled features. Residual coupling between features still limits the model's cross-domain performance.

[0004] Furthermore, existing methods for large-scale fine-tuning of pre-trained backbone networks can easily disrupt their cross-modal semantic geometry, leading to the "catastrophic forgetting" problem. Traditional convolutional networks are limited by local receptive fields and have difficulty capturing global long-range dependencies. While the Transformer attention mechanism is good at global modeling, it weakens the characterization of local details and boundary structures. A single feature modeling mechanism cannot achieve a comprehensive expression of image features.

[0005] Low-rank adaptation (LoRA), as a representative method of parameter efficient fine-tuning (PEFT), achieves incremental parameter updates by inserting a low-rank matrix into the attention layer, reducing fine-tuning costs while maintaining the capabilities of the pre-trained model. The CLIP vision-language pre-trained model possesses rich cross-modal semantic knowledge, and its image-text aligned semantic structure provides an important foundation for cross-domain generalization. Convolution and attention mechanisms are complementary in feature modeling, and their fusion can achieve collaborative capture of local and global information. Based on this, this invention proposes a feature decoupling and recombination domain generalization method combining low-rank adaptation to address the aforementioned shortcomings of existing technologies. Summary of the Invention

[0006] Purpose of the invention: This invention provides a domain generalization method that combines low-rank adaptation with feature decoupling and recombination to solve the problems in existing domain generalization methods, such as blindly stripping domain features, resulting in loss of discriminative information, lack of purity verification of decoupled class-domain features, large-scale fine-tuning of the backbone network, which can easily lead to catastrophic forgetting, and the limitations of single convolution or attention mechanisms in feature modeling, ultimately resulting in insufficient generalization ability of the model for unknown target domains.

[0007] Technical Solution: To address the aforementioned technical problems, according to one aspect of the present invention, more specifically, it is a feature decoupling and recombination domain generalization method combining low-rank adaptation. This method constructs a student-teacher distillation framework with CLIP as the teacher model, integrating three core components: a LoRA-DG feature decoupling module, a cross-modal semantic distillation mechanism, and a CAFM convolution-attention fusion module. Through feature decoupling and recombination, cross-modal semantic transfer, and local-global feature fusion, it enhances the model's cross-domain generalization ability. Simultaneously, it employs an efficient parameter fine-tuning strategy to avoid catastrophic forgetting. The specific technical solution is as follows: 1. Preliminary Knowledge The problem of domain generalization is defined as follows: Let the input feature space be... The tag space is A domain is formalized as a joint distribution of inputs and labels. It means that, among them This represents the set of joint probability distributions defined on the Cartesian product of the input and output. During the training phase, a set of samples from K source domains is available for access. Each of these fields The samples are given by the corresponding joint distribution. Generation, i.e. Access during the training phase Sample set from the source domain The goal is to train a mapping function. This puts it in a completely unknown target area. and It has good generalization performance.

[0008] 2. Framework Overview This invention constructs a student-teacher distillation framework that combines low-rank adaptation with multi-branch feature decoupling and semantic alignment fusion. The overall architecture is as follows: Figure 2 As shown, the input image is divided into two paths: one path is fed into the frozen CLIP image encoder (embedded with a CAFM module) and connected to two parallel LoRA-DG sub-modules (class branch and domain branch) to generate decoupled class-domain semantic vectors; the other path is fed into the student image encoder, whose output is mapped to the same feature space as CLIP through a projection module and receives knowledge distillation from the teacher model.

[0009] During training, the image's class and domain labels are filled into the natural language template "a {domain} of {class}" to form a text prompt. This prompt is then processed by the CLIP text encoder to obtain text features, providing semantic supervision for the student model. The model is jointly optimized using classification loss, distillation loss, recombination loss, and semantic loss. Only the newly added LoRA-DG and CAFM modules and the student encoder parameters are trained, while the CLIP backbone parameters are frozen. During inference, domain information is not used; a unified text prompt "a photo of {class}" is used to predict the class.

[0010] 3 LoRA-DG Feature Decoupling Module The core of the LoRA-DG module is to decouple image features into class-related features (domain-invariant, used for class discrimination) and domain-related features (such as style, which provides assistance for semantic alignment) using two parallel LoRA structures, and verify the purity of the decoupled features through a feature recombination and comparison strategy.

[0011] The core of the LoRA structure is to use low-rank matrix factorization to simulate parameter shifts, avoiding large-scale updates of pre-trained weights. Its weight updates and forward propagation satisfy formulas (1) and (2): Where W is the original high-dimensional weight matrix of the CLIP pre-trained Transformer layer. and For trainable low-rank matrices x represents the input feature, and h represents the output feature.

[0012] For samples within a training batch, the LoRA-DG class branch outputs class features. Domain branch output domain features First, the two types of features are classified and constrained by cross-entropy loss, satisfying formulas (3) and (4): in, For the class label and domain label of the sample, calculate the predicted probability of the corresponding feature.

[0013] To verify the purity of the decoupled features, the original binding relationship between the class features and domain features of the intra-batch samples was broken, and a reconstructed feature vector was obtained by concatenation. Find true class-domain combination features with the same class label and domain label in the training set. The L2 distance loss between the two in the feature space is calculated, satisfying formula (5): This recombination and comparison strategy enhances the independence and composability of class-domain features and eliminates residual coupling between features.

[0014] 4. Cross-modal semantic distillation mechanism To transfer CLIP's cross-modal semantic knowledge to the student image encoder while avoiding catastrophic forgetting, this invention constructs a two-layer distillation mechanism of image distillation and text semantic distillation, with corresponding losses including basic classification loss, KL divergence distillation loss, and text semantic distance loss.

[0015] Basic classification loss: The output of the student model is subject to classification constraints to ensure its basic discriminative ability and satisfy formula (6): in, Given the sample class labels, calculate the degree of matching between the student model's classification prediction probability and the true labels.

[0016] Image distillation loss: The output distribution of the student model is aligned with the output distribution of the CLIP image encoder using KL divergence, satisfying formula (7): in, The output distributions are for the student model and the CLIP image encoder, respectively.

[0017] Text semantic loss: Constructing language anchors for each category c and training domain d By using distance loss, the image embeddings of student models are encouraged to be semantically aligned with the corresponding language anchors, thus reproducing the semantic space geometry of CLIP and satisfying formula (8): Where C represents the total number of categories. This is an indicator function (1 if the sample belongs to class c, 0 otherwise). Image embedding for student models, This represents the text feature of class c.

[0018] 5 CAFM Convolution-Attention Fusion Module To overcome the limitations of feature modeling using single convolutional or attention mechanisms, this invention designs a CAFM module, consisting of local and global branches. These branches capture local structural features and global long-range dependency features of the image, respectively, and the fused features output enhanced visual features. The module architecture is as follows: Figure 3 As shown.

[0019] Local branch: By using the structure of "1×1 convolution + channel shuffle + 3×3 convolution", the interaction between channels and the modeling of local spatial features are enhanced, and the output satisfies formula (9): in, These are the input features for the CAFM module. and These are 1×1 and 3×3 convolution operations, respectively. The channel shuffling operation enables the rearrangement and fusion of feature channels.

[0020] Global branch: A lightweight self-attention mechanism is adopted, which generates Query, Key, and Value vectors through "1×1 convolution + 3×3 depthwise convolution", calculates attention weights and aggregates global context features, and adds residual connections to improve feature propagation efficiency. The output satisfies formula (10): in, As a self-attention mechanism, the similarity matrix of spatial locations is calculated by matrix multiplication and then normalized by Softmax to obtain the attention weights.

[0021] Feature fusion: The output features of the local branch and the global branch are added element by element to obtain the final output of the CAFM module, which satisfies formula (11): 6. Total Loss Function and Model Training To achieve multi-dimensional collaborative optimization of the model, this invention performs a weighted summation of the above loss terms to construct a total loss function that satisfies formula (12): in, For hyperparameters, considering both model performance and stability, set... =0.2, =0.2, =0.1, =0.5.

[0022] The model was trained using the AdamW optimizer, with a uniform image input size of 224×224. Standard data augmentation strategies were employed, including random cropping, horizontal flipping, color perturbation, and grayscale conversion. The initial learning rate was set to 0.002 for the Terra-Incognita dataset and 0.001 for the other datasets. The training cycle consisted of 20 epochs with a batch size of 128. During training, the CLIP backbone parameters were frozen, and only the parameters of the LoRA-DG module, CAFM module, and student image encoder were trained.

[0023] Beneficial effects: The problem of information loss due to blindly stripping domain features is solved: the LoRA-DG module is used to decouple class-domain features instead of simply removing them, which preserves the auxiliary value of domain features for semantic alignment. At the same time, the purity of decoupled features is verified by feature recombination and comparison strategy, eliminating residual coupling between features and improving the model's ability to model features in a fine-grained manner.

[0024] It avoids the catastrophic loss of backbone network fine-tuning: using CLIP as the teacher model, its backbone parameters are frozen, and parameters are fine-tuned efficiently only through LoRA, transferring cross-modal semantic knowledge to the student model, thus preserving CLIP's graph-text alignment semantic geometry to the greatest extent.

[0025] It achieves collaborative modeling of local and global features: The CAFM module combines the advantages of local structure modeling of convolution with the global dependency capture capability of attention mechanism, outputting richer and more robust visual features and improving the transferability of feature representation.

[0026] The model's generalization performance is significantly improved: the classification accuracy on the four classic domain generalization benchmark datasets—PACS, VLCS, Office-Home, and Terra-Incognita—reaches 91.5%, 84.3%, 76.8%, and 57.8%, respectively, with an average accuracy of 77.6%. This significantly outperforms existing mainstream methods such as DANN, CORAL, SWAD, and SMIDG, with the most significant improvement on the Terra-Incognita dataset, which has high domain dissimilarity. Attached Figure Description

[0027] Figure 1 This is a diagram of the LoRA framework of the present invention; Figure 2 This is a schematic diagram of the overall framework of the model of the present invention; Figure 3 This is a diagram of the CAFM module framework of the present invention; Figure 4 This is an example diagram of the dataset of the present invention. Detailed Implementation

[0028] To make the technical solution, objective, and beneficial effects of this invention clearer, the invention will be further described in detail below with reference to Figures 1-4 and specific embodiments. These embodiments are only for explaining the invention and are not intended to limit the scope of protection of the invention. Example 1

[0029] refer to Figures 1-4This invention proposes a domain generalization method that combines low-rank adaptation with feature decoupling and recombination. This embodiment verifies the effectiveness of the method in domain generalization tasks through comparative experiments, ablation experiments, and hyperparameter analysis experiments. The experiments are conducted on four classic domain generalization benchmark datasets: PACS, VLCS, Office-Home, and Terra-Incognita. The overall experimental framework is shown in Figure 2, the LoRA structure is shown in Figure 1, the CAFM module structure is shown in Figure 3, and the dataset sample examples are shown in Figure 4.

[0030] 1. Experimental Setup 1.1 Experimental Dataset Four domain generalization benchmark datasets—PACS, VLCS, Office-Home, and Terra-Incognita—were selected. Basic information about each dataset is as follows: PACS: 4 fields × 7 categories × 9991 images. The fields include Photo, Artpainting, Cartoon, and Sketch. VLCS: 4 domains × 5 classes × 10729 images, including VOC2007, LabelMe, Caltech101, and SUN09; Office-Home: 4 domains × 65 categories × 15,500 images. The domains include Art, Clipart, Product, and Real-World. Terra-Incognita: 4 shooting locations × 10 categories × 24,000 images, including Loc38, Loc43, Loc46, and Loc100.

[0031] The experiment uses the leave-one-domain-out method for training, meaning that each time a domain is selected as the target test domain, and the remaining domains are used as the training set. An example dataset is shown below. Figure 4 As shown.

[0032] 1.2 Experimental Setup Using CLIP as the visual encoding backbone, its parameters are frozen, and the LoRA-DG module and CAFM module of this invention are introduced; during the training phase, a text prompt "a {domain} of {class}" is constructed, and CLIP text features are pre-calculated and fixed; during the inference phase, a unified text prompt "a photo of {class}" is used.

[0033] Hardware and software configuration for model training: NVIDIA GPU is used for acceleration, and PyTorch is used as the deep learning framework; the image input size is 224×224, and data augmentation includes random cropping, horizontal flipping, color perturbation, and grayscale conversion; the optimizer is AdamW, with an initial learning rate (0.002 for Terra and 0.001 for the rest), 20 training epochs, and a batch size of 128.

[0034] 2. Comparison of experimental results The method of this invention was compared with 12 existing mainstream domain generalization methods, including DANN, CORAL, MTL, ERM, SWAD, PCL, CoOP, and SMIDG. The experimental results are shown in Tables 1-5. Table 1 Comparison of cross-domain classification accuracy of different methods As shown in Table 1, the method of this invention achieves an average classification accuracy of 77.6% on the four datasets, which is significantly better than all the comparison methods and 3.7 percentage points higher than the second-best method SMIDG.

[0035] Table 2 Results on the PACS dataset Table 3 Results on the Office-Home dataset Table 4. Results on the VLCS dataset Table 5 Results on the Terra dataset As shown in Tables 2-5, the method of the present invention achieves optimal or near-optimal classification accuracy in each subdomain of each dataset. The improvement is particularly significant in subdomains with large style differences and significant domain shifts (such as the Cartoon domain of PACS, the Clipart domain of Office-Home, and the Loc100 domain of Terra), which verifies the robustness of the method under different domain shift scenarios.

[0036] 3 Ablation Experiment Results To verify the effectiveness of the LoRA-DG module, CAFM module, and feature recombination strategy in this invention, ablation experiments were conducted on the PACS dataset. The results are shown in Tables 6 and 7. Table 6 Ablation results of different modules on the PACS dataset The results in Table 6 show that the LoRA-DG module and the CAFM module are highly complementary in function. Introducing either module alone can improve model performance, and the model performance reaches its optimal level when the two are combined, which verifies the effectiveness of the two core modules.

[0037] Table 7 Ablation results of feature branches on the PACS dataset The results in Table 7 show that class branching is the core of class discrimination, domain branching can provide assistance for semantic alignment, and feature reorganization strategy is the key to improving the purity of decoupled features and eliminating residual coupling. The model performance is significantly improved after the introduction of the reorganization strategy, which verifies the necessity of the strategy.

[0038] 4. Hyperparameter Analysis Results To evaluate the sensitivity of the method of this invention to the hyperparameters in the total loss function, the hyperparameters were tested on the PACS dataset. The results of the univariate scan are shown in Table 8. Table 8 Hyperparameter Analysis Table 8 shows that all hyperparameters exhibit a pattern of "optimal moderate values ​​and a decline when excessively large." The hyperparameters set in this invention... =0.2, =0.2, =0.1, =0.5 is the optimal value; the accuracy of the model fluctuates by only 0.4-0.6 percentage points under different hyperparameter values, which verifies the robustness of the method to hyperparameters.

[0039] The embodiments described above are merely illustrative of several implementations of the present invention, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of the present invention. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of the present invention, and these modifications and improvements all fall within the scope of protection of the present invention. Therefore, the scope of protection of this patent should be determined by the appended claims.

Claims

1. A method combining low-rank adaptation feature decoupling and recombination domain generalization, characterized in that, A feature decoupling and recombination domain generalization method combining low-rank adaptation is characterized by the following steps: S1. Construct a student-teacher distillation domain generalization framework based on CLIP. Feed the input image into the frozen CLIP image encoder and the student image encoder respectively. The CLIP image encoder embeds the convolution-attention fusion module CAFM and is connected to two parallel LoRA-DG feature decoupling sub-modules. The student image encoder is mapped to the same feature space as CLIP through the projection module. S2. Use the LoRA-DG module to decouple class-related features and domain-related features. Calculate the feature space difference between the recombined features and the real class-domain combination features through a feature recombination comparison strategy to verify and strengthen the purity of the decoupled features. S3. Introduce CLIP text encoder to construct cross-modal semantic distillation mechanism, and transfer CLIP's cross-modal semantic knowledge to student image encoder through image distillation and distance loss of text semantic anchors; S4. Utilize the CAFM module to fuse local convolution and global attention mechanisms, extracting local structural features and global dependency features of the image through local branches and global branches respectively, and outputting enhanced visual features after fusion. S5. Construct a total loss function that includes classification loss, image distillation loss, text semantic loss, feature recombination loss, etc., and perform joint optimization training on the model to obtain a model with cross-domain generalization ability. S6. Apply the trained model to an image recognition task in an unknown target domain. During the inference phase, use unified text prompts to complete category prediction.

2. The feature decoupling and recombination domain generalization method combining low-rank adaptation as described in claim 1, characterized in that, The LoRA-DG module described in step S2 uses a low-rank adaptive LoRA structure as a trainable decoupled encoder. The weight update and forward propagation of the LoRA structure satisfy formulas (1) and (2): Where W is the original weight matrix of the pre-trained Transformer layer, A and B are trainable low-rank matrices, x is the input feature, and h is the output feature.

3. The feature decoupling and recombination domain generalization method combining low-rank adaptation as described in claim 2, characterized in that, The specific process of the feature recombination comparison strategy described in step S2 is as follows: 1) The class features and domain features of the samples within the training batch are constrained by cross-entropy loss, and the loss function satisfies formulas (3) and (4): in, For the class label and domain label of the sample, calculate the predicted probability of the corresponding feature.

4. Disrupt the binding relationship between class features and domain features of samples within the batch, and concatenate them to obtain a reconstructed feature vector. ; 3) In the training set, find true feature combinations that have the same class and domain labels as the recombined features. The difference loss between the two in the feature space is calculated, satisfying formula (5): Basic classification loss: The output of the student model is subject to classification constraints, satisfying formula (6): in, Given the sample class labels, calculate the degree of matching between the student model's classification prediction probability and the true labels.

5. The feature decoupling and recombination domain generalization method combining low-rank adaptation as described in claim 1, characterized in that, The loss of the cross-modal semantic distillation mechanism in step S3 includes the KL divergence loss of image distillation and the distance loss of text semantic anchors, which satisfy formulas (7) and (8) respectively: in, The output distributions are for the student model and the CLIP image encoder, respectively. This is an indicator function (1 if the sample belongs to class c, 0 otherwise). Image embedding for student models, This represents the text feature of class c.

6. The feature decoupling and recombination domain generalization method combining low-rank adaptation as described in claim 1, characterized in that, The local branch output of the CAFM module in step S4 satisfies formula (9), the global branch output satisfies formula (10), and the final output of the module satisfies formula (11): in, These are the input features of the CAFM module. and These are 1×1 and 3×3 convolution operations, respectively. For channel shuffling operations, Attention is a lightweight self-attention mechanism. These are the output features of the local branch and the global branch, respectively.

7. The feature decoupling and recombination domain generalization method combining low-rank adaptation according to claim 1, characterized in that, The total loss function mentioned in step S5 is a weighted sum of the loss terms, satisfying formula (12): in, =0.2, =0.2, =0.1, =0.5 is a hyperparameter.

8. The feature decoupling and recombination domain generalization method combining low-rank adaptation according to claim 1, characterized in that, In step S1, the parameters of the CLIP image encoder are frozen throughout the process, and only the parameters of the newly added LoRA-DG module, CAFM module, and student image encoder are trained. During the training phase, a text prompt is constructed based on the target category as "a {domain} of {class}", and during the inference phase, a unified text prompt is used as "a photo of {class}".

9. The feature decoupling and recombination domain generalization method combining low-rank adaptation according to claim 1, characterized in that, The model was trained using the AdamW optimizer, with a uniform image input size of 224×224. The training employed data augmentation strategies including random cropping, horizontal flipping, color perturbation, and grayscale conversion. The initial learning rate was set to 0.002 on the Terra-Incognita dataset and 0.001 on other datasets. The training cycle consisted of 20 rounds, with a batch size of 128.

10. The feature decoupling and recombination domain generalization method combining low-rank adaptation according to claim 1, characterized in that, The method achieved classification accuracies of 91.5%, 84.3%, 76.8%, and 57.8% on four benchmark datasets: PACS, VLCS, Office-Home, and Terra-Incognita, respectively, with an average accuracy of 77.6%.

Images