ViT-based unsupervised domain adaptive image classification method
By introducing the domain cross attention module and the convolution-transformer feature interaction module in ViT, the shortcomings of domain-level information capture and local information interaction in the unsupervised domain adaptation image classification method are solved, cross-domain relationship representation and multi-scale feature interaction are realized, and image classification performance is improved.
Patent Information
- Application Number
- CN202510326308.3
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-03-19
- Publication Date
- 2025-06-20
- Estimated Expiration
- 2045-03-19
AI Technical Summary
The existing unsupervised domain adaptation image classification methods lack effective processing of domain-level information capture and local information interaction, resulting in insufficient relationship characterization between cross-domain samples and multi-scale feature interaction.
The domain cross attention module and the convolution-transformer feature interaction module are introduced in ViT. The global correspondence between cross-domain sample features is captured through the domain-level cross-domain feature mechanism, and the fused features are obtained to enhance semantic representation through the bidirectional interaction between the convolutional features and the Transformer feature.
Cross-domain long-range dependency representation is realized, the model's performance on the target dataset is improved, the problem of lack of local information and multi-scale feature interaction is alleviated, and the gap between patch-based ViT processing and convolution processing is bridged.
Smart Images

Figure CN120182710A_ABST
Abstract
Description
Technical Field
[0001] The present invention belongs to the field of image classification, and particularly relates to an unsupervised domain adaptation image classification method based on ViT. Background Art
[0002] The field of deep learning has made remarkable progress in various recognition tasks, especially in image classification. Unsupervised domain adaptation image classification methods mainly involve the fields of computer vision, machine learning, and image processing technologies. It particularly focuses on how a model trained on a source domain can maintain good performance on a target domain with similar features but different distributions without the label information of the target domain. In the field of computer vision (CV), a large number of methods have been extensively studied. This includes Convolutional Neural Networks (CNNs) and the recent Vision Transformers (ViTs). ViT has shown remarkable generalization ability in supervised learning scenarios, often outperforming CNN-based techniques. Despite these advancements, deep neural networks (DNNs) have addressed the phenomenon of domain shift, where the model is trained on data with different distributions in the source dataset and the target dataset. Traditional solutions involve annotating the target domain data, which is not always feasible due to the associated costs. Unsupervised Domain Adaptation (UDA) has become a strategic option that relies on a labeled source dataset and an unlabeled target dataset, usually showing significant domain differences. UDA aims to narrow the gap between these two domains, thereby improving the performance of the model on this target dataset. Category-level alignment-based solutions reduce the domain gap and generate pseudo-labels for target samples, such as metric learning, adversarial training, and optimal transport. In addition, some works have explored the potential of ViT in non-trivial UDA tasks and achieved better results than CNNs.
[0003] Vision Transformer (ViT) is a pioneering work that applies a non-convolutional transformation structure to image classification. Some works have used ViT in the field of unsupervised domain adaptation. TVT proposed an adaptive module to capture the transferable and discriminative features of domain data. SSRT proposed a framework with a transformer backbone and a secure self-refinement strategy to handle problems in the case of large domain gaps. CDTrans proposed a two-step framework that uses cross-attention in ViT for direct feature alignment and uses pre-generated pseudo-labels for target samples. PMTrans interprets the process of unsupervised domain adaptation as a max-min game and finds an intermediate domain.
[0004] In terms of domain adaptation, strict transfer theory shows that the target error of a classifier is mainly bounded by its source error plus the divergence between domains. Guided by this theory, a series of CNN-based methods have attempted to learn domain-invariant features by explicitly minimizing the source error and the statistical distribution difference between domains. With the great success of Transformers in computer vision, this has further inspired the latest progress in UDA. However, simply applying Transformers can only establish the interaction relationship between image patches and cannot represent the relationship between cross-domain samples. Therefore, simply learning patch-level information is not enough.
[0005] Secondly, in terms of feature representation, in order to process 2D images using Transformers, the input image is first converted into a sequence of tokens corresponding to image patches in the image. Then the attention module attends to all the tokens and calculates a weighted sum as the tokens for the next layer. In this way, the effective receptive field is extended to the entire image through a single self-attention layer. However, the problem with the conformational change of vision Transformers is that the global connectivity conflicts with the convolutional concept. Considering the advantages of CNNs over Transformers, a natural question is whether it is possible to effectively combine the locality of CNNs and the global connectivity of vision transformers to improve performance without increasing the complexity of the model. Summary of the Invention
[0006] The technical problem to be solved by the present invention is to provide an unsupervised domain adaptation image classification method based on ViT to solve the problems of lack of capture of domain-level information and local information interaction in existing unsupervised domain adaptation image classification methods.
[0007] Based on the above objectives, one or more embodiments in the present application provide an unsupervised domain adaptation image classification method based on ViT, which includes the following steps:
[0008] Step 1, introduce a domain cross-attention module based on global correspondence in the original ViT to interact the sample features between different domains at the domain level and capture the global correspondence between cross-domain sample features; obtain the sample-level global features based on the captured global correspondence between cross-domain sample features, input the sample-level global features into the original ViT, and the original ViT outputs multi-level global features;
[0009] Step 2, introduce a convolutional-transformer feature interaction module in the original ViT, input the multi-level global features into the convolutional-transformer feature interaction module to obtain local features; first interact and then superimpose the obtained local features with the multi-level global features to obtain fused features;
[0010] Step 3: After the original ViT incorporates the cross-domain attention module and the convolutional-transformer feature interaction module to form the fused ViT, use the multi-level global features obtained in Step 1 as the input of the fused ViT, and the fused features obtained in Step 2 as the output of the fused ViT; Iteratively train the fused ViT using adversarial training to obtain the trained fused ViT;
[0011] Step 4: Use the trained ViT to perform image classification tasks.
[0012] Based on the above technical solutions of the present invention, the following improvements can also be made:
[0013] Optionally, in Step 1, the global correspondence relationship between cross-domain sample features is characterized by an attention score matrix, and the attention score matrix is calculated based on the query generated from the source-domain sample features and the key-value generated from the target-domain sample features. Reconstruct the target-domain features according to the attention score matrix; The reconstructed target-domain features continue to use the labels of the source-domain features, and the reconstructed target-domain features are used as new source-domain features, that is, sample-level global features. Input the new source-domain features into the original ViT, and the original ViT outputs multi-level global features.
[0014] Optionally, in Step 2, the process of the convolutional-transformer feature interaction module obtaining local features is as follows:
[0015] First, divide the input image into several image patches, and use the orthographic projection operator to project the flattened feature sequence of the ViT back into the 2D image space;
[0016] Then, repeatedly apply the two-dimensional convolution operator to the local patches in the flattened feature sequence of the ViT projected back into the 2D space to generate local convolution features;
[0017] For the obtained local convolution features, use the inverse projection operator in the feature space to project back to the original dimension to obtain new features, and the new features are local features.
[0018] Optionally, the size of the convolution kernel in the two-dimensional convolution operator is k, and zero padding of size (k - 1) / 2 is added around the perimeter of the convolution kernel coverage area to keep the size of the local convolution features unchanged.
[0019] Optionally, the process of obtaining the fused features is as follows: After obtaining the local features, first use the self-attention mechanism to interact the local features and the multi-level global features, and then use the residual connection to stack the local features and the multi-level global features to obtain the fused features.
[0020] Optionally, before Step 3 is executed, divide the fused ViT into several stages, and each stage processes patches of different sizes.
[0021] Optionally, the loss function for adversarial training is as follows:
[0022]
[0023] where the i-th sample and its corresponding one-hot label represent the labeled source set The j-th sample represents the unlabeled target set n s and n t respectively represent the sizes of samples in the source domain and the target domain. D is the union of D s and D t d i represents the domain label, M is the prediction result of the model for the input sample, λ is the trade-off coefficient, F is the feature extraction network, C is the classifier, is the cross-entropy loss on the labeled source domain samples, is the classical domain alignment loss based on domain adversarial learning:
[0024]
[0025] where represents the i-th sample in the source domain, represents the i-th sample in the target domain, x i represents the i-th input sample, y i represents the true label of the i-th sample in the source domain.
[0026] According to the second aspect of the present invention, an unsupervised domain adaptation image classification system based on ViT is provided. The system includes a VIT architecture and:
[0027] A domain cross-attention module that introduces a domain cross-attention module based on global correspondence in the original ViT to interact sample features between different domains at the domain level, capture the global correspondence between cross-domain sample features, and obtain global features;
[0028] A convolutional-transformer feature interaction module that obtains local features by introducing a convolutional-transformer feature interaction module in the original ViT; the obtained local features are interacted and superimposed with the global features to obtain fused features;
[0029] A training module integrates a domain cross-attention module and a convolutional-transformer feature interaction module into ViT to form a fused ViT. The input of the fused ViT is global features, and the output is fused features. The fused ViT is iteratively trained in an adversarial training manner to obtain a trained fused ViT.
[0030] An execution module performs an image classification task based on the trained fused ViT.
[0031] According to the third aspect of the present invention, an electronic device is provided, and the device includes:
[0032] A memory storing executable program code;
[0033] A processor coupled to the memory;
[0034] The processor calls the executable program code stored in the memory and executes the steps of the above-mentioned ViT-based unsupervised domain adaptation image classification method.
[0035] According to the fourth aspect of the present invention, a computer-readable storage medium is provided. The computer-readable storage medium stores computer instructions, and when the computer instructions are called, they are used to execute the steps of the above-mentioned ViT-based unsupervised domain adaptation image classification method.
[0036] The beneficial effects of the present invention are as follows: The present invention provides a ViT-based unsupervised domain adaptation image classification method, which forms cross-domain long-range dependence relationships instead of explicitly modeling domain differences. A domain-level cross-attention mechanism based on global correspondence is introduced under the ViT framework to learn features with cross-domain transfer capabilities. This method directly operates at the domain level, which helps to establish sample correspondence relationships between different domains. A convolutional-transformer feature interaction module is introduced to introduce local features into the vision Transformer, realizing the interaction and integration of global information and local information, and introducing convolutional features without changing the ViT structure. At the same time, through two-way interaction, while further enhancing the semantic representation of the visual odometer, the problem of lack of local information and multi-scale feature interaction in the vision transformer is alleviated. It bridges the gap between patch-based ViT processing and convolutional processing, and this interaction method allows the advantages of both models to be utilized. BRIEF DESCRIPTION OF THE DRAWINGS
[0037] Figure 1 It is a schematic flowchart of a ViT-based unsupervised domain adaptation image classification method according to an embodiment of the present invention.
[0038] Figure 2Schematic diagram of attention of an unsupervised domain adaptation image classification method based on ViT according to an embodiment of the present invention.
[0039] Figure 3 Schematic diagram of feature transformation of an unsupervised domain adaptation image classification method based on ViT according to an embodiment of the present invention. Detailed implementation manners
[0040] To make the objectives, technical solutions and advantages of the present disclosure clearer and more understandable, the following further describes the present disclosure in detail with reference to specific embodiments and the accompanying drawings.
[0041] It should be noted that unless otherwise defined, the technical terms or scientific terms used in one or more embodiments of this application should have the ordinary meanings understood by those of ordinary skill in the art to which the present disclosure belongs. The "first", "second" and similar terms used in one or more embodiments of this application do not denote any order, quantity or importance, but are only used to distinguish different components. Words such as "including" or "comprising" mean that the elements or items appearing before this word cover the elements or items listed after this word and their equivalents, without excluding other elements or items. Words such as "connected" or "coupled" are not limited to physical or mechanical connections, but may include electrical connections, whether direct or indirect. "Upper", "lower", "left", "right", etc. are only used to represent relative positional relationships, and when the absolute position of the object being described changes, the relative positional relationship may also change accordingly.
[0042] Refer to Figures 1 - 3 , an unsupervised domain adaptation image classification method based on ViT in one or more embodiments of this application, the content includes:
[0043] As Figure 1 shown, this embodiment actually provides an unsupervised domain adaptation image classification method based on ViT that fuses domain attention and local features. In UDA (unsupervised domain adaptation), the i-th sample and its corresponding one-hot label are used to represent the labeled source set The j-th sample is used to represent the unlabeled target set n s and n t respectively represent the sizes of samples in the source domain and the target domain. Note that the data in the two domains are sampled from two different distributions, and it is assumed that the two domains share the same label space. The goal is to solve the significant domain discrepancy problem and smoothly transfer knowledge from the source domain to the target domain.
[0044] To extend Transformer to image-based learning tasks, ViT (Vision Transformer) and its variants usually divide each image into N image patches and represent them as feature moments Apply different projections on Z to obtain Q, K and d and d n Denote their dimensions. The self-attention mechanism in ViT is formulated as:
[0045]
[0046] Attention(Z) = AV
[0047] This formula establishes long-range dependencies between each image patch, which enables ViT to identify image regions relevant to classification. Additionally, when Q and K are calculated from different images, the above formula introduces the cross-attention mechanism.
[0048] First, we propose the Domain-level Cross-attention Module (DoM), which forms long-range dependencies across domains instead of explicitly modeling domain differences. Specifically, a domain-level cross-attention module based on global correspondences is introduced under the original ViT framework to learn features with cross-domain transfer ability. This method operates directly at the domain level, helping to establish sample correspondences between different domains. Different from traditional ViT-based UDA methods that directly interact at the patch level or explicitly model distribution differences, we introduce a domain attention module to interact features of different domains at the domain level, directly characterizing cross-domain sample correspondences and capturing the global correspondences between cross-domain sample features, thereby learning transferable features. Among them, the global features are obtained by the Tranformer Encoder. The purpose of the domain-level cross-attention module is to utilize the advantage of the long-range correspondence of the attention mechanism to characterize the correspondences between cross-domain samples.
[0049] Under the UDA assumption, samples from different domains may have potentially significant distribution differences due to domain shift. In this embodiment, we refer to the similarity of samples in the same latent class in different domains as global correspondence. Additionally, we construct domain-level sample correspondences by proposing a domain attention module, where the query comes from the source domain and the key-value pairs come from the target domain. To simulate the independent and learnable feature embeddings of the query and key-value pairs, we use projection operators for projection. The projection operators for the source domain and the target domain are denoted as Proj s (·) and Proj t (·). The query and the key (value) can be expressed as:
[0050]
[0051] Among them, and are the outputs of the i-th stage of ViT.
[0052] As Figure 2 shown, DoM forms the at attention map by cross-domain exchanging sample features between different domains through a domain-level attention mechanism (also known as the attention score matrix), as the cross-domain expression coefficient:
[0053]
[0054] The scaled similarity matrix A characterizes the pairwise similarity between the source sample X s and the target sample X t in the feature space. Reconstruct the target domain feature F according to the attention score matrix A t . If the attention score is high, it indicates that the target domain feature is more similar to the source domain feature, and the target domain feature will receive more attention. The reconstructed target domain feature can be expressed as:
[0055]
[0056] The reconstructed feature is the weighted sum of the keys, and the target domain feature more similar to the source domain feature will receive higher attention. Although there are differences between the source domain and the target domain, the features in different domains of the same category are expected to be more similar, reflecting the global correspondence and the correspondence between samples of the same category. Since is reconstructed based on similarity, so can continue to use the label of F s and take as the new source domain feature. The module obtains the new source domain feature, that is, the sample-level global feature, through the cross-attention mechanism between the source domain and the target domain features. The new source domain feature will replace the original source domain feature and enter the original ViT and subsequent stages. Specifically, the new source domain feature will replace the source domain feature of the previous stage and enter the original ViT to obtain multi-level global features.
[0057] Secondly, we introduce a convolutional-transformer feature interaction module (CoT) to introduce local features into ViT to achieve the interaction and integration of global information and local information. Specifically, the 2D convolutional operator in the interaction module is used to obtain local features. The obtained local features are superimposed on the multi-level global features output by the original ViT. Since the local features and the multi-level global features come from different architectures, the self-attention mechanism is used for integration and interaction and used as the input of the next stage.
[0058] As shown in Figure 3. It introduces convolutional features without changing the ViT structure. At the same time, through bidirectional interaction, while further enhancing the semantic representation of visual odometry, it alleviates the problems of lack of local information and multi-scale feature interaction in ViT.
[0059] In a vision transformer, an input image of size H×W is divided into image patches of size P×P, and the number of patches is N = HW / P 2 . To obtain convolutional features, we first use a projection operator to project the flattened feature sequence of ViT back into the 2D image space.
[0060]
[0061] where D is the feature dimension, C is the number of color channels of the image, and B represents the number of data samples input to the model at one time.
[0062] Then, for the local patches in X ′ , the two-dimensional convolution operator is repeatedly applied to generate local convolutional features. This is achieved by sliding the convolution kernel over the patches, calculating the dot product at each position, and generating a feature map that captures the spatial hierarchy and local patterns within the patches.
[0063]
[0064] The size of the convolution kernel is k (k≥3), and zero padding of size (k - 1) / 2 is added around the perimeter of the small patches. It should be noted that this zero padding is important for keeping the feature size unchanged.
[0065] For the obtained local convolutional features, we further adopt an inverse projection operator in the feature space to project the features back to the original dimension:
[0066]
[0067] After the newly obtained feature map F passes through the convolution operator, it maximally retains the local information of the image. However, due to the differences in architectures, F and Transformer features will show differences in modal representations. To solve this problem, we adopt the self-attention mechanism to unify the convolutional features and the transformed features, enhancing the invariance of the representation to modal differences. Here, the transformed features are the Transformer features, that is, the multi-level global features, which are the features after the new source domain features pass through the original ViT; the convolutional features are the local features.
[0068] Through CoT, we effectively bridge the gap between patch-based ViT processing and convolutional processing. This interaction method allows leveraging the advantages of both models. To further facilitate the interaction between convolutional features and Transformer features, we use residual connections to fuse the outputs of the CoT and ViT layers, addressing the weaknesses of ViT in lacking local information and multi-scale features. By combining global and local information, the semantic information of the output features is further enriched and enhanced.
[0069] After integrating the DoM and CoT modules into ViT, as shown in Figure 1. ViT is divided into N stages, with different patch sizes in each stage.
[0070] For the training process, we adopt a relatively traditional adversarial training method for training, and the loss function is as follows:
[0071]
[0072] Among them, the i-th sample and its corresponding one-hot label represent the labeled source set The j-th sample represents the unlabeled target set n s and n t represent the sizes of samples in the source domain and the target domain respectively, D is the union of D s and D t d i represents the domain label, M is the prediction result of the model for the input sample, λ is the trade-off coefficient, F is the feature extraction network, C is the classifier, is the cross-entropy loss on the labeled source domain samples, is the domain alignment loss based on classical domain adversarial learning:
[0073]
[0074] Among them, represents the i-th sample in the source domain, represents the i-th sample in the target domain, x i represents the i-th input sample, y i represents the true label of the i-th sample in the source domain.
[0075] In another embodiment, an unsupervised domain adaptation image classification system based on ViT is provided, and the system includes a VIT architecture and:
[0076] Domain Cross-Attention Module: Introduce a domain cross-attention module based on global correspondence relationships into the original ViT to interact with sample features between different domains at the domain level, capture the global correspondence relationships between cross-domain sample features, and obtain global features.
[0077] Convolution-Transformer Feature Interaction Module: Obtain local features by introducing a convolution-Transformer feature interaction module into the original ViT; the obtained local features are interacted and superimposed with the global features to obtain fused features.
[0078] Training Module: Integrate the domain cross-attention module and the convolution-Transformer feature interaction module into ViT to form a fused ViT. The input of the fused ViT is the global feature, and the output is the fused feature. The fused ViT is trained in an adversarial training manner to obtain a trained fused ViT.
[0079] Execution Module: Based on the trained fused ViT, perform an image classification task.
[0080] In another embodiment, an electronic device is provided. The device includes:
[0081] A memory storing executable program code;
[0082] A processor coupled to the memory;
[0083] The processor calls the executable program code stored in the memory and executes the steps of the above-mentioned unsupervised domain adaptation image classification method based on ViT.
[0084] In another embodiment, a computer-readable storage medium is provided. The computer-readable storage medium stores computer instructions that, when called, are used to execute the steps of the above-mentioned unsupervised domain adaptation image classification method based on ViT.
[0085] Those skilled in the art should understand that the embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware aspects. Moreover, the present invention can take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk memory, CD-ROM, optical memory, etc.) containing computer-usable program code.
[0086] The present invention will be described with reference to the flowcharts and / or block diagrams of methods, apparatuses (systems), and computer program products according to embodiments of the present invention. It should be understood that each flow and / or block in the flowchart and / or block diagram, and the combination of flows and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to the processor of a general-purpose computer, a special-purpose computer, an embedded computer, or other programmable data processing devices to generate a machine, such that the instructions executed by the processor of the computer or other programmable data processing devices generate a system for implementing the functions specified in one or more flows Figure 1 one or more flows and / or blocks Figure 1 or in one or more blocks.
[0087] These computer program instructions can also be stored in a computer-readable memory that can direct a computer or other programmable data processing device to work in a specific manner, such that the instructions stored in the computer-readable memory generate a manufacture including an instruction system that implements the functions specified in one or more flows Figure 1 one or more flows and / or blocks Figure 1 or in one or more blocks.
[0088] These computer program instructions can also be loaded onto a computer or other programmable data processing device, such that a series of operation steps are executed on the computer or other programmable device to generate a computer-implemented process, so that the instructions executed on the computer or other programmable device provide steps for implementing the functions specified in one or more flows Figure 1 one or more flows and / or blocks Figure 1 or in one or more blocks.
[0089] Although the preferred embodiments of the present invention have been described, those skilled in the art can make additional changes and modifications once they learn the basic inventive concept. Therefore, the appended claims are intended to be construed to include the preferred embodiments as well as all changes and modifications that fall within the scope of the present invention.
[0090] Obviously, those skilled in the art can make various changes and variations to the present invention without departing from the spirit and scope of the present invention. Thus, if these modifications and variations of the present invention fall within the scope of the claims of the present invention and their equivalent technologies, the present invention is also intended to include these modifications and variations.
Claims
1. A ViT-based unsupervised domain adaptation image classification method, characterized in that: It includes the following steps: Step 1: Introduce a domain cross-attention module based on global correspondence into the original ViT, which is used to interact sample features between different domains at the domain level and capture the global correspondence between cross-domain sample features; obtain sample-level global features based on the captured global correspondence between cross-domain sample features, input the sample-level global features into the original ViT, and the original ViT outputs multi-level global features; Step 2: Introduce the convolution-transformer feature interaction module into the original ViT, input the multi-level global features into the convolution-transformer feature interaction module to obtain local features; first interact the obtained local features with the multi-level global features, and then superimpose them to obtain fused features; Step 3: After introducing the domain cross-attention module and the convolution-transformer feature interaction module, the original ViT forms a fused ViT; the fused ViT is iteratively trained using adversarial training to obtain a trained fused ViT; Step 4: Use the trained fusion ViT to perform image classification tasks.
2. The ViT-based unsupervised domain adaptation image classification method according to claim 1, characterized in that: In the step 1, the global correspondence between cross-domain sample features is characterized by an attention score matrix, and the attention score matrix is calculated based on the query generated by the source domain sample features and the key value generated by the target domain sample features, and the target domain features are reconstructed according to the attention score matrix; the reconstructed target domain features continue to use the labels of the source domain features, and the reconstructed target domain features are used as new source domain features, i.e., sample-level global features, and the new source domain features are input into the original ViT, and the original ViT outputs multi-level global features.
3. The unsupervised domain adaptation image classification method based on ViT as claimed in claim 1, characterized in that In the step 2, the process of obtaining the local features is as follows: First, the input image is divided into several image blocks, and the flattened feature sequence of ViT is projected back to the 2D image space using the forward projection operator; Then, the 2D convolution operator is repeatedly applied to the local blocks in the flattened feature sequence of ViT projected back to the 2D space to produce local convolution features; For the obtained local convolution features, a back projection operator is used in the feature space to project them back to the original dimension to obtain new features, which are local features.
4. The ViT-based unsupervised domain adaptation image classification method as claimed in claim 3, characterized in that: The convolution kernel size in the two-dimensional convolution operator is k, and zero padding of size (k-1) / 2 is added around the perimeter of the convolution kernel coverage area to keep the local convolution feature size unchanged.
5. The unsupervised domain adaptation image classification method based on ViT as claimed in claim 3, characterized in that: The feature acquisition process is as follows: after obtaining the local features, the self-attention mechanism is used to interact the local features with the multi-level global features, and then the residual connection is used to superimpose the local features with the multi-level global features to obtain the fused features.
6. The ViT-based unsupervised domain adaptation image classification method according to claim 1, characterized in that: Before executing step 3, the fused ViT is divided into several stages, each of which processes patches of different sizes.
7. The ViT-based unsupervised domain adaptation image classification method according to claim 1, characterized in that: The loss function of adversarial training is as follows: Among them, use the i-th sample and its corresponding one-hot label Represents a labeled source set Use the jth sample represents the unlabeled target set n s and n t Denote the size of samples in the source domain and target domain respectively, and D is D s and D t The union of i represents the domain label, M is the model's prediction result for the input sample, λ is the weight coefficient, F is the feature extraction network, C is the classifier, is the cross entropy loss on labeled source domain samples, is the classic domain alignment loss based on domain adversarial learning: in, represents the i-th sample in the source domain, represents the i-th sample in the target domain, x i represents the i-th input sample, y i represents the true label of the i-th sample in the source domain.
8. An unsupervised domain adaptation image classification system based on ViT, characterized in that: The system includes the VIT architecture and: Domain cross-attention module: In the original ViT, a domain cross-attention module based on global correspondence is introduced to interact sample features between different domains at the domain level, capture the global correspondence between cross-domain sample features, and obtain global features; Convolution-transformer feature interaction module, which introduces the convolution-transformer feature interaction module into the original ViT to obtain local features; the obtained local features interact and superimpose with the global features to obtain fused features; The training module integrates the domain cross-attention module and the convolution-transformer feature interaction module into ViT to form a fused ViT. The input of the fused ViT is the global feature, and the output is the fused feature. The fused ViT is iteratively trained to obtain the trained fused ViT. The execution module performs the image classification task based on the trained fusion ViT.
9. An electronic device, characterized in that: The device comprises: A memory storing executable program code; a processor coupled to the memory; The processor calls the executable program code stored in the memory to execute the steps of the ViT-based unsupervised domain adaptation image classification method as described in any one of claims 1 to 7.
10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer instructions, which, when called, are used to execute the steps of the ViT-based unsupervised domain adaptation image classification method as described in any one of claims 1 to 7.
Citation Information
Patent Citations
Unsupervised domain adaptation method for image classification based on category adaptive model
CN113011456A
Infrared and visible light image double-domain self-attention interactive fusion method
CN117788989A
Cross-domain image classification method based on few-sample unsupervised domain adaptation
CN119649155A
System and method for self-distilled vision transformer for domain generalization
US20240203098A1
