A fully test-time adaptive method based on self-attention scale shift recalibration

Abstract

This invention provides a fully test-time adaptive method based on self-attention scale shift recalibration. The method includes: acquiring a target domain image; inputting the target domain image into a pre-trained source domain model, where each encoder layer of the source domain model contains a factor generation module and a self-attention module; using the embedding layer in the source domain model to convert the target domain image into several image tags, which are then used by the factor generation module to generate scale factors and shift factors; inputting the scale factors and shift factors into the self-attention module to calibrate the query vector, key vector, and value vector in the self-attention module to obtain the target domain model; and inputting the target domain image into the target domain model to obtain the corresponding image classification result. This invention improves the model's adaptability and classification accuracy in the target domain by introducing scale factors and shift factors to dynamically adjust the query vector, key vector, and value vector in the self-attention mechanism.

Description

Technical Field

[0001] This invention relates to the fields of deep learning and computer vision, and more particularly to a fully test-time adaptive method based on self-attention scale offset recalibration. Background Technology

[0002] In many practical applications, machine learning models are typically trained on a specific data distribution (source domain), usually exhibiting good performance in this environment. However, this often requires labeled data and consumes significant computational resources. When these models are deployed to an environment with a different data distribution than during training (target domain), their performance often degrades significantly. This performance degradation is called domain shift, or domain drift. The existence of this phenomenon indicates insufficient robustness and generalization ability of the model when dealing with new environments, especially when the model may fail to handle data distributions or conditions unseen in the target domain. For example, in the field of autonomous driving, a vehicle's perception system may perform well in clear daytime weather, but its performance may drop drastically at night or in foggy environments, leading to potential safety hazards. This problem is particularly prominent in many practical applications, especially in tasks involving complex environmental changes, such as computer vision, natural language processing, and speech recognition.

[0003] To improve the robustness and adaptability of models, various unsupervised domain adaptation methods have been proposed in recent years as effective means of transfer learning. Unsupervised domain adaptation methods do not rely on labeled data from the target domain; instead, they utilize unlabeled data from the target domain to optimize and adjust the model, thereby adapting to the data distribution of the target domain, mitigating the negative impact of domain shift on model performance, and enhancing the model's performance in unknown environments. Currently, unsupervised domain adaptation is mainly divided into two categories: active domain adaptation and passive domain adaptation. Active domain adaptation methods require labeled data from the source domain, but in practical applications, they often face difficulties in data transmission and privacy protection issues. Passive domain adaptation methods, on the other hand, do not rely on source domain data for training, thus requiring the use of the entire target domain test dataset for model adjustment. Full-test-time adaptation methods can dynamically fine-tune the model during the inference phase, adjusting only based on the data of the current batch. This method has greater flexibility and superiority because it can adapt to changes in the target domain data in real time, improving the model's performance in practical applications.

[0004] In practical applications, many scenarios require models to adapt to changes in the target data in real time during the testing phase, without relying on the source domain data or making extensive adjustments to the target domain during training. Adaptive testing during online testing aims to provide a method for dynamically adjusting the model during testing, enabling it to quickly adapt to the data distribution when receiving new test samples and improve the model's performance in the target domain.

[0005] In existing online test-time adaptive methods, models often fail to effectively handle distributional differences caused by domain shifts when processing target domain data. These differences lead to a significant drop in model performance, especially when the query, key, and value vectors in the self-attention mechanism are highly sensitive to specific features of the source domain data. Query vectors, key vectors, and value vectors may change in the target domain, causing the model to fail to accurately capture the contextual relationships of the input data, resulting in lower adaptability and classification accuracy in the target domain.

[0006] Therefore, existing technologies have shortcomings and need to be improved and developed. Summary of the Invention

[0007] The technical problem to be solved by this invention is to provide a fully test-time adaptive method based on self-attention scale offset recalibration, which addresses the above-mentioned deficiencies of the prior art. This method aims to solve the problem that in the prior art, during domain adaptation, the query vector, key vector, and value vector may change in the target domain, causing the model to be unable to accurately capture the contextual relationships of the input data, thus resulting in low adaptability and classification accuracy of the model in the target domain.

[0008] The technical solution adopted by this invention to solve the technical problem is as follows:

[0009] A fully test-time adaptive method based on self-attention scale shift recalibration, wherein the method includes:

[0010] The target domain image is acquired and input into a pre-trained source domain model. Each encoder layer of the source domain model has a factor generation module and a self-attention module.

[0011] The target domain image is converted into several image tags using the embedding layer in the source domain model, and the several image tags are used by the factor generation module to generate a scale factor and an offset factor.

[0012] The scaling factor and offset factor are input into the self-attention module to calibrate the query vector, key vector and value vector in the self-attention module, thereby obtaining the target domain model;

[0013] The target domain image is input into the target domain model to obtain the corresponding image classification result.

[0014] In one embodiment of this application, acquiring a target domain image and inputting the target domain image into a pre-trained source domain model includes:

[0015] Obtain a pre-constructed target domain dataset, which includes several target domain images;

[0016] All target domain images in the target domain dataset are input into the pre-trained source domain model in batches.

[0017] In one embodiment of this application, the factor generation module includes a domain separation network and a factor generation network; the target domain image is converted into several image tags using the embedding layer in the source domain model, and the several image tags are used by the factor generation module to generate scale factors and offset factors, including:

[0018] The target domain image is converted into several image labels using the embedding layer in the source domain model. After the several image labels are processed by the domain separation network, the average value of the processed image labels is calculated to obtain the domain label. The domain label is used to represent the domain offset noise information.

[0019] Obtain the pre-trained class labels in the source domain model, input the domain labels and the class labels into the factor generation network, and obtain the scale factor and offset factor corresponding to the current encoder layer.

[0020] In one embodiment of this application, the domain separation network is trained with the goal of minimizing a pre-built similarity loss function, so as to minimize the distance between the generated domain tags and all image tags.

[0021] In one embodiment of this application, the factor generation network is a lightweight fully connected layer with input and output dimensions.

[0022] In one embodiment of this application, the scaling factor and offset factor are input into the self-attention module to calibrate the query vector, key vector, and value vector in the self-attention module, thereby obtaining a target domain model, including:

[0023] Obtain a pre-constructed total loss function, which includes an entropy loss function and a similarity loss function. The similarity loss function is used to guide the source domain model to capture features of the target domain.

[0024] The scaling factor and offset factor are input into the self-attention module, and the query vector, key vector, and value vector in the self-attention module are calibrated according to the total loss function.

[0025] In one embodiment of this application, several target domain images in each batch are simultaneously input into a pre-trained source domain model, and the source domain model used in the current batch is the target domain model updated in the previous batch.

[0026] The target domain image is input into the target domain model to obtain the corresponding image classification result, including:

[0027] Input all the target domain images in the current batch into the target domain model to obtain the image classification results corresponding to each target domain image in the current batch.

[0028] This application also provides a fully time-adaptive device based on self-attention scale offset recalibration, wherein the device includes:

[0029] The input module is used to acquire the target domain image and input the target domain image into the pre-trained source domain model. Each encoder layer of the source domain model has a factor generation module and a self-attention module.

[0030] The generation module is used to convert the target domain image into several image tags using the embedding layer in the source domain model, and the several image tags are used by the factor generation module to generate a scale factor and a shift factor.

[0031] A calibration module is used to input the scaling factor and offset factor into the self-attention module, calibrate the query vector, key vector and value vector in the self-attention module, and obtain the target domain model.

[0032] The classification module is used to input the target domain image into the target domain model to obtain the corresponding image classification result.

[0033] This application also provides a terminal, comprising: a memory, a processor, and a fully adaptive test-time program based on self-attention scale offset recalibration stored in the memory and executable on the processor, wherein the fully adaptive test-time program based on self-attention scale offset recalibration implements the steps of the fully adaptive test-time method based on self-attention scale offset recalibration as described above when executed by the processor.

[0034] This application also provides a computer-readable storage medium storing a computer program that can be executed to implement the steps of the fully test-time adaptive method based on self-attention scale offset recalibration as described above.

[0035] This invention provides a fully test-time adaptive method based on self-attention scale shift recalibration. The method includes: acquiring a target domain image; inputting the target domain image into a pre-trained source domain model, wherein each encoder layer of the source domain model has a factor generation module and a self-attention module; using the embedding layer in the source domain model to convert the target domain image into several image tags, wherein the several image tags are used by the factor generation module to generate scale factors and shift factors; inputting the scale factors and shift factors into the self-attention module to calibrate the query vector, key vector, and value vector in the self-attention module to obtain the target domain model; and inputting the target domain image into the target domain model to obtain the corresponding image classification result. This invention improves the model's adaptability and classification accuracy in the target domain by introducing scale factors and shift factors to dynamically adjust the query vector, key vector, and value vector in the self-attention mechanism. Attached Figure Description

[0036] Figure 1 This is a flowchart of a preferred embodiment of the fully adaptive test-time method based on self-attention scale offset recalibration in this invention;

[0037] Figure 2 This is a framework illustration of the fully adaptive test-time method based on self-attention scale offset recalibration in this invention;

[0038] Figure 3 This is a functional principle block diagram of a preferred embodiment of the fully adaptive test-time device based on self-attention scale offset recalibration in this invention;

[0039] Figure 4 This is a functional principle block diagram of a preferred embodiment of the terminal in this invention. Detailed Implementation

[0040] To make the objectives, technical solutions, and advantages of this invention clearer and more explicit, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the invention and are not intended to limit the invention.

[0041] Existing methods typically rely on source domain data, generated pseudo-labels, or require access to the entire test dataset for model tuning.

[0042] The problems with these methods include:

[0043] First, dependence on source domain data: Existing methods still need to access source domain data during the testing phase, which increases the dependence on source domain data. This not only raises data privacy issues but also increases storage and transmission costs. Model updates and adjustments are relatively complex, and the model is more susceptible to the negative impact of distribution differences between the source and target domains, thereby reducing the model's performance in the target domain.

[0044] Second, the accuracy of pseudo-label generation is crucial: existing methods rely on generating pseudo-labels to fine-tune the model. However, due to changes in the distribution of data in the target domain, the pseudo-label generation process may not be accurate enough and may even introduce noise, which limits the model's adaptability and may even reduce the overall performance of the model.

[0045] Third, lack of real-time adjustment capability: Existing models lack flexibility when facing domain drift during the inference stage and cannot adjust parameters in real time to adapt to the data distribution of the target domain, resulting in limited adaptability and coping ability.

[0046] In other words, in many real-world scenarios, the distributions of training and testing data differ significantly. This difference directly impacts the model's classification performance, leading to instability on new data. To address this issue, this invention proposes a network model for adaptively handling domain shift during testing based on scale-offset factor calibration, aiming to address the common domain drift problem in practical applications. Utilizing the target domain's distribution offset information, the scale and offset factors of the query, key, and value in the Transformer model's self-attention module are dynamically adjusted during testing. Through this calibration mechanism, the model can better adapt to the target domain's data distribution, effectively mitigating the impact of domain shift on classification accuracy, thereby significantly improving the performance and robustness of the image classification model during testing, especially when there are significant differences in data distribution.

[0047] Please see Figure 1 , Figure 1 This is a flowchart of the fully test-time adaptive method based on self-attention scale offset recalibration in this invention. Figure 1 As shown in the embodiments of the present invention, the fully adaptive test-time method based on self-attention scale offset recalibration includes:

[0048] Step S100: Obtain the target domain image and input the target domain image into the pre-trained source domain model. Each encoder layer of the source domain model has a factor generation module and a self-attention module.

[0049] Step S200: The target domain image is converted into several image tags using the embedding layer in the source domain model. The several image tags are then used by the factor generation module to generate a scale factor and an offset factor.

[0050] Step S300: Input the scaling factor and offset factor into the self-attention module, calibrate the query vector, key vector and value vector in the self-attention module, and obtain the target domain model;

[0051] Step S400: Input the target domain image into the target domain model to obtain the corresponding image classification result.

[0052] This application presents an online testing-time adaptive method for Transformer network models, addressing the domain shift problem between the source and target domains, primarily for image classification. This method eliminates the need for source domain data to aid unsupervised domain adaptation, as well as the need to generate pseudo-labels and access all test data; it processes data based solely on the current batch during testing. Specifically, in each Transformer network layer, a domain separation network is learned to extract domain tokens, representing domain shift information learned from all image tokens. These domain tokens, along with class tokens, are used to learn the conditional scale and shift factor in the self-attention module of the factor generation network. When applying the Transformer network model to a new target domain, local linear variations parameterized by the conditional scale and shift factor can be used to calibrate the query, key, and value vectors in the self-attention module of each layer, thereby improving the Transformer model's adaptability and classification accuracy in the target domain. The self-attention module is the neural network structure that implements the self-attention mechanism.

[0053] The fully adaptive method provided in this application, by introducing a scaling factor and a shift factor, dynamically adjusts the query vector, key vector, and value vector in the self-attention mechanism, which significantly improves the classification performance and robustness of the Transformer model when facing domain drift, ensuring that the model can process data from different domains more stably and reliably in real-world application scenarios.

[0054] In one embodiment of this application, step S100 specifically includes:

[0055] Step S110: Obtain a pre-constructed target domain dataset, which includes several target domain images;

[0056] Step S120: Input all target domain images in the target domain dataset into the pre-trained source domain model in batches.

[0057] Specifically, a base model that has already been trained on the source dataset is loaded from a pre-trained model library. The model can be a Vision Transformer network with a standard architecture and parameters θ. S The model can be represented as: Where y represents the predicted output, X s Represents source domain data, Representation model.

[0058] The embodiments of this application use weights trained on the source dataset to initialize the model, ensuring that the model can perform effective reasoning based on pre-trained knowledge when processing target data.

[0059] In this embodiment, the factor generation module includes a domain separation network and a factor generation network. Step S200 specifically includes:

[0060] Step S210: The target domain image is converted into several image tags using the embedding layer in the source domain model. After the several image tags are processed by the domain separation network, the average value of the processed several image tags is calculated to obtain the domain tag. The domain tag is used to represent the domain offset noise information.

[0061] Step S220: Obtain the pre-trained class labels in the source domain model, input the domain labels and the class labels into the factor generation network to obtain the scale factor and offset factor corresponding to the current encoder layer.

[0062] Specifically, by adding a domain separation network to each Transformer layer, this application effectively extracts domain drift information from the current batch by learning a domain label in each Transformer layer. This eliminates the need to access source domain data or generate pseudo-labels, avoiding privacy issues and storage costs, reducing dependence on source domain data, and improving model performance in the target domain. Furthermore, by using domain and class labels, the factor generator network learns conditional scaling and shift factors in the self-attention module and applies these factors to the self-attention module of each layer. This method enables the model to dynamically adjust to domain changes during the inference phase, thereby enhancing its flexibility and adaptability in new target domains.

[0063] In one embodiment of this application, the domain separation network is trained with the goal of minimizing a pre-built similarity loss function, so as to minimize the distance between the generated domain tags and all image tags.

[0064] Specifically, such as Figure 2As shown, a domain separation network is trained to extract representative domain features from a set of image labels; these features are called domain labels. Domain labels aggregate domain-specific information in each layer to capture domain drift. Specifically, a domain label is obtained by averaging multiple image labels passed through the same domain separation network, resulting in a label that represents the domain shift noise information, as shown in the following formula:

[0065]

[0066] Where l represents the l-th layer in the Transformer network, D 1 This represents obtaining the domain label of the l-th level. The nth image label feature represents the l-th layer of the model, Avg[.] represents average pooling, and Φ 1 Representative domain separation network.

[0067] To ensure that the domain labels effectively represent the domain shift information obtained from each image label, the network is trained by minimizing the similarity loss function, thus minimizing the distance between the generated domain labels and all image labels. The model's l-th layer is used to compute the distance between two image labels. and Similarity between

[0068]

[0069] Loss function that minimizes similarity loss It can be represented as:

[0070]

[0071] in, It is the sign of the similarity loss function, which depends on the parameter θ. t Given input data x. The goal of the loss function is to minimize the similarity between labels, thus ensuring that the domain labeling generated by the domain separation network represents the domain drift information learned from all image labels. External summation term. This term represents the summation and averaging of all layers *l* in the model, where *L* is the total number of self-attention modules in the model. This term ensures that the loss function considers the similarity of all layers, not just a single layer, thus providing a more comprehensive representation of domain drift. (Intermediate summation term) This represents the summation and averaging of the similarities among all image tags in each layer. Here, N is the number of image tags in each layer. This is used to calculate the average similarity between all label pairs in each layer. denoted as represents the cosine similarity between the j-th and k-th image labels in layer l. A higher cosine similarity indicates greater similarity between the two labels in the feature space. To ensure that the neighborhood labels represent all image labels as comprehensively as possible, the loss function aims to minimize the average of these similarity values. The negative sign at the beginning indicates the desire to maximize the cosine similarity. The average value of the domain is used to minimize (i.e., maximize its negative value), effectively increasing similarity so that the domain labels better reflect the relationships between image labels. This method allows samples from the same domain, even those belonging to different categories, to cluster together in the feature space. This learned domain label will then be used to generate scale and shift factors.

[0072] In one embodiment of this application, the factor generation network is a lightweight fully connected layer with input and output dimensions.

[0073] Specifically, such as Figure 2 As shown. The factor generation network is a key component in this method, dynamically adjusting the visual Transformer in each batch to adapt to domain changes. This network mitigates the impact of domain changes by generating a scaling factor γ and a shift factor β to recalibrate the query vector, key vector, and value vector in the self-attention module. Specifically, in the implementation, each layer of the factor generation network Ψ... l It is a lightweight fully connected layer with an input dimension d and an output dimension 2d. Let D be the domain. l and class tag C l As input, the output is the scale factor γ of the l-th layer. l and offset factor β l :

[0074] [γ l ,β l ]=Ψ l (D l C l )

[0075] The output scaling factor and offset factor are then applied to the query vector, key vector, and value vector of the self-attention module.

[0076] In this embodiment of the application, step S300 specifically includes:

[0077] Step S310: Obtain the pre-constructed total loss function, which includes an entropy loss function and a similarity loss function. The similarity loss function is used to guide the source domain model to capture the features of the target domain.

[0078] Step S320: Input the scaling factor and offset factor into the self-attention module, and calibrate the query vector, key vector and value vector in the self-attention module according to the total loss function.

[0079] Specifically, to address the impact of domain changes on the self-attention module and make the query, key, and value vectors adapt to domain changes, this method introduces a scale factor γ and an offset factor β for adjusting the query, key, and value vectors. The specific calculations are as follows (self-attention recalibration in Figure 2 ):

[0080]

[0081] where γ q , γ k , γ v are the scale factors for the query component, key component, and value component, and β q , β k , β v are the corresponding offset factors. Based on the scale factor γ and the offset factor β, the query vector Q, key vector K, and value vector V are adjusted to Q ′ , K ′ , V ′ . In this method, the scale factor and the offset factor are shared among the query, key, and value components to address the domain change problem. Then, the self-attention operation is performed: where d represents the dimension of the features.

[0082]

[0083] In the test phase, the widely used entropy loss is selected to guide the adaptation process. For the samples x in the current mini-batch B t , the reliable entropy loss is defined as:

[0084]

[0085] where I is an exponential mask defined as I = I[E(θ t ; x) < E0], indicating that only samples with entropy greater than the threshold E0 are optimized. E(θ t ; x) represents the entropy value of the sample x, measuring the uncertainty of the model prediction. E0 is a manually set threshold parameter used to control which samples will be used to update the model to avoid the model deviating too much from the original pre-trained model. In addition to the entropy loss, the embodiments of this application also combine the similarity loss to promote the domain separation network to learn domain-specific information, thus forming the following total loss function:

[0086]

[0087] where, as a balancing parameter, represents the weight relationship between the entropy loss and the similarity loss. ∑I represents the sum of the entropy masks for the entire batch. The similarity loss This is used to facilitate domain-separated networks in learning domain-specific information, ensuring that the model can better capture the features of the target domain, thereby improving its adaptability.

[0088] By introducing scale and offset factors at each Transformer layer, the effects of domain variations are gradually mitigated. This incremental adjustment enables the model to recover the original feature representation, thereby maintaining performance across different domains.

[0089] In one embodiment of this application, several target domain images in each batch are simultaneously input into a pre-trained source domain model, and the source domain model used in the current batch is the target domain model updated in the previous batch. Step S400 specifically involves: inputting all the target domain images in the current batch into the target domain model to obtain the image classification results corresponding to each target domain image in the current batch.

[0090] This application embodiment uses the target domain model obtained from the previous batch update as the source domain model used in the current batch, enabling the visual Transformer to be dynamically adjusted in each batch to adapt to domain changes.

[0091] Thus, this application improves the performance of the Transformer model in online test-time adaptation by calibrating the adaptive scaling and shift factors in the self-attention module. Specifically, firstly, a lightweight neural network called a domain separation network is introduced, which generates a domain label representing the domain shift from all image labels in each layer. This domain label effectively captures domain-specific features, facilitating the learning of improved scaling and shift factors. Secondly, a factor generator network is added, which generates these recalibration factors conditioned on the domain label and class label. Thirdly, a novel Transformer network self-attention module is proposed, which recalibrates the query features, key features, and value features of test samples online during target domain inference.

[0092] Experimental results show that the embodiments of this application significantly improve the domain adaptation performance of the Transformer model, outperforming current methods in domain adaptation when tested online across multiple commonly used benchmark datasets.

[0093] In one embodiment, such as Figure 3 As shown, based on the above-described fully adaptive test-time method based on self-attention scale shift recalibration, the present invention also provides a fully adaptive test-time device based on self-attention scale shift recalibration, comprising:

[0094] The input module 100 is used to acquire a target domain image and input the target domain image into a pre-trained source domain model. Each encoder layer of the source domain model has a factor generation module and a self-attention module.

[0095] The generation module 200 is used to convert the target domain image into several image tags using the embedding layer in the source domain model, and the several image tags are used by the factor generation module to generate a scale factor and an offset factor.

[0096] The calibration module 300 is used to input the scaling factor and offset factor into the self-attention module, calibrate the query vector, key vector and value vector in the self-attention module, and obtain the target domain model.

[0097] The classification module 400 is used to input the target domain image into the target domain model to obtain the corresponding image classification result.

[0098] Figure 4 A schematic diagram of the structure of a terminal provided in an embodiment of this application. The terminal may include:

[0099] The memory 501, the processor 502, and the computer program stored on the memory 501 and capable of running on the processor 502.

[0100] When the processor 502 executes the program, it implements the fully adaptive test-time method based on self-attention scale offset recalibration provided in the above embodiments.

[0101] Furthermore, the terminal also includes:

[0102] Communication interface 503 is used for communication between memory 501 and processor 502.

[0103] The memory 501 is used to store computer programs that can run on the processor 502.

[0104] The memory 501 may include high-speed RAM memory, and may also include non-volatile memory, such as at least one disk storage device.

[0105] If the memory 501, processor 502, and communication interface 503 are implemented independently, they can be interconnected via a bus to communicate with each other. The bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus, etc. Buses can be categorized as address buses, data buses, control buses, etc. For ease of representation, only one line is used in the diagram, but this does not imply that there is only one bus or one type of bus.

[0106] Optionally, in a specific implementation, if the memory 501, processor 502, and communication interface 503 are integrated on a single chip, then the memory 501, processor 502, and communication interface 503 can communicate with each other through an internal interface.

[0107] Processor 502 may be a central processing unit (CPU), an application specific integrated circuit (ASIC), or one or more integrated circuits configured to implement the embodiments of this application.

[0108] This embodiment also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the above-described fully test-time adaptive method based on self-attention scale offset recalibration.

[0109] In the description of this specification, the references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of this application. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples. Moreover, without contradiction, those skilled in the art can combine and integrate the different embodiments or examples described in this specification, as well as the features of different embodiments or examples.

[0110] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one of that feature. In the description of this application, "N" means at least two, such as two, three, etc., unless otherwise explicitly specified.

[0111] Any process or method description in the flowchart or otherwise herein can be understood as representing a module, segment, or portion of code comprising one or N executable instructions for implementing custom logic functions or processes, and the scope of the preferred embodiments of this application includes additional implementations in which functions may be performed not in the order shown or discussed, including substantially simultaneously or in reverse order depending on the functions involved, as should be understood by those skilled in the art to which embodiments of this application pertain.

[0112] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as a ordered list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by, or in conjunction with, an instruction execution system, apparatus, or device (such as a computer-based system, a processor-included system, or other system that can read and execute instructions from or in conjunction with such an instruction execution system, apparatus, or device). For the purposes of this specification, "computer-readable medium" can be any means that can contain, store, communicate, propagate, or transmit programs for use by or in conjunction with an instruction execution system, apparatus, or device. More specific examples (a non-exhaustive list) of computer-readable media include: an electrical connection having one or more wires (electronic device), a portable computer disk drive (magnetic device), random access memory (RAM), read-only memory (ROM), erasable and editable read-only memory (EPROM or flash memory), fiber optic devices, and portable optical disc read-only memory (CDROM). In addition, computer-readable media can even be paper or other suitable media on which programs can be printed, because programs can be obtained electronically by optically scanning paper or other media, then editing, interpreting or otherwise processing them as necessary, and then storing them in computer memory.

[0113] It should be understood that the various parts of this application can be implemented using hardware, software, firmware, or a combination thereof. In the above embodiments, the N steps or methods can be implemented using software or firmware stored in memory and executed by a suitable instruction execution system. If implemented in hardware, as in another embodiment, it can be implemented using any one or a combination of the following techniques known in the art: discrete logic circuits having logic gates for implementing logical functions on data signals, application-specific integrated circuits (ASICs) having suitable combinational logic gates, programmable gate arrays (PGAs), field-programmable gate arrays (FPGAs), etc.

[0114] Those skilled in the art will understand that all or part of the steps of the methods described in the above embodiments can be implemented by a program instructing related hardware. The program can be stored in a computer-readable storage medium, and when executed, it includes one or a combination of the steps of the method embodiments.

[0115] Furthermore, the functional units in the various embodiments of this application can be integrated into a processing module, or each unit can exist physically separately, or two or more units can be integrated into a module. The integrated module can be implemented in hardware or as a software functional module. If the integrated module is implemented as a software functional module and sold or used as an independent product, it can also be stored in a computer-readable storage medium.

[0116] The storage medium mentioned above can be a read-only memory, a disk, or an optical disk, etc. Although embodiments of this application have been shown and described above, it is understood that the above embodiments are exemplary and should not be construed as limiting this application. Those skilled in the art can make changes, modifications, substitutions, and variations to the above embodiments within the scope of this application.

[0117] In summary, this invention discloses a fully test-time adaptive method based on self-attention scale shift recalibration. The method includes: acquiring a target domain image; inputting the target domain image into a pre-trained source domain model, wherein each encoder layer of the source domain model has a factor generation module and a self-attention module; using the embedding layer in the source domain model to convert the target domain image into several image tags, wherein the several image tags are used by the factor generation module to generate scale factors and shift factors; inputting the scale factors and shift factors into the self-attention module to calibrate the query vector, key vector, and value vector in the self-attention module to obtain the target domain model; and inputting the target domain image into the target domain model to obtain the corresponding image classification result. This invention improves the model's adaptability and classification accuracy in the target domain by introducing scale factors and shift factors to dynamically adjust the query vector, key vector, and value vector in the self-attention mechanism.

[0118] It should be understood that the application of the present invention is not limited to the examples above. Those skilled in the art can make improvements or modifications based on the above description, and all such improvements and modifications should fall within the protection scope of the appended claims.

Claims

1. A fully test-time adaptive method based on self-attention scale shift recalibration, characterized in that, The method includes: The target domain image is acquired and input into a pre-trained source domain model. Each encoder layer of the source domain model has a factor generation module and a self-attention module. The target domain image is converted into several image tags using the embedding layer in the source domain model, and the several image tags are used by the factor generation module to generate a scale factor and an offset factor. The scaling factor and offset factor are input into the self-attention module to calibrate the query vector, key vector and value vector in the self-attention module, thereby obtaining the target domain model; The target domain image is input into the target domain model to obtain the corresponding image classification result; The factor generation module includes a domain separation network and a factor generation network; it uses the embedding layer in the source domain model to convert the target domain image into several image tags, and the factor generation module generates scale factors and offset factors from these image tags, including: The target domain image is converted into several image labels using the embedding layer in the source domain model. After the several image labels are processed by the domain separation network, the average value of the processed image labels is calculated to obtain the domain label. The domain label is used to represent the domain offset noise information. Obtain the pre-trained class labels in the source domain model, input the domain labels and the class labels into the factor generation network, and obtain the scale factor and offset factor corresponding to the current encoder layer.

2. The fully adaptive test-time method based on self-attention scale offset recalibration according to claim 1, characterized in that, Acquiring the target domain image and inputting the target domain image into a pre-trained source domain model includes: Obtain a pre-constructed target domain dataset, which includes several target domain images; All target domain images in the target domain dataset are input into the pre-trained source domain model in batches.

3. The fully adaptive test-time method based on self-attention scale offset recalibration according to claim 1, characterized in that, The domain separation network is trained with the goal of minimizing a pre-built similarity loss function, so as to minimize the distance between the generated domain labels and all image labels.

4. The fully adaptive test-time method based on self-attention scale offset recalibration according to claim 1, characterized in that, The factor generation network is a lightweight fully connected layer with input and output dimensions.

5. The fully adaptive test-time method based on self-attention scale offset recalibration according to claim 3, characterized in that, The scaling factor and offset factor are input into the self-attention module to calibrate the query vector, key vector, and value vector in the self-attention module, thereby obtaining the target domain model, including: Obtain a pre-constructed total loss function, which includes an entropy loss function and a similarity loss function. The similarity loss function is used to guide the source domain model to capture features of the target domain. The scaling factor and offset factor are input into the self-attention module, and the query vector, key vector, and value vector in the self-attention module are calibrated according to the total loss function.

6. The fully adaptive test-time method based on self-attention scale offset recalibration according to claim 2, characterized in that, Several target domain images in each batch are simultaneously input into a pre-trained source domain model. The source domain model used in the current batch is the target domain model updated in the previous batch. The target domain image is input into the target domain model to obtain the corresponding image classification result, including: Input all the target domain images in the current batch into the target domain model to obtain the image classification results corresponding to each target domain image in the current batch.

7. A fully adaptive test-time device based on self-attention scale offset recalibration, characterized in that, The device includes: The input module is used to acquire the target domain image and input the target domain image into the pre-trained source domain model. Each encoder layer of the source domain model has a factor generation module and a self-attention module. The generation module is used to convert the target domain image into several image tags using the embedding layer in the source domain model, and the several image tags are used by the factor generation module to generate a scale factor and a shift factor. A calibration module is used to input the scaling factor and offset factor into the self-attention module, calibrate the query vector, key vector and value vector in the self-attention module, and obtain the target domain model. The classification module is used to input the target domain image into the target domain model to obtain the corresponding image classification result; The factor generation module includes a domain separation network and a factor generation network; it uses the embedding layer in the source domain model to convert the target domain image into several image tags, and the factor generation module generates scale factors and offset factors from these image tags, including: The target domain image is converted into several image labels using the embedding layer in the source domain model. After the several image labels are processed by the domain separation network, the average value of the processed image labels is calculated to obtain the domain label. The domain label is used to represent the domain offset noise information. Obtain the pre-trained class labels in the source domain model, input the domain labels and the class labels into the factor generation network, and obtain the scale factor and offset factor corresponding to the current encoder layer.

8. A terminal, characterized in that, include: The memory, the processor, and a fully adaptive test-time program based on self-attention scale offset recalibration stored in the memory and executable on the processor, wherein the fully adaptive test-time program based on self-attention scale offset recalibration, when executed by the processor, implements the steps of the fully adaptive test-time method based on self-attention scale offset recalibration as described in any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that can be executed to implement the steps of the fully test-time adaptive method based on self-attention scale offset recalibration as described in any one of claims 1 to 6.

Images