Full-test adaptive method based on self-attention scale offset recalibration

Abstract

The invention provides a self-attention scale offset recalibration-based self-adaption method during a complete test, which comprises the following steps of: acquiring a target domain image, inputting the target domain image into a pre-trained source domain model, and arranging a factor generation module and a self-attention module in each encoder layer of the source domain model; converting the target domain image into a plurality of picture marks by using an embedded layer in the source domain model, and generating scale factors and offset factors by the plurality of picture marks through a factor generation module; inputting the scale factor and the offset factor into a self-attention module, and calibrating a query vector, a key vector and a value vector in the self-attention module to obtain a target domain model; and inputting the target domain image into the target domain model to obtain a corresponding image classification result. According to the method, the scale factor and the offset factor are introduced, the polling vector, the key vector and the value vector in the self-attention mechanism are dynamically adjusted, and the adaptability and the classification accuracy of the model in the target domain are improved.

Description

Technical Field

[0001] The present invention relates to the fields of deep learning and computer vision, and in particular to a fully test-time adaptive method based on self-attention scale shift recalibration. Background Art

[0002] In many practical application scenarios, machine learning models are usually trained on a specific data distribution (source domain) and usually show good performance in this environment, but this often requires labeled data and consumes a lot of computing resources. However, when these models are deployed to another environment (target domain) with a different data distribution from that during training, their performance often degrades significantly. This performance degradation phenomenon is called domain shift, or domain drift. The existence of this phenomenon indicates that the model is insufficient in robustness and generalization ability when dealing with new environments, especially in data distributions or conditions that have not been seen in the target domain, the model may not be able to handle it correctly. For example, in the field of autonomous driving, the vehicle's perception system may perform well in clear weather during the day, but at night or in foggy environments, the performance of the perception system may drop significantly, leading to potential safety hazards. This problem is extremely prominent in many practical applications, especially in tasks involving complex environmental changes, such as computer vision, natural language processing, and speech recognition.

[0003] In order to improve the robustness and adaptability of the model, a variety of unsupervised domain adaptation methods have been proposed in recent years as an effective means of transfer learning. Unsupervised domain adaptation methods do not rely on labeled data in the target domain, but use unlabeled data in the target domain to optimize and adjust the model, so as to adapt to the data distribution of the target domain, alleviate the negative impact of domain shift on model performance, and enhance the performance of the model in unknown environments. At present, unsupervised domain adaptation is mainly divided into two categories: active domain adaptation and passive domain adaptation. Active domain adaptation methods require labeled data in the source domain, but often face data transmission difficulties and privacy protection issues in practical applications. Passive domain adaptation methods do not rely on source domain data for training, so they need to use the entire target domain test data set to adjust the model. The fully test-time adaptive method can dynamically fine-tune the model in the inference stage, relying only on the current batch of data for adjustment. This method has higher flexibility and superiority because it can adapt to changes in target domain data in real time and improve the performance of the model in practical applications.

[0004] In practical applications, many application scenarios require that the model can adapt to changes in the target data in real time during the testing phase, without relying on the source domain data or making a lot of adjustments to the target domain during the training phase. Online test-time adaptation aims to provide a method to dynamically adjust the model during the testing process, which can quickly adapt to the data distribution when receiving new test samples and improve the performance of the model in the target domain.

[0005] In existing online test-time adaptation methods, the model is usually unable to effectively cope with the distribution difference caused by domain shift when processing target domain data. This difference can lead to a significant drop in model performance, especially when the query, key, and value vectors in the self-attention mechanism are highly sensitive to specific features of the source domain data. The query vector, key vector, and value vector may change in the target domain, causing the model to be unable to accurately capture the contextual relationship of the input data, which in turn leads to low adaptability and classification accuracy of the model in the target domain.

[0006] Therefore, the prior art has defects and needs to be improved and developed. Summary of the invention

[0007] The technical problem to be solved by the present invention is that, in view of the above-mentioned defects of the prior art, a fully test-time adaptive method based on self-attention scale offset recalibration is provided, aiming to solve the problem in the prior art that during domain adaptation, the query vector, key vector and value vector may change in the target domain, resulting in the model being unable to accurately capture the contextual relationship of the input data, thereby resulting in low adaptability and classification accuracy of the model in the target domain.

[0008] The technical solution adopted by the present invention to solve the technical problem is as follows:

[0009] A fully test-time adaptive method based on self-attention scale shift recalibration, wherein the method comprises:

[0010] Acquire a target domain image, and input the target domain image into a pre-trained source domain model, wherein each encoder layer of the source domain model has a factor generation module and a self-attention module;

[0011] The target domain image is converted into a plurality of picture tags by using the embedding layer in the source domain model, and the plurality of picture tags are generated into a scale factor and an offset factor by the factor generation module;

[0012] Inputting the scale factor and the offset factor into the self-attention module, calibrating the query vector, the key vector, and the value vector in the self-attention module, and obtaining a target domain model;

[0013] The target domain image is input into the target domain model to obtain a corresponding image classification result.

[0014] In one embodiment of the present application, obtaining a target domain image and inputting the target domain image into a pre-trained source domain model includes:

[0015] Acquire a pre-constructed target domain dataset, where the target domain dataset includes a plurality of target domain images;

[0016] All target domain images in the target domain dataset are input into the pre-trained source domain model in batches.

[0017] In one embodiment of the present application, the factor generation module includes a domain separation network and a factor generation network; the target domain image is converted into a plurality of picture tags using the embedding layer in the source domain model, and the plurality of picture tags are generated into a scale factor and an offset factor by the factor generation module, including:

[0018] The target domain image is converted into a plurality of picture tags by using the embedding layer in the source domain model, and after the plurality of picture tags are processed by the domain separation network, the processed plurality of picture tags are averaged to obtain domain tags, where the domain tags are used to represent domain offset noise information;

[0019] The pre-trained class label in the source domain model is obtained, and the domain label and the class label are input into the factor generation network to obtain the scale factor and the offset factor corresponding to the current encoder layer.

[0020] In one embodiment of the present application, the domain separation network is trained with the goal of minimizing a pre-constructed similarity loss function, so as to minimize the distance between the generated domain label and all the image labels.

[0021] In one embodiment of the present application, the factor generation network is a lightweight fully connected layer having an input dimension and an output dimension.

[0022] In one embodiment of the present application, the scale factor and the offset factor are input into the self-attention module, and the query vector, the key vector, and the value vector in the self-attention module are calibrated to obtain the target domain model, including:

[0023] Obtaining a pre-built total loss function, where the total loss function includes: an entropy loss function and a similarity loss function, where the similarity loss function is used to guide a source domain model to capture features of a target domain;

[0024] The scale factor and the offset factor are input into the self-attention module, and the query vector, the key vector, and the value vector in the self-attention module are calibrated according to the total loss function.

[0025] In one embodiment of the present application, a plurality of target domain images of each batch are simultaneously input into a pre-trained source domain model, and the source domain model used in the current batch is the target domain model updated in the previous batch;

[0026] Inputting the target domain image into the target domain model to obtain a corresponding image classification result includes:

[0027] All the target domain images in the current batch are input into the target domain model to obtain image classification results corresponding to each target domain image in the current batch.

[0028] The present application also provides a fully test-time adaptive device based on self-attention scale shift recalibration, wherein the device comprises:

[0029] An input module, used for acquiring a target domain image, and inputting the target domain image into a pre-trained source domain model, wherein each encoder layer of the source domain model has a factor generation module and a self-attention module;

[0030] A generating module, configured to convert the target domain image into a plurality of picture tags by using the embedding layer in the source domain model, wherein the plurality of picture tags are generated into a scale factor and an offset factor by the factor generating module;

[0031] A calibration module, used for inputting the scale factor and the offset factor into the self-attention module, calibrating the query vector, the key vector and the value vector in the self-attention module, and obtaining a target domain model;

[0032] The classification module is used to input the target domain image into the target domain model to obtain a corresponding image classification result.

[0033] The present application also provides a terminal, which includes: a memory, a processor, and a fully test-time adaptive program based on self-attention scale offset recalibration stored in the memory and executable on the processor, wherein the fully test-time adaptive program based on self-attention scale offset recalibration implements the steps of the fully test-time adaptive method based on self-attention scale offset recalibration as described above when executed by the processor.

[0034] The present application also provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and the computer program can be executed to implement the steps of the fully test-time adaptive method based on self-attention scale offset recalibration as described above.

[0035] The present invention provides a fully test-time adaptive method based on self-attention scale offset recalibration, the method comprising: obtaining a target domain image, inputting the target domain image into a pre-trained source domain model, each encoder layer of the source domain model having a factor generation module and a self-attention module; using the embedding layer in the source domain model to convert the target domain image into a number of picture tags, the number of picture tags are generated by the factor generation module. The scale factor and the offset factor are input into the self-attention module, and the query vector, the key vector and the value vector in the self-attention module are calibrated to obtain the target domain model; the target domain image is input into the target domain model to obtain the corresponding image classification result. The present invention introduces the scale factor and the offset factor to dynamically adjust the query vector, the key vector and the value vector in the self-attention mechanism, thereby improving the adaptability and classification accuracy of the model in the target domain. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] Figure 1 is a flow chart of a preferred embodiment of a fully test-time adaptive method based on self-attention scale shift recalibration in the present invention;

[0037] Figure 2 is a framework diagram illustrating a fully test-time adaptive method based on self-attention scale shift recalibration in the present invention;

[0038] Figure 3 is a functional principle block diagram of a preferred embodiment of a fully test-time adaptive device based on self-attention scale shift recalibration in the present invention;

[0039] Figure 4 It is a functional principle block diagram of a preferred embodiment of the terminal in the present invention. DETAILED DESCRIPTION

[0040] In order to make the purpose, technical solution and advantages of the present invention clearer and more specific, the present invention is further described in detail below with reference to the accompanying drawings and examples. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention.

[0041] Existing methods usually rely on source domain data, rely on generated pseudo labels, or require access to the entire test dataset for model tuning.

[0042] Problems with these approaches include:

[0043] First, dependence on source domain data: Existing methods still need to access source domain data during the testing phase, which increases the dependence on source domain data. Not only does it cause data privacy issues, it also increases storage and transmission costs. The update and adjustment of the model is relatively complex, and it makes the model more susceptible to the negative impact of the distribution differences between the source domain and the target domain, thereby reducing the performance of the model in the target domain.

[0044] Second, reliance on the accuracy of pseudo-label generation: Existing methods rely on generating pseudo-labels to fine-tune the model. However, due to changes in the distribution of target domain data, the pseudo-label generation process may not be accurate enough and may introduce noise, which limits the adaptability of the model and may even reduce the overall performance of the model.

[0045] Third, lack of real-time adjustment capabilities: Existing models lack flexibility when facing domain drift during the inference phase and are unable to adjust parameters in real time to adapt to the data distribution of the target domain, resulting in limited adaptability and response capabilities.

[0046] That is to say, in many practical scenarios, there are significant differences in the distribution of training data and test data. This difference will directly affect the classification performance of the model, resulting in unstable performance on new data. In order to solve this problem, the present invention proposes a network model that adaptively handles domain shift during testing based on scale-offset factor calibration, which aims to deal with the common domain drift problem in practical applications. Utilizing the distribution offset information of the target domain, the scale and offset factor of the query (Query), key (Key), and value (Value) in the self-attention module of the Transformer model are dynamically adjusted during testing. Through this calibration mechanism, the model can better adapt to the data distribution of the target domain, effectively reduce the impact of domain shift on classification accuracy, thereby significantly improving the performance and robustness of the image classification model during testing, especially when there are significant differences in data distribution.

[0047] See also Figure 1 , Figure 1 , which is a flow chart of a fully test-time adaptive method based on self-attention scale shift recalibration in the present invention. Figure 1 As shown, the fully test-time adaptive method based on self-attention scale shift recalibration according to the embodiment of the present invention includes:

[0048] Step S100: acquiring a target domain image, and inputting the target domain image into a pre-trained source domain model, wherein each encoder layer of the source domain model has a factor generation module and a self-attention module;

[0049] Step S200, converting the target domain image into a plurality of picture tags by using the embedding layer in the source domain model, and generating scale factors and offset factors by the factor generation module for the plurality of picture tags;

[0050] Step S300: input the scale factor and the offset factor into the self-attention module, calibrate the query vector, key vector and value vector in the self-attention module, and obtain a target domain model;

[0051] Step S400: input the target domain image into the target domain model to obtain a corresponding image classification result.

[0052] The embodiment of the present application is an adaptive method for online testing of the Transformer network model, which is used to deal with the domain shift problem between the source domain and the target domain, and mainly solves the image classification problem. The embodiment of the present application does not require source domain data to assist unsupervised domain adaptation, nor does it need to generate pseudo labels and access all test data. It can be processed according to the data of the current batch in the test. Specifically, in each network layer in the Transformer, a domain separation network can be learned to extract a domain tag (Domain Token), which can represent the domain drift information learned from all picture tags (Patch Token). The domain tag (Domain Token) can be used together with the class tag (Class Token) to learn the conditional scale and shift factor of the self-attention module in the factor generation network. When the Transformer network model is applied to a new target domain, the query (Query), key (Key) and value (Value) vectors in the self-attention module of each layer can be calibrated using local linear changes parameterized by the conditional scale and shift factor, thereby improving the adaptability and classification accuracy of the Transformer model in the target domain. Among them, the self-attention module is a neural network structure that implements the self-attention mechanism.

[0053] The fully test-time adaptive method provided in the embodiment of the present application introduces a scale factor and an offset factor to dynamically adjust the query vector, key vector, and value vector in the self-attention mechanism, thereby significantly improving the classification performance and robustness of the Transformer model in the face of domain drift, and ensuring that the model can process data from different fields more stably and reliably in actual application scenarios.

[0054] In one embodiment of the present application, the step S100 specifically includes:

[0055] Step S110, obtaining a pre-constructed target domain dataset, wherein the target domain dataset includes a plurality of target domain images;

[0056] Step S120: input all target domain images in the target domain dataset into the pre-trained source domain model in batches.

[0057] Specifically, load the basic model that has been trained on the source dataset from the pre-trained model library. The model can be a Vision Transformer network with a standard architecture and parameters θ S . The model can be expressed as: Among them, y represents the predicted output, X s Represents source domain data, Represents a model.

[0058] The embodiment of the present application uses the weight initialization model trained on the source data set to ensure that the model can perform effective reasoning based on pre-trained knowledge when processing target data.

[0059] In the embodiment of the present application, the factor generation module includes a domain separation network and a factor generation network. The step S200 specifically includes:

[0060] Step S210: using the embedding layer in the source domain model to convert the target domain image into a plurality of picture tags, after the plurality of picture tags are processed by the domain separation network, averaging the processed plurality of picture tags to obtain a domain tag, wherein the domain tag is used to represent domain offset noise information;

[0061] Step S220: obtaining the pre-trained class label in the source domain model, inputting the domain label and the class label into the factor generation network, and obtaining the scale factor and the offset factor corresponding to the current encoder layer.

[0062] Specifically, since the present application adds a domain separation network in each Transformer layer, by learning a domain label in each Transformer layer, the domain drift information in the current batch is effectively extracted without accessing the source domain data or generating pseudo labels, which not only avoids privacy issues and storage costs, but also reduces dependence on source domain data and improves the performance of the model in the target domain. In addition, by using domain labels and class labels, the factor generator network learns conditional scales and shift factors in the self-attention module and applies these factors to the self-attention module of each layer. This method enables the model to dynamically adjust to domain changes during the inference phase, thereby enhancing its flexibility and adaptability in new target domains.

[0063] In one embodiment of the present application, the domain separation network is trained with the goal of minimizing a pre-constructed similarity loss function, so as to minimize the distance between the generated domain label and all the image labels.

[0064] Specifically, Figure 2As shown in Figure 1, a domain separation network is trained to extract representative domain features from a set of image tags. These features are called domain tags. Domain tags aggregate domain-specific information in each layer to capture domain drift. Specifically, domain tags are obtained by averaging multiple image tags through the same domain separation network to obtain a tag that can represent domain shift noise information, as shown in the following formula:

[0065]

[0066] Among them, l represents the lth layer in the Transformer network, D 1 Represents the domain label of the first layer, represents the nth image tag feature of the lth layer of the model, Avg[.] represents average pooling, Φ 1 Stands for Domain Separation Network.

[0067] In order to ensure that the domain label can effectively represent the domain drift information obtained from each image label, the network is trained by minimizing the similarity loss function so that the distance between the generated domain label and all image labels is minimized. and Similarity

[0068]

[0069] Constrained loss function to minimize similarity loss It can be expressed as:

[0070]

[0071] in, is the sign of the similarity loss function, which depends on the parameter θ t and input data x. The goal of the loss function is to minimize the similarity between the labels, thereby ensuring that the domain labels generated by the domain separation network can represent the domain drift information learned from all image labels. represents the summation and averaging of all layers l in the model, where L is the total number of self-attention modules in the model. This term ensures that the loss function considers the similarity of all layers, not just a single layer, thus providing a more comprehensive representation of domain drift. It means summing and averaging the similarities between all the image tags in each layer. Here N is the number of image tags in each layer. It is to calculate the average similarity between all tag pairs in each layer. It represents the cosine similarity between the jth and kth image tags in the lth layer. The higher the cosine similarity, the more similar the two tags are in the feature space. In order to ensure that the domain tags are as representative as possible of all image tags, the goal of the loss function is to minimize the average of these similarity values. The minus sign in front is because we want to maximize the cosine similarity. By minimizing it (i.e. maximizing its negative value), we are actually increasing the similarity so that the domain label can better reflect the relationship between the image labels. This method makes samples from the same domain, even if they belong to different categories, cluster together in the feature space. This learned domain label will be used to generate scale and shift factors later.

[0072] In one embodiment of the present application, the factor generation network is a lightweight fully connected layer having an input dimension and an output dimension.

[0073] Specifically, Figure 2 As shown in Figure 2. The factor generation network is a key component used in this method. It dynamically adjusts the visual Transformer in each batch to adapt to domain changes. The network recalibrates the query vector, key vector, and value vector in the self-attention module by generating a scale factor γ and an offset factor β to mitigate the impact of domain changes. In the specific implementation, the factor generation network Ψ of each layer l is a lightweight fully connected layer with input dimension d and output dimension 2d. The domain is labeled D l and class marker C l As input, the scale factor γ of the output layer l is l and the offset factor β l :

[0074] [γ l ,β l ]=Ψ l (D l ,C l )

[0075] The output scale and offset factors are then applied to the query vector, key vector, and value vector of the self-attention module.

[0076] In the embodiment of the present application, the step S300 specifically includes:

[0077] Step S310: obtaining a pre-constructed total loss function, wherein the total loss function includes: an entropy loss function and a similarity loss function, wherein the similarity loss function is used to guide the source domain model to capture the characteristics of the target domain;

[0078] Step S320: input the scale factor and the offset factor into the self-attention module, and calibrate the query vector, key vector and value vector in the self-attention module according to the total loss function.

[0079] Specifically, to address the impact of domain changes on the self-attention module and make the query, key, and value vectors adapt to domain changes, this method introduces a scale factor γ and an offset factor β to adjust the query, key, and value vectors. The specific calculation is as follows (self-attention recalibration in Figure 2 ):

[0080]

[0081] where γ q , γ k , γ v are the scale factors of the query component, key component, and value component, and β q , β k , β v are the corresponding offset factors. Based on the scale factor γ and the offset factor β, the query vector Q, key vector K, and value vector V are adjusted to Q ′ , K ′ , V ′ . In this method, the scale factor and the offset factor are shared among the query, key, and value components to address the domain change problem. Then, the self-attention operation is performed: where d represents the dimension of the feature.

[0082]

[0083] In the test phase, the widely used entropy loss is selected to guide the adaptation process. For the samples x in the current mini-batch B t , the reliable entropy loss is defined as:

[0084]

[0085] where I is an exponential mask, which is defined as I = I[E(θ t ; x) < E0], indicating that only the samples with entropy greater than the threshold E0 are optimized. E(θ t ; x) represents the entropy value of the sample x, measuring the uncertainty of the model prediction. E0 is a manually set threshold parameter used to control which samples will be used to update the model to avoid the model deviating too much from the original pre-trained model. In addition to the entropy loss, the embodiment of this application also combines the similarity loss to promote the domain separation network to learn domain-specific information, thus forming the following total loss function:

[0086]

[0087] where, as a balance parameter, represents the weight relationship between the entropy loss and the similarity loss. ∑I represents the sum of the entropy masks of the entire batch. The similarity loss It is used to promote the domain separation network to learn domain-specific information, ensuring that the model can better capture the characteristics of the target domain, thereby improving the adaptability.

[0088] By introducing scale and offset factors at each Transformer layer, the impact of domain changes is gradually mitigated. This incremental adjustment enables the model to recover the original feature representation and thus maintain performance across different domains.

[0089] In one embodiment of the present application, several target domain images of each batch are simultaneously input into the pre-trained source domain model, and the source domain model used in the current batch is the target domain model updated in the previous batch. The step S400 is specifically as follows: input all the target domain images in the current batch into the target domain model to obtain the image classification results corresponding to each target domain image in the current batch.

[0090] In the embodiment of the present application, the target domain model updated in the previous batch is used as the source domain model for the current batch, so that the visual Transformer can be dynamically adjusted in each batch to adapt to domain changes.

[0091] In this way, the present application improves the performance of the Transformer model in online test time adaptation by calibrating the adaptive scale and offset factors in the self-attention module. Specifically, first, a lightweight neural network called a domain separation network is introduced, which generates a domain label representing the domain shift from all image labels in each layer. This domain label effectively captures domain-specific features and promotes the learning of improved scale and transfer factors. Second, a factor generator network is added, which generates these recalibration factors conditioned on the domain label and class label. Third, a new Transformer network self-attention module is proposed, which recalibrates the query features, key features, and value features of the test samples online during the target domain reasoning process.

[0092] Experimental results show that the embodiments of the present application significantly improve the domain adaptation performance of the Transformer model and outperform current methods in domain adaptation during online testing across multiple commonly used benchmark datasets.

[0093] In one embodiment, if Figure 3 As shown, based on the above-mentioned fully test-time adaptive method based on self-attention scale shift recalibration, the present invention also provides a fully test-time adaptive device based on self-attention scale shift recalibration, comprising:

[0094] An input module 100 is used to obtain a target domain image and input the target domain image into a pre-trained source domain model, wherein each encoder layer of the source domain model has a factor generation module and a self-attention module;

[0095] A generating module 200, configured to convert the target domain image into a plurality of picture tags by using the embedding layer in the source domain model, wherein the plurality of picture tags are generated into a scale factor and an offset factor by the factor generating module;

[0096] A calibration module 300, configured to input the scale factor and the offset factor into the self-attention module, calibrate the query vector, the key vector and the value vector in the self-attention module, and obtain a target domain model;

[0097] The classification module 400 is used to input the target domain image into the target domain model to obtain a corresponding image classification result.

[0098] Figure 4 A schematic diagram of the structure of a terminal provided in an embodiment of the present application. The terminal may include:

[0099] A memory 501 , a processor 502 , and a computer program stored in the memory 501 and executable on the processor 502 .

[0100] When the processor 502 executes the program, the fully test-time adaptive method based on self-attention scale shift recalibration provided in the above embodiment is implemented.

[0101] Furthermore, the terminal further includes:

[0102] The communication interface 503 is used for communication between the memory 501 and the processor 502 .

[0103] The memory 501 is used to store computer programs that can be executed on the processor 502 .

[0104] The memory 501 may include a high-speed RAM memory, and may also include a non-volatile memory (non-volatile memory), such as at least one disk memory.

[0105] If the memory 501, the processor 502 and the communication interface 503 are implemented independently, the communication interface 503, the memory 501 and the processor 502 can be connected to each other through a bus and communicate with each other. The bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component (PCI) bus or an Extended Industry Standard Architecture (EISA) bus. The bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, only one line is used in the figure, but it does not mean that there is only one bus or one type of bus.

[0106] Optionally, in a specific implementation, if the memory 501, the processor 502 and the communication interface 503 are integrated on a chip, the memory 501, the processor 502 and the communication interface 503 can communicate with each other through an internal interface.

[0107] The processor 502 may be a central processing unit (CPU), or an application specific integrated circuit (ASIC), or one or more integrated circuits configured to implement the embodiments of the present application.

[0108] This embodiment also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the above-mentioned fully test-time adaptive method based on self-attention scale shift recalibration.

[0109] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" etc. means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present application. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or N embodiments or examples in a suitable manner. In addition, those skilled in the art may combine and combine the different embodiments or examples described in this specification and the features of the different embodiments or examples, without contradiction.

[0110] In addition, the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of technical features indicated. Therefore, a feature defined as "first" or "second" may explicitly or implicitly include at least one of the features. In the description of this application, "N" means at least two, such as two, three, etc., unless otherwise clearly and specifically defined.

[0111] Any process or method description in a flowchart or otherwise described herein may be understood to represent a module, fragment or portion of code comprising one or N executable instructions for implementing the steps of a custom logical function or process, and the scope of the preferred embodiments of the present application includes additional implementations, in which the order shown or discussed may not be followed, including performing functions in a substantially simultaneous manner or in reverse order depending on the functions involved, which should be understood by technicians in the technical field to which the embodiments of the present application belong.

[0112] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as an ordered list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by an instruction execution system, device or apparatus (such as a computer-based system, a system including a processor, or other system that can read instructions from an instruction execution system, device or apparatus and execute instructions), or in combination with these instruction execution systems, devices or apparatuses. For the purpose of this specification, "computer-readable medium" can be any device that can contain, store, communicate, propagate or transmit a program for use by an instruction execution system, device or apparatus, or in combination with these instruction execution systems, devices or apparatuses. More specific examples of computer-readable media (a non-exhaustive list) include the following: an electrical connection with one or N wirings (electronic devices), a portable computer disk box (magnetic device), a random access memory (RAM), a read-only memory (ROM), an erasable and programmable read-only memory (EPROM or flash memory), a fiber optic device, and a portable compact disk read-only memory (CDROM). In addition, the computer-readable medium may even be paper or other suitable medium on which the program is printed, since the program may be obtained electronically by optically scanning the paper or other medium and then editing, interpreting or otherwise processing in a suitable manner if necessary and then storing it in a computer memory.

[0113] It should be understood that the various parts of the present application can be implemented by hardware, software, firmware or a combination thereof. In the above embodiment, the N steps or methods can be implemented by software or firmware stored in a memory and executed by a suitable instruction execution system. If implemented by hardware, as in another embodiment, it can be implemented by any one of the following technologies known in the art or their combination: a discrete logic circuit having a logic gate circuit for implementing a logic function for a data signal, a dedicated integrated circuit having a suitable combination of logic gate circuits, a programmable gate array (PGA), a field programmable gate array (FPGA), etc.

[0114] A person skilled in the art may understand that all or part of the steps in the method for implementing the above-mentioned embodiment may be completed by instructing related hardware through a program, and the program may be stored in a computer-readable storage medium. When the program is executed, it includes one or a combination of the steps of the method embodiment.

[0115] In addition, each functional unit in each embodiment of the present application may be integrated into a processing module, or each unit may exist physically separately, or two or more units may be integrated into one module. The above-mentioned integrated module may be implemented in the form of hardware or in the form of a software functional module. If the integrated module is implemented in the form of a software functional module and sold or used as an independent product, it may also be stored in a computer-readable storage medium.

[0116] The storage medium mentioned above may be a read-only memory, a magnetic disk or an optical disk, etc. Although the embodiments of the present application have been shown and described above, it can be understood that the above embodiments are exemplary and cannot be understood as limiting the present application. A person of ordinary skill in the art may change, modify, replace and modify the above embodiments within the scope of the present application.

[0117] In summary, the present invention discloses a fully test-time adaptive method based on self-attention scale offset recalibration, the method comprising: obtaining a target domain image, inputting the target domain image into a pre-trained source domain model, each encoder layer of the source domain model having a factor generation module and a self-attention module; using the embedding layer in the source domain model to convert the target domain image into a number of picture tags, the number of picture tags are generated by the factor generation module. The scale factor and the offset factor are input into the self-attention module, and the query vector, the key vector and the value vector in the self-attention module are calibrated to obtain the target domain model; the target domain image is input into the target domain model to obtain the corresponding image classification result. The present invention introduces the scale factor and the offset factor to dynamically adjust the query vector, the key vector and the value vector in the self-attention mechanism, thereby improving the adaptability and classification accuracy of the model in the target domain.

[0118] It should be understood that the application of the present invention is not limited to the above examples. For ordinary technicians in this field, improvements or changes can be made based on the above description. All these improvements and changes should fall within the scope of protection of the claims attached to the present invention.

Claims

1. A fully test-time adaptive method based on self-attention scale shift recalibration, characterized in that The method comprises: Acquire a target domain image, and input the target domain image into a pre-trained source domain model, wherein each encoder layer of the source domain model has a factor generation module and a self-attention module; The target domain image is converted into a plurality of picture tags by using the embedding layer in the source domain model, and the plurality of picture tags are generated into a scale factor and an offset factor by the factor generation module; Inputting the scale factor and the offset factor into the self-attention module, calibrating the query vector, the key vector, and the value vector in the self-attention module, and obtaining a target domain model; The target domain image is input into the target domain model to obtain a corresponding image classification result.

2. The fully test-time adaptive method based on self-attention scale shift recalibration according to claim 1, characterized in that Obtaining a target domain image and inputting the target domain image into a pre-trained source domain model includes: Acquire a pre-constructed target domain dataset, where the target domain dataset includes a plurality of target domain images; All target domain images in the target domain dataset are input into the pre-trained source domain model in batches.

3. The fully test-time adaptive method based on self-attention scale shift recalibration according to claim 1, characterized in that The factor generation module includes a domain separation network and a factor generation network; The target domain image is converted into a plurality of picture tags by using the embedding layer in the source domain model, and the plurality of picture tags are generated into a scale factor and an offset factor by the factor generation module, including: The target domain image is converted into a plurality of picture tags by using the embedding layer in the source domain model, and after the plurality of picture tags are processed by the domain separation network, the processed plurality of picture tags are averaged to obtain domain tags, where the domain tags are used to represent domain offset noise information; The pre-trained class label in the source domain model is obtained, and the domain label and the class label are input into the factor generation network to obtain the scale factor and the offset factor corresponding to the current encoder layer.

4. The fully test-time adaptive method based on self-attention scale shift recalibration according to claim 3, characterized in that The domain separation network is trained with the goal of minimizing a pre-constructed similarity loss function so as to minimize the distance between the generated domain labels and all image labels.

5. The fully test-time adaptive method based on self-attention scale shift recalibration according to claim 3, characterized in that The factor generation network is a lightweight fully connected layer with an input dimension and an output dimension.

6. The fully test-time adaptive method based on self-attention scale shift recalibration according to claim 4, characterized in that Inputting the scale factor and the offset factor into the self-attention module, calibrating the query vector, the key vector, and the value vector in the self-attention module, and obtaining a target domain model, including: Obtaining a pre-built total loss function, where the total loss function includes: an entropy loss function and a similarity loss function, where the similarity loss function is used to guide a source domain model to capture features of a target domain; The scale factor and the offset factor are input into the self-attention module, and the query vector, the key vector, and the value vector in the self-attention module are calibrated according to the total loss function.

7. The fully test-time adaptive method based on self-attention scale shift recalibration according to claim 2, characterized in that Several target domain images of each batch are simultaneously input into the pre-trained source domain model. The source domain model used in the current batch is the target domain model updated in the previous batch. Inputting the target domain image into the target domain model to obtain a corresponding image classification result includes: All the target domain images in the current batch are input into the target domain model to obtain image classification results corresponding to each target domain image in the current batch.

8. A fully test-time adaptive device based on self-attention scale shift recalibration, characterized in that The device comprises: An input module, used for acquiring a target domain image, and inputting the target domain image into a pre-trained source domain model, wherein each encoder layer of the source domain model has a factor generation module and a self-attention module; A generating module, configured to convert the target domain image into a plurality of picture tags by using the embedding layer in the source domain model, wherein the plurality of picture tags are generated into a scale factor and an offset factor by the factor generating module; A calibration module, used for inputting the scale factor and the offset factor into the self-attention module, calibrating the query vector, the key vector and the value vector in the self-attention module, and obtaining a target domain model; The classification module is used to input the target domain image into the target domain model to obtain a corresponding image classification result.

9. A terminal, characterized in that: include: A memory, a processor, and a fully test-time adaptive program based on self-attention scale shift recalibration stored in the memory and executable on the processor, wherein the fully test-time adaptive program based on self-attention scale shift recalibration, when executed by the processor, implements the steps of the fully test-time adaptive method based on self-attention scale shift recalibration as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, which can be executed to implement the steps of the fully test-time adaptive method based on self-attention scale shift recalibration as described in any one of claims 1 to 7.

Images