Adaptive method and device during testing based on dual-path adversarial lifting denoising

Abstract

The invention provides a self-adaption method and device during testing based on double-path adversarial lifting denoising, and the method comprises the steps: inputting a target domain image into a pre-trained source domain model, adding a learnable domain drift mark into the source domain model, adding a lifting denoising module into each encoder layer, and obtaining a target domain image; the domain drift mark is used for learning category inseparable features in the target domain image, and the lifting denoising module is used for denoising image features of the target domain image; inputting the image block vector, the domain drift mark and a pre-trained class mark in the source domain model into a plurality of encoder layers of the source domain model, updating the domain drift mark through back propagation and improving network parameters in a denoising module to obtain a target domain model; and inputting the target domain image into the target domain model to obtain a corresponding image classification result. According to the method, the model can be updated online in real time without transmitting a large amount of data, accessing the data of the whole test set and performing multiple rounds of training.

Description

Technical Field

[0001] The present invention relates to the technical field of transfer learning, and in particular to a test-time adaptive method and device based on dual-path adversarial boosting denoising. Background Art

[0002] The application of deep neural networks in various computer vision tasks relies on a large number of labeled training data sets and a large amount of computing resources. When the training samples and test samples come from different environments, the problem of domain drift will occur. Test-time domain adaptation is a set of transfer learning methods that aims to migrate network models trained with labeled data sets (source domain) to new unlabeled data sets (target domain). It performs sequence analysis on input samples during the inference phase, which can effectively adapt the source domain model during the test process to solve the cross-domain performance degradation problem of deep neural networks.

[0003] Unsupervised domain adaptation includes: active domain adaptation methods and passive domain adaptation methods. Active domain adaptation methods mainly include statistical distribution matching and generative adversarial network methods to minimize domain differences, but active domain adaptation methods must rely on labeled data in the source domain. In practical applications, due to data privacy issues or transmission difficulties, source domain samples are usually not accessible. Passive domain adaptation methods directly perform unsupervised transfer learning in the target domain without the need for labeled data in the source domain. However, passive domain adaptation methods require access to the entire test set data and multiple rounds of training.

[0004] Therefore, the two existing domain adaptation methods require the transmission of large amounts of data and fail to protect data privacy, or require access to the entire test set data and undergo multiple rounds of training and fail to update the model online in real time.

[0005] Therefore, the prior art has defects and needs to be improved and developed. Summary of the invention

[0006] The technical problem to be solved by the present invention is that, in view of the above-mentioned defects of the prior art, a test-time adaptive method and device based on dual-path adversarial boosting denoising is provided, aiming to solve the problem that the domain adaptive method in the prior art requires the transmission of a large amount of data and cannot protect data privacy, or requires access to the entire test set data and undergoes multiple rounds of training and cannot update the model online in real time.

[0007] The technical solution adopted by the present invention to solve the technical problem is as follows:

[0008] A test-time adaptive method based on dual-path adversarial boosting denoising, wherein the method comprises:

[0009] Acquire a target domain image, and input the target domain image into a pre-trained source domain model, wherein a learnable domain drift marker is added to the source domain model, and a lifting and denoising module is added to each encoder layer of the source domain model, wherein the domain drift marker is used to learn the category-inseparable features in the target domain image, and the lifting and denoising module is used to denoise the image features of the target domain image;

[0010] The target domain image is converted into an image block vector by using an embedding layer in the source domain model, and the image block vector, the domain drift mark and the class mark pre-trained in the source domain model are input into several encoder layers of the source domain model, and the domain drift mark and the network parameters in the lifting and denoising module are updated by back propagation to obtain a target domain model;

[0011] The target domain image is input into the target domain model to obtain a corresponding image classification result.

[0012] In one embodiment of the present application, acquiring a target domain image and inputting the target domain image into a pre-trained source domain model includes:

[0013] Acquire a target domain dataset for model domain adaptation, wherein the target domain dataset includes a plurality of target domain images;

[0014] All target domain images in the target domain dataset are input into the pre-trained source domain model in batches.

[0015] In one embodiment of the present application, the target domain image is converted into an image block vector using an embedding layer in the source domain model, and the image block vector, the domain drift mark and the pre-trained class mark in the source domain model are input into several encoder layers of the source domain model, and the domain drift mark and the network parameters in the lifting and denoising module are updated by back propagation to obtain the target domain model, including:

[0016] Using the embedding layer in the source domain model, all the target domain images in the current batch are divided into a number of image blocks, and each of the image blocks is converted into an image block vector;

[0017] Inputting the image patch vector, the domain drift label and the pre-trained class label in the source domain model into several encoder layers of the source domain model;

[0018] In each encoder layer, based on the domain drift mark, the class mark and the lifting denoising module of the previous layer, the domain drift features of the current encoder layer and the noise in the image features corresponding to the target domain image are separated;

[0019] Using a preset information entropy loss function to back-propagate, the domain drift marker and the network parameters in the boost denoising module are updated to obtain an optimized model;

[0020] The optimization model is adjusted using a preset similarity loss function to obtain a target domain model.

[0021] In one embodiment of the present application, the lifting and denoising module includes a prediction network and an update network;

[0022] In each encoder layer, based on the domain drift mark, class mark and the lifting denoising module of the previous layer, the domain drift feature of the current encoder layer and the noise in the image feature corresponding to the target domain image are separated, including:

[0023] In each encoder layer, the domain drift mark of the previous layer is input into the prediction network to predict the domain drift feature of the current encoder layer;

[0024] Inputting the domain drift feature and the class label into the update network to separate the noise in the image feature corresponding to the target domain image;

[0025] Wherein, the initialization network parameter of the prediction network is 0.

[0026] In one embodiment of the present application, the ratio of the weight occupied by the similarity loss function to the weight occupied by the information entropy loss function is a preset balance parameter.

[0027] In one embodiment of the present application, several target domain images of each batch are simultaneously input into a pre-trained source domain model, and the source domain model used in the current batch is the target domain model updated in the previous batch.

[0028] In one embodiment of the present application, the target domain image is input into the target domain model to obtain a corresponding image classification result, including:

[0029] All the target domain images in the current batch are input into the target domain model to obtain image classification results corresponding to each target domain image in the current batch.

[0030] The present application also provides a test-time adaptive device based on dual-path adversarial boosting denoising, wherein the device comprises:

[0031] An input module, used to obtain a target domain image, input the target domain image into a pre-trained source domain model, wherein a learnable domain drift marker is added to the source domain model, and a lifting and denoising module is added to each encoder layer of the source domain model, wherein the domain drift marker is used to learn the category-inseparable features in the target domain image, and the lifting and denoising module is used to denoise the image features of the target domain image;

[0032] An updating module, configured to convert the target domain image into an image block vector by using an embedding layer in the source domain model, input the image block vector, the domain drift mark and the pre-trained class mark in the source domain model into several encoder layers of the source domain model, and back-propagate and update the domain drift mark and the network parameters in the lifting and denoising module to obtain a target domain model;

[0033] The classification module is used to input the target domain image into the target domain model to obtain a corresponding image classification result.

[0034] The present application also provides a terminal, comprising: a memory, a processor, and a test-time adaptive program based on dual-path adversarial boosting denoising stored in the memory and executable on the processor, wherein the test-time adaptive program based on dual-path adversarial boosting denoising implements the steps of the test-time adaptive method based on dual-path adversarial boosting denoising as described above when executed by the processor.

[0035] The present application also provides a computer-readable storage medium, which stores a computer program, and the computer program can be executed to implement the steps of the test-time adaptive method based on dual-path adversarial boosting denoising as described above.

[0036] The present invention provides a test-time adaptive method and device based on dual-path adversarial lifting denoising, the method comprising: obtaining a target domain image, inputting the target domain image into a pre-trained source domain model, adding a learnable domain drift marker to the source domain model, and adding a lifting denoising module to each encoder layer of the source domain model, the domain drift marker is used to learn the category-inseparable features in the target domain image, and the lifting denoising module is used to denoise the image features of the target domain image; using an embedding layer in the source domain model to convert the target domain image into an image block vector, and inputting the image block vector, the domain drift marker and the pre-trained class marker in the source domain model into several encoder layers of the source domain model, back-propagating to update the domain drift marker and the network parameters in the lifting denoising module to obtain the target domain model; inputting the target domain image into the target domain model to obtain a corresponding image classification result. By adding a learnable domain drift marker and a boosting denoising module, in a fully online test-time adaptive process, the boosting denoising module of each encoder layer can use the domain drift marker for denoising and gradually eliminate the influence of domain offset, thereby improving the test performance of the model and avoiding the need to transmit a large amount of data and fail to protect data privacy, or the need to access the entire test set data and undergo multiple rounds of training and fail to update the model online in real time. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] Figure 1 It is a flow chart of a preferred embodiment of the test-time adaptive method based on dual-path confrontation lifting denoising in the present invention;

[0038] Figure 2 This is a diagram illustrating the dual-path adversarial boosting denoising Transformer framework in the present invention;

[0039] Figure 3 It is a schematic block diagram of the specific structure of the lifting and denoising module in the present invention;

[0040] Figure 4 It is a functional principle block diagram of a preferred embodiment of a test-time adaptive device based on dual-path confrontation lifting and denoising in the present invention;

[0041] Figure 5 It is a functional principle block diagram of a preferred embodiment of the terminal in the present invention. DETAILED DESCRIPTION

[0042] In order to make the purpose, technical solution and advantages of the present invention clearer and more specific, the present invention is further described in detail below with reference to the accompanying drawings and examples. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention.

[0043] In the transfer learning image classification task, the performance of the model trained in the source domain is often greatly reduced when it is transferred to the target domain. Most of the existing transfer learning domain adaptation methods require access to the training data of the source domain, which is not applicable in some scenarios where data transmission is difficult or privacy is sensitive. In order to solve this data transmission or privacy problem, passive domain adaptation that does not require access to source domain data has been proposed, while current passive domain adaptation methods require access to the entire test set data and multiple rounds of training. Full test-time adaptation only requires access to the real-time stream of test samples, which can dynamically adapt the source model during the test process.

[0044] That is to say, the existing methods have the following problems:

[0045] First, existing methods need to transfer a large amount of data to access source domain data and cannot protect data privacy. Most existing transfer learning domain adaptation methods need to access both source and target domain data to improve the accuracy in the target domain.

[0046] Second, the existing methods need to transmit a large amount of data to access the source domain data, which consumes huge transmission resources and cannot protect data privacy.

[0047] Third, the existing methods access the entire test set data and undergo multiple rounds of training, but cannot update the model online in real time during testing, which is inefficient in actual test scenarios;

[0048] Fourth, existing methods do not have a test-time adaptive method for the Transformer model.

[0049] The present invention finds that when the Transformer model is applied to a new domain, the self-attention map of the image sample in the target domain has a lot of noise and does not focus on the image target object. Self-attention is one of the core modules in the Transformer network design. Once this self-attention distribution is disturbed by domain changes or noise, the performance of the Transformer model will be significantly reduced.

[0050] The embodiment of the present application solves the problem of decreased test set accuracy caused by domain drift in transfer learning by improving the denoising module, taking the newly added learnable domain drift marker as input, predicting noise and updating features between Transformer layers, and thus separating the noise in the original features. The source model is effectively adapted during the test process to solve the cross-domain performance degradation problem of deep neural networks.

[0051] See also Figure 1 , Figure 1 Flow chart of the test-time adaptive method based on dual-path adversarial boosting denoising in the present invention. Figure 1 As shown, the test-time adaptive method based on dual-path adversarial boosting denoising described in the embodiment of the present invention includes:

[0052] Step S100, obtaining a target domain image, and inputting the target domain image into a pre-trained source domain model, wherein a learnable domain drift marker is added to the source domain model, and a lifting and denoising module is added to each encoder layer of the source domain model, wherein the domain drift marker is used to learn the category-inseparable features in the target domain image, and the lifting and denoising module is used to denoise the image features of the target domain image;

[0053] Step S200, using the embedding layer in the source domain model to convert the target domain image into an image block vector, and inputting the image block vector, the domain drift mark and the pre-trained class mark in the source domain model into several encoder layers of the source domain model, back-propagating and updating the domain drift mark and the network parameters in the lifting and denoising module to obtain the target domain model;

[0054] Step S300: input the target domain image into the target domain model to obtain a corresponding image classification result.

[0055] Specifically, the present application provides a dual-path adversarial lifting denoising (Dual-path AdversarialLifting), which aims to separate the noise in the feature signal, thereby improving the model classification test performance. When testing the Transformer model, a new lifting denoising module is added to each layer, and a new learnable domain drift marker (token) is added before the original class token. The newly added domain drift marker and the original class marker form a dual path, in which the class marker hopes to output class-separable features for better classification prediction, and the domain drift marker hopes to output the common noise of all samples, that is, the class-inseparable features, thus forming adversarial learning, which can also be called an adversarial lifting denoising module.

[0056] The embodiment of the present application adds a learnable domain drift marker and a boosting denoising module. In a fully online test-time adaptation process, the boosting denoising module of each Transformer encoder layer can gradually eliminate the impact of domain shift, thereby improving the test performance of the model to solve the cross-domain performance degradation problem of deep neural networks.

[0057] In an embodiment of the present application, step S100 specifically includes: obtaining a target domain dataset for model domain adaptation, wherein the target domain dataset includes a plurality of target domain images; and inputting all target domain images in the target domain dataset into a pre-trained source domain model in batches.

[0058] Specifically, unlike existing domain adaptation methods, the embodiments of the present application do not need to access the source domain data or the entire target domain test set data. They only need to access the current batch of data to perform domain adaptation during test inference, and use the adversarial boosting denoising module to separate the noise and restore the self-attention of the Transformer to solve the problem of decreased test set accuracy caused by domain drift in transfer learning, and effectively adapt the source model during the test process to solve the cross-domain performance degradation problem of deep neural networks. However, existing technical solutions require access to the entire test set data and undergo multiple rounds of training. They cannot update the model online in real time during testing, and are inefficient in actual test scenarios.

[0059] In the embodiment of the present application, the step S200 specifically includes:

[0060] Step S210, using the embedding layer in the source domain model to divide all the target domain images in the current batch into a number of image blocks, and converting each of the image blocks into an image block vector;

[0061] Step S220, inputting the image block vector, the domain drift mark and the pre-trained class mark in the source domain model into several encoder layers of the source domain model;

[0062] Step S230: In each encoder layer, based on the domain drift mark, class mark and the lifting denoising module of the previous layer, separate the domain drift features of the current encoder layer and the noise in the image features corresponding to the target domain image;

[0063] Step S240, using a preset information entropy loss function to back propagate and update the domain drift mark and the network parameters in the boosting and denoising module to obtain an optimized model;

[0064] Step S250: Use a preset similarity loss function to adjust the optimization model to obtain a target domain model.

[0065] like Figure 2 As shown, Figure 2 This is a diagram illustrating the framework of the Dual-path Adversarial Lifting Transformer. The embodiment of the present application first inputs the test samples of the current batch, uses the information entropy loss function of the output probability distribution to back-propagate the domain shift token and the network parameters in the lifting denoising module, and finally uses the adaptive model to output the predicted classification results.

[0066] Specifically, referring to the unsupervised domain adaptation setting of transfer learning, a source domain model is first trained in the source domain, and then unsupervised domain adaptation is performed in the target domain. The source domain model can use any pre-trained Transformer model.

[0067] Information entropy is a measure of the uncertainty of random variables. In machine learning, information entropy loss functions (such as cross entropy loss functions) are often used for classification problems to measure the difference between the model's predicted distribution and the true distribution. The goal is to optimize the model parameters by minimizing the loss function so that the model's predicted results are closer to the true results. The similarity loss function is used to evaluate the similarity between two images. The embodiment of the present application is a multi-task learning scenario, and the model needs to process two tasks at the same time: a classification task and a similarity assessment task. The cross entropy loss function is used for the classification task, and the cosine similarity loss function is used for the similarity assessment task. In the hierarchical model, the model first performs coarse-grained classification, and then performs fine-grained similarity assessment based on the classification results.

[0068] In the embodiment of the present application, the information entropy loss function is used for training at the classification layer, and then the similarity loss function is used for fine-tuning at the similarity evaluation layer. This hierarchical training method helps the model to better understand the data structure and task requirements.

[0069] In one embodiment of the present application, the lifting and denoising module includes a prediction network and an update network. The step S230 specifically includes: in each layer of the encoder, inputting the domain drift mark of the previous layer into the prediction network to predict the domain drift feature of the current encoder layer; inputting the domain drift feature and the class mark into the update network to separate the noise in the image feature corresponding to the target domain image; wherein the initialization network parameter of the prediction network is 0.

[0070] Specifically, when testing the model, the embodiment of the present application adds a new learnable domain drift token before the original class token, inputs the prediction network with the new domain drift token at each Transformer encoder layer, and inputs the predicted domain drift features (i.e., noise) of the current encoder layer and the original class token into the update network. The updated denoised class features are then input into the next layer of Transformer encoder, thereby separating the noise and restoring the Transformer's self-attention, so as to solve the problem of decreased test set accuracy caused by domain drift in transfer learning. That is, this gradual adjustment process helps to restore the original noise-free self-attention map, so that the model can maintain its performance in different domains.

[0071] like Figure 3 As shown, Figure 3 The specific structure diagram of the lifting denoising module is shown in Figure 2. The lifting denoising module includes the prediction network (Prediction NetworkΦ l ) and Update Networkψ l ), predict the domain shift feature of each layer of the network output (Domain shift featureΠ l ). Initialize the prediction network Φ l The parameter matrix is ​​0, so that the initialization will not affect the performance of the original model, and the domain shift token P of the previous layer is input. l-1 Given the prediction network as a condition, the prediction network Φ l Predict the domain drift feature Π of the current layer l , and then input into the update network ψ l Separate the noise from the features, l represents the lth layer of the visual Transformer encoder.

[0072] In an embodiment of the present application, when a pre-trained Transformer model is migrated to a downstream test scenario, the newly added domain drift markers, prediction network, and update network are updated in real time according to different input images, thereby separating the noise in the image features.

[0073] In the embodiment of the present application, in step S250, the ratio between the weight occupied by the similarity loss function and the weight occupied by the information entropy loss function is a preset balance parameter.

[0074] The prediction results of the embodiment of the present application require both accurate classification and maintaining certain similarity constraints. Therefore, the information entropy loss function and the similarity loss function are combined as the optimization goals of the model. By adjusting the weights of the two loss functions, the best balance between classification accuracy and similarity can be found.

[0075] Specifically, by inserting the above-mentioned lifting and denoising module in each layer of the Transformer network, the noise in the features can be separated layer by layer. During the test, a batch of test sample images x is input, and the domain shift token and the prediction network Φ are updated respectively using gradient descent. l and update the network ψ l After updating the parameters, the prediction results of the current batch are output. The first information entropy loss function is:

[0076]

[0077] in E0 is the information entropy threshold to filter out samples with excessive noise, E is the information entropy function, θ t is the parameter to be updated. The second similarity loss function is:

[0078]

[0079] Where L represents the number of network layers, B represents the number of samples in each batch, and M represents the domain drift feature Π of all samples in the batch class. l The calculated cosine similarity matrix. The final total loss function is:

[0080]

[0081] Where λ is a balancing hyperparameter.

[0082] In one embodiment of the present application, several target domain images of each batch are simultaneously input into a pre-trained source domain model, and the source domain model used in the current batch is the target domain model updated in the previous batch.

[0083] The embodiments of the present application do not need to access the source domain data or use the entire target domain test set data for multiple iterative training. It only needs to access the current batch data during test inference to perform domain adaptation and dynamically fine-tune the model to improve the robustness of the model across domains.

[0084] In the embodiment of the present application, the step S300 is specifically: inputting all the target domain images in the current batch into the target domain model to obtain the image classification results corresponding to each target domain image in the current batch.

[0085] The present invention can update the source domain model by only accessing one batch of data, thus achieving real-time update. It is not necessary to use all test data for multiple iterative training, thus improving update efficiency.

[0086] In one embodiment, adversarial training and autoencoder mechanisms can be introduced simultaneously in the source domain model. Adversarial training is used to reduce the differences between domains, while the autoencoder is used to remove image noise during the encoding process, further improving the performance of the model in the target domain. On the basis of transfer learning, data enhancement technology can also be combined to expand the training samples of the target domain. By increasing the diversity of target domain images, the adaptability of the model to target domain images can be improved, thereby obtaining more stable classification results.

[0087] The embodiments of the present application achieve the following beneficial effects:

[0088] First, since the self-attention of image samples in the target domain of the Transformer network contains noise, the performance is greatly reduced when the domain changes. Therefore, the embodiment of the present application adds a denoising module to separate the noise in the features layer by layer, which solves the problem that most of the existing domain adaptation methods need to access the source domain data for transfer learning domain adaptation, or the adaptation accuracy is low.

[0089] Second, the lifting and denoising module of the embodiment of the present application includes a prediction network and an update network, which can online predict and update the noise of the input sample image to separate the noise in the feature and make the self-attention map closer to the source domain, thereby significantly improving the performance of domain adaptation during testing.

[0090] In one embodiment, if Figure 4 As shown, based on the above-mentioned test-time adaptive method based on dual-path confrontation lifting denoising, the present invention also provides a test-time adaptive device based on dual-path confrontation lifting denoising, including:

[0091] An input module 100 is used to obtain a target domain image, and input the target domain image into a pre-trained source domain model, wherein a learnable domain drift marker is added to the source domain model, and a lifting and denoising module is added to each encoder layer of the source domain model, wherein the domain drift marker is used to learn the category-inseparable features in the target domain image, and the lifting and denoising module is used to denoise the image features of the target domain image;

[0092] An updating module 200 is used to convert the target domain image into an image block vector by using the embedding layer in the source domain model, and input the image block vector, the domain drift mark and the pre-trained class mark in the source domain model into several encoder layers of the source domain model, and back-propagate to update the domain drift mark and the network parameters in the lifting and denoising module to obtain a target domain model;

[0093] The classification module 300 is used to input the target domain image into the target domain model to obtain a corresponding image classification result.

[0094] Figure 5 A schematic diagram of the structure of a terminal provided in an embodiment of the present application. The terminal may include:

[0095] A memory 501 , a processor 502 , and a computer program stored in the memory 501 and executable on the processor 502 .

[0096] When the processor 502 executes the program, the test-time adaptive method based on dual-path adversarial boosting denoising provided in the above embodiment is implemented.

[0097] Furthermore, the terminal further includes:

[0098] The communication interface 503 is used for communication between the memory 501 and the processor 502 .

[0099] The memory 501 is used to store computer programs that can be executed on the processor 502 .

[0100] The memory 501 may include a high-speed RAM memory, and may also include a non-volatile memory (non-volatile memory), such as at least one disk memory.

[0101] If the memory 501, the processor 502 and the communication interface 503 are implemented independently, the communication interface 503, the memory 501 and the processor 502 can be connected to each other through a bus and communicate with each other. The bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component (PCI) bus or an Extended Industry Standard Architecture (EISA) bus. The bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, only one line is used in the figure, but it does not mean that there is only one bus or one type of bus.

[0102] Optionally, in a specific implementation, if the memory 501, the processor 502 and the communication interface 503 are integrated on a chip, the memory 501, the processor 502 and the communication interface 503 can communicate with each other through an internal interface.

[0103] The processor 502 may be a central processing unit (CPU), or an application specific integrated circuit (ASIC), or one or more integrated circuits configured to implement the embodiments of the present application.

[0104] This embodiment also provides a computer-readable storage medium on which a computer program is stored. When the program is executed by a processor, the test-time adaptive method based on dual-path adversarial boosting denoising as described above is implemented.

[0105] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" etc. means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present application. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or N embodiments or examples in a suitable manner. In addition, those skilled in the art may combine and combine the different embodiments or examples described in this specification and the features of the different embodiments or examples, without contradiction.

[0106] In addition, the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of technical features indicated. Therefore, a feature defined as "first" or "second" may explicitly or implicitly include at least one of the features. In the description of this application, "N" means at least two, such as two, three, etc., unless otherwise clearly and specifically defined.

[0107] Any process or method description in a flowchart or otherwise described herein may be understood to represent a module, fragment or portion of code comprising one or N executable instructions for implementing the steps of a custom logical function or process, and the scope of the preferred embodiments of the present application includes additional implementations, in which the order shown or discussed may not be followed, including performing functions in a substantially simultaneous manner or in reverse order depending on the functions involved, which should be understood by technicians in the technical field to which the embodiments of the present application belong.

[0108] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as an ordered list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by an instruction execution system, device or apparatus (such as a computer-based system, a system including a processor, or other system that can read instructions from an instruction execution system, device or apparatus and execute instructions), or in combination with these instruction execution systems, devices or apparatuses. For the purpose of this specification, "computer-readable medium" can be any device that can contain, store, communicate, propagate or transmit a program for use by an instruction execution system, device or apparatus, or in combination with these instruction execution systems, devices or apparatuses. More specific examples of computer-readable media (a non-exhaustive list) include the following: an electrical connection with one or N wirings (electronic devices), a portable computer disk box (magnetic device), a random access memory (RAM), a read-only memory (ROM), an erasable and programmable read-only memory (EPROM or flash memory), a fiber optic device, and a portable compact disk read-only memory (CDROM). In addition, the computer-readable medium may even be paper or other suitable medium on which the program is printed, since the program may be obtained electronically by optically scanning the paper or other medium and then editing, interpreting or otherwise processing in a suitable manner if necessary and then storing it in a computer memory.

[0109] It should be understood that the various parts of the present application can be implemented by hardware, software, firmware or a combination thereof. In the above embodiment, the N steps or methods can be implemented by software or firmware stored in a memory and executed by a suitable instruction execution system. If implemented by hardware, as in another embodiment, it can be implemented by any one of the following technologies known in the art or their combination: a discrete logic circuit having a logic gate circuit for implementing a logic function for a data signal, a dedicated integrated circuit having a suitable combination of logic gate circuits, a programmable gate array (PGA), a field programmable gate array (FPGA), etc.

[0110] A person skilled in the art may understand that all or part of the steps in the method for implementing the above-mentioned embodiment may be completed by instructing related hardware through a program, and the program may be stored in a computer-readable storage medium. When the program is executed, it includes one or a combination of the steps of the method embodiment.

[0111] In addition, each functional unit in each embodiment of the present application may be integrated into a processing module, or each unit may exist physically separately, or two or more units may be integrated into one module. The above-mentioned integrated module may be implemented in the form of hardware or in the form of a software functional module. If the integrated module is implemented in the form of a software functional module and sold or used as an independent product, it may also be stored in a computer-readable storage medium.

[0112] The storage medium mentioned above may be a read-only memory, a magnetic disk or an optical disk, etc. Although the embodiments of the present application have been shown and described above, it can be understood that the above embodiments are exemplary and cannot be understood as limiting the present application. A person of ordinary skill in the art may change, modify, replace and modify the above embodiments within the scope of the present application.

[0113] In summary, the present invention discloses a test-time adaptive method and device based on dual-path adversarial boosting denoising, the method comprising: obtaining a target domain image, inputting the target domain image into a pre-trained source domain model, the source domain model having a learnable domain drift marker added thereto, and a boosting denoising module added to each encoder layer of the source domain model, the domain drift marker being used to learn the category-inseparable features in the target domain image, and the boosting denoising module being used to denoise the image features of the target domain image; using an embedding layer in the source domain model to convert the target domain image into an image block vector, and inputting the image block vector, the domain drift marker and the pre-trained class marker in the source domain model into several encoder layers of the source domain model, back-propagating and updating the domain drift marker and the network parameters in the boosting denoising module to obtain a target domain model; inputting the target domain image into the target domain model to obtain a corresponding image classification result. By adding a learnable domain drift marker and a boosting denoising module, in a fully online test-time adaptive process, the boosting denoising module of each encoder layer can use the domain drift marker for denoising and gradually eliminate the influence of domain offset, thereby improving the test performance of the model and avoiding the need to transmit a large amount of data and fail to protect data privacy, or the need to access the entire test set data and undergo multiple rounds of training and fail to update the model online in real time.

[0114] It should be understood that the application of the present invention is not limited to the above examples. For ordinary technicians in this field, improvements or changes can be made based on the above description. All these improvements and changes should fall within the scope of protection of the claims attached to the present invention.

Claims

1. A test-time adaptive method based on dual-path adversarial boosting denoising, characterized in that: The method comprises: Acquire a target domain image, and input the target domain image into a pre-trained source domain model, wherein a learnable domain drift marker is added to the source domain model, and a lifting and denoising module is added to each encoder layer of the source domain model, wherein the domain drift marker is used to learn the category-inseparable features in the target domain image, and the lifting and denoising module is used to denoise the image features of the target domain image; The target domain image is converted into an image block vector by using an embedding layer in the source domain model, and the image block vector, the domain drift mark and the class mark pre-trained in the source domain model are input into several encoder layers of the source domain model, and the domain drift mark and the network parameters in the lifting and denoising module are updated by back propagation to obtain a target domain model; The target domain image is input into the target domain model to obtain a corresponding image classification result.

2. The test-time adaptive method based on dual-path adversarial boosting denoising according to claim 1, characterized in that: The acquiring of the target domain image and inputting the target domain image into the pre-trained source domain model includes: Acquire a target domain dataset for model domain adaptation, wherein the target domain dataset includes a plurality of target domain images; All target domain images in the target domain dataset are input into the pre-trained source domain model in batches.

3. The test-time adaptive method based on dual-path adversarial boosting denoising according to claim 2, characterized in that: The target domain image is converted into an image block vector by using an embedding layer in the source domain model, and the image block vector, the domain drift mark and the pre-trained class mark in the source domain model are input into several encoder layers of the source domain model, and the domain drift mark and the network parameters in the lifting and denoising module are updated by back propagation to obtain a target domain model, including: Using the embedding layer in the source domain model, all the target domain images in the current batch are divided into a number of image blocks, and each of the image blocks is converted into an image block vector; Inputting the image block vector, the domain drift label and the pre-trained class label in the source domain model into several encoder layers of the source domain model; In each encoder layer, based on the domain drift mark, the class mark and the lifting denoising module of the previous layer, the domain drift features of the current encoder layer and the noise in the image features corresponding to the target domain image are separated; Using a preset information entropy loss function to back-propagate, the domain drift marker and the network parameters in the boost denoising module are updated to obtain an optimized model; The optimization model is adjusted using a preset similarity loss function to obtain a target domain model.

4. The test-time adaptive method based on dual-path adversarial boosting denoising according to claim 3, characterized in that: The lifting and denoising module includes a prediction network and an update network; In each encoder layer, based on the domain drift mark, class mark and the lifting denoising module of the previous layer, the domain drift feature of the current encoder layer and the noise in the image feature corresponding to the target domain image are separated, including: In each encoder layer, the domain drift mark of the previous layer is input into the prediction network to predict the domain drift feature of the current encoder layer; Inputting the domain drift feature and the class label into the update network to separate the noise in the image feature corresponding to the target domain image; Wherein, the initialization network parameter of the prediction network is 0.

5. The test-time adaptive method based on dual-path adversarial boosting denoising according to claim 3, characterized in that: The ratio of the weight occupied by the similarity loss function to the weight occupied by the information entropy loss function is a preset balance parameter.

6. The test-time adaptive method based on dual-path adversarial boosting denoising according to claim 3, characterized in that: Several target domain images in each batch are simultaneously input into the pre-trained source domain model, and the source domain model used in the current batch is the target domain model updated in the previous batch.

7. The test-time adaptive method based on dual-path adversarial boosting denoising according to claim 3, characterized in that: Inputting the target domain image into the target domain model to obtain a corresponding image classification result includes: All the target domain images in the current batch are input into the target domain model to obtain image classification results corresponding to each target domain image in the current batch.

8. A test-time adaptive device based on dual-path adversarial boosting denoising, characterized in that: The device comprises: An input module, used to obtain a target domain image, input the target domain image into a pre-trained source domain model, wherein a learnable domain drift marker is added to the source domain model, and a lifting and denoising module is added to each encoder layer of the source domain model, wherein the domain drift marker is used to learn the category-inseparable features in the target domain image, and the lifting and denoising module is used to denoise the image features of the target domain image; An updating module, configured to convert the target domain image into an image block vector by using an embedding layer in the source domain model, input the image block vector, the domain drift mark and the pre-trained class mark in the source domain model into several encoder layers of the source domain model, and back-propagate and update the domain drift mark and the network parameters in the lifting and denoising module to obtain a target domain model; The classification module is used to input the target domain image into the target domain model to obtain a corresponding image classification result.

9. A terminal, characterized in that: include: A memory, a processor, and a test-time adaptive program based on dual-path adversarial boosting denoising stored in the memory and executable on the processor, wherein the test-time adaptive program based on dual-path adversarial boosting denoising, when executed by the processor, implements the steps of the test-time adaptive method based on dual-path adversarial boosting denoising as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, which can be executed to implement the steps of the test-time adaptive method based on dual-path adversarial boosting denoising as described in any one of claims 1 to 7.

Images