Image super-resolution method and device based on passive domain adaptation

Abstract

The application provides an image super-resolution method and device based on passive domain adaptation, which comprises the following steps: inputting an unlabeled image to be processed into a pre-trained source domain model to obtain a pseudo label corresponding to the image to be processed, wherein the source domain model is trained based on labeled sample images; inputting the image to be processed into a target domain model to obtain a target super-resolution image and a target low-resolution image, wherein the target domain model is trained based on the pseudo label. The image super-resolution method based on passive domain adaptation can improve the generalization performance of the model and reduce the gap between the source domain and the target domain.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of deep learning, and in particular to an image super-resolution method and device based on passive domain adaptation. BACKGROUND

[0002] Single image super-resolution is a basic task in low-level vision, and has high commercial value. In recent years, image super-resolution has been an active research topic because of its high practical value in enhancing image details and textures, and it is widely used in various computer vision applications, including security image monitoring, medical image reconstruction, video enhancement and image segmentation, etc.

[0003] Image super-resolution is an image conversion task aiming to reconstruct a high-resolution image from a degraded low-resolution image. Benefiting from the booming development of deep learning, many deep learning-based methods have been applied to single image super-resolution and have achieved great success in this task. Most of the methods utilize advanced deep neural network architectures, such as convolutional neural networks or Vision Transformers (ViT), etc. However, the rapid development of image super-resolution relies on a large number of low-resolution and high-resolution image pairs, which are usually fed into the network for supervised learning. Due to the difficulty in obtaining real image pairs, most of the existing methods are trained on synthetic datasets, which synthesize low-resolution images through simple and deterministic degradation kernels. However, there is a big difference between synthetic low-resolution images and real low-resolution images, and the degradation in real scenarios is much more complex. There is also a significant gap between the degradation kernels of different cameras when taking low-resolution and high-resolution images in the same scene, i.e., the domain gap across devices. Therefore, the model trained on synthetic datasets or datasets with domain gaps may be difficult to generalize to real-world practical applications. SUMMARY

[0004] The present application provides an image super-resolution method and device based on passive domain adaptation to solve the problem of domain gap between different image data in the prior art.

[0005] The present application provides an image super-resolution method based on passive domain adaptation, comprising:

[0006] inputting an unlabeled image to be processed into a pre-trained source domain model to obtain a pseudo label corresponding to the image to be processed, the source domain model being trained based on labeled sample images;

[0007] inputting the image to be processed into a target domain model to obtain a target super-resolution image and a target low-resolution image, the target domain model being trained based on the pseudo label.

[0008] The source domain model is constructed based on a head structure, a feature extractor, and a tail structure.

[0009] The target domain model includes a first model, a second model, and a third model, the structure of the first model is the same as the structure of the source domain model, the second model is constructed based on the first model and a plurality of batch enhancement layers, the third model is constructed based on a down-sampling network, the batch enhancement layer is used for data enhancement, and the down-sampling network is used to obtain the target low-resolution image based on the target super-resolution image.

[0010] In some embodiments, the inputting the unlabeled to-be-processed image into the pre-trained source domain model to obtain a pseudo label corresponding to the to-be-processed image comprises:

[0011] The to-be-processed image is input into the head structure to obtain a shallow feature of the to-be-processed image.

[0012] The to-be-processed image is input into the feature extractor to obtain a deep feature of the to-be-processed image.

[0013] The shallow feature and the deep feature are input into the tail structure to obtain the pseudo label.

[0014] In some embodiments, the target super-resolution image includes a first super-resolution image and a second super-resolution image, and the target low-resolution image includes a first low-resolution image and a second low-resolution image.

[0015] The inputting the to-be-processed image into the target domain model to obtain a target super-resolution image and a target low-resolution image comprises:

[0016] The to-be-processed image is input into the first model to obtain the first super-resolution image.

[0017] The to-be-processed image is input into the second model to obtain the second super-resolution image.

[0018] The first super-resolution image is input into the third model to obtain the first low-resolution image.

[0019] The second super-resolution image is input into the third model to obtain the second low-resolution image.

[0020] In some embodiments, the target domain model is trained in the following manner:

[0021] A first loss function is constructed based on the first super-resolution image, the second super-resolution image, and the pseudo label.

[0022] constructing a second loss function based on the first low-resolution image and the second low-resolution image;

[0023] constructing a third loss function based on the image to be processed, a discriminator, the target low-resolution image and the target domain model, the discriminator being used for distinguishing the source of the target low-resolution image;

[0024] training the target domain model based on the first loss function, the second loss function and the third loss function.

[0025] In some embodiments, the batch enhancement layer is constructed based on a batch enhancement attention mechanism, layer normalization and a multi-layer perception mechanism, and the down-sampling network is constructed based on a plurality of convolution layers with down-sampling, a plurality of convolution layers without down-sampling and a plurality of residual blocks.

[0026] In some embodiments, the head structure comprises one convolution layer, the feature extractor comprises six residual blocks and one convolution layer, and the tail structure comprises an up-sampling layer and a plurality of convolution layers.

[0027] The application further provides an image super-resolution device based on passive domain adaptation, comprising:

[0028] a first processing module configured to input an unlabeled image to be processed into a pre-trained source domain model to obtain a pseudo label corresponding to the image to be processed, the source domain model being trained based on labeled sample images;

[0029] a second processing module configured to input the image to be processed into a target domain model to obtain a target super-resolution image and a target low-resolution image, the target domain model being trained based on the pseudo label;

[0030] The source domain model is constructed based on a head structure, a feature extractor and a tail structure.

[0031] The target domain model comprises a first model, a second model and a third model, the structure of the first model being the same as that of the source domain model, the second model being constructed based on the first model and a plurality of batch enhancement layers, and the third model being constructed based on a down-sampling network, the batch enhancement layers being used for data enhancement, and the down-sampling network being used for obtaining the target low-resolution image based on the target super-resolution image.

[0032] The application further provides an electronic device comprising a memory, a processor and a computer program stored in the memory and executable on the processor, the processor implementing the above-mentioned image super-resolution method based on passive domain adaptation when executing the program.

[0033] The application further provides a non-transitory computer-readable storage medium, which stores a computer program, and the computer program is executed by a processor to implement the image super-resolution method based on source domain self-adaption according to any one of the above.

[0034] The application further provides a computer program product, which comprises a computer program, and the computer program is executed by a processor to implement the image super-resolution method based on source domain self-adaption according to any one of the above.

[0035] The image super-resolution method based on source domain self-adaption provided by the application can obtain a target domain model with good performance by using the unlabeled image to be processed and the pre-trained source domain model to perform knowledge transfer without accessing the source domain data, and can generate a target super-resolution image and a target low-resolution image by using a first model, a second model and a third model in the target domain model, so as to further improve the performance of the target domain model by completing knowledge transfer through a self-training mechanism and reducing the gap between the source domain and the target domain, and improve the robustness of the target domain model by performing data enhancement through a batch enhancement layer. BRIEF DESCRIPTION OF DRAWINGS

[0036] In order to more clearly illustrate the technical solutions of the application or the prior art, the following will briefly introduce the drawings needed to be used in the embodiments or prior art description. Obviously, the drawings in the following description are some embodiments of the application, and other drawings can be obtained by those skilled in the art without creative labor.

[0037] Figure 1 is a flowchart of the image super-resolution method based on source domain self-adaption provided by the application;

[0038] Figure 2 is a schematic diagram of the source domain self-adaption framework structure of the image super-resolution method based on source domain self-adaption provided by the application;

[0039] Figure 3 is a schematic diagram of the source domain model structure of the image super-resolution method based on source domain self-adaption provided by the application;

[0040] Figure 4 is a schematic diagram of the residual block structure in the source domain model of the image super-resolution method based on source domain self-adaption provided by the application;

[0041] Figure 5 is a schematic diagram of the STL structure of the image super-resolution method based on source domain self-adaption provided by the application;

[0042] Figure 6A flowchart of a batch enhancement layer and a batch enhancement attention mechanism of an image super-resolution method based on passive domain adaptation provided by the present application is shown in FIG. 1.

[0043] Figure 7 A down-sampling network structure diagram of the image super-resolution method based on passive domain adaptation provided by the present application is shown in FIG. 2.

[0044] Figure 8 A residual block structure diagram in the down-sampling network of the image super-resolution method based on passive domain adaptation provided by the present application is shown in FIG. 3.

[0045] Figure 9 A structure diagram of the image super-resolution device based on passive domain adaptation provided by the present application is shown in FIG. 4.

[0046] Figure 10 A structure diagram of the electronic device provided by the present application is shown in FIG. 5. DETAILED DESCRIPTION

[0047] To make the objectives, technical solutions and advantages of the present application clearer, the technical solutions in the present application will be described clearly and completely below with reference to the drawings in the present application. Obviously, the described embodiments are only some of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without any creative work fall within the scope of protection of the present application.

[0048] In related art, there are some existing technologies to explore the relationship between low-resolution and high-resolution image pairs in real scenes, such as unsupervised learning, domain adaptation, etc. In addition, there are some data collected in real scenes to train and test image super-resolution models for real scenes. These data sets include low-resolution and high-resolution image pairs taken by different cameras in the same scene. Since there is a significant gap between the degradation kernels of different cameras, this can be regarded as a cross-device domain gap. To solve the problem of domain gap between different data, the current main method is unsupervised domain adaptation method, which uses source domain data and unlabeled target domain data to improve the model in realizing knowledge transfer and improve the performance of the model in the target domain. However, this kind of method assumes that the source domain data is accessible and can be obtained, because the source domain data can make the model retain valuable source domain knowledge in the training process, and plays an important role in the process of iteratively reducing cross-domain differences. In some real scene applications, due to some privacy policies or transmission restrictions, the source domain data cannot be obtained, only the trained source domain model and the unlabeled target domain data set are available. Due to the lack of source domain supervision and the uncertainty of pseudo-labels, these unsupervised domain adaptation methods cannot work in this source-free scenario.

[0049] In order to solve the above problems in the prior art, the present application provides an image super-resolution method based on passive domain adaptation, which can effectively utilize the information in the source domain model and the unlabeled target domain data by using a specific data enhancement method and a self-training mechanism, realize knowledge transfer, and improve the robustness of the model.

[0050] The execution subject of the image super-resolution method based on passive domain adaptation provided by the present application can be an electronic device, a component in the electronic device, an integrated circuit, or a chip. The electronic device can be a mobile electronic device or a non-mobile electronic device. Illustratively, the mobile electronic device can be a mobile phone, a tablet computer, a notebook computer, a palm computer, a vehicle-mounted electronic device, a wearable device, an ultra-mobile personal computer (UMPC), a netbook, or a personal digital assistant (PDA), etc., and the non-mobile electronic device can be a server, a network attached storage (NAS), a personal computer (PC), a television (TV), a teller machine, or a self-service machine, etc., which are not limited by the present application.

[0051] The technical solutions of the present application will be described in detail below with the computer executing the image super-resolution method based on passive domain adaptation provided by the present application as an example.

[0052] Figure 1 is a flowchart of the image super-resolution method based on passive domain adaptation provided by the present application. Referring to Figure 1 The image super-resolution method based on passive domain adaptation provided by the present application can include steps 110 and 120.

[0053] In step 110, the unlabeled image to be processed is input into the pre-trained source domain model to obtain the pseudo label corresponding to the image to be processed, and the source domain model is trained based on the labeled sample image;

[0054] In step 120, the image to be processed is input into the target domain model to obtain the target super-resolution image and the target low-resolution image, and the target domain model is trained based on the pseudo label;

[0055] The source domain model is constructed based on a head structure, a feature extractor, and a tail structure.

[0056] The target domain model comprises a first model, a second model and a third model, the structure of the first model is the same as that of the source domain model, the second model is constructed based on the first model and a plurality of batch enhancement layers, and the third model is constructed based on a down-sampling network; the batch enhancement layer is used for data enhancement, and the down-sampling network is used for obtaining a target low-resolution image based on a target super-resolution image.

[0057] In step 110, the source domain model is pre-trained by supervised learning of labeled source domain data, and a pre-trained source domain model is obtained. The source domain data is a sample image, and the parameters of the source domain model are also updated by supervised training.

[0058] After the pre-training of the source domain model is completed, no source domain data will be obtained. Then, the pre-trained source domain model can be used to predict unlabeled images.

[0059] In actual execution, an unlabeled image to be processed is obtained as an initial input image, and the image to be processed is input into the pre-trained source domain model for super-resolution processing to obtain pseudo-labels output by the source domain model, which are super-resolution images.

[0060] It should be noted that the image to be processed is a low-resolution image, and the image to be processed is used for self-training of the target domain model. The source domain model comprises a head structure, a feature extractor and a tail structure, and the head structure and the tail structure are additionally connected by a branch. The source domain model can convert a low-resolution image into a super-resolution image.

[0061] In step 120, the image to be processed is input into the target domain model for image reconstruction, and a reconstructed target super-resolution image and a target low-resolution image can be obtained.

[0062] The parameters of the target domain model are updated by self-training.

[0063] The target domain model in the embodiment of the application comprises a first model, a second model and a third model, the target domain model performs knowledge transfer by unsupervised learning, improves the performance of the model in the target domain, and reduces the gap between the model in the source domain and the model in the target domain.

[0064] The first model is used to indicate a real stream, the second model is used to indicate an enhanced stream, and the third model is used to indicate a down-sampling network.

[0065] The structure of the first model is the same as that of the source domain model, and the first model comprises a head structure, a feature extractor and a tail structure. The model parameters of the first model are initialized as the model parameters of the pre-trained source domain model. The model parameters of the first model and the model parameters of the second model can be shared.

[0066] The second model is based on the first model, and a plurality of batch enhancement layers are added between the feature extractor and the tail structure to convert the input image to be processed into a data-enhanced super-resolution image. The batch enhancement layer calculates self-attention in the batch dimension based on the batch self-attention mechanism, and performs batch mixing in the feature space of the image, so as to obtain implicitly enhanced data. The batch enhancement layer is used to learn a proper data enhancement method to improve the robustness of the target domain model

[0067] In some embodiments, the down-sampling network is configured as a series of convolutional layers with and without down-sampling and residual blocks. The down-sampling network is used to extract features of the target super-resolution image and predict the corresponding target low-resolution image.

[0068] In actual execution, the image to be processed is subjected to super-resolution processing by the pre-trained source domain model to obtain pseudo labels. The image to be processed is input into the target domain model composed of the first model, the second model and the third model, and subjected to super-resolution processing and down-sampling processing to obtain the target super-resolution image and the reconstructed target low-resolution image.

[0069] The image super-resolution method based on passive domain adaptation provided by the application can perform knowledge transfer by using the unlabeled image to be processed and the pre-trained source domain model without accessing the source domain data, obtain a target domain model with good performance, and generate the target super-resolution image and the target low-resolution image by using the first model, the second model and the third model in the target domain model, complete knowledge transfer by the self-training mechanism, further improve the performance of the target domain model, reduce the gap between the source domain and the target domain, and improve the robustness of the target domain model by the batch enhancement layer.

[0070] In some embodiments, the unlabeled image to be processed is input into the pre-trained source domain model to obtain the pseudo label corresponding to the image to be processed, including:

[0071] The image to be processed is input into the head structure to obtain the shallow feature of the image to be processed.

[0072] The image to be processed is input into the feature extractor to obtain the deep feature of the image to be processed.

[0073] The shallow feature and the deep feature are input into the tail structure to obtain the pseudo label.

[0074] In actual execution, the head structure adopts a convolutional layer to extract the shallow feature of the image to be processed, and directly transmits the shallow feature to the tail structure through a residual connection, so as to retain the low-frequency information contained in the image to be processed. The shallow feature contains more information of pixel points, which can be some fine-grained information, such as color, texture, edge and corner information of the image.

[0075] The feature extractor is configured to extract deep features of the image to be processed. The deep features can be some coarse-grained information, which contains more abstract information, such as semantic information, etc.

[0076] The feature extractor is composed of a plurality of Swin Transformer blocks, each block is composed of a plurality of Swin Transformer layers to calculate local self-attention and realize information interaction across windows, a convolution layer is added at the end of each block for feature enhancement, and residual connection is used for feature aggregation, so as to extract deep features of the image to be processed.

[0077] The tail structure can perform pixel-level addition operation on the shallow features and the deep features of the image to be processed to obtain high-resolution image features, and then restore a high-resolution image, i.e., a pseudo label. The sub-pixel convolution layer used in the tail structure up-samples the extracted features to predict a super-resolution result image.

[0078] In some embodiments, the head structure includes one convolution layer, the feature extractor includes six residual blocks and one convolution layer, and the tail structure includes an up-sampling layer and a plurality of convolution layers.

[0079] In actual execution, the head structure is configured as a 3x3 convolution layer to map the image to a higher-dimensional feature space and extract shallow features of the image.

[0080] The feature extractor is configured as six residual Swin Transformer blocks, each block is composed of six Swin Transformer layers and one convolution layer, residual connection is used to complete the aggregation of features, the Transformer can be regarded as a kind of spatially varying convolution, and the convolution layer can use a spatially invariant filter to improve the translational invariance of the model, residual connection is used to aggregate features at different levels, and deep features of the image are extracted.

[0081] The tail structure is configured as a series of stacked convolution layers and up-sampling layers, and the extracted features are used to predict a super-resolution image, i.e., a pseudo label.

[0082] In some embodiments, the target super-resolution image includes a first super-resolution image and a second super-resolution image, and the target low-resolution image includes a first low-resolution image and a second low-resolution image.

[0083] The image to be processed is input into the target domain model to obtain a target super-resolution image and a target low-resolution image, including:

[0084] The image to be processed is input into the first model to obtain a first super-resolution image.

[0085] inputting the image to be processed into the second model to obtain a second super-resolution image;

[0086] inputting the first super-resolution image into a third model to obtain a first low-resolution image;

[0087] inputting the second super-resolution image into the third model to obtain a second low-resolution image.

[0088] In actual execution, the first model is the same as the source domain model in structure, and the image to be processed is input into the first model for super-resolution processing to obtain a real super-resolution image r(x), that is, the first super-resolution image. It should be noted that the target domain model only uses the output of the real stream as the super-resolution result in the test process, that is, the first super-resolution image output by the first model.

[0089] The second model contains the structure of the first model, and the difference between the structure of the second model and the structure of the first model is that a plurality of batch enhancement layers are added between the feature extractor and the tail structure, and the deep features extracted by the feature extractor can be input into the batch enhancement layers. Therefore, the image to be processed can be converted into a super-resolution image a(x) after data enhancement, that is, the second super-resolution image.

[0090] In some embodiments, the batch enhancement layer is built based on a batch enhancement attention mechanism, layer normalization and a multi-layer perception mechanism. The multi-layer perception mechanism can be a fully connected layer.

[0091] Firstly, the batch enhancement attention mechanism is used to realize the feature interaction and aggregation between different samples in a batch, so as to implicitly realize data enhancement in the hidden space. Then, the layer normalization and the fully connected layer are used to finally obtain the enhanced features.

[0092] The batch enhancement attention mechanism can be configured to first exchange the dimensions of all input features in a batch, then calculate the query feature matrix, the key feature matrix and the value feature matrix in the batch dimension, multiply the query feature matrix and the key feature matrix through a softmax layer for classification, multiply the output result with the value feature matrix, exchange the dimensions, and finally realize the feature aggregation between different samples and realize data enhancement.

[0093] inputting the first super-resolution image r(x) into a downsampling network in the third model to obtain a reconstructed real low-resolution image l r (x), that is, the first low-resolution image.

[0094] inputting the second super-resolution image a(x) into the downsampling network in the third model to obtain a reconstructed low-resolution image l a (x) after data enhancement, that is, the second low-resolution image.

[0095] In some embodiments, the down-sampling network is constructed based on a plurality of down-sampling convolution layers, a plurality of convolution layers without down-sampling, and a plurality of residual blocks.

[0096] In actual implementation, the down-sampling network can be configured to output the super-resolution images of the real stream and the enhanced stream through 2 convolution layers without down-sampling, 8 residual blocks, and 2 convolution layers with down-sampling, and finally reconstruct the first low-resolution image and the second low-resolution image.

[0097] The image super-resolution method based on passive domain adaptation provided by the present application can realize the addition of the enhanced attention mechanism as a plug-and-play module to the existing model through the batch enhancement layer in the second model, realize data enhancement at the feature level, improve the robustness of the model to various images, and also improve the generalization performance of the target domain model, improve the generation effect of the super-resolution image of the target domain, and ensure the stability of the super-resolution quality.

[0098] In order to more clearly illustrate the present application, the image super-resolution method based on passive domain adaptation is described from two aspects of a passive domain adaptation framework and an image super-resolution passive domain adaptation method in combination with the accompanying drawings.

[0099] 1. Passive domain adaptation framework

[0100] As shown in Figure 2 , the passive domain adaptation framework in the embodiment of the present application includes a source domain model and a target domain model composed of a real stream, an enhanced stream, and a down-sampling network. Among them, the real stream corresponds to the first model in the above embodiment, the enhanced stream corresponds to the second model in the above embodiment, and the down-sampling network corresponds to the third model in the above embodiment.

[0101] (1) Source domain model

[0102] As shown in Figure 3 , the source domain model structure is composed of a head structure, a feature extractor, and a tail structure. Among them, the head structure is composed of one convolution layer, and the feature extractor is composed of 6 residual Swin Transformer blocks (RSTB) and one convolution layer.

[0103] As shown in Figure 4 , each residual Swin Transformer block (RSTB) is composed of 6 Swin Transformer layers (STL) and one convolution layer. As shown in Figure 5As shown, the STL is composed of a standard multi-head self-attention mechanism (Multi-Head Self-Attention, MSA), layer normalization (LayerNorm), and a multi-layer perceptron (Multi-Layer Perceptron, MLP), and the multi-head self-attention is calculated by moving the window. The tail structure is composed of a series of convolutional layers and the last sampling layer, and the extracted features are used to realize the super-resolution processing of the image.

[0104] In actual execution, the shallow features extracted from the head structure and the deep features extracted from the feature extractor are fused in the tail structure in the order of the head structure, the feature extractor, and the tail structure, to realize super-resolution image reconstruction.

[0105] For the input image to be processed x∈R 3×H×W For ×4 super-resolution, the source domain model can convert a low-resolution image to a super-resolution image ∈R 3×4H×4W The calculation process is as follows:

[0106] f source (x)=f tail (f head (x)+f fe (f head (x)))

[0107] Where x is the input image to be processed, f head represents the head structure, ff fe represents the feature extractor, and f tail represents the tail structure.

[0108] (2) Target domain model

[0109] As shown in Figure 2 , the target domain model is mainly composed of a real stream, an enhanced stream, and a down-sampling network. The structure of the real stream is exactly the same as that of the source domain model, the structure of the enhanced stream additionally includes a number of batch enhancement layers, and the down-sampling network is composed of a series of convolutional layers with and without down-sampling and residual blocks. For the target domain model, the following steps are included:

[0110] The image to be processed is passed through the real stream, and the shallow features extracted from the head and the deep features extracted from the feature extractor are fused in the tail in the order of the head structure, the feature extractor, and the tail structure, to realize real super-resolution image reconstruction. The calculation process is as follows:

[0111] f real (x)=f tail (f head (x)+f fe (f head (x)))

[0112] where x is the input image to be processed, f head represents the head structure, f fe represents the feature extractor, f tail represents the tail structure.

[0113] The image to be processed is passed through the enhanced flow, and the shallow features extracted by the head structure and the deep features after data enhancement are fused in the tail structure in the order of the head structure, the feature extractor, the batch enhancement layer and the tail structure, to realize the reconstruction of the super-resolution image after data enhancement. As shown in Figure 6 , the batch enhancement layer is composed of a batch enhancement attention mechanism, layer normalization and a multi-layer perception.

[0114] The calculation process of the batch enhancement attention mechanism is as follows:

[0115] For the input feature z∈R B×N×C , where N=P h ×P w , B is the batch size, P h is the image block height, P w is the image block width, and C is the number of channels. First, the dimensions are exchanged so that z∈R N×B×C , the query feature, key feature and value feature matrices Q, K, V∈R N×B×C are calculated, and then the output matrix ∈N×B×C is calculated, the process being as follows:

[0116]

[0117] Then the dimensions of the output matrix are exchanged to obtain the final output ∈B×N×C. For the batch enhancement layer, the input feature z∈R B×N×C is calculated after batch enhancement attention and layer normalization, and then passed through a multi-layer perception and layer normalization, and each module is connected with a residual connection. The calculation process is as follows:

[0118] f BAL (z)=f LN (f MLP (f LN (f BAA (z)+z))+f LN (f BAA (z)+z))

[0119] where z is the input feature, f LN represents layer normalization, f MLP represents a multi-layer perception, and f BAA represents a batch enhancement attention mechanism.

[0120] For the entire enhanced stream, the calculation process is as follows:

[0121] f augmentation (x)=f tail (f head (x)+f BdL (f fe (f head (x))))

[0122] Wherein, x is the input picture, f head represents the head, f fe represents the feature extractor, f tail represents the tail, f BAL represents the batch enhancement layer.

[0123] In actual execution, the target super-resolution images output by the real stream and the enhanced stream are all output through the down-sampling network, such as Figure 7 As shown in the figure, the down-sampling network is composed of a series of convolution layers with down-sampling, convolution layers without down-sampling and residual blocks, and the structure of the residual block is as shown in Figure 8 Finally, the reconstructed real low-resolution image and the enhanced low-resolution image are obtained, and the calculation process is as follows:

[0124] [x real ,x augmentation ]=[f DSN (y real ),f DSN (y augnentation )]

[0125] Wherein, x real is the reconstructed real low-resolution image, i.e. the first low-resolution image; x augmentation is the reconstructed enhanced low-resolution image, i.e. the second low-resolution image; f DSN is the down-sampling network, y real is the super-resolution image output by the real stream, i.e. the first super-resolution image; y augmentation is the super-resolution image output by the enhanced stream, i.e. the second super-resolution image.

[0126] In the teacher-student architecture in the self-training mechanism, the student network is the real stream in the target domain model, and the teacher network is the source domain model. After each iteration is completed, the teacher network is updated by the way of exponential moving average, and the teacher network parameters are updated by the exponential moving average of the student network parameters. The calculation process is as follows:

[0127] θ tea =ηθ tea +(1-η)θ stu

[0128] Wherein, θ teaθ represents all parameters of the teacher network stu η represents all parameters of the student network, and η is a decay rate and is set to 0.999.

[0129] The application provides an image super-resolution method based on passive field adaptation, which performs knowledge transfer by using a pre-trained source domain model and unlabeled target domain data without accessing source domain data. The input image enters the source domain model to provide pseudo-labels for unsupervised learning as a supervision signal, and then enters the source domain model composed of a real flow, an enhanced flow and a down-sampling network. The proposed batch enhancement attention mechanism performs implicit data enhancement in the image feature hidden space, so that the model is more robust. The application uses a self-training mechanism to transfer the knowledge learned in the student network to the teacher network through a teacher-student architecture, so that the two networks can progress together and improve the super-resolution performance of the model.

[0130] (3) Model training process

[0131] In order to enable the target domain model to effectively train in the knowledge transfer process, a series of loss functions are introduced in the training process to make the model have better super-resolution effect.

[0132] In some embodiments, a first loss function is constructed based on the first super-resolution image, the second super-resolution image and the pseudo-label.

[0133] A pixel-by-pixel reconstruction loss is used to constrain the training process of the target domain model, so that the generated super-resolution image is close to the pseudo-label pixel by pixel, and a perception loss is added to improve the perception quality of the image. The loss of the real flow and the enhanced flow is represented as the first loss function:

[0134]

[0135]

[0136] wherein y real is a super-resolution image output by the real flow, i.e. the first super-resolution image; y augmentation is a super-resolution image output by the enhanced flow, i.e. the second super-resolution image; y pseudo is a pseudo-label, and φ() represents a VGG-19 feature extractor, the output features of the conv 5_3 layer of which will be used to calculate the perception loss.

[0137] In some embodiments, a second loss function is constructed based on the first low-resolution image and the second low-resolution image.

[0138] Whether the image sent into the real stream or the enhanced stream, the image will pass through a cycle reconstruction process: low resolution image→high resolution image→low resolution image, after the low resolution image is reconstructed, the pixel level L1 loss between the low resolution images is calculated, and the second loss function is represented as:

[0139] L rec =||x real -x augmentation ||1

[0140] Wherein, x real is the reconstructed real low resolution image, that is, the first low resolution image;x augmentation It is the reconstructed enhanced low resolution image, that is, the second low resolution image.

[0141] In some embodiments, based on the image to be processed, the discriminator, the target low resolution image and the target domain model, a third loss function is constructed, and the discriminator is used to distinguish the source of the target low resolution image.

[0142] In order to further improve the visual perception quality of the super resolution image, the adversarial learning is introduced in the training process for the real stream and the enhanced stream. Two additional discriminator networks are designed to be against the generator network, and the generator is the real stream / enhanced stream and the down-sampling network, which generates the reconstructed low resolution image. The discriminator is used to distinguish whether the generated low resolution image is from the target domain, and the generator is used to deceive the discriminator. Wherein, the third loss function of the generator and the discriminator is respectively represented as:

[0143]

[0144]

[0145] Wherein, x is the input image to be processed;G is the generator;G(x) is the generated low resolution image, that is, the target low resolution image;D is the discriminator.

[0146] In actual execution, based on the first loss function, the second loss function and the third loss function, the target domain model is trained.

[0147] 2、Image super resolution method based on passive domain adaptation

[0148] Based on the passive domain adaptation framework proposed in the above embodiments, the image super resolution method based on passive domain adaptation provided by the application can include:

[0149] Obtain the input image to be processed and the pre-trained source domain model;

[0150] The pre-trained source domain model is used for image super resolution processing of the image to be processed, and pseudo label is generated.

[0151] The target super-resolution image and the reconstructed target low-resolution image are generated by using the real flow, the enhanced flow and the down-sampling network in the target domain model, the knowledge migration is completed through the self-training mechanism, and the performance of the target domain model is improved.

[0152] The image super-resolution method based on passive domain adaptation provided by the embodiment of the application is tested on the DRealSR database, and the test results are shown in the following table.

[0153] Method Panasonic → Sony Sony → Panasonic Olympus → Panasonic SODA-SR 32.16 31.25 31.28

[0154] The image super-resolution method based on passive domain adaptation provided by the application has the following advantages

[0155] Advantages:

[0156] 1. The generalization performance of the model is improved, the generation effect of the target domain high-resolution image is improved, and the stability of the super-resolution quality is ensured.

[0157] 2. Through a passive domain adaptation method, unsupervised training is performed on the target domain data and the pre-trained source domain model without accessing the source domain data, thereby improving the performance of the model in the target domain.

[0158] 3. The batch enhancement attention mechanism can be added to the existing model as a plug-and-play module to perform data enhancement at the feature level and improve the robustness of the model to various images.

[0159] 4. The self-training mechanism can transfer the knowledge learned by the target domain model to the source domain model, improve the accuracy of the generated pseudo label, and thereby improve the quality of the super-resolution.

[0160] The image super-resolution device based on passive domain adaptation provided by the application is described below, and the image super-resolution device based on passive domain adaptation described below can be correspondingly referred to the image super-resolution method based on passive domain adaptation described above.

[0161] Figure 9 It is a structural schematic diagram of the image super-resolution device based on passive domain adaptation provided by the application. Referring to Figure 9 The image super-resolution device based on passive domain adaptation provided by the application can include a first processing module 910 and a second processing module 920.

[0162] The first processing module 910 is configured to input the unlabeled image to be processed into the pre-trained source domain model to obtain the pseudo label corresponding to the image to be processed, and the source domain model is trained based on the labeled sample image.

[0163] The second processing module 920 is configured to input the image to be processed into a target domain model to obtain a target super-resolution image and a target low-resolution image, wherein the target domain model is trained based on the pseudo label.

[0164] The source domain model is constructed based on a head structure, a feature extractor and a tail structure.

[0165] The target domain model comprises a first model, a second model and a third model, the structure of the first model is the same as that of the source domain model, the second model is constructed based on the first model and a plurality of batch enhancement layers, the third model is constructed based on a down-sampling network, the batch enhancement layer is used for data enhancement, and the down-sampling network is used for obtaining the target low-resolution image based on the target super-resolution image.

[0166] The image super-resolution device based on passive field adaptation provided by the application can perform knowledge transfer by using the image to be processed without label and the pre-trained source domain model without accessing the source domain data, obtain a target domain model with good performance, generate a target super-resolution image and a target low-resolution image by using the first model, the second model and the third model in the target domain model, complete knowledge transfer through a self-training mechanism, further improve the performance of the target domain model, reduce the gap between the source domain and the target domain, perform data enhancement through the batch enhancement layer, and improve the robustness of the target domain model.

[0167] In some embodiments, the first processing module 910 is further configured to:

[0168] input the image to be processed into the head structure to obtain shallow features of the image to be processed;

[0169] input the image to be processed into the feature extractor to obtain deep features of the image to be processed;

[0170] input the shallow features and the deep features into the tail structure to obtain the pseudo label.

[0171] In some embodiments, the target super-resolution image comprises a first super-resolution image and a second super-resolution image, and the target low-resolution image comprises a first low-resolution image and a second low-resolution image.

[0172] The second processing module 920 is further configured to:

[0173] input the image to be processed into the first model to obtain the first super-resolution image;

[0174] input the image to be processed into the second model to obtain the second super-resolution image.

[0175] The first super-resolution image is input into the third model to obtain the first low-resolution image;

[0176] The second super-resolution image is input into the third model to obtain the second low-resolution image.

[0177] In some embodiments, the target domain model is trained in the following manner:

[0178] Based on the first super-resolution image, the second super-resolution image, and the pseudo-label, a first loss function is constructed;

[0179] Based on the first low-resolution image and the second low-resolution image, a second loss function is constructed;

[0180] Based on the image to be processed, the discriminator, the target low-resolution image, and the target domain model, a third loss function is constructed, wherein the discriminator is used to distinguish the source of the target low-resolution image;

[0181] The target domain model is trained based on the first loss function, the second loss function, and the third loss function.

[0182] In some embodiments, the batch enhancement layer is constructed based on a batch enhancement attention mechanism, layer normalization, and a multilayer perceptron, and the downsampling network is constructed based on multiple convolutional layers with downsampling, multiple convolutional layers without downsampling, and multiple residual blocks.

[0183] In some embodiments, the head structure includes one convolutional layer, the feature extractor includes six residual blocks and one convolutional layer, and the tail structure includes an upsampling layer and multiple convolutional layers.

[0184] Figure 10 An example is a schematic diagram of the physical structure of an electronic device, such as... Figure 10 As shown, the electronic device may include: a processor 1010, a communications interface 1020, a memory 1030, and a communication bus 1040, wherein the processor 1010, the communications interface 1020, and the memory 1030 communicate with each other via the communication bus 1040. The processor 1010 can call logical instructions in the memory 1030 to execute a passive-domain adaptive image super-resolution method, which includes:

[0185] The unlabeled image to be processed is input into the pre-trained source domain model to obtain the pseudo label corresponding to the image to be processed. The source domain model is trained based on labeled sample images.

[0186] inputting the image to be processed into a target domain model to obtain a target super-resolution image and a target low-resolution image, the target domain model being obtained based on the pseudo label;

[0187] The source domain model is constructed based on a head structure, a feature extractor, and a tail structure.

[0188] The target domain model comprises a first model, a second model, and a third model, the structure of the first model being the same as that of the source domain model, the second model being constructed based on the first model and a plurality of batch enhancement layers, the third model being constructed based on a down-sampling network, the batch enhancement layers being used for data enhancement, and the down-sampling network being used for obtaining the target low-resolution image based on the target super-resolution image.

[0189] In addition, the logic instructions in the memory 1030 described above can be implemented in the form of a software function unit and sold or used as an independent product, and can be stored in a computer readable storage medium. Based on such understanding, the technical solutions of the present application essentially or the parts that contribute to the prior art or parts of the technical solutions can be embodied in the form of a software product. The computer software product is stored in a storage medium, and includes a plurality of instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present application. The aforementioned storage medium includes: a U disk, a mobile hard disk, a read-only memory (ROM, Read-Only Memory), a random access memory (RAM, Random Access Memory), a magnetic disk or an optical disk, and various media that can store program codes.

[0190] On the other hand, the present application also provides a computer program product, which comprises a computer program, the computer program can be stored on a non-transitory computer readable storage medium, and the computer program can be executed by a processor to enable a computer to execute the image super-resolution method based on passive field adaptation provided by the above-mentioned methods, the method comprising:

[0191] inputting the image to be processed into a target domain model to obtain a target super-resolution image and a target low-resolution image, the target domain model being obtained based on the pseudo label;

[0192] inputting the image to be processed into a target domain model to obtain a target super-resolution image and a target low-resolution image, the target domain model being obtained based on the pseudo label;

[0193] The source domain model is constructed based on a head structure, a feature extractor, and a tail structure.

[0194] The target domain model comprises a first model, a second model and a third model, a structure of the first model is same as a structure of the source domain model, the second model is constructed based on the first model and a plurality of batch enhancement layers, the third model is constructed based on a down-sampling network, the batch enhancement layers are used for data enhancement, and the down-sampling network is used for obtaining the target low-resolution image based on the target super-resolution image.

[0195] In yet another aspect, the present application also provides a non-transitory computer readable storage medium having a computer program stored thereon, the computer program being executed by a processor to implement the image super-resolution method based on passive domain adaptation provided by the above method, the method comprising: inputting an unlabeled to-be-processed image into a pre-trained source domain model to obtain a pseudo label corresponding to the to-be-processed image, the source domain model being trained based on a labeled sample image;

[0196] inputting the to-be-processed image into a target domain model to obtain a target super-resolution image and a target low-resolution image, the target domain model being trained based on the pseudo label;

[0197] The source domain model is constructed based on a head structure, a feature extractor and a tail structure.

[0198] The target domain model comprises a first model, a second model and a third model, a structure of the first model is same as a structure of the source domain model, the second model is constructed based on the first model and a plurality of batch enhancement layers, the third model is constructed based on a down-sampling network, the batch enhancement layers are used for data enhancement, and the down-sampling network is used for obtaining the target low-resolution image based on the target super-resolution image.

[0199] The device embodiments described above are only schematic, wherein the units shown as separate components can or can not be physically separate, and the components shown as units can or can not be physical units, i.e., can be located in one place or distributed on a plurality of network units. Some or all of the modules can be selected according to actual needs to achieve the purpose of the present embodiment scheme. Those skilled in the art can understand and implement without creative labor.

[0200] Those skilled in the art can clearly understand the technical solutions of the various embodiments from the above description of the embodiments, and the various embodiments can be implemented by means of software with the necessary general hardware platforms, and of course, can also be implemented by hardware. Based on such understanding, the above technical solutions, essentially or in other words, the part of the prior art that makes a contribution, can be embodied in the form of a software product, which can be stored in a computer readable storage medium, such as a ROM / RAM, a magnetic disk, an optical disk, and the like, and includes a number of instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0201] Finally, it should be noted that: the above embodiments are only used to illustrate the technical solutions of the present application, and not to limit them; although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that: it can still modify the technical solutions recorded in the foregoing embodiments, or make equivalent replacement for some technical features therein; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A passive-domain adaptive image super-resolution method, characterized in that, include: The unlabeled image to be processed is input into the pre-trained source domain model to obtain the pseudo label corresponding to the image to be processed. The source domain model is trained based on labeled sample images. The image to be processed is input into the target domain model to obtain a target super-resolution image and a target low-resolution image. The target domain model is trained based on the pseudo-label. The source domain model is constructed based on the head structure, feature extractor, and tail structure. The target domain model includes a first model, a second model, and a third model. The structure of the first model is the same as that of the source domain model. The second model is constructed based on the first model and multiple batch enhancement layers. The third model is constructed based on a downsampling network. The batch enhancement layers are used for data enhancement, and the downsampling network is used to obtain the target low-resolution image based on the target super-resolution image.

2. The image super-resolution method based on passive neighborhood adaptation according to claim 1, characterized in that, The step of inputting the unlabeled image to be processed into the pre-trained source domain model to obtain the pseudo-label corresponding to the image to be processed includes: The image to be processed is input into the head structure to obtain the shallow features of the image to be processed; The image to be processed is input into the feature extractor to obtain the deep features of the image to be processed; The shallow features and the deep features are input into the tail structure to obtain the pseudo-label.

3. The image super-resolution method based on passive neighborhood adaptation according to claim 1, characterized in that, The target super-resolution image includes a first super-resolution image and a second super-resolution image, and the target low-resolution image includes a first low-resolution image and a second low-resolution image; The step of inputting the image to be processed into the target domain model to obtain the target super-resolution image and the target low-resolution image includes: The image to be processed is input into the first model to obtain the first super-resolution image; The image to be processed is input into the second model to obtain the second super-resolution image; The first super-resolution image is input into the third model to obtain the first low-resolution image; The second super-resolution image is input into the third model to obtain the second low-resolution image.

4. The image super-resolution method based on passive neighborhood adaptation according to claim 3, characterized in that, The target domain model is trained in the following manner: Based on the first super-resolution image, the second super-resolution image, and the pseudo-label, a first loss function is constructed; Based on the first low-resolution image and the second low-resolution image, a second loss function is constructed; Based on the image to be processed, the discriminator, the target low-resolution image, and the target domain model, a third loss function is constructed, wherein the discriminator is used to distinguish the source of the target low-resolution image; The target domain model is trained based on the first loss function, the second loss function, and the third loss function.

5. The image super-resolution method based on passive neighborhood adaptation according to any one of claims 1-4, characterized in that, The batch enhancement layer is constructed based on the batch enhancement attention mechanism, layer normalization, and multilayer perceptron. The downsampling network is constructed based on multiple convolutional layers with downsampling, multiple convolutional layers without downsampling, and multiple residual blocks.

6. The image super-resolution method based on passive neighborhood adaptation according to any one of claims 1-4, characterized in that, The head structure includes one convolutional layer, the feature extractor includes six residual blocks and one convolutional layer, and the tail structure includes an upsampling layer and multiple convolutional layers.

7. A passive-domain adaptive image super-resolution device, characterized in that, include: The first processing module is used to input the unlabeled image to be processed into the pre-trained source domain model to obtain the pseudo label corresponding to the image to be processed. The source domain model is trained based on labeled sample images. The second processing module is used to input the image to be processed into the target domain model to obtain the target super-resolution image and the target low-resolution image, wherein the target domain model is trained based on the pseudo-label. The source domain model is constructed based on the head structure, feature extractor, and tail structure. The target domain model includes a first model, a second model, and a third model. The structure of the first model is the same as that of the source domain model. The second model is constructed based on the first model and multiple batch enhancement layers. The third model is constructed based on a downsampling network. The batch enhancement layers are used for data enhancement, and the downsampling network is used to obtain the target low-resolution image based on the target super-resolution image.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the passive neighborhood-adaptive image super-resolution method as described in any one of claims 1 to 6.

9. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the passive neighborhood-adaptive image super-resolution method as described in any one of claims 1 to 6.

10. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by the processor, it implements the passive neighborhood-adaptive image super-resolution method as described in any one of claims 1 to 6.

Images