Image reconstruction model training method, device, electronic device and storage medium

Abstract

The present application discloses a training method for an image reconstruction model, an image super-resolution reconstruction method, an apparatus, an electronic device and a storage medium, wherein the training method for the image reconstruction model comprises: using the same training image to simultaneously train a first neural network model and a second neural network model; wherein, when training the first neural network model, the first neural network model is supervised and trained based on first information; the first information represents image features of at least one level of the intermediate output of the second neural network model; the first neural network model and the second neural network model are both used for image super-resolution reconstruction; the first parameter corresponding to the second neural network model is greater than the first parameter corresponding to the first neural network model; the first parameter represents the structural complexity of the first network structure in the neural network model; the first network structure represents a nonlinear mapping structure.

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and in particular to a training method for an image reconstruction model, an image super-resolution reconstruction method, a device, an electronic device and a storage medium. Background Art

[0002] Image super-resolution reconstruction is a technology that uses a group of low-quality, low-resolution images to generate a single high-quality, high-resolution image, thereby improving the recognition ability and recognition accuracy of the image. In related technologies, a single convolutional layer is used as a nonlinear mapping module for image super-resolution reconstruction. In this way, the structure of the nonlinear mapping network is single, and it is difficult to extract rich feature information from the image, resulting in poor reconstruction effect. Summary of the invention

[0003] To solve the related technical problems, the embodiments of the present application provide a training method for an image reconstruction model, an image super-resolution reconstruction method, a device, an electronic device and a storage medium.

[0004] The present application embodiment provides a method for training an image reconstruction model, including:

[0005] The first neural network model and the second neural network model are trained simultaneously using the same training image; wherein,

[0006] When training the first neural network model, supervised training is performed on the first neural network model based on first information; the first information represents image features of at least one level of an intermediate output of the second neural network model;

[0007] The first neural network model and the second neural network model are both used for image super-resolution reconstruction; the first parameter corresponding to the second neural network model is greater than the first parameter corresponding to the first neural network model; the first parameter represents the structural complexity of the first network structure in the neural network model; the first network structure represents a nonlinear mapping structure.

[0008] In the above scheme, the nonlinear mapping structure of the second neural network model includes M cascaded residual blocks; the nonlinear mapping structure of the first neural network model includes N cascaded residual blocks; wherein,

[0009] Each residual block is used to extract corresponding image features; the M and the N are both integers greater than 1, and the M is greater than the N.

[0010] In the above scheme, a jump connection is made between two adjacent residual blocks.

[0011] In the above scheme, each of the N residual blocks of the first neural network model adopts depth-separable convolution.

[0012] In the above scheme, the method further comprises:

[0013] The training error of the first neural network model is calculated based on the set loss function; wherein,

[0014] The set loss function consists of a first loss function and at least one second loss function; the function value of the first loss function represents the difference between the output image of the first neural network model and the target image; the function value of each second loss function in the at least one second loss function represents the difference between the image features output by the first neural network model at the corresponding layer and the image features output by the second neural network model at the same corresponding layer.

[0015] In the above solution, the first loss function includes one of the following:

[0016] Mean Absolute Error (MAE) loss function;

[0017] Perceptual loss function.

[0018] In the above scheme, each of the at least one second loss function includes a pair-wise loss function.

[0019] In the above scheme, the first neural network model and the second neural network model both include a second network structure for processing and obtaining a shallow feature map of the input image; wherein,

[0020] The second network structure includes a convolutional layer and a nonlinear activation layer.

[0021] In the above solution, the first neural network model and the second neural network model both include a third network structure for image reconstruction; wherein,

[0022] The third network structure reconstructs the super-resolution image by pixel recombination.

[0023] An embodiment of the present application also provides an image super-resolution reconstruction method, which uses an image reconstruction model trained by any of the above-mentioned image reconstruction model training methods to perform image super-resolution reconstruction.

[0024] The present application also provides a training device for an image reconstruction model, including:

[0025] A training unit is used to simultaneously train the first neural network model and the second neural network model using the same training image; wherein,

[0026] When training the first neural network model, supervised training is performed on the first neural network model based on first information; the first information represents image features of at least one level of an intermediate output of the second neural network model;

[0027] The first neural network model and the second neural network model are both used for image super-resolution reconstruction; the first parameter corresponding to the second neural network model is greater than the first parameter corresponding to the first neural network model; the first parameter represents the structural complexity corresponding to the first network structure in the neural network model; the first network structure represents a nonlinear mapping structure.

[0028] The present application also provides an image super-resolution reconstruction device, comprising:

[0029] A reconstruction unit is used to perform image super-resolution reconstruction using an image reconstruction model trained by any of the above-mentioned image reconstruction model training methods.

[0030] The embodiment of the present application further provides a first electronic device, comprising: a first processor and a first communication interface; wherein,

[0031] The first processor is used to simultaneously train the first neural network model and the second neural network model using the same training image; wherein,

[0032] When training the first neural network model, supervised training is performed on the first neural network model based on first information; the first information represents image features of at least one level of an intermediate output of the second neural network model;

[0033] The first neural network model and the second neural network model are both used for image super-resolution reconstruction; the first parameter corresponding to the second neural network model is greater than the first parameter corresponding to the first neural network model; the first parameter represents the structural complexity corresponding to the first network structure in the neural network model; the first network structure represents a nonlinear mapping structure.

[0034] The embodiment of the present application further provides a second electronic device, including: a second processor and a second communication interface; wherein,

[0035] The second processor is used to perform image super-resolution reconstruction using an image reconstruction model trained by any of the above-mentioned image reconstruction model training methods.

[0036] The embodiment of the present application further provides a first electronic device, comprising: a first processor and a first memory for storing a computer program that can be run on the processor,

[0037] Wherein, the first processor is used to execute the steps of any of the above-mentioned image reconstruction model training methods when running the computer program.

[0038] The embodiment of the present application further provides a second electronic device, comprising: a second processor and a second memory for storing a computer program that can be run on the processor,

[0039] Wherein, the second processor is used to execute the steps of the above-mentioned image super-resolution reconstruction method when running the computer program.

[0040] An embodiment of the present application also provides a storage medium having a computer program stored thereon, wherein when the computer program is executed by a processor, the steps of the training method of the image reconstruction model described in any one of the above items are implemented, or the steps of the method of image super-resolution reconstruction described in the above item are implemented.

[0041] The training method of the image reconstruction model, the image super-resolution reconstruction method, the device, the electronic device and the storage medium provided by the embodiment of the present application, in the training method of the image reconstruction model, the first neural network model and the second neural network model are both used for image super-resolution reconstruction, and the first parameter corresponding to the second neural network model is greater than the first parameter corresponding to the first neural network model, wherein the first parameter represents the structural complexity of the first network structure in the neural network model; the first network structure represents the nonlinear mapping structure; the first neural network model and the second neural network model are trained simultaneously using the same training image, and when training the first neural network model, the first neural network model is supervised and trained based on the first information, wherein the first information represents the image features of at least one level of the intermediate output of the second neural network model. In this way, the training of the first neural network model is supervised by using the intermediate features of the second neural network model with a more complex network structure, so that the first neural network model can learn the rich feature information of the second neural network model, and the generalization ability of the first neural network model is enhanced. On this basis, the trained first neural network model is used to reconstruct the image super-resolution, which can extract rich feature information from the image and improve the reconstruction effect. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] Figure 1 A flowchart of a training method for an image reconstruction model of related technology;

[0043] Figure 2 A schematic diagram of a flow chart of a training method for an image reconstruction model according to an embodiment of the present application;

[0044] Figure 3 This is an example diagram of the implementation process of the training method of the image reconstruction model in the embodiment of the present application;

[0045] Figure 4 This is a schematic diagram of the structure of a training device for an image reconstruction model according to an embodiment of the present application;

[0046] Figure 5 This is a schematic diagram of the structure of an image super-resolution reconstruction device according to an embodiment of the present application;

[0047] Figure 6 This is a schematic diagram of the first device structure of the embodiment of the present application;

[0048] Figure 7 This is a schematic diagram of the second device structure of an embodiment of the present application. DETAILED DESCRIPTION

[0049] Image super-resolution reconstruction is a technology that uses a group of low-quality, low-resolution images to generate a single high-quality, high-resolution image, thereby improving the image recognition ability and recognition accuracy. Figure 1 The related technology of image super-resolution reconstruction shown in the figure, in the related technology, the input low-resolution image is first preprocessed, including converting the image in RGB format into an image in YUV format, and then the image is feature extracted and upsampled to complete the reconstruction of the image super-resolution, and the high-resolution image is output. In the process of training the image reconstruction model, the MAE loss between the output high-resolution image and the real high-resolution image is calculated to determine whether the model converges. The image feature extraction part in the above reconstruction process is usually implemented based on convolutional neural networks (CNN, Convolutional Neural Networks). CNN is a type of feedforward neural network with a deep structure that includes convolution calculations. It is one of the representative algorithms of deep learning. It is widely used in computer vision fields such as image recognition and image super-resolution reconstruction. At the same time, the network structure of deep convolutional neural networks is becoming deeper and deeper, and the amount of calculation is also increasing. Since the storage space and computing power of mobile devices such as mobile phones and vehicle-mounted devices are limited, in order to embed deep neural networks in the application of mobile devices, it is necessary to compress the network model. Therefore, the related technology uses a single convolution layer for nonlinear mapping when implementing it. However, the structure of the nonlinear mapping network is single, which makes it difficult to extract rich feature information from the image.

[0050] Based on the training method of the image reconstruction model, the image super-resolution reconstruction method, device, electronic device and storage medium provided by the embodiment of the present application, when training the image super-resolution model, a teacher network with a more complex network structure is used to supervise the learning of the student network. Compared with the student network, the network structure used by the teacher network when performing image feature extraction is more complex, and the number of feature extraction modules is greater. Therefore, while training the student network, the teacher network is introduced for supervised learning, and the intermediate feature information output by the multiple feature extraction modules of the teacher network is passed to the student network. In this way, the model generalization ability of the trained student network is improved, and better reconstruction effect can be obtained when applied to image super-resolution reconstruction.

[0051] The present application is further described in detail below in conjunction with the accompanying drawings and embodiments.

[0052] The present application embodiment provides a method for training an image reconstruction model. Figure 2 As shown, the method includes:

[0053] Step 201: Use the same training image to simultaneously train a first neural network model and a second neural network model; when training the first neural network model, perform supervised training on the first neural network model based on first information.

[0054] Among them, the first information represents the image features of at least one level of the intermediate output of the second neural network model; the first neural network model and the second neural network model are both used for image super-resolution reconstruction; the first parameter corresponding to the second neural network model is greater than the first parameter corresponding to the first neural network model; the first parameter represents the structural complexity of the first network structure in the neural network model; the first network structure represents a nonlinear mapping structure.

[0055] Here, both the first neural network model and the second neural network model are used for image super-resolution reconstruction, that is, a low-resolution (LR, Low Resolution) image is input into the neural network model for image super-resolution reconstruction, and the neural network model performs image preprocessing, image feature extraction, image reconstruction and other processing on the input image in sequence, and finally outputs a high-resolution (HR, High Resolution) image. In practical application, the structural complexity of the nonlinear mapping structure of the second neural network model is greater than the structural complexity of the nonlinear mapping structure of the first neural network model. In other words, the number of feature extraction modules of the second neural network model is greater than the number of feature extraction modules of the first neural network model, and the complexity of each feature extraction module in the second neural network model is also higher than the complexity of the feature extraction module of the first neural network model. In this way, compared with the first neural network, the second neural network model can extract more in-depth image features. Based on this, when conducting model training, the same training image is used to input the first neural network model and the second neural network model at the same time, that is, the first neural network model and the second neural network model are trained at the same time, and in the training process, the first neural network model is supervised and trained based on the deeper image features of at least one level of the intermediate output of the second neural network model. In this way, the generalization ability of the first neural network model can be improved while ensuring the lightweight of the network structure of the first neural network model and without increasing the computational complexity of the first neural network model.

[0056] In one embodiment, the nonlinear mapping structure of the second neural network model includes M cascaded residual blocks; the nonlinear mapping structure of the first neural network model includes N cascaded residual blocks; wherein,

[0057] Each residual block is used to extract corresponding image features; the M and the N are both integers greater than 1, and the M is greater than the N.

[0058] The nonlinear mapping structure of the first neural network model includes N cascaded residual blocks, and the nonlinear mapping structure of the second neural network model includes M cascaded residual blocks, where M is greater than N, that is, the number of residual blocks in the nonlinear mapping structure of the second neural network model is greater than the number of residual blocks in the nonlinear mapping structure of the first neural network model. Here, each residual block is used to extract image features, which means that the number of modules used for image feature extraction in the second neural network model is greater than the number of modules used for image feature extraction in the first neural network model. In actual application, multi-scale residual blocks are used in both the first neural network model and the second neural network model. In this way, image features of different scales can be extracted when extracting image features, thereby improving the effect of image reconstruction.

[0059] In one embodiment, two adjacent residual blocks are skip-connected.

[0060] Here, in the nonlinear mapping structure of the neural network model, a skip connection is used for adjacent residual blocks. In this way, the original pixel information of the input image can be retained while the neural network model can focus on learning the residual information of the image during training, thereby reducing the training complexity of the model. The trained image reconstruction model can have good reconstruction performance.

[0061] In one embodiment, each of the N residual blocks of the first neural network model adopts depthwise separable convolution.

[0062] In actual application, the depthwise separable convolution splits a convolution kernel into two independent convolution kernels, and performs two convolutions respectively: depthwise convolution and pointwise convolution. Compared with conventional convolution operations, the number of parameters and computational cost of the depthwise separable convolution are lower. Therefore, the use of depthwise separable convolution in each residual block in the first neural network model can make the network structure of the first neural network model lightweight and more suitable for image processing scenarios of mobile devices.

[0063] In one embodiment, the method further comprises:

[0064] The training error of the first neural network model is calculated based on the set loss function; wherein,

[0065] The set loss function consists of a first loss function and at least one second loss function; the function value of the first loss function represents the difference between the output image of the first neural network model and the target image; the function value of each second loss function in the at least one second loss function represents the difference between the image features output by the first neural network at the corresponding layer and the image features output by the second neural network at the same corresponding layer.

[0066] In one embodiment, the first loss function includes one of the following:

[0067] MAE loss function;

[0068] Perceptual loss function, such as SRGAN network.

[0069] In one embodiment, each of the at least one second loss function comprises a pair-wise loss function.

[0070] In the related art, the MAE loss function or the perceptual loss function is usually used in the training process to calculate the difference between the actual value and the target value of the model output. Here, since the second neural network model is introduced in the model training process to supervise the learning of the first neural network model, the intermediate feature information output by the multiple feature extraction modules of the second neural network model is transmitted to the first neural network model. Therefore, when calculating the loss function, the loss value calculation of the corresponding intermediate feature layer is added. In practical application, the loss function can be loss = MAE loss + n*pair-wise loss, wherein loss is the loss function of the first neural network model, MAE loss is the loss value calculated based on the MAE function, and the corresponding loss value represents the difference between the output image after the first neural network model performs super-resolution reconstruction on the training image and the target image corresponding to the training image, and n*pair-wise loss represents the sum of the corresponding loss values ​​calculated by all pair-wise functions in the n pair-wise functions, wherein the loss value of each pair-wise function in the n pair-wise functions represents the difference between the image features output by the first neural network model at the corresponding layer and the image features output by the second neural network model at the same corresponding layer.

[0071] In practical applications, when calculating the loss value, the intermediate features output by each corresponding layer of the first neural network model and the second neural network model correspond to the image features from shallow to deep layers in turn. For example, the loss function is set to loss = loss0 + loss1 + loss2 + loss3, where loss0 represents the difference between the output image of the first neural network model after super-resolution reconstruction of the training image and the target image corresponding to the training image, loss1 represents the difference between the shallow image features of the intermediate output of the first neural network model and the shallow image features of the intermediate output of the second neural network model, loss2 represents the difference between the middle image features of the intermediate output of the first neural network model and the middle image features of the intermediate output of the second neural network model, and loss3 represents the difference between the deep image features of the intermediate output of the first neural network model and the deep image features of the intermediate output of the second neural network model. In practical applications, the loss function can also be represented as the weighted sum of the four loss functions of loss0, loss1, loss2, and loss3.

[0072] Here, regarding the above pair-wise function, combined with image feature extraction, in practical application, let Characterize the correlation between the i-th pixel and the j-th pixel in the output image of the second neural network model, Represents the correlation between the i-th pixel and the j-th pixel in the output image of the first neural network model. Then, for the i-th pixel and the j-th pixel in the output image, the pair-wise loss function is used to characterize the difference between the two as follows:

[0073]

[0074] Here, since a pair-wise loss function is used to calculate the loss value corresponding to the intermediate feature, that is, the correlation between paired pixels is used instead of using a single piece of information for knowledge distillation, the trained first neural network model has better generalization ability while achieving model compression.

[0075] In one embodiment, the first neural network model and the second neural network model both include a second network structure for processing and obtaining a shallow feature map of the input image; wherein,

[0076] The second network structure includes a convolutional layer and a nonlinear activation layer.

[0077] The first neural network model and the second neural network model both include a third network structure for image reconstruction; wherein,

[0078] The third network structure reconstructs the super-resolution image through pixel shuffle.

[0079] Here, pixel reorganization is used for image reconstruction. Compared with the image reconstruction by deconvolution in the related art, the computational complexity is small and no parameters are involved.

[0080] An embodiment of the present application also provides an image super-resolution reconstruction method, which uses an image reconstruction model trained by any of the above-mentioned image reconstruction model training methods to perform image super-resolution reconstruction.

[0081] Figure 3 The following is a schematic diagram of the implementation process of the training method of the image reconstruction model in the embodiment of the present application. Figure 3 , the low-resolution image as the training image is simultaneously input into the first neural network model as the student network and the second neural network model as the teacher network. After using multiple multi-scale feature extraction modules in each neural network model, the image is reconstructed based on the extracted multi-scale image features, and a high-resolution image is output. In addition, when calculating the loss function during the training process, it includes calculating the real high-resolution image, that is, the MAE loss function between the real high-resolution image (that is, the target image) and the output high-resolution image, and also includes the pair-wise loss function between the teacher network and the student network corresponding to each layer.

[0082] The following are application examples of the present application:

[0083] In the first application example, the network structure of the second neural network model includes the following three parts:

[0084] 1. Feature extraction and representation: This part consists of a convolutional layer and a nonlinear activation. Its main purpose is to extract features from the input image and obtain a shallow feature map.

[0085] 2. Non-linear mapping: This part consists of 10 residual blocks, each of which contains 2 convolutional layers, and each convolutional layer is followed by a non-linear activation. In addition, each residual block has a skip connection to connect the input features of each residual block with the input features.

[0086] 3. Image reconstruction: This part uses the deconvolution layer to reconstruct the high-resolution image and output it. The deconvolution layer can be considered as the inverse operation of the convolution layer and is usually stacked at the end of the image super-resolution reconstruction network.

[0087] The network structure of the first neural network model includes the following three parts:

[0088] 1. Feature extraction and representation: This part has the same structure as the feature extraction and representation part of the teacher network. It consists of a convolutional layer and a nonlinear activation. Its main purpose is to extract features from the input image and obtain a shallow feature map.

[0089] 2. Nonlinear mapping: This part uses 3 depth-wise separable convolution modules, each of which contains a depth-wise convolution layer and a point-wise convolution layer. The depth-wise convolution layer applies a single 3×3 filter to each input channel, and the point-wise convolution layer applies a 1×1 convolution to combine the outputs of the depth-wise convolution.

[0090] 3. Image reconstruction: This part has the same structure as the image reconstruction part of the teacher network, and uses the deconvolution layer to reconstruct the high-resolution image and output it. The deconvolution layer can be considered as the inverse operation of the convolution layer and is usually stacked at the end of the image super-resolution reconstruction network.

[0091] In the second application example, the network structure of the second neural network model includes the following three parts:

[0092] 1. Feature extraction and representation: This part consists of a convolutional layer and a nonlinear activation. Its main purpose is to extract features from the input image and obtain a shallow feature map.

[0093] 2. Nonlinear mapping: This part consists of 8 multi-scale residual blocks. Each residual block consists of a 3×3 convolution layer and a 5×5 convolution layer in parallel. In each residual block, the input feature map first passes through a 3×3 convolution layer and a 5×5 convolution layer respectively to obtain two sets of intermediate output feature maps. The two sets of intermediate output feature maps are cascaded as input to another set of 3×3 and 5×5 convolution layers, and the output feature maps of the two convolutions are cascaded again after passing through the activation function respectively. Then a 1×1 convolution layer is used to compress the feature map channel to obtain the output feature map of the multi-scale residual module.

[0094] 3. Image reconstruction: This part cascades the output feature maps of the above 8 multi-scale residual blocks, redistributes the weights of the cascaded feature maps through the channel attention module, and then reconstructs the image using the pixel shuffle module. The final reconstructed image is obtained through a 1×1 convolution.

[0095] The network structure of the first neural network model includes the following three parts:

[0096] 1. Feature extraction and representation: This part has the same structure as the feature extraction and representation part of the teacher network. It consists of a convolutional layer and a nonlinear activation. Its main purpose is to extract features from the input image and obtain a shallow feature map.

[0097] 2. Nonlinear mapping: This part uses two multi-scale residual modules and uses depthwise separable convolution to optimize the convolution structure of the module to reduce the number of parameters and computational complexity.

[0098] 3. Image reconstruction: This part has the same structure as the image reconstruction part of the teacher network. The output feature maps of the above two multi-scale residual blocks are cascaded, and the weights of the cascaded feature maps are redistributed through the channel attention module. The pixel shuffle module is then used to reconstruct the image, and the final reconstructed image is obtained after a 1×1 convolution.

[0099] In order to implement the method of the embodiment of the present application, the embodiment of the present application also provides a training device for an image reconstruction model, which is arranged on a first electronic device, such as Figure 4 As shown, the device comprises:

[0100] The training unit 401 is used to simultaneously train the first neural network model and the second neural network model using the same training image; wherein,

[0101] When training the first neural network model, supervised training is performed on the first neural network model based on first information; the first information represents image features of at least one level of an intermediate output of the second neural network model;

[0102] The first neural network model and the second neural network model are both used for image super-resolution reconstruction; the first parameter corresponding to the second neural network model is greater than the first parameter corresponding to the first neural network model; the first parameter represents the structural complexity corresponding to the first network structure in the neural network model; the first network structure represents a nonlinear mapping structure.

[0103] In one embodiment, the nonlinear mapping structure of the second neural network model includes M cascaded residual blocks; the nonlinear mapping structure of the first neural network model includes N cascaded residual blocks;

[0104] Each residual block is used to extract corresponding image features; the M and the N are both integers greater than 1, and the M is greater than the N.

[0105] In one embodiment, two adjacent residual blocks are skipped.

[0106] In one embodiment, each of the N residual blocks of the first neural network model adopts depthwise separable convolution.

[0107] In one embodiment, the device further comprises:

[0108] A calculation unit, used to calculate the training error of the first neural network model based on a set loss function; wherein,

[0109] The set loss function consists of a first loss function and at least one second loss function; the function value of the first loss function represents the difference between the output image of the first neural network model and the target image; the function value of each second loss function in the at least one second loss function represents the difference between the image features output at the corresponding layer of the first neural network model and the image features output at the same corresponding layer of the second neural network.

[0110] In one embodiment, the first loss function includes one of the following:

[0111] MAE loss function;

[0112] Perceptual loss function.

[0113] In one embodiment, each of the at least one second loss function comprises a pair-wise loss function.

[0114] In one embodiment, the first neural network model and the second neural network model both include a second network structure for processing and obtaining a shallow feature map of the input image; wherein,

[0115] The second network structure includes a convolutional layer and a nonlinear activation layer.

[0116] In one embodiment, the first neural network model and the second neural network model both include a third network structure for image reconstruction; wherein,

[0117] The third network structure reconstructs the super-resolution image by pixel recombination.

[0118] In actual application, the training unit 401 and the calculation unit can be implemented by a processor in a training device for an image reconstruction model.

[0119] It should be noted that: the training device for the image reconstruction model provided in the above embodiment only uses the division of the above program modules as an example when performing synchronization. In actual applications, the above processing can be assigned to different program modules as needed, that is, the internal structure of the device is divided into different program modules to complete all or part of the processing described above. In addition, the training device for the image reconstruction model provided in the above embodiment and the training method embodiment of the image reconstruction model belong to the same concept. The specific implementation process is detailed in the method embodiment and will not be repeated here.

[0120] In order to implement the method of the embodiment of the present application, the embodiment of the present application also provides an image super-resolution reconstruction device, which is arranged on a second electronic device, such as Figure 5 As shown, the device comprises:

[0121] The reconstruction unit 501 is used to perform image super-resolution reconstruction using an image reconstruction model trained by any of the above-mentioned image reconstruction model training methods.

[0122] In actual application, the reconstruction unit 501 can be implemented by a processor in an image super-resolution reconstruction device.

[0123] It should be noted that: when the image super-resolution reconstruction device provided in the above embodiment is synchronized, only the division of the above program modules is used as an example. In actual applications, the above processing can be assigned to different program modules as needed, that is, the internal structure of the device is divided into different program modules to complete all or part of the processing described above. In addition, the image super-resolution reconstruction device provided in the above embodiment and the image super-resolution reconstruction method embodiment belong to the same concept, and the specific implementation process is detailed in the method embodiment, which will not be repeated here.

[0124] Based on the hardware implementation of the above program modules, and in order to implement the training method of the image reconstruction model of the embodiment of the present application, the embodiment of the present application also provides a first electronic device, such as Figure 6 As shown, the first electronic device 600 includes:

[0125] The first communication interface 601 is capable of exchanging information with other network nodes;

[0126] The first processor 602 is connected to the first communication interface 601 to implement information exchange with other network nodes, and is used to execute the method provided by one or more technical solutions of the first electronic device side when running a computer program. The computer program is stored in the first memory 603.

[0127] Specifically, the first processor 602 is used to simultaneously train the first neural network model and the second neural network model using the same training image; wherein,

[0128] When training the first neural network model, supervised training is performed on the first neural network model based on first information; the first information represents image features of at least one level of an intermediate output of the second neural network model;

[0129] The first neural network model and the second neural network model are both used for image super-resolution reconstruction; the first parameter corresponding to the second neural network model is greater than the first parameter corresponding to the first neural network model; the first parameter represents the structural complexity of the first network structure in the neural network model; the first network structure represents a nonlinear mapping structure.

[0130] In one embodiment, the nonlinear mapping structure of the second neural network model includes M cascaded residual blocks; the nonlinear mapping structure of the first neural network model includes N cascaded residual blocks;

[0131] Each residual block is used to extract corresponding image features; the M and the N are both integers greater than 1, and the M is greater than the N.

[0132] In one embodiment, two adjacent residual blocks are skip-connected.

[0133] In one embodiment, each of the N residual blocks of the first neural network model adopts depthwise separable convolution.

[0134] In one embodiment, the first processor 602:

[0135] The training error of the first neural network model is calculated based on the set loss function; wherein,

[0136] The set loss function consists of a first loss function and at least one second loss function; the function value of the first loss function represents the difference between the output image of the first neural network model and the target image; the function value of each second loss function in the at least one second loss function represents the difference between the image features output at the corresponding layer of the first neural network model and the image features output at the same corresponding layer of the second neural network.

[0137] In one embodiment, the first loss function includes one of the following:

[0138] MAE loss function;

[0139] Perceptual loss function.

[0140] In one embodiment, each of the at least one second loss function comprises a pair-wise loss function.

[0141] In one embodiment, the first neural network model and the second neural network model both include a second network structure for processing and obtaining a shallow feature map of the input image; wherein,

[0142] The second network structure includes a convolutional layer and a nonlinear activation layer.

[0143] In one embodiment, the first neural network model and the second neural network model both include a third network structure for image reconstruction; wherein,

[0144] The third network structure reconstructs the super-resolution image by pixel recombination.

[0145] It should be noted that the specific processing process of the first processor 602 and the first communication interface 601 can be understood by referring to the above method.

[0146] Of course, in actual application, the various components in the first electronic device 600 are coupled together through the bus system 604. It can be understood that the bus system 604 is used to realize the connection and communication between these components. In addition to the data bus, the bus system 604 also includes a power bus, a control bus, and a status signal bus. However, for the sake of clarity, Figure 6 Various buses are labeled as bus system 604 .

[0147] The first memory 603 in the embodiment of the present application is used to store various types of data to support the operation of the first electronic device 600. Examples of such data include: any computer program used to operate on the first electronic device 600.

[0148] The method disclosed in the above embodiment of the present application can be applied to the first processor 602, or implemented by the first processor 602. The first processor 602 may be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the above method can be completed by the hardware integrated logic circuit or software instructions in the first processor 602. The above-mentioned first processor 602 may be a general-purpose processor, a digital signal processor (DSP, Digital Signal Processor), or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components, etc. The first processor 602 can implement or execute the various methods, steps and logic block diagrams disclosed in the embodiments of the present application. A general-purpose processor may be a microprocessor or any conventional processor, etc. In combination with the steps of the method disclosed in the embodiment of the present application, it can be directly embodied as a hardware decoding processor to execute, or it can be executed by a combination of hardware and software modules in the decoding processor. The software module can be located in a storage medium, which is located in the first memory 603, and the first processor 602 reads the information in the first memory 603 and completes the steps of the above method in combination with its hardware.

[0149] In an exemplary embodiment, the first electronic device 600 can be implemented by one or more application specific integrated circuits (ASIC), DSP, programmable logic device (PLD), complex programmable logic device (CPLD), field programmable gate array (FPGA), general processor, controller, microcontroller (MCU), microprocessor, or other electronic components to execute the aforementioned method.

[0150] Based on the hardware implementation of the above program modules, and in order to implement the image super-resolution reconstruction method of the embodiment of the present application, the embodiment of the present application also provides a second electronic device, such as Figure 7 As shown, the second electronic device 700 includes:

[0151] The second communication interface 701 is capable of exchanging information with other network nodes;

[0152] The second processor 702 is connected to the second communication interface 701 to implement information exchange with other network nodes, and is used to execute the image super-resolution reconstruction method provided by one or more technical solutions when running a computer program. The computer program is stored in the second memory 703.

[0153] Specifically, the second processor 702 is used to perform image super-resolution reconstruction using an image reconstruction model trained using any of the above-mentioned image reconstruction model training methods.

[0154] It should be noted that the specific processing process of the second processor 702 and the second communication interface 701 can be understood by referring to the above method.

[0155] Of course, in actual application, the various components in the second electronic device 700 are coupled together through the bus system 704. It can be understood that the bus system 704 is used to realize the connection and communication between these components. In addition to the data bus, the bus system 704 also includes a power bus, a control bus, and a status signal bus. However, for the sake of clarity, Figure 7 Various buses are labeled as bus system 704 .

[0156] The second memory 703 in the embodiment of the present application is used to store various types of data to support the operation of the second electronic device 700. Examples of such data include: any computer program used to operate on the second electronic device 700.

[0157] The method disclosed in the above embodiment of the present application can be applied to the second processor 702, or implemented by the second processor 702. The second processor 702 may be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the above method can be completed by an integrated logic circuit of the hardware in the second processor 702 or an instruction in the form of software. The above-mentioned second processor 702 may be a general-purpose processor, a DSP, or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components, etc. The second processor 702 can implement or execute the various methods, steps and logic block diagrams disclosed in the embodiment of the present application. A general-purpose processor may be a microprocessor or any conventional processor, etc. In combination with the steps of the method disclosed in the embodiment of the present application, it can be directly embodied as a hardware decoding processor to execute, or it can be executed by a combination of hardware and software modules in the decoding processor. The software module may be located in a storage medium, which is located in the second memory 703, and the second processor 702 reads the information in the second memory 703 and completes the steps of the above method in combination with its hardware.

[0158] In an exemplary embodiment, the second electronic device 700 may be implemented by one or more ASICs, DSPs, PLDs, CPLDs, FPGAs, general purpose processors, controllers, MCUs, Microprocessors, or other electronic components to perform the aforementioned methods.

[0159] It can be understood that the memory (first memory 603, second memory 703) of the embodiment of the present application can be a volatile memory or a non-volatile memory, and can also include both volatile and non-volatile memories. Among them, the non-volatile memory can be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), a magnetic random access memory (FRAM), a ferromagnetic random access memory, a flash memory, a magnetic surface memory, an optical disc, or a compact disc read-only memory (CD-ROM); the magnetic surface memory can be a disk memory or a tape memory. The volatile memory can be a random access memory (RAM), which is used as an external cache. By way of example and not limitation, many forms of RAM are available, such as static random access memory (SRAM), synchronous static random access memory (SSRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDRSDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link dynamic random access memory (SLDRAM), and direct RAM bus random access memory (DRRAM).The memories described in the embodiments of the present application are intended to include, but are not limited to, these and any other suitable types of memories.

[0160] In an exemplary embodiment, the embodiment of the present application further provides a storage medium, namely a computer storage medium, specifically a computer-readable storage medium, for example, including a first memory 603 storing a computer program, and the computer program can be executed by a first processor 602 of a first electronic device 600 to complete the steps of the method on the first electronic device side. For another example, a second memory 703 storing a computer program can be executed by a second processor 702 of a second electronic device 700 to complete the steps of the method on the second electronic device side. The computer-readable storage medium can be a memory such as FRAM, ROM, PROM, EPROM, EEPROM, Flash Memory, magnetic surface memory, optical disk, or CD-ROM.

[0161] It should be noted that: "first", "second", etc. are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence.

[0162] In addition, the technical solutions described in the embodiments of the present application can be combined arbitrarily without conflict.

[0163] The above description is only a preferred embodiment of the present application and is not intended to limit the protection scope of the present application.

Claims

1. A training method for an image reconstruction model, characterized in that: include: The first neural network model and the second neural network model are trained simultaneously using the same training image; wherein, When training the first neural network model, supervised training is performed on the first neural network model based on first information; the first information represents image features of at least one level of an intermediate output of the second neural network model; The step of performing supervised training on the first neural network model based on the first information includes: The training error of the first neural network model is calculated based on the set loss function; wherein, The set loss function consists of a first loss function and at least one second loss function; the function value of the first loss function represents the difference between the output image of the first neural network model and the target image; the function value of each second loss function in the at least one second loss function represents the difference between the image features output at the corresponding layer of the first neural network model and the image features output at the same corresponding layer of the second neural network; The first neural network model and the second neural network model are both used for image super-resolution reconstruction; the first parameter corresponding to the second neural network model is greater than the first parameter corresponding to the first neural network model; the first parameter represents the structural complexity of the first network structure in the neural network model; the first network structure represents a nonlinear mapping structure.

2. The method according to claim 1, characterized in that The nonlinear mapping structure of the second neural network model includes M cascaded residual blocks; the nonlinear mapping structure of the first neural network model includes N cascaded residual blocks; wherein, Each residual block is used to extract corresponding image features; both M and N are integers greater than 1, and M is greater than N.

3. The method according to claim 1, characterized in that There is a skip connection between two adjacent residual blocks.

4. The method according to claim 2, characterized in that: Each of the N residual blocks of the first neural network model adopts depth-wise separable convolution.

5. The method according to claim 1, characterized in that: The first loss function includes one of the following: Mean absolute error MAE loss function; Perceptual loss function.

6. The method according to claim 1, characterized in that Each of the at least one second loss function comprises a pair-wise loss function.

7. The method according to claim 1, characterized in that The first neural network model and the second neural network model both include a second network structure for processing and obtaining a shallow feature map of an input image; wherein, The second network structure includes a convolutional layer and a nonlinear activation layer.

8. The method according to claim 1, characterized in that The first neural network model and the second neural network model both include a third network structure for image reconstruction; wherein, The third network structure reconstructs the super-resolution image by pixel recombination.

9. A method for super-resolution image reconstruction, characterized in that: Image super-resolution reconstruction is performed using the image reconstruction model trained by the method according to any one of claims 1 to 8.

10. A training device for an image reconstruction model, characterized in that: include: A training unit is used to simultaneously train the first neural network model and the second neural network model using the same training image; wherein, When training the first neural network model, supervised training is performed on the first neural network model based on first information; the first information represents image features of at least one level of an intermediate output of the second neural network model; A calculation unit, configured to calculate a training error of a first neural network model based on a set loss function; wherein the set loss function is composed of a first loss function and at least one second loss function; a function value of the first loss function represents a difference between an output image of the first neural network model and a target image; a function value of each second loss function in the at least one second loss function represents a difference between an image feature output at a corresponding layer of the first neural network model and an image feature output at the same corresponding layer of the second neural network; The first neural network model and the second neural network model are both used for image super-resolution reconstruction; the first parameter corresponding to the second neural network model is greater than the first parameter corresponding to the first neural network model; the first parameter represents the structural complexity corresponding to the first network structure in the neural network model; the first network structure represents a nonlinear mapping structure.

11. An image super-resolution reconstruction device, characterized in that: include: A reconstruction unit, used for performing image super-resolution reconstruction using an image reconstruction model trained by the method according to any one of claims 1 to 8.

12. A first electronic device, characterized in that: include: A first processor and a first communication interface; wherein, The first processor is used to simultaneously train the first neural network model and the second neural network model using the same training image; wherein, When training the first neural network model, supervised training is performed on the first neural network model based on the first information; the first information represents the image features of at least one level of the intermediate output of the second neural network model; and the training error of the first neural network model is calculated based on the set loss function; wherein the set loss function is composed of a first loss function and at least one second loss function; the function value of the first loss function represents the difference between the output image of the first neural network model and the target image; the function value of each second loss function in the at least one second loss function represents the difference between the image features output at the corresponding layer of the first neural network model and the image features output at the same corresponding layer of the second neural network; The first neural network model and the second neural network model are both used for image super-resolution reconstruction; the first parameter corresponding to the second neural network model is greater than the first parameter corresponding to the first neural network model; the first parameter represents the structural complexity corresponding to the first network structure in the neural network model; the first network structure represents a nonlinear mapping structure.

13. A second electronic device, characterized in that: include: A second processor and a second communication interface; wherein, The second processor is used to perform image super-resolution reconstruction using the image reconstruction model trained by the method according to any one of claims 1 to 8.

14. A first electronic device, characterized in that: include: a first processor and a first memory for storing a computer program executable on the processor, Wherein, when the first processor is used to run the computer program, the steps of the method described in any one of claims 1 to 8 are executed.

15. A second electronic device, characterized in that: include: a second processor and a second memory for storing a computer program executable on the processor, Wherein, when the second processor is used to run the computer program, it executes the steps of the method according to claim 9.

16. A storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the computer program implements the steps of the method according to any one of claims 1 to 8, or implements the steps of the method according to claim 9.

Images