An image reconstruction method, apparatus, device and medium

Abstract

The present invention discloses an image reconstruction method, apparatus, device and medium, relating to the field of computer vision. The method includes: inputting an image with a first resolution into a trained super-resolution convolutional neural network model to obtain an image with a second resolution output by the super-resolution convolutional neural network model; all convolutional layers, batch normalization layers, point addition layers and upsampling layers in the super-resolution convolutional neural network model perform compression of the model calculation bit width through quantization processing; during the training process of the super-resolution convolutional neural network model, the error between the data before and after quantization is within a preset range through a self-supervised loss function. The present invention greatly reduces the computational complexity of the model, and can bring advantages in terms of the running speed, power consumption, memory access bandwidth, etc. of the super-resolution convolutional neural network model during actual application; the quantization coefficient is constrained by the self-supervised loss function to stabilize the training process and effectively improve the quantization accuracy of the super-resolution convolutional neural network model.

Description

Technical Field

[0001] The present invention relates to the field of computer vision, and in particular to an image reconstruction method, device, equipment and medium. Background Art

[0002] Image super-resolution (SR) algorithm is an important research field in the field of computer vision. The super-resolution algorithm aims to convert or reconstruct the first low-resolution (LR) image into a high-resolution (HR) image. The super-resolution algorithm has important application value in monitoring equipment, satellite images, medical imaging and other fields.

[0003] As deep neural networks, especially deep convolutional neural networks (DCNN), are widely used in various fields of computer vision. Studies have shown that deep learning algorithms can achieve better results than traditional algorithms in the field of super-resolution. However, the high computational complexity of deep learning algorithms hinders the deployment of such algorithms in the field of super-resolution, especially since the resolution of each feature map in the super-resolution network is generally large, which places high demands on the system's computing power, storage, and data transmission bandwidth. Therefore, it is necessary to use technologies such as model quantization to compress the super-resolution network structure based on deep learning algorithms, so that these algorithms can be deployed more efficiently on actual platforms.

[0004] In summary, a model that can significantly reduce the computational complexity of deep learning algorithms for super-resolution tasks is an important issue that needs to be solved urgently. Summary of the invention

[0005] In view of this, embodiments of the present invention provide an image reconstruction method, apparatus, device and medium to solve the problem of high system resource requirements when a super-resolution network model based on a deep learning algorithm is actually deployed.

[0006] According to a first aspect, an embodiment of the present invention provides an image reconstruction method, the method comprising:

[0007] Inputting the image of the first resolution into the trained super-resolution convolutional neural network model to obtain an image of the second resolution output by the super-resolution convolutional neural network model; the second resolution is higher than the first resolution;

[0008] All convolutional layers, batch normalization layers, point addition layers, and upsampling layers in the super-resolution convolutional neural network model compress the model calculation bit width through quantization processing; during the training process of the super-resolution convolutional neural network model, the error between the data before and after quantization is within a preset range through the self-supervised loss function, and the final loss function of the super-resolution convolutional neural network model is obtained by updating the initial loss function based on the self-supervised loss function.

[0009] Combined with the first aspect, in the first implementation manner of the first aspect, the input data of the convolutional layer is quantized through the formula (2):

[0010] (2)

[0011] Among them, represents the quantization value of the data ; represents the quantization coefficient; represents the input data of the convolutional layer, which is an eigenvalue or a weight; represents the quantization bit width, represents the minimum range of quantization, represents the maximum range of quantization; represents the rounding integer function; , represents the integer part of the data after quantization; , represents the floating-point part of the data after quantization.

[0012] Combined with the first implementation manner of the first aspect, after the input data is quantized, the convolutional layer performs a convolution operation through the formula (3):

[0013] (3)

[0014] Among them, represents the eigenvalue after quantization of the convolutional layer; represents the weight after quantization of the convolutional layer; represents the output data of the convolutional layer; represents the integer part of the data ; represents the floating-point part of the data ; represents the integer part of the data ; represents the floating-point part of the data ; , represents the data The integer part; , represents the floating-point part of the data .

[0015] Combined with the first implementation of the first aspect, in the third implementation of the first aspect, the input data of the batch normalization layer is quantized by formula (5):

[0016] (5)

[0017] where ; ; represents the output data of the batch normalization layer after quantization; represents the floating-point part of the output data of the nearest preceding convolutional layer of the batch normalization layer after quantization; represents the integer part of the output data of the nearest preceding convolutional layer of the batch normalization layer after quantization; represents the mean of the batch normalization layer; represents the variance of the batch normalization layer; represents a constant used by the batch normalization layer to avoid the division overflow error caused; represents the normalization coefficient of the batch normalization layer; represents the bias of the batch normalization layer; represents the floating-point part of the data after quantization; represents the integer part of the data after quantization; represents the rounding integer function.

[0018] Combined with the third implementation of the first aspect, in the fourth implementation of the first aspect, the input data of the point addition layer is quantized by formula (6):

[0019] ) = (6)

[0020] where represents one of the input data of the point addition layer after quantization; represents the other input data of the point addition layer after quantization; represents the output data of the point addition layer after quantization; represents the floating-point part of the data after quantization; represents the integer part of the data after quantization; represents the data The floating-point part after quantization; Indicates data The integer part after quantization; Indicates data The quantized floating-point part; Indicates data The integer part after quantization; .

[0021] Combined with any one of the first to fourth embodiments of the first aspect, in the fifth embodiment of the first aspect, the self-supervised loss function is:

[0022]

[0023] wherein, represents the p-norm; indicates data the quantization value of.

[0024] Combined with the fifth embodiment of the first aspect, in the sixth embodiment of the first aspect, the loss function is:

[0025]

[0026] wherein, represents the hyperparameter coefficient of the self-supervised loss function ; represents the final loss function; represents the original loss function.

[0027] In a second aspect, an embodiment of the present invention further provides an image reconstruction device, and the device includes:

[0028] An image reconstruction module, configured to input an image with a first resolution into a trained super-resolution convolutional neural network model to obtain an image with a second resolution output by the super-resolution convolutional neural network model; the second resolution is higher than the first resolution;

[0029] All convolutional layers, batch normalization layers, point addition layers, and upsampling layers in the super-resolution convolutional neural network model are compressed in network structure through quantization processing; during the training process of the super-resolution convolutional neural network model, the difference between the data before and after quantization is within a preset range by means of a self-supervised loss function, and the loss function of the super-resolution convolutional neural network model is obtained based on the self-supervised loss function.

[0030] In a third aspect, an embodiment of the present invention further provides an electronic device, including a memory, a processor, and a computer program stored on the memory and executable on the processor. When the processor executes the program, the steps of any one of the above-mentioned image reconstruction methods are implemented.

[0031] In a fourth aspect, an embodiment of the present invention further provides a non-transitory computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps of any one of the above-mentioned image reconstruction methods are implemented.

[0032] In a fifth aspect, an embodiment of the present invention further provides a computer program product, including a computer program. When the computer program is executed by a processor, the steps of any one of the above-mentioned image reconstruction methods are implemented.

[0033] The image reconstruction method, device, equipment, and medium provided by the present invention are different from other compression schemes for super-resolution algorithms. In this application, all layers in the super-resolution convolutional neural network model based on deep learning are quantized. This not only realizes the compression of the model's computational bit width as low as possible but also end-to-end realizes the mapping from low-resolution images to high-resolution images, greatly reducing the computational complexity of the model and bringing advantages in aspects such as the running speed, power consumption, and memory access bandwidth of the super-resolution convolutional neural network model in practical applications. Additionally, this application also constrains the quantization coefficients during the quantization process through a self-supervised loss function, making the quantization process smoother, stabilizing the model training process, and effectively improving the quantization accuracy of the super-resolution convolutional neural network model. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] The features and advantages of the present invention will be more clearly understood by referring to the accompanying drawings. The drawings are schematic and should not be construed as imposing any limitation on the present invention. In the drawings:

[0035] Figure 1 A flowchart showing the image reconstruction method provided by the present invention is shown;

[0036] Figure 2 A structural diagram showing the image reconstruction device provided by the present invention is shown;

[0037] Figure 3 A structural diagram showing the electronic device provided by the present invention is shown. DETAILED DESCRIPTION

[0038] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the following will clearly and completely describe the technical solutions in the embodiments of the present invention with reference to the accompanying drawings in the embodiments of the present invention. Apparently, the described embodiments are some, but not all, of the embodiments of the present invention. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without creative efforts shall fall within the protection scope of the present invention.

[0039] The SR algorithm is an important research field in the field of computer vision. The SR algorithm aims to convert or reconstruct a low-resolution image into a high-resolution image. Due to information loss, the problem of image resolution conversion belongs to an ill-posed problem, posing a huge challenge to the design of the SR algorithm. Deep neural networks (Convolution Neural Network, DNN), especially DCNN, are widely applied to various fields in computer vision. Of course, deep neural networks can also be applied to the field of super-resolution algorithms.

[0040] The high computational complexity of deep learning algorithms hinders the deployment of such algorithms in the field of super-resolution algorithms. Since the super-resolution algorithm is a pixel-level reconstruction algorithm, there is no pooling layer in the corresponding deep learning network to reduce the resolution of the intermediate feature maps like algorithms such as image classification and object detection. The intermediate feature maps required by the deep learning model for super-resolution are of the original input size or the network output size, that is, the resolution of each feature map in the super-resolution network is generally relatively large, which means that the deep learning model for super-resolution has high requirements for the system's computing power, storage capacity, and data transmission bandwidth. In actual deployment, it is necessary to compress the model to reduce the resource requirements of the system. Therefore, it is necessary to use techniques such as model quantization to compress the computational bit width of the SR model based on deep learning, which is crucial for the deployment of such algorithms on actual platforms.

[0041] Currently, the compression schemes for the SR network model of deep learning generally have two directions. One is to perform structured pruning to reduce the width and depth of the network, and use low-resolution feature maps in most layers of the network, and only increase the resolution of the feature maps at the network output layer; the other is to perform model quantization on the deep learning super-resolution algorithm, converting the original 32-bit floating-point calculation into 8-bit or lower bit width. In some schemes, these two directions can be used in combination without affecting each other.

[0042] However, the second direction or the scheme combining the first and second directions only quantifies some layers in the super-resolution algorithm, and there are still some operators that are not quantized, especially the first and last layers of the network structure. The remaining full-precision computational complexity is still very large and cumbersome, causing great difficulties for the actual deployment of the network structure.

[0043] The following combines with Figure 1 to describe the image reconstruction method of the present invention. The method includes the following steps:

[0044] S10. Input an image of a first resolution into a trained super-resolution convolutional neural network model to obtain an image of a second resolution output by the super-resolution convolutional neural network model. In this embodiment, the second resolution is higher than the first resolution, the first resolution is a low resolution, and the second resolution is a high resolution.

[0045] All convolutional layers (conv), batch normalization layers (BatchNormalization), element-wise addition layers (Element-wise Add), and upsampling layers (Upsample or Pixelshuffle) in the super-resolution convolutional neural network model are compressed in terms of network structure through quantization processing.

[0046] During the training process of the super-resolution convolutional neural network model, the error between the data before and after quantization is within a preset range through a self-supervised loss function, and the final loss function of the super-resolution convolutional neural network model is obtained by updating the initial loss function based on the self-supervised loss function.

[0047] The image reconstruction method provided by the present invention is different from other compression schemes for super-resolution algorithms. In this application, all layers in the super-resolution convolutional neural network model based on deep learning are subjected to quantization processing, which not only realizes the compression of the model's computing bit width as low as possible but also realizes the mapping from a low-resolution image to a high-resolution image end-to-end, greatly reducing the computing complexity of the model and bringing advantages in terms of the running speed, power consumption, memory access bandwidth, etc. of the super-resolution convolutional neural network model during actual application; in addition, this application also constrains the quantization coefficients during the quantization process through a self-supervised loss function, making the quantization process smoother, which not only stabilizes the training process of the model but also effectively improves the quantization accuracy of the super-resolution convolutional neural network model.

[0048] A super-resolution network model applied to the field of computer vision mainly works as follows: receiving a low-resolution input image and predicting and outputting a high-resolution output image by the network model. When the super-high-resolution network model uses deep learning to process super-resolution tasks, its network structure generally includes three layers (modules): a feature extraction layer (FeatureExtraction), a non-linear mapping layer (Non-linear Mapping), and a super-resolution reconstruction layer (SR ImageReconstruction). The feature extraction layer is used for extracting image patches and feature representation, the non-linear mapping layer is used for non-linear feature mapping, and the super-resolution reconstruction layer is used for final image reconstruction.

[0049] In the simplest super-resolution network model, these three layers are respectively a convolutional layer, and the convolutional kernel sizes of these three convolutional layers are different. More complexly, each of these three layers can be a composite structure, that is, having multiple sub-network layers. In addition, there can also be many cross-layer connections in the network structure.

[0050] For example, the Super-Resolution Convolutional Neural Network (SRCNN) first uses bicubic interpolation to enlarge the input low-resolution image to the target size, then fits the non-linear mapping through three convolutional layers (feature extraction layer, non-linear mapping layer, and super-resolution reconstruction layer), and finally outputs a high-resolution image.

[0051] As shown in formula (1), formally, given a low-resolution input , after being processed by these three layers: the feature extraction layer, the non-linear mapping layer, and the super-resolution reconstruction layer, a high-resolution image output is generated. Formula (1) is:

[0052] (1)

[0053] Among them, represents the feature extraction module; represents the non-linear mapping module; represents the super-resolution reconstruction module.

[0054] When using quantization to compress the super-resolution network model, only the non-linear mapping layer is quantized, while the other two head and tail layers are retained for full-precision calculation. This compression method requires the hardware platform to have both full-precision processing and quantization bit-width calculation in the deployment of the super-resolution network model on the actual platform. However, the proportion of the computational amount of the feature extraction module and the super-resolution reconstruction layer in the total computational amount of the entire model cannot be ignored in many scenarios. For example, for a 2x magnification task, these two modules respectively account for about 15.6% and 11.4% of the computational amount in the Super-Resolution Residual network (SRResnet) and the Enhanced Deep Super-Resolution network (EDSR). In a 4x magnification task, these two modules respectively account for about 45.1% and 38.7% of SRResnet and EDSR.

[0055] Therefore, the solution of quantifying all layers (modules) in the model in this application will improve the deployment efficiency of the model on the hardware platform. More specifically, in the super-resolution network model based on a deep neural network, the network structure usually includes a convolutional layer, a batch normalization layer, an addition layer, and an upsampling layer. The full-quantization super-resolution network structure provided in this application quantifies all types of layers into integer calculations.

[0056] Specifically, the input data of the convolutional layer is quantized through formula (2):

[0057] (2)

[0058] where, represents the quantization value of the data ; represents the quantization coefficient; represents the input data of the convolutional layer, which is an eigenvalue or a weight; represents the quantization bit width, represents the minimum quantization range, represents the maximum quantization range; represents the rounding integer function; , represents the integer part of the data after quantization; , represents the floating-point part of the data after quantization;

[0059] and constitute the quantization range. When is a value in an unsigned or non-negative interval, is 0, is 1. When is a signed value, is -1, is 1.

[0060] After that, after the input data of the convolutional layer is quantized, the convolutional layer performs a convolution operation through formula (3):

[0061] (3)

[0062] where, represents the eigenvalue after quantization of the convolutional layer; represents the weight after quantization of the convolutional layer; represents the output data of the convolutional layer; represents the integer part of the data ; Represents the floating-point part of the data ; Represents the integer part of the data ; Represents the floating-point part of the data ; , Represents the integer part of the data ; , Represents the floating-point part of the data .

[0063] In the batch normalization layer, the accurate value of the BN layer is calculated through (4), and the formula (4) is:

[0064]

[0065]

[0066] (4)

[0067] In this application, the input data of the batch normalization layer is quantized through formula (5):

[0068] (5)

[0069] Among them, ; ; Represents the output data of the batch normalization layer after quantization; Represents the floating-point part of the output data of the nearest preceding convolutional layer to this batch normalization layer after quantization; Represents the integer part of the output data of the nearest preceding convolutional layer to this batch normalization layer after quantization; Represents the mean of the batch normalization layer; Represents the variance of the batch normalization layer; Represents the constant used by the batch normalization layer to avoid the division overflow error caused; Represents the normalization coefficient of the batch normalization layer; Represents the bias of the batch normalization layer; Represents the data after quantization of the floating-point part; Represents the data after quantization of the integer part; Represents the rounding integer function.

[0070] That is, when quantizing the BN layer, only the bias term in the BN layer needs to be rounded.

[0071] The input data of the dot addition layer is quantized through formula (6):

[0072] )= (6)

[0073] Wherein, represents one of the quantized input data of the dot addition layer; represents another quantized input data of the dot addition layer; represents the quantized output data of the dot addition layer; represents the data after quantization of the floating-point part; represents the data after quantization of the integer part; represents the data after quantization of the floating-point part; represents the data after quantization of the integer part; represents the data after quantization of the floating-point part; represents the data after quantization of the integer part; .

[0074] Since the upsampling layer only adjusts the shape of the data, that is, only changes the arrangement order and does not perform specific numerical calculations, quantization is not required.

[0075] In this embodiment, the quantized data of the super-resolution convolutional neural network model has two components. The floating-point part is used when adjusting the range of the quantized data and is only used for auxiliary calculation during the training process to assist in the gradient optimization of the neural network. In actual deployment, the calculation only involves the integer part, that is, only the integer part of the quantized data is used to ensure that the calculation of the entire super-resolution convolutional neural network model is all-integer calculation.

[0076] Since only a part of the operations in the super-resolution algorithm are quantized in the related compression work in the prior art, after quantization, a large amount of calculations still maintain full-precision calculations, which brings huge challenges to the deployment of the algorithm in the actual environment.

[0077] To reduce the computational complexity of deep learning-based super-resolution algorithms, this application proposes a fully quantized quantization strategy. For the super-resolution convolutional neural network model, this strategy performs bit-width quantization on all layers of the model instead of some of them, and by giving quantization formulas for all layers involved in numerical calculations, the entire super-resolution algorithm can run end-to-end on low-bit integer computing units, which can reduce the computational complexity and storage complexity of the algorithm by an order of magnitude, resulting in an order-of-magnitude reduction in the computational complexity of the model, bringing great convenience to the deployment of the super-resolution network on mobile platforms, enabling the entire super-resolution algorithm to run end-to-end on integer computing components, and effectively improving the execution efficiency.

[0078] In this application, the self-supervised loss function is:

[0079]

[0080] where, represents the -th norm; represents the quantized value of the data , which can be the quantized value of the input data of any layer among the convolutional layer, batch normalization layer, point addition layer, or upsampling layer after quantization; the self-supervised loss function can make the quantization training of the model more stable by constraining the data error before and after quantization of a data, and the accuracy of the finally trained model will also be improved. Preferably, the value of

[0081] is 1. Therefore, the loss function

[0082]

[0083] of the entire model finally is: where represents the hyperparameter coefficient of the self-supervised loss function ; represents the final loss function;

[0084] represents the original loss function. It can be seen that the final loss function is obtained by updating the initial loss function based on the self-supervised loss function. p To improve the task accuracy of the super-resolution convolutional neural network model, this application proposes a self-supervised loss function (SQCL), which constrains the data before and after quantization of a data, making the data before and after quantization as consistent as possible, that is, the difference is within a preset range. Specifically, this self-supervised loss function uses pThe p-norm is used as a constraint means. Theoretically, the self-supervised loss function proposed in this application can use any order of norm. In order to obtain a better constraint effect, in this embodiment, when actually deployed, p= 1, that is, the first-order norm has the best effect.

[0085] Since in the prior art, the relevant compression work still uses the original training data and the loss function unique to super-resolution to train the quantization parameters. However, for low-bit quantization, especially after quantizing all layers in the network, the network training becomes very unstable, and ultimately may affect the accuracy of network quantization.

[0086] In order to improve the problem of loss of accuracy after quantization of the super-resolution convolutional neural network model, this application proposes a self-supervised loss function. Through the setting of the self-supervised loss function, the data distribution during the training process of the super-resolution convolutional neural network model is more stable, the data ranges before and after data quantization are closer, the effect of improving quantization accuracy is achieved, and the quantization accuracy of the model has been greatly improved.

[0087] The image reconstruction device provided by the present invention will be described below. The image reconstruction device described below can be correspondingly referred to the image reconstruction method described above.

[0088] Next, in combination with Figure 1 The image reconstruction device of the present invention is described. The device includes:

[0089] An image reconstruction module 10, configured to input an image with a first resolution into a trained super-resolution convolutional neural network model, and obtain an image with a second resolution output by the super-resolution convolutional neural network model. In this embodiment, the second resolution is higher than the first resolution, the first resolution is a low resolution, and the second resolution is a high resolution.

[0090] All convolutional layers (conv), batch normalization layers (BatchNormalization), element-wise addition layers (Element-wise Add), and upsampling layers (Upsample or Pixelshuffle) in the super-resolution convolutional neural network model are compressed in terms of network structure through quantization processing.

[0091] During the training process of the super-resolution convolutional neural network model, the difference between the data before and after quantization is within a preset range through the self-supervised loss function, and the loss function of the super-resolution convolutional neural network model is obtained based on the self-supervised loss function.

[0092] The image reconstruction device provided by the present invention is different from other compression schemes for super-resolution algorithms. In this application, all layers in the super-resolution convolutional neural network model based on deep learning are quantized. This not only realizes the compression of the model's computational bit width as low as possible, but also end-to-end realizes the mapping from low-resolution images to high-resolution images, greatly reducing the computational complexity of the model, and bringing advantages in terms of the running speed, power consumption, memory access bandwidth, etc. of the super-resolution convolutional neural network model during actual application. In addition, this application also constrains the quantization coefficients during the quantization process through a self-supervised loss function, making the quantization process more stable, stabilizing the model training process, and effectively improving the quantization accuracy of the super-resolution convolutional neural network model.

[0093] Since only a part of the operations in the super-resolution algorithm are quantized in the related compression work in the prior art, after quantization, a large amount of calculations still maintain full-precision calculations, bringing huge challenges to the deployment of the algorithm in the actual environment.

[0094] In order to reduce the computational complexity of the super-resolution algorithm based on deep learning, this application proposes a full-quantization strategy. For the super-resolution convolutional neural network model, this strategy quantizes the bit width of all layers in the model, rather than some of the layers, and gives quantization formulas for all layers involved in numerical calculations. The entire super-resolution algorithm can run end-to-end on low-bit integer computing units, which can reduce the computational complexity and storage complexity of the algorithm by an order of magnitude, making the computational complexity of the model reduced by an order of magnitude, bringing great convenience to the deployment of the super-resolution network on mobile platforms, and enabling the entire super-resolution algorithm to run end-to-end on integer computing components, effectively improving the execution efficiency.

[0095] Since the related compression work in the prior art still uses the original training data and the loss function inherent in super-resolution to train the quantization parameters. However, for low-bit quantization, especially after quantizing all layers in the network, the network training becomes very unstable, and ultimately may affect the quantization accuracy of the network.

[0096] In order to improve the problem of loss of quantization accuracy of the super-resolution convolutional neural network model after quantization, this application proposes a self-supervised loss function. Through the setting of the self-supervised loss function, the data distribution during the training process of the super-resolution convolutional neural network model is more stable, the data ranges before and after quantization are closer, and the effect of improving the quantization accuracy is achieved, and the quantization accuracy of the model has been greatly improved.

[0097] Figure 3 Illustrates a schematic diagram of the physical structure of an electronic device, such as Figure 3As shown in the figure, the electronic device may include: a processor 210, a communications interface 220, a memory 230, and a communication bus 240. Among them, the processor 210, the communications interface 220, and the memory 230 complete communication with each other through the communication bus 240. The processor 210 can call the logical commands in the memory 230 to execute an image reconstruction method, which includes:

[0098] Input an image with a first resolution into a trained super-resolution convolutional neural network model to obtain an image with a second resolution output by the super-resolution convolutional neural network model; the second resolution is higher than the first resolution;

[0099] For all convolutional layers, batch normalization layers, point addition layers, and upsampling layers in the super-resolution convolutional neural network model, the bit width of model calculation is compressed through quantization processing; during the training process of the super-resolution convolutional neural network model, the error between the data before and after quantization is within a preset range through a self-supervised loss function, and the final loss function of the super-resolution convolutional neural network model is obtained by updating the initial loss function based on the self-supervised loss function.

[0100] In addition, when the logical commands in the above-mentioned memory 230 are implemented in the form of software functional units and sold or used as an independent product, they can be stored in a computer-readable storage medium. Based on such an understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or a part of this technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several commands for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods described in various embodiments of the present invention. The aforementioned storage medium includes: USB flash drives, mobile hard disks, read-only memories (ROM, Read-Only Memory), random access memories (RAM, Random Access Memory), magnetic disks, or optical discs, etc., which can store program codes.

[0101] On the other hand, the present invention also provides a computer program product. The computer program product includes a computer program. The computer program can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the image reconstruction method provided by the above-mentioned various methods. The method includes:

[0102] Input an image with a first resolution into a trained super-resolution convolutional neural network model to obtain an image with a second resolution output by the super-resolution convolutional neural network model; the second resolution is higher than the first resolution;

[0103] In the super-resolution convolutional neural network model, all convolutional layers, batch normalization layers, point addition layers, and upsampling layers perform compression of the model calculation bit width through quantization processing; during the training process of the super-resolution convolutional neural network model, the self-supervised loss function is used to keep the error between the data before and after quantization within a preset range, and the final loss function of the super-resolution convolutional neural network model is obtained by updating the initial loss function based on the self-supervised loss function.

[0104] In another aspect, the present invention also provides a non-transitory computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, it implements the image reconstruction method provided by the above-mentioned various methods, and the method includes:

[0105] Input an image with a first resolution into the trained super-resolution convolutional neural network model to obtain an image with a second resolution output by the super-resolution convolutional neural network model; the second resolution is higher than the first resolution.

[0106] In the super-resolution convolutional neural network model, all convolutional layers, batch normalization layers, point addition layers, and upsampling layers perform compression of the model calculation bit width through quantization processing; during the training process of the super-resolution convolutional neural network model, the self-supervised loss function is used to keep the error between the data before and after quantization within a preset range, and the final loss function of the super-resolution convolutional neural network model is obtained by updating the initial loss function based on the self-supervised loss function.

[0107] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place, or may be distributed to multiple network units. Some or all of the modules can be selected according to actual needs to achieve the purpose of the solution of this embodiment. Those of ordinary skill in the art can understand and implement it without creative efforts.

[0108] Through the description of the above embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus a necessary general hardware platform, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence, or the part that contributes to the prior art can be embodied in the form of a software product. The computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disc, etc., and includes several instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0109] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit them. Although the present invention has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art should understand that they can still modify the technical solutions described in the foregoing embodiments, or perform equivalent replacements for some of the technical features. And these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. An image reconstruction method, characterized in that, The method includes: Inputting an image with a first resolution into a trained super-resolution convolutional neural network model to obtain an image with a second resolution output by the super-resolution convolutional neural network model; the second resolution is higher than the first resolution; All convolutional layers, batch normalization layers, point addition layers, and upsampling layers in the super-resolution convolutional neural network model perform compression of the model calculation bit width through quantization processing; during the training process of the super-resolution convolutional neural network model, the error between the data before and after quantization is within a preset range through a self-supervised loss function, and the final loss function of the super-resolution convolutional neural network model is obtained by updating the initial loss function based on the self-supervised loss function; The batch normalization layer performs quantization processing through formula (5): Among them, represents the output data of the batch normalization layer; α conv represents the floating-point part of the output data of the nearest preceding convolutional layer to this batch normalization layer; η conv represents the integer part of the output data of the nearest preceding convolutional layer to this batch normalization layer; μ represents the mean of the batch normalization layer; σ represents the variance of the batch normalization layer; ∈ represents the constant used by the batch normalization layer to avoid division overflow errors caused by σ; γ represents the normalization coefficient of the batch normalization layer; β represents the bias of the batch normalization layer; α z represents the floating-point part of the output data of the batch normalization layer that processes the data z; η z represents the integer part of the output data of the batch normalization layer that processes the data z; represents the rounding function for rounding.

2. The image reconstruction method according to claim 1, wherein The convolutional layer performs quantization processing through formula (2): Among them, Q(v) represents the quantization value of data v; I represents the quantization coefficient; v represents the input data of the convolutional layer, which is an eigenvalue or a weight; M represents the quantization bit width, Q min represents the minimum range of quantization, Q max represents the maximum range of quantization; represents the rounding integer function; η v represents the integer part of the data v after quantization; α v = 1 / (2 M -1), α v represents the floating-point part of the data v after quantization.

3. The image reconstruction method according to claim 2, characterized in that, After the input data is quantized, the convolutional layer performs a convolution operation through formula (3): y = x·w = (η x ·α x )·( η w ·α w ) = η conv ·α conv (3) Wherein, x represents the feature value after quantization of the convolutional layer; w represents the weight after quantization of the convolutional layer; y represents the output data of the convolutional layer; η x represents the integer part of the data x; α x represents the floating-point part of the data x; η w represents the integer part of the data w; α w represents the floating-point part of the data w; η conv = η x · η w ,η conv represents the integer part of the data y; α conv = α x · α w ,α conv represents the floating-point part of the data y.

4. The image reconstruction method according to claim 1, wherein The point addition layer performs quantization processing through formula (6): t = Q(h) + Q(u) = η h ·α h +η u ·α u =η t ·α t (6) Among them, h represents one of the quantized input data of the dot addition layer; u represents the other quantized input data of the dot addition layer; y represents the quantized output data of the dot addition layer; α h represents the floating-point part of the quantized data h; η h represents the integer part of the quantized data h; α u represents the floating-point part of the quantized data u; η u represents the integer part of the quantized data u; α t represents the floating-point part of the quantized data t; η t represents the integer part of the quantized data t; α t = α h = α u .

5. The image reconstruction method according to any one of claims 1-4, characterized in that The self-supervised loss function L q is as follows: L q = ‖Q(s) - s‖ p where, ||·||p represents the p-norm; Q(s) represents the quantization value of data s.

6. The image reconstruction method according to claim 5, wherein The loss function L is: L = L sr + αL q Among them, α represents the hyperparameter coefficient of the self-supervised loss function L q ; L sr represents the final loss function; L represents the original loss function.

7. An image reconstruction device, characterized in that, The apparatus includes: An image reconstruction module, configured to input an image with a first resolution into a trained super-resolution convolutional neural network model to obtain an image with a second resolution output by the super-resolution convolutional neural network model; the second resolution is higher than the first resolution; All convolutional layers, batch normalization layers, point addition layers, and upsampling layers in the super-resolution convolutional neural network model perform compression of the model calculation bit width through quantization processing; during the training process of the super-resolution convolutional neural network model, the error between the data before and after quantization is within a preset range through a self-supervised loss function, and the final loss function of the super-resolution convolutional neural network model is obtained by updating the initial loss function based on the self-supervised loss function; The batch normalization layer performs quantization processing through formula (5): Floating-point part; η conv represents the integer part of the output data of the previous convolutional layer closest to this batch normalization layer; μ represents the mean of the batch normalization layer; σ represents the variance of the batch normalization layer; ∈ represents the constant used by the batch normalization layer to avoid division overflow errors caused by σ; γ represents the normalization coefficient of the batch normalization layer; β represents the bias of the batch normalization layer; α z represents the floating-point part of the output data of the batch normalization layer that processes the data z; η z represents the integer part of the output data of the batch normalization layer that processes the data z; represents the rounding function for rounding.

8. An electronic device, comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, wherein, When the processor executes the program, it implements the steps of the image reconstruction method according to any one of claims 1 to 6.

9. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the steps of the image reconstruction method according to any one of claims 1 to 6.

Images