Image super-resolution reconstruction method, model distillation method, device and storage medium
By employing deep learning-based feature extraction, alignment, fusion, and reconstruction on multiple frames, the method enhances image super-resolution reconstruction efficiency and quality using a simpler network structure.
Patent Information
- Application Number
- CN202110039733.6
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2021-01-13
- Publication Date
- 2024-12-20
- Estimated Expiration
- 2041-01-13
AI Technical Summary
Existing image super-resolution methods struggle to effectively utilize inter-frame information and achieve high efficiency with a simple network structure, leading to suboptimal reconstruction results.
A method involving feature extraction, data alignment, fusion, and reconstruction using deep learning networks on multiple consecutive frames to maximize inter-frame information utilization, resulting in a simpler network design with improved efficiency and reconstruction quality.
The proposed method maximizes inter-frame information use, achieving better output effects with fewer network parameters and higher operating efficiency in image super-resolution reconstruction.
Smart Images
Figure CN112767247B_ABST
Abstract
Description
Technical Field
[0001] The embodiments of the present disclosure relate to the field of image processing technology, and specifically to an image super-resolution reconstruction method, a model distillation method, a device, and a computer-readable storage medium. Background Art
[0002] Image super-resolution is the process of improving the resolution of the original image through hardware or software methods, and obtaining a high-resolution image through a series of low-resolution images. Summary of the invention
[0003] The embodiments of the present disclosure provide a super-resolution reconstruction method, a model distillation method, a device and a computer-readable storage medium, which have good reconstruction effects.
[0004] On the one hand, the present disclosure provides an image super-resolution reconstruction method, comprising:
[0005] Performing feature extraction on at least three temporally continuous input original image frames to obtain at least three images to be processed, each of which corresponds to one original image frame;
[0006] Taking one of the at least three images to be processed as a reference image, using an alignment processing network based on deep learning to perform data alignment between the image to be processed and the reference image, and outputting a plurality of aligned frames;
[0007] Using a deep learning-based fusion processing network to perform data fusion on the multiple aligned frames;
[0008] A deep learning-based reconstruction processing network is used to upsample the data fused image to achieve image super-resolution reconstruction.
[0009] On the other hand, an embodiment of the present disclosure provides a model distillation method, including using the aforementioned image super-resolution reconstruction method to obtain a reconstructed image after super-resolution reconstruction, using a teacher model to obtain a predicted image, and calculating the loss function of the reconstructed image and the predicted image. When the loss function meets the preset loss function requirements, the model distillation is completed.
[0010] On the other hand, an embodiment of the present disclosure also provides an image processing device, including a processor and a memory storing a computer program that can be run on the processor, wherein when the processor executes the program, the steps of the above-mentioned image super-resolution reconstruction method or model distillation method are implemented.
[0011] On the other hand, an embodiment of the present disclosure further provides a computer-readable storage medium storing a computer program executable on a processor, wherein the computer program, when executed by the processor, is used to implement the above-mentioned image super-resolution reconstruction method or model distillation method.
[0012] The super-resolution reconstruction method based on multi-frame information alignment proposed in the present invention performs data alignment on three consecutive frames of input information, fuses the aligned information, and finally realizes super-resolution reconstruction through a reconstruction processing network. The method of the embodiment of the present invention can maximize the use of inter-frame information to obtain better output effects, has few network parameters, simple structure, and high operating efficiency, and the network design can simultaneously meet the super-resolution reconstruction effect and operating efficiency.
[0013] Of course, it is not necessary to achieve all the advantages described above at the same time for any product or method of the present disclosure. Other features and advantages of the present disclosure will be described in the subsequent description embodiments, and partly become apparent from the description embodiments, or be understood by practicing the present disclosure. The purposes and other advantages of the embodiments of the present disclosure can be achieved and obtained by the structures specifically pointed out in the description, claims and drawings. BRIEF DESCRIPTION OF THE DRAWINGS
[0014] The accompanying drawings are used to provide a further understanding of the technical solution of the present disclosure and constitute a part of the specification. Together with the embodiments of the present disclosure, they are used to explain the technical solution of the present disclosure and do not constitute a limitation on the technical solution of the present disclosure. The shapes and sizes of the components in the accompanying drawings do not reflect the actual proportions and are only intended to illustrate the contents of the present disclosure.
[0015] Figure 1 This is a flow chart of the image super-resolution reconstruction method according to an embodiment of the present disclosure;
[0016] Figure 2 This is a schematic diagram of the preprocessing network structure of an embodiment of the present disclosure;
[0017] Figure 3 This is a schematic diagram of the alignment processing network structure according to an embodiment of the present disclosure;
[0018] Figure 4 This is a schematic diagram of the fusion processing network structure of the embodiment of the present disclosure;
[0019] Figure 5 A schematic diagram of a reconstruction processing network structure according to an embodiment of the present disclosure;
[0020] Figure 6 A schematic diagram of a convolutional neural network structure;
[0021] Figure 7 This is a schematic diagram of the distillation process of the model of the embodiment of the present disclosure;
[0022] Figure 8 Schematic diagram of the structure of the image processing device according to the embodiment of the present disclosure. DETAILED DESCRIPTION
[0023] The present disclosure describes multiple embodiments, but the description is exemplary rather than restrictive, and it is obvious to those skilled in the art that there may be more embodiments and implementations within the scope of the embodiments described in the present disclosure. Although many possible feature combinations are shown in the drawings and discussed in the embodiments, many other combinations of the disclosed features are also possible. Unless specifically limited, any feature or element of any embodiment may be used in combination with any other feature or element in any other embodiment, or may replace any other feature or element in any other embodiment.
[0024] The present disclosure includes and contemplates combinations of features and elements known to those of ordinary skill in the art. The embodiments, features, and elements disclosed in the present disclosure may also be combined with any conventional features or elements to form a unique invention scheme defined by the claims. Any features or elements of any embodiment may also be combined with features or elements from other invention schemes to form another unique invention scheme defined by the claims. Therefore, it should be understood that any feature shown and / or discussed in the present disclosure may be implemented individually or in any appropriate combination. Therefore, except for the limitations made according to the attached claims and their equivalents, the embodiments are not subject to other limitations. In addition, various modifications and changes may be made within the scope of protection of the attached claims.
[0025] In addition, when describing representative embodiments, the specification may have presented the method and / or process as a specific sequence of steps. However, to the extent that the method or process does not rely on the specific order of the steps described herein, the method or process should not be limited to the steps in the specific order described. As will be appreciated by those of ordinary skill in the art, other orders of steps are also possible. Therefore, the specific order of the steps set forth in the specification should not be interpreted as a limitation to the claims. In addition, the claims for the method and / or process should not be limited to the steps performed in the order written, and those skilled in the art can easily understand that these orders can be changed and still remain within the spirit and scope of the disclosed embodiments.
[0026] Unless otherwise defined, the technical terms or scientific terms used in this disclosure are the usual meanings understood by people with ordinary skills in the field to which this disclosure belongs. The "first", "second" and similar words used in this disclosure do not indicate any order, quantity or importance, but are only used to distinguish different components. In this disclosure, "plurality" can mean two or more numbers. "Include" or "comprising" and similar words mean that the elements or objects appearing before the word cover the elements or objects listed after the word and their equivalents, without excluding other elements or objects.
[0027] In order to keep the following description of the embodiments of the present disclosure clear and concise, the present disclosure omits detailed descriptions of some known functions and known components. The drawings of the embodiments of the present disclosure only relate to the structures related to the embodiments of the present disclosure, and other structures can refer to the general design.
[0028] The present disclosure provides a super-resolution reconstruction method. Figure 1 As shown, the following steps are included:
[0029] Step 11, preprocessing step: extract features from at least three temporally continuous input original image frames to obtain at least three images to be processed, each of which corresponds to one original image frame.
[0030] The purpose of feature extraction is to facilitate subsequent alignment processing;
[0031] In an exemplary embodiment, the input multiple frames of images may be three consecutive frames of images in a video stream. By using three temporally consecutive frames of images as original images, it is possible to utilize inter-frame information to obtain a better output effect.
[0032] In the embodiment of the present disclosure, a preprocessing network based on deep learning can be used to preprocess the input multiple frames of images.
[0033] Step 12, an alignment step: using one of the at least three images to be processed as a reference image, using an alignment processing network based on deep learning to perform data alignment between the image to be processed and the reference image, and outputting a plurality of aligned frames;
[0034] In an exemplary embodiment, when the number of original images is three, that is, the number of preprocessed images is three, the intermediate image to be processed, that is, the image obtained after preprocessing the intermediate frame original image, can be used as a reference image, so that the two frames before and after the intermediate frame are aligned to the intermediate image respectively, and at least the aligned frame of the first image and the second image, and the aligned frame of the second image and the third image are obtained. Optionally, the second image can also be input into the alignment processing network to obtain enhanced second image information.
[0035] Step 13, fusion step: using a fusion processing network based on deep learning to perform data fusion on the multiple aligned frames;
[0036] In the embodiment of the present disclosure, a fusion processing network based on deep learning may be used for data fusion, and the fused output includes information of the original frames of the aligned multiple inputs.
[0037] Step 14, reconstruction step: use a deep learning-based reconstruction processing network to upsample the data fused image to achieve image super-resolution reconstruction.
[0038] The image after data fusion is upsampled to obtain a super-resolution reconstruction result. The super-resolution multiple can be 2 times, 3 times, 4 times, 8 times, 16 times, etc.
[0039] The disclosed embodiment proposes a super-resolution reconstruction method based on multi-frame information alignment, in which data alignment is performed on three consecutive frames of input information respectively, and the aligned information is fused, and finally super-resolution reconstruction is achieved through a reconstruction processing network. The method of the disclosed embodiment can maximize the use of inter-frame information to obtain better output effects, with fewer network parameters, simple structure, and high operating efficiency. The network design can simultaneously meet the super-resolution reconstruction effect and operating efficiency.
[0040] In an exemplary embodiment, a deep learning based preprocessing network such as Figure 2 As shown, it includes a preprocessing concatenation layer, a preprocessing convolution layer and a preprocessing activation layer, where:
[0041] The preprocessing splicing layer is used to splice the input original images in one dimension, for example, using the concat function. The preprocessing splicing layer is as follows: Figure 2 As shown in concat1;
[0042] The preprocessing convolution layer is used to extract local features of the spliced image, for example, using the conv2d function. The preprocessing splicing layer is as follows: Figure 2 As shown in conv2d1.
[0043] The preprocessing activation layer is used to perform nonlinear mapping on the feature map obtained after convolution of the preprocessing convolution layer. In this embodiment, the linear rectified linear unit (ReLu) function is used to implement the preprocessing activation layer. Figure 2 relu1 as shown in .
[0044] Taking the input image as three consecutive frames of images as an example, the above preprocessing network is explained.
[0045] The input images of the preprocessing network include the first frame image (P1), the second frame image (P2) and the third frame image (P3). The size of each frame image has four dimensions, including batch, channel, width and height. In this embodiment, the size of each frame image is expressed as (1, 3, W, H). Among them, batch is 1; channel is 3, representing R (red), G (green) and B (blue). After the three input frames are spliced in the batch dimension through the preprocessing splicing layer, an output image of size (3, 3, W, H) is formed;
[0046] In this embodiment, the preprocessing convolution layer is implemented using the conv2d function, the convolution kernel size is 3×3, and the number of channels is 64, that is, there are 64 convolution kernels, and accordingly, the output of the convolution layer is 64 feature maps. In other embodiments, the number of channels can also be 96 or 128. The smaller the number of channels, the faster the processing speed, and the larger the number of channels, the better the processing effect.
[0047] After the preprocessing activation layer performs nonlinear mapping on the feature map, the output image size is (3, 64, W, H). The output image will be split into three groups of output images from the batch dimension, namely the preprocessed first frame image (hereinafter referred to as the first image) F1, the preprocessed second frame image (hereinafter referred to as the second image) F2 and the preprocessed third frame image (hereinafter referred to as the third image) F3, where the size of the first image F1 is (1, 64, W, H), the size of the second image F2 is (1, 64, W, H), and the size of the third image F3 is (1, 64, W, H).
[0048] In an exemplary embodiment, the deep learning-based alignment processing network is as follows: Figure 3 As shown, it includes multiple convolution blocks, the first convolution block includes a first splicing layer, a first convolution layer and a first activation layer, and the input of the first convolution block includes a first input image (inx1) and a second input image (inx2); the second convolution block includes a second splicing layer, a second convolution layer and a second activation layer, and the input of the second convolution block includes the output of the first convolution block and the first input image; the third convolution block includes a third splicing layer, a third convolution layer and a third activation layer, and the input of the third convolution block includes the output of the second convolution block and the first input image; the fourth convolution block includes a fourth splicing layer, a fourth convolution layer and a fourth activation layer, and the input of the fourth convolution block includes the output of the third convolution block and the second input image; the fifth convolution block includes a fifth splicing layer, a fifth convolution layer and a fifth activation layer, and the input of the fifth convolution block includes the output of the fourth convolution block and the second input image; the output of the fifth convolution block is the output of the alignment processing network (Align out).
[0049] In an exemplary embodiment, the alignment network model may further include a sixth convolution block after the fifth convolution block, the sixth convolution block includes a sixth convolution layer, a sixth convolution layer and a sixth activation layer, the input of the sixth convolution block includes the output of the fifth convolution block and the second input image; the output of the sixth convolution block is the output of the alignment processing network.
[0050] Optionally, the first splicing layer, the second splicing layer, the third splicing layer, the fourth splicing layer, the fifth splicing layer and the sixth splicing layer (if any) are implemented using a concat function.
[0051] Optionally, the first convolution layer, the second convolution layer, the third convolution layer, the fourth convolution layer, the fifth convolution layer and the sixth convolution layer (if any) are implemented using a conv2d function, with a convolution kernel size of 3×3 and a number of channels of 64.
[0052] Optionally, the first activation layer, the second activation layer, the third activation layer, the fourth activation layer, the fifth activation layer and the sixth activation layer (if any) are implemented using a relu function.
[0053] The first image, the second image, and the third image obtained by preprocessing are grouped and input into the alignment processing network, and multiple alignment processes are performed to obtain multiple aligned frames. The grouped input into the alignment processing network, and the multiple alignment processes, for example, include:
[0054] First alignment: the first image and the second image are used as a set of input alignment processing networks, wherein the first image is used as the first input image and the second image is used as the second input image. The output image of the alignment processing network is the first calibration image (or the first alignment image, the first alignment frame) A1, which can be understood as obtaining the error information of the motion between the first image and the second image. Obtaining the error information can perform motion compensation on the output image in the subsequent reconstruction step.
[0055] Second alignment: the second image and the second image are used as a set of input alignment processing networks, wherein the second image is used as the first input image and the second input image respectively, and the alignment processing network outputs a second calibration image (or second alignment image, second alignment frame) A2, which is used to extract the second image information. By extracting the second image information, it can be understood as the information of the enhanced reference frame (i.e., the reference image);
[0056] The third alignment: the second image and the third image are used as a set of input alignment processing networks, wherein the second image is used as the first input image and the third image is used as the second input image. The alignment processing network outputs a third calibration image (or third alignment image, third alignment frame) A3, which can be understood as obtaining the error information of the motion between the second image and the third image. Obtaining the error information can perform motion compensation on the output image in the subsequent reconstruction step.
[0057] In this embodiment, the execution order of the three alignments is not limited.
[0058] In this embodiment, the second frame image after preprocessing the middle frame of three temporally continuous original frames is used as a reference frame, which can better achieve alignment. The purpose of the alignment process is to align the input images data. Since there is motion between frames, data alignment is more conducive to learning the motion information and the reference frame information.
[0059] In an exemplary embodiment, a deep learning based fusion processing network is Figure 4 As shown, it includes a seventh convolution block, an eighth convolution block, a ninth convolution block, a tenth convolution block and an eleventh convolution block, wherein the seventh convolution block and the eighth convolution block are two parallel branches. The input of the seventh convolution block includes the first calibration image A1 and the second calibration image A2, and the output image of the seventh convolution block serves as the first input of the ninth convolution block. The input of the eighth convolution block includes the second calibration image A2 and the third calibration image A3, the output image of the eighth convolution block serves as the second input of the ninth convolution block, the output of the ninth convolution block serves as the input of the tenth convolution block, the output of the tenth convolution block serves as the input of the eleventh convolution block, and the output of the eleventh convolution block serves as the output (out) of the fusion processing network model.
[0060] In this embodiment, the structure of each convolution block in the seventh convolution block, the eighth convolution block and the ninth convolution block includes a convolution layer, a convolution layer and an activation layer. The structure of each convolution block in the tenth convolution block and the eleventh convolution block includes a convolution layer and an activation layer. Among them, the convolution layer of each convolution block in the seventh convolution block to the ninth convolution block is implemented by the concat function. The convolution layer of each convolution block in the seventh convolution block to the eleventh convolution block is implemented by the conv2d function, the convolution kernel is 3×3, and the number of channels is 64. The activation layer of each convolution block in the seventh convolution block to the eleventh convolution block adopts the relu function.
[0061] The first calibration image A1 is fused with the second calibration image A2 through the seventh convolution block to obtain a first fused image A12. The second calibration image A2 is fused with the third calibration image A3 through the eighth convolution block to obtain a second fused image A23. The first fused image A12 is fused with the second fused image A23 through the ninth convolution block to obtain a third fused image A123. The third fused image is learned by the tenth convolution block and the eleventh convolution block to obtain the fused output O1.
[0062] In an exemplary embodiment, the tenth convolution block and the eleventh convolution block are optional convolution blocks, that is, the third fused image A123 can be used as the fused output O1. Alternatively, in an exemplary embodiment, the fusion processing network includes the tenth convolution block, that is, the third fused image A123 is learned once by the tenth convolution block, and the output obtained is the fused output O1. Alternatively, in an exemplary embodiment, the fusion processing network may further include a twelfth convolution block, the structure of which is the same as that of the tenth convolution block and the eleventh convolution block, and learning is performed again through the twelfth convolution block.
[0063] In an exemplary embodiment, the deep learning based reconstruction processing network is as follows Figure 5 As shown, it includes a thirteenth convolution block, a fourteenth convolution block, a fifteenth convolution block, a sixteenth convolution block and a seventeenth convolution block. The output O1 of the fusion processing network is the input of the thirteenth convolution block, the output of the thirteenth convolution block is the input of the fourteenth convolution block, the sum of the output of the fourteenth convolution block and the output of the thirteenth convolution block is the input of the fifteenth convolution block, the sum of the output of the fifteenth convolution block, the output of the thirteenth convolution block and the output of the fourteenth convolution block is the input of the sixteenth convolution block, the output of the sixteenth convolution block is the input of the seventeenth convolution block, and the output of the seventeenth convolution block is the output O2 of the reconstruction processing network.
[0064] Among them, the structure of each convolution block in the thirteenth convolution block, the fourteenth convolution block and the fifteenth convolution block includes a convolution layer and an activation layer. Among them, the convolution layer is implemented by the conv2d function, the convolution kernel is 3×3, the number of channels is 64, and the activation layer is implemented by the relu function. The sixteenth convolution block includes the sixteenth convolution layer and the sixteenth activation layer, among which the sixteenth convolution layer adopts the deconvolution (convTranspose2d) function, and super-resolution is achieved by up-sampling. The seventeenth convolution block includes the seventeenth convolution layer and the seventeenth activation layer, the seventeenth convolution layer is implemented by the conv2d function, the convolution kernel is 3×3, the number of channels is 64, and the seventeenth activation layer is implemented by the relu function.
[0065] In this embodiment, upsampling by 2 times is taken as an example. In other embodiments, it can also be 4 times, 8 times or 16 times, and it only needs to be upsampled several times. In some exemplary embodiments, it can be 3 times super-resolution, and it can be processed by 4 times super-resolution first, and then reduced by traditional methods, or the original image can be super-resolved by 1.5 times traditional algorithm first, and then 2 times super-resolution is achieved by the method of the embodiment of the present disclosure.
[0066] In other embodiments, the reconstruction processing network may include two thirteenth convolution blocks and one fourteenth convolution block before the sixteenth convolution block, that is, the fifteenth convolution block is not included. Or the reconstruction processing network may include two thirteenth convolution blocks, one fourteenth convolution block and one fifteenth convolution block before the sixteenth convolution block. Or the reconstruction processing network may include one thirteenth convolution block, one fourteenth convolution block and two fifteenth convolution blocks before the sixteenth convolution block.
[0067] The convolution block mentioned in this embodiment can be considered as a convolutional neural network. A convolutional neural network is a special structure of a neural network that uses images as the input and output of the system and uses convolution kernels to replace scalar weights. A three-layer convolutional neural network is as follows: Figure 6 shown. Figure 6 The network shown has 4 inputs, 3 outputs in the hidden layer, and 2 outputs in the output layer. Finally, the system outputs two images. Represents a convolution kernel. k represents the input layer number, i and j represent the input and output unit numbers. It is a set of scalars superimposed on the output of the convolution layer. The output of the convolution layer with the bias superimposed will enter the activation layer (such as relu or sigmoid function). After training, the convolution kernel and bias of the system are fixed. Training is to tune the parameters of the convolution kernel and bias through a set of matching input and output and optimization algorithms. Usually each convolution layer can contain dozens or hundreds of convolution kernels. In this embodiment, the number of convolution kernels is 64 as an example.
[0068] The disclosed embodiment also provides a model distillation method, which refers to migrating the knowledge learned from a large model or multiple models to a lightweight single model for easy deployment. That is, using a new small model to learn the prediction results of the large model. Figure 7 As shown, the model distillation method comprises the following steps:
[0069] Step 21, using the aforementioned super-resolution reconstruction method to obtain a reconstructed image after super-resolution reconstruction;
[0070] As mentioned above, the input of the super-resolution reconstruction method is three temporally consecutive frames, and the middle frame of the three frames is used as a reference frame to obtain the super-resolution reconstructed image. The neural network that implements the above super-resolution reconstruction method is the student model.
[0071] Step 22, use the teacher model to obtain the predicted image;
[0072] The teacher model is a native complex network model, which can be any native model used to achieve super-resolution reconstruction. Depending on the native model, the input can be 1 frame or multiple frames.
[0073] The execution order of the above steps 21 and 22 is not limited.
[0074] Step 23, calculate the loss function of the reconstructed image and the predicted image. When the loss function meets the preset loss function requirements, the model distillation is completed, that is, the student model training is completed.
[0075] The output of the Student model and the output of the Teacher model are used to calculate the loss function, and the parameters of the student model are updated to realize the training of the student model. The loss function can adopt the L2 loss function. When the loss function meets the preset loss function requirements, for example, the loss function result is less than the preset loss function threshold, the student model training is completed.
[0076] In an exemplary embodiment, the present disclosure also provides an image processing device, which may include a processor and a memory, wherein the memory stores a computer program that can be run on the processor, wherein when the processor executes the computer program, the steps of the image super-resolution reconstruction method or model distillation method in any of the above embodiments of the present disclosure are implemented.
[0077] In an exemplary embodiment, Figure 8 FIG. 1 is a schematic diagram of the structure of an image processing device in an embodiment of the present disclosure. Figure 8 As shown, the device 60 includes: at least one processor 601; and at least one memory 602 and a bus 603 connected to the processor 601; wherein the processor 601 and the memory 602 communicate with each other through the bus 603; the processor 601 is used to call the program instructions in the memory 602 to execute the steps of the method in any of the above embodiments.
[0078] The processor can be a central processing unit (CPU), a microprocessor (MPU), a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA), a transistor logic device, etc., and the present disclosure does not limit this.
[0079] The memory may include a read-only memory (ROM) and a random access memory (RAM), and provides instructions and data to the processor. A portion of the memory may also include a non-volatile random access memory. For example, the memory may also store information about the device type.
[0080] In addition to the data bus, the bus may also include a power bus, a control bus, and a status signal bus. However, for the sake of clarity, Figure 8 In the text file, various buses are marked as buses.
[0081] In the implementation process, the processing performed by the processing device can be completed by the hardware integrated logic circuit in the processor or the instructions in the form of software. That is, the method steps of the embodiment of the present disclosure can be embodied as being executed by a hardware processor, or being executed by a combination of hardware and software modules in the processor. The software module can be located in a storage medium such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory or an electrically erasable programmable memory, a register, etc. The storage medium is located in the memory, and the processor reads the information in the memory and completes the steps of the above method in combination with its hardware. To avoid repetition, it will not be described in detail here.
[0082] In an exemplary embodiment, the present disclosure also provides a non-volatile computer-readable storage medium on which is stored a computer program that can be run on a processor, and when the computer program is executed by the processor, the steps of the aforementioned image super-resolution reconstruction method or model distillation method are implemented.
[0083] It will be appreciated by those skilled in the art that all or some of the steps, systems, and functional modules / units in the methods disclosed above may be implemented as software, firmware, hardware, and appropriate combinations thereof. In hardware implementations, the functional modules / units mentioned in the above description are not divided equally; for example, a physical component may have multiple functions, or a function or step may be performed by several physical components in cooperation. Some or all components may be implemented as software executed by a processor, such as a digital signal processor or a microprocessor, or implemented as hardware, or implemented as an integrated circuit, such as an application-specific integrated circuit. Such software may be distributed on a computer-readable medium, which may include a computer storage medium (or non-transitory medium) and a communication medium (or temporary medium). As known to those skilled in the art, the term computer storage medium includes volatile and non-volatile, removable and non-removable media implemented in any method or technology for storing information (such as computer-readable instructions, data structures, program modules, or other data). Computer storage media include, but are not limited to, RAM, ROM, EEPROM, flash memory or other memory technology, CD-ROM, digital versatile disks (DVD) or other optical disk storage, magnetic cassettes, magnetic tapes, magnetic disk storage or other magnetic storage devices, or any other medium that can be used to store the desired information and can be accessed by a computer. In addition, it is well known to those of ordinary skill in the art that communication media typically contain computer-readable instructions, data structures, program modules, or other data in a modulated data signal such as a carrier wave or other transport mechanism, and may include any information delivery media.
[0084] Although the embodiments disclosed in the present disclosure are as above, the contents described are only embodiments adopted to facilitate understanding of the present disclosure and are not intended to limit the present disclosure. Any technician in the field to which the present disclosure belongs can make any modifications and changes in the form and details of implementation without departing from the spirit and scope disclosed in the present disclosure, but the scope of patent protection of the present disclosure shall still be subject to the scope defined in the attached claims.
Claims
1. A method for super-resolution image reconstruction, characterized in that: include: Performing feature extraction on at least three temporally continuous input original image frames to obtain at least three images to be processed, each of which corresponds to one original image frame; Taking one of the at least three images to be processed as a reference image, using an alignment processing network based on deep learning to perform data alignment between the image to be processed and the reference image, and outputting multiple aligned frames; wherein, taking the middle image to be processed among the at least three images to be processed as a reference image, the middle image to be processed corresponds to the middle frame among the at least three frames of original images that are continuous in time, inputting the first image to be processed and the second image to be processed into the alignment processing network to obtain a first aligned frame, inputting the second image to be processed and the second image to be processed into the alignment processing network to obtain a second aligned frame, and inputting the second image to be processed and the third image to be processed into the alignment processing network to obtain a third aligned frame; Using a deep learning-based fusion processing network to perform data fusion on the multiple aligned frames; A deep learning-based reconstruction processing network is used to upsample the data fused image to achieve image super-resolution reconstruction.
2. The method according to claim 1, characterized in that The alignment processing network based on deep learning includes: a first convolution block, a second convolution block, a third convolution block, a fourth convolution block and a fifth convolution block, wherein: The first convolution block includes a first concatenation layer, a first convolution layer and a first activation layer, and the input of the first convolution block includes a first input image and a second input image; The second convolution block includes a second concatenation layer, a second convolution layer, and a second activation layer, and the input of the second convolution block includes the output of the first convolution block and the first input image; The third convolution block includes a third concatenation layer, a third convolution layer and a third activation layer, and the input of the third convolution block includes the output of the second convolution block and the first input image; The fourth convolution block includes a fourth concatenation layer, a fourth convolution layer and a fourth activation layer, and the input of the fourth convolution block includes the output of the third convolution block and the second input image; The fifth convolution block includes a fifth splicing layer, a fifth convolution layer and a fifth activation layer. The input of the fifth convolution block includes the output of the fourth convolution block and the second input image. The output of the fifth convolution block is the output of the alignment processing network.
3. The method according to claim 2, characterized in that in, The first splicing layer, the second splicing layer, the third splicing layer, the fourth splicing layer and the fifth splicing layer are implemented using the concat function; The first convolutional layer, the second convolutional layer, the third convolutional layer, the fourth convolutional layer and the fifth convolutional layer are implemented using the conv2d function, with a convolution kernel size of 3×3 and a number of channels of 64; The first activation layer, the second activation layer, the third activation layer, the fourth activation layer and the fifth activation layer are implemented using the relu function.
4. The method according to claim 1, characterized in that: in, The deep learning-based fusion processing network includes a seventh convolution block, an eighth convolution block, a ninth convolution block, a tenth convolution block and an eleventh convolution block, wherein: The input of the seventh convolution block includes the first aligned frame and the second aligned frame, and the output of the seventh convolution block serves as the first input of the ninth convolution block; The input of the eighth convolution block includes the second aligned frame and the third aligned frame, and the output of the eighth convolution block serves as the second input of the ninth convolution block; The output of the ninth convolution block is the input of the tenth convolution block, the output of the tenth convolution block is the input of the eleventh convolution block, and the output of the eleventh convolution block is the output of the fusion processing network.
5. The method according to claim 4, characterized in that in, The structure of each convolution block in the seventh convolution block, the eighth convolution block and the ninth convolution block includes a convolution layer, a convolution layer and an activation layer; the structure of each convolution block in the tenth convolution block and the eleventh convolution block includes a convolution layer and an activation layer; wherein, the convolution layer of each convolution block in the seventh convolution block, the eighth convolution block and the ninth convolution block is implemented by the concat function, and the convolution layer of each convolution block in the seventh convolution block, the eighth convolution block, the ninth convolution block, the tenth convolution block and the eleventh convolution block is implemented by the conv2d function, the convolution kernel is 3×3, and the number of channels is 64; the activation layer of each convolution block in the seventh convolution block, the eighth convolution block, the ninth convolution block, the tenth convolution block and the eleventh convolution block adopts the relu function.
6. The method according to claim 1, characterized in that in, The deep learning-based fusion processing network includes a thirteenth convolution block, a fourteenth convolution block, a fifteenth convolution block, a sixteenth convolution block and a seventeenth convolution block, wherein: The output of the fusion processing network is the input of the thirteenth convolution block, the output of the thirteenth convolution block is the input of the fourteenth convolution block, the sum of the output of the fourteenth convolution block and the output of the thirteenth convolution block is the input of the fifteenth convolution block, the sum of the output of the fifteenth convolution block, the output of the thirteenth convolution block and the output of the fourteenth convolution block is the input of the sixteenth convolution block, the output of the sixteenth convolution block is the input of the seventeenth convolution block, and the output of the seventeenth convolution block is the output of the reconstruction processing network.
7. The method according to claim 6, characterized in that in, The structure of each convolution block in the thirteenth convolution block, the fourteenth convolution block and the fifteenth convolution block includes a convolution layer and an activation layer, wherein the convolution layer is implemented by using a conv2d function, the convolution kernel is 3×3, and the number of channels is 64; the activation layer is implemented by using a relu function; The sixteenth convolution block includes a sixteenth convolution layer and a sixteenth activation layer, wherein the sixteenth convolution layer uses a deconvolution function to achieve super-resolution; The seventeenth convolution block includes the seventeenth convolution layer and the seventeenth activation layer. The seventeenth convolution layer is implemented by using the conv2d function. The convolution kernel is 3×3 and the number of channels is 64. The seventeenth activation layer is implemented by using the relu function.
8. The method according to claim 1, characterized in that: in, Performing feature extraction on at least three temporally continuous frames of original images inputted, including: using a preprocessing network based on deep learning to perform feature extraction on at least three temporally continuous frames of original images, wherein the preprocessing network based on deep learning includes a preprocessing concatenation layer, a preprocessing convolution layer, and a preprocessing activation layer, wherein: The preprocessing splicing layer is used to splice the input original image in one dimension; Preprocessing convolution layer, used to extract local features of the spliced image; The preprocessing activation layer is used to perform nonlinear mapping on the feature map obtained after convolution of the preprocessing convolution layer.
9. The method according to claim 8, characterized in that in, The preprocessing concatenation layer is implemented by the concat function; the preprocessing convolution layer is implemented by the conv2d function; and the preprocessing activation layer is implemented by the relu function.
10. A model distillation method, comprising: A reconstructed image after super-resolution reconstruction is obtained by the method as described in any one of claims 1 to 9, a predicted image is obtained by using a teacher model, and a loss function of the reconstructed image and the predicted image is calculated. When the loss function meets the preset loss function requirements, the model distillation is completed.
11. An image processing device, comprising a processor and a memory storing a computer program executable on the processor, wherein: When the processor executes the program, the steps of the method according to any one of claims 1 to 9 or 10 are implemented.
12. A computer-readable storage medium, characterized in that: A computer program that can be run on a processor is stored thereon, and when the computer program is executed by the processor, the steps of the method according to any one of claims 1 to 9 or 10 are implemented.