Multi-frame image super-resolution method based on recurrent neural network

Abstract

A kind of multi-frame image super-resolution method based on recurrent neural network belongs to image restoration technical field.The present low-resolution image is recovered by single image, and the problem of insufficient details of the recovery result is solved.The present application comprises: setting recurrent neural network as the basic architecture of super-resolution model, and training recurrent neural network to obtain multi-frame image super-resolution network;The recurrent neural network comprises N branches, wherein the first branch comprises the first encoder, the first fusion module and the first upsampling module, and the remaining N-1 branches comprise the i-th encoder, the i-1 alignment module, the i-th fusion module and the i-th upsampling module;N is a positive integer;I=2, 3, 4, …, N;High-resolution image is output through the Nth branch;During the training process of recurrent neural network, network parameters are modified according to the loss function.The method of the present application is based on the recovery of low-resolution image of multiple input low-resolution images of the same scene.

Description

Technical Field

[0001] This invention relates to a multi-frame image super-resolution method based on recurrent neural networks, belonging to the field of image restoration technology. Background Technology

[0002] Image super-resolution aims to process low-resolution images to obtain high-resolution images with richer and more realistic details. If only a single image is used as input, the resulting image often lacks sufficient detail.

[0003] Currently, multi-frame photography modes on imaging devices such as mobile phones can quickly capture multiple images of the same natural scene. However, due to sub-pixel shifts caused by camera shake or the movement of objects themselves, different images may contain different sampling information of the same natural scene. Summary of the Invention

[0004] To address the problem that existing low-resolution image restoration methods rely on single-image reconstruction, resulting in insufficient detail, this invention provides a multi-frame image super-resolution method based on recurrent neural networks.

[0005] The present invention provides a multi-frame image super-resolution method based on a recurrent neural network, comprising:

[0006] A recurrent neural network is set as the basic architecture of the super-resolution model, and the recurrent neural network is trained to obtain a multi-frame image super-resolution network.

[0007] The recurrent neural network includes N branches, where the first branch includes the first encoder, the first fusion module, and the first upsampling module, and the remaining N-1 branches each include the i-th encoder, the (i-1)-th alignment module, the i-th fusion module, and the i-th upsampling module; N is a positive integer; i = 2, 3, 4, ..., N;

[0008] Methods for training recurrent neural networks include:

[0009] For the first branch, the low-resolution image of the first frame is used as the base frame and input into the first encoder to obtain the base frame coding features. The base frame coding features are then processed by the first fusion module to obtain the base frame fusion features. The base frame fusion features are then processed by the first upsampling module to obtain the base frame upsampled image. The image obtained by upsampling the base frame is used as the reference upsampled image. The base frame upsampled image and the reference upsampled image are added together to obtain the high-resolution image of the first branch.

[0010] For the second branch, the low-resolution image of the second frame is input into the second encoder to obtain the coded features of the second frame. The coded features of the second frame are aligned with the coded features of the base frame by the first alignment module to obtain the aligned features of the second frame. The aligned features of the second frame, the coded features of the base frame, and the fused features of the base frame are then processed by the second fusion module to obtain the fused features of the second frame. The fused features of the second frame are processed by the second upsampling module to obtain the upsampled image of the second frame. The upsampled image of the second frame is added to the reference upsampled image to obtain the high-resolution image of the second branch.

[0011] For the third branch, the low-resolution image of the third frame is input into the third encoder to obtain the coded features of the third frame. The coded features of the third frame are aligned with the coded features of the base frame by the second alignment module to obtain the aligned features of the third frame. The aligned features of the third frame, the coded features of the base frame, and the fused features of the second frame are then processed by the third fusion module to obtain the fused features of the third frame. The fused features of the third frame are processed by the third upsampling module to obtain the upsampled image of the third frame. The upsampled image of the third frame is added to the reference upsampled image to obtain the high-resolution image of the third branch; ...

[0012] And so on, until a high-resolution image of the Nth branch is obtained;

[0013] Recurrent neural networks, during training, are based on the loss function Modify network parameters:

[0014]

[0015] In the formula For the high-resolution image of the Nth branch, y HR For true high-resolution images, For the i-th branch, the high-resolution image is represented by RS, which indicates random sampling.

[0016] After the above training process reaches the preset termination condition, a multi-frame image super-resolution network is obtained.

[0017] A multi-frame image super-resolution network is used to super-resolution the M low-resolution images to be restored, where M is a positive integer. If M≤N, the restored high-resolution image is directly output on the Mth branch; if M>N, the Nth branch is used as a loop branch until the loop outputs the restored high-resolution image on the Mth branch.

[0018] In the multi-frame image super-resolution method based on recurrent neural networks according to the present invention, all alignment modules have the same structure;

[0019] The alignment module includes an optical flow estimation network, an offset estimation convolutional layer, and a deformable convolutional layer. The optical flow estimation network is used to calculate the optical flow information of the coding features of the i-th frame and the coding features of the base frame, and to perform a distortion operation on the coding features of the i-th frame according to the optical flow information to obtain the coarsely aligned features of the i-th frame.

[0020] Then, the offset estimation convolutional layer is used to calculate the offset between the coarsely aligned features of the i-th frame and the coded features of the base frame. The deformable convolutional layer uses the offset and optical flow information to finely align the coded features of the i-th frame to the coded features of the base frame, thus obtaining the aligned features of the i-th frame.

[0021] In the multi-frame image super-resolution method based on recurrent neural networks according to the present invention, all fusion modules have the same structure;

[0022] The fusion module consists of convolutional layers, activation layers, and F residual blocks, where F is a positive integer;

[0023] After the i-th frame is aligned, the base frame encoded features and the (i-1)-th frame fused features are concatenated along the channel dimension. Then, the number of channels is reduced by sequentially passing through convolutional and activation layers. Finally, the features obtained by sequentially passing through F residual blocks are added to the (i-1)-th frame fused features to obtain the i-th frame fused features.

[0024] According to the multi-frame image super-resolution method based on recurrent neural networks of the present invention, the alignment module aligns the coded features of the i-th frame to the coded features of the base frame in an optical flow-guided manner, obtaining the aligned features of the i-th frame, represented as:

[0025]

[0026] f i =ε(x i ),

[0027] f base =ε(x1),

[0028] In the formula This represents the features after alignment in the i-th frame. Indicates the alignment operation, f i Let x represent the coding features of the i-th frame, ε represent the coding operation, and x represent the coding features of the i-th frame. i f represents the low-resolution image of the i-th frame. base This represents the basic frame coding features, where x1 represents the low-resolution image of the first frame.

[0029] The fused feature h of the i-th frame i for:

[0030]

[0031] In the formula Indicates the fusion operation, h i-1 This represents the fused features of the (i-1)th frame.

[0032] According to the multi-frame image super-resolution method based on recurrent neural networks of the present invention, the recurrent neural network is trained end-to-end using the AdamW optimization algorithm.

[0033] In the multi-frame image super-resolution method based on recurrent neural networks according to the present invention, all encoders have the same structure;

[0034] The encoder sequentially includes a first encoding convolutional layer, a first encoding activation layer, a first encoding residual block, a second encoding residual block, a third encoding residual block, a fourth encoding residual block, and a fifth encoding residual block. After the low-resolution image enters the first encoding convolutional layer, the encoded features are output by the fifth encoding residual block.

[0035] According to the multi-frame image super-resolution method based on recurrent neural networks of the present invention, the offset estimation convolutional layer of the alignment module includes an offset first convolutional layer, an offset first activation layer, an offset second convolutional layer, an offset second activation layer, an offset third convolutional layer, an offset third activation layer, and an offset fourth convolutional layer in sequence. The features after coarse alignment of the i-th frame and the encoded features of the base frame are input into the offset first convolutional layer and the offset fourth convolutional layer outputs the offset amount.

[0036] The fusion module consists of 40 residual blocks.

[0037] In the multi-frame image super-resolution method based on recurrent neural networks according to the present invention, all upsampling modules have the same structure;

[0038] The upsampling module includes an upsampling first convolutional layer, an upsampling first activation layer, an upsampling first residual block, an upsampling second residual block, an upsampling third residual block, an upsampling fourth residual block, an upsampling fifth residual block, a first upsampling layer, a second upsampling layer, a third upsampling layer, an upsampling second convolutional layer, an upsampling second activation layer, and an upsampling third convolutional layer.

[0039] According to the multi-frame image super-resolution method based on recurrent neural networks of the present invention, the activation function of all activation layers is the LeakyReLU function;

[0040] All upsampling operations for all upsampling layers use the PixelShuffle function;

[0041] Each residual block performs a first convolution operation, a first ReLU activation operation, and a second convolution operation on the input features at its input end, and then adds the output features of the residual block to the input features at its input end.

[0042] According to the multi-frame image super-resolution method based on recurrent neural networks of the present invention, the first and second convolution operations of the residual block both include 64 3×3 convolution kernels, with a stride of 1 and padding of 1.

[0043] The encoder's first convolutional layer consists of four 3×3 convolutional kernels with a stride of 1 and padding of 1.

[0044] The offset estimation of the alignment module is as follows: The first convolutional layer consists of 130 3×3 convolutional kernels with a stride of 1 and padding of 1; the second to fourth offset convolutional layers each consist of 64 3×3 convolutional kernels with a stride of 1 and padding of 1.

[0045] The deformable convolutional layer of the alignment module consists of 64 3×3 deformable convolutional kernels grouped into 8s;

[0046] The convolutional layers of the fusion module consist of 192 3×3 convolutional kernels with a stride of 1 and padding of 1.

[0047] The upsampling module's first upsampling convolutional layer consists of 128 3×3 convolutional kernels with a stride of 1 and padding of 1; the second and third upsampling convolutional layers each consist of 64 3×3 convolutional kernels with a stride of 1 and padding of 1.

[0048] The beneficial effects of this invention are as follows: The method of this invention relies on the ability of recurrent neural networks to flexibly handle variable-length input sequences, fully utilizing the complementary information between multiple low-resolution input images to assist in the image super-resolution process. During image super-resolution, due to the significant information loss in low-resolution images, relying on a single image as input often makes it difficult to restore details consistent with the true high-resolution image. However, multiple consecutively captured images contain different sampling information of the same scene, which can assist in the image super-resolution process. This difference can aid in the development of super-resolution models to overcome the practical problem of insufficient detail in super-resolution results relying solely on a single image. This invention's method utilizes the complementarity between multiple misaligned images for image restoration, that is, it combines multiple misaligned images as model input to restore higher-quality high-resolution images. To more flexibly and efficiently model the complementary information between multiple input images and address the practical situation of variable-length frame restoration, this invention selects a recurrent neural network as the basic architecture of the model.

[0049] The method of this invention utilizes the complementarity between multiple misaligned images to develop an image super-resolution model. By combining multiple misaligned images as model input, a higher quality high-resolution image corresponding to the base frame can be restored.

[0050] This invention not only leverages the complementarity between multiple misaligned images to improve image super-resolution performance, but also utilizes the flexibility of recurrent neural networks to design a supervision method that enables the super-resolution model to handle the realities of restoring variable-length frames. Compared to single-image super-resolution methods, this invention can better accomplish the super-resolution task by utilizing different sampling information from the same scene. Compared to other multi-frame image super-resolution methods, this invention further emphasizes the role of the base frame and leverages the flexibility of the recurrent neural network architecture to enable the model to handle variable-length inputs.

[0051] The restoration results obtained by the method of this invention are more realistic and lifelike in terms of the visual effect of real-world images, effectively alleviating the problems of excessive artifacts and unrealistic results from other methods. The multi-frame image super-resolution network obtained by the method of this invention can be directly deployed on imaging devices such as mobile phones to obtain higher quality, higher resolution images. Attached Figure Description

[0052] Figure 1 This is a network architecture diagram of the multi-frame image super-resolution method based on recurrent neural networks described in this invention;

[0053] Figure 2 This is the network structure diagram of the alignment module;

[0054] Figure 3 This is the network structure diagram of the fusion module;

[0055] Figure 4 This is the first set of comparison images showing the restoration of low-resolution images on simulation data using the method of this invention and other existing methods.

[0056] Figure 5 This is the second set of comparison images showing the restoration of low-resolution images on simulation data using the method of this invention and other existing methods.

[0057] Figure 6 This is the first set of comparison images showing the restoration of low-resolution images on real data using the method of this invention and other existing methods.

[0058] Figure 7 This is a second set of comparison images showing the restoration of low-resolution images on real data using the method of this invention and other existing methods. Detailed Implementation

[0059] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0060] It should be noted that, unless otherwise specified, the embodiments and features described in the present invention can be combined with each other.

[0061] The present invention will be further described below with reference to the accompanying drawings and specific embodiments, but this is not intended to limit the scope of the invention.

[0062] Specific Implementation Method 1: Combination Figures 1 to 3As shown, this invention provides a multi-frame image super-resolution method based on a recurrent neural network, including:

[0063] A recurrent neural network is set as the basic architecture of the super-resolution model, and the recurrent neural network is trained to obtain a multi-frame image super-resolution network.

[0064] The recurrent neural network includes N branches, where the first branch includes the first encoder, the first fusion module, and the first upsampling module, and the remaining N-1 branches each include the i-th encoder, the (i-1)-th alignment module, the i-th fusion module, and the i-th upsampling module; N is a positive integer; i = 2, 3, 4, ..., N;

[0065] Methods for training recurrent neural networks include:

[0066] For the first branch, the low-resolution image of the first frame is used as the base frame and input into the first encoder to obtain the base frame coding features. The base frame coding features are then processed by the first fusion module to obtain the base frame fusion features. The base frame fusion features are then processed by the first upsampling module to obtain the base frame upsampled image. The image obtained by upsampling the base frame is used as the reference upsampled image. The base frame upsampled image and the reference upsampled image are added together to obtain the high-resolution image of the first branch.

[0067] For the second branch, the low-resolution image of the second frame is input into the second encoder to obtain the coded features of the second frame. The coded features of the second frame are aligned with the coded features of the base frame by the first alignment module to obtain the aligned features of the second frame. The aligned features of the second frame, the coded features of the base frame, and the fused features of the base frame are then processed by the second fusion module to obtain the fused features of the second frame. The fused features of the second frame are processed by the second upsampling module to obtain the upsampled image of the second frame. The upsampled image of the second frame is added to the reference upsampled image to obtain the high-resolution image of the second branch.

[0068] For the third branch, the low-resolution image of the third frame is input into the third encoder to obtain the coded features of the third frame. The coded features of the third frame are aligned with the coded features of the base frame by the second alignment module to obtain the aligned features of the third frame. The aligned features of the third frame, the coded features of the base frame, and the fused features of the second frame are then processed by the third fusion module to obtain the fused features of the third frame. The fused features of the third frame are processed by the third upsampling module to obtain the upsampled image of the third frame. The upsampled image of the third frame is added to the reference upsampled image to obtain the high-resolution image of the third branch; ...

[0069] And so on, until a high-resolution image of the Nth branch is obtained;

[0070] In real-world scenarios, to ensure model flexibility, the model must not only achieve satisfactory restoration results with a fixed number of input frames, but also flexibly handle variable-length frames. Leveraging the flexibility of recurrent networks in processing variable-length data, this implementation method designs a corresponding variable-length input training method.

[0071] During training, the model takes N frames of low-resolution images as input and outputs the super-resolution result at frame number i. With high-resolution images y HR To further constrain this, considering that users prefer using more frames when using multi-frame super-resolution systems, a loss function that favors a higher number of frames is used. During training, the recurrent neural network adjusts the loss function accordingly. Modify network parameters:

[0072]

[0073] In the formula For the high-resolution image of the Nth branch, y HR For true high-resolution images, For the i-th branch, the high-resolution image is represented by RS, which indicates random sampling.

[0074] After the above training process reaches the preset termination condition, a multi-frame image super-resolution network is obtained.

[0075] A multi-frame image super-resolution network is used to super-resolution the M low-resolution images to be restored, where M is a positive integer. If M≤N, the restored high-resolution image is directly output on the Mth branch; if M>N, the Nth branch is used as a loop branch until the loop outputs the restored high-resolution image on the Mth branch.

[0076] In order to model the complementary information between multiple input images more flexibly and efficiently and to deal with the actual situation of restoration of variable-length frames, this implementation chooses a recurrent neural network as the basic architecture of the model.

[0077] The multi-frame image super-resolution network described in this embodiment can not only achieve satisfactory restoration results with a fixed number of input frames, but also flexibly handle cases with variable-length frames. In specific implementations, all encoder functions can be implemented by the same encoder, all alignment module functions can be implemented by the same alignment module, all fusion module functions can be implemented by the same fusion module, and all upsampling module functions can be implemented by the same upsampling module. In this case, each module saves the current processing result after completing the current image processing function for use in subsequent image processing.

[0078] Therefore, for the case where M > N, the Nth branch of the recurrent neural network can be used repeatedly to achieve a function similar to branches N+1, N+2, etc.

[0079] Furthermore, combined with Figure 2 As shown, all alignment modules have the same structure;

[0080] The alignment module is used to handle misalignment between multiple input frames. In actual shooting scenarios, there is varying degrees of spatial misalignment between multiple frames. Convolutional neural networks cannot directly utilize the complementary information between spatially misaligned images; therefore, an alignment module is needed to address the misalignment problem. This implementation addresses the spatial misalignment problem at the feature level rather than the image level. For the input frame that needs to be super-resolution, i.e., the base frame, after a series of processing steps, the high-resolution image output by the Nth branch serves as the restored image of the base frame.

[0081] The alignment module includes an optical flow estimation network, an offset estimation convolutional layer, and a deformable convolutional layer. The optical flow estimation network is used to calculate the optical flow information of the coding features of the i-th frame and the coding features of the base frame, and to perform a distortion operation on the coding features of the i-th frame according to the optical flow information to obtain the coarsely aligned features of the i-th frame.

[0082] Then, the offset estimation convolutional layer is used to calculate the offset between the coarsely aligned features of the i-th frame and the coded features of the base frame. The deformable convolutional layer uses the offset and optical flow information to finely align the coded features of the i-th frame to the coded features of the base frame, thus obtaining the aligned features of the i-th frame.

[0083] The optical flow estimation network uses a pre-trained SpyNet lightweight optical flow estimation network, and the model parameters are fine-tuned during training.

[0084] Furthermore, combining Figure 3 As shown, all fusion modules have the same structure;

[0085] This implementation fully models the complementary information between frames to complete the fusion of inter-frame information; the fusion module gradually aggregates the feature information of multiple frames and adopts a residual structure during fusion.

[0086] The fusion module consists of convolutional layers, activation layers, and F residual blocks, where F is a positive integer;

[0087] After the i-th frame is aligned, the base frame encoded features and the (i-1)-th frame fused features are concatenated along the channel dimension. Then, they are sequentially processed through convolutional and activation layers to reduce the number of channels and thus the number of parameters. Finally, the features obtained by passing through F residual blocks are added to the (i-1)-th frame fused features to obtain the i-th frame fused features.

[0088] Furthermore, the alignment module aligns the coded features of the i-th frame to the coded features of the base frame using optical flow guidance, resulting in the aligned features of the i-th frame, represented as:

[0089]

[0090] f i =ε(x i ),

[0091] f base =ε(x1),

[0092] In the formula This represents the features after alignment in the i-th frame. Indicates the alignment operation, f i Let x represent the coding features of the i-th frame, ε represent the coding operation, and x represent the coding features of the i-th frame. i f represents the low-resolution image of the i-th frame. base This represents the basic frame coding features, where x1 represents the low-resolution image of the first frame.

[0093] The fused feature h of the i-th frame i for:

[0094]

[0095] In the formula Indicates the fusion operation, h i-1 This represents the fused features of the (i-1)th frame.

[0096] In this embodiment, the i-th frame fusion feature h i By fully utilizing the reference information contained in the base frame when gradually fusing information from multiple frames, and providing the base frame information for each frame as a reference during fusion, the reliability of the restoration result will be higher.

[0097] In practice, the fusion features can be determined according to actual needs, such as the fusion feature h of the i-th frame. i The method of obtaining it can also be:

[0098]

[0099] At this point, the fusion module does not utilize the basic frame coding features when performing channel-level connections.

[0100] In this embodiment, the recurrent neural network is trained end-to-end using the AdamW optimization algorithm.

[0101] As an example, all encoders have the same structure;

[0102] The encoder sequentially includes a first convolutional layer, a first activation layer, a first residual block, a second residual block, a third residual block, a fourth residual block, and a fifth residual block. After the low-resolution image enters the first convolutional layer, the encoded features are output by the fifth residual block. The residual block configuration in the encoder effectively mitigates the mode collapse problem during training.

[0103] As an example, the offset estimation convolutional layer of the alignment module includes an offset first convolutional layer, an offset first activation layer, an offset second convolutional layer, an offset second activation layer, an offset third convolutional layer, an offset third activation layer, and an offset fourth convolutional layer in sequence. The coarse alignment features of the i-th frame and the encoded features of the base frame are input into the offset first convolutional layer and the offset fourth convolutional layer outputs the offset amount.

[0104] The fusion module consists of 40 residual blocks. The stacking of residual blocks in the fusion module can further enhance the fused features, which is beneficial for recovering richer image details.

[0105] As an example, all upsampling modules have the same structure;

[0106] The upsampling module includes an upsampling first convolutional layer, an upsampling first activation layer, an upsampling first residual block, an upsampling second residual block, an upsampling third residual block, an upsampling fourth residual block, an upsampling fifth residual block, a first upsampling layer, a second upsampling layer, a third upsampling layer, an upsampling second convolutional layer, an upsampling second activation layer, and an upsampling third convolutional layer. The upsampling module does not directly learn from high-resolution images; instead, it learns the residual between high-resolution and low-resolution images, reducing the difficulty of model learning.

[0107] The upsampling module in this embodiment can further improve the resolution of the restored image. The output of the upsampling module can be expressed as:

[0108]

[0109] In the formula This represents the upsampled image of the Nth frame. Indicates the fusion feature h of the Nth frame N Perform an upsampling operation.

[0110] In this embodiment, the preferred encoder includes a combination of 5 residual blocks, the fusion module includes 40 residual blocks, and the upsampling module includes 5 residual blocks.

[0111] As an example, the activation function for all activation layers is the LeakyReLU function;

[0112] All upsampling operations for all upsampling layers use the PixelShuffle function;

[0113] Each residual block performs a first convolution operation, a first ReLU activation operation, and a second convolution operation on the input features at its input end, and then adds the output features of the residual block to the input features at its input end.

[0114] In this embodiment, the activation operation in the residual block is the ReLU function, and the other activation functions are LeakyReLU functions. Compared with ReLU, the LeakyReLU activation function retains some negative axis values, which can alleviate the problem of neuron parameters not being able to be updated due to excessively small activation values.

[0115] As an example, the first and second convolution operations of the residual block both consist of 64 3×3 convolution kernels, with a stride of 1 and padding of 1.

[0116] The encoder's first convolutional layer consists of four 3×3 convolutional kernels with a stride of 1 and padding of 1.

[0117] The offset estimation of the alignment module is as follows: The first convolutional layer consists of 130 3×3 convolutional kernels with a stride of 1 and padding of 1; the second to fourth offset convolutional layers each consist of 64 3×3 convolutional kernels with a stride of 1 and padding of 1.

[0118] The deformable convolutional layer of the alignment module consists of 64 3×3 deformable convolutional kernels grouped into 8s;

[0119] The convolutional layers of the fusion module consist of 192 3×3 convolutional kernels with a stride of 1 and padding of 1.

[0120] The upsampling module's first upsampling convolutional layer consists of 128 3×3 convolutional kernels with a stride of 1 and padding of 1; the second and third upsampling convolutional layers each consist of 64 3×3 convolutional kernels with a stride of 1 and padding of 1.

[0121] Figure 4 and Figure 5 The comparison involves examining the visual quality of high-resolution images obtained using existing models and the method of this invention on simulation data. Figure 6 and Figure 7 It compares the visual quality of high-resolution images obtained using existing models and the method of this invention on real data; where low resolution represents the input low-resolution image, DBSR, MFIR and BIPnet are the current mainstream advanced multi-frame super-resolution methods, BIPnet is the current state-of-the-art multi-frame super-resolution method, and high resolution corresponds to the paired high-resolution image, which can approximately represent the upper limit of the super-resolution algorithm.

[0122] Depend on Figures 4 to 7This demonstrates that the method of the present invention can restore high-resolution images with richer details. Compared with current mainstream multi-frame image super-resolution methods, the method of the present invention achieves good restoration performance, significantly improves visual effects, and further enhances the flexibility of the model.

[0123] Current mainstream methods often suffer from problems such as loss of high-frequency details, detail distortion, and speckled highlights in the restored image. The method of this invention effectively alleviates these problems in both simulation and real-world data, achieving higher-quality image restoration. Furthermore, mainstream methods such as BIPnet are only applicable to a fixed number of input frames, while this invention can adapt to a variable number of input frames, making it more flexible and practical.

[0124] While the invention has been described herein with reference to specific embodiments, it should be understood that these embodiments are merely examples of the principles and applications of the invention. Therefore, it should be understood that many modifications can be made to the exemplary embodiments, and other arrangements can be designed without departing from the spirit and scope of the invention as defined by the appended claims. It should be understood that different dependent claims and features described herein can be combined in ways different from those described in the original claims. It is also understood that features described in conjunction with individual embodiments can be used in other described embodiments.

Claims

1. A multi-frame image super-resolution method based on recurrent neural networks, characterized in that... include, A recurrent neural network is set as the basic architecture of the super-resolution model, and the recurrent neural network is trained to obtain a multi-frame image super-resolution network. The recurrent neural network includes N branches, where the first branch includes the first encoder, the first fusion module, and the first upsampling module, and the remaining N-1 branches each include the i-th encoder, the (i-1)-th alignment module, the i-th fusion module, and the i-th upsampling module; N is a positive integer; i = 2, 3, 4, ..., N; Methods for training recurrent neural networks include: For the first branch, the low-resolution image of the first frame is used as the base frame and input into the first encoder to obtain the base frame coding features. The base frame coding features are then processed by the first fusion module to obtain the base frame fusion features. The base frame fusion features are then processed by the first upsampling module to obtain the base frame upsampled image. The image obtained by upsampling the base frame is used as the reference upsampled image. The base frame upsampled image and the reference upsampled image are added together to obtain the high-resolution image of the first branch. For the second branch, the low-resolution image of the second frame is input into the second encoder to obtain the coded features of the second frame. The coded features of the second frame are aligned with the coded features of the base frame by the first alignment module to obtain the aligned features of the second frame. The aligned features of the second frame, the coded features of the base frame, and the fused features of the base frame are then processed by the second fusion module to obtain the fused features of the second frame. The fused features of the second frame are processed by the second upsampling module to obtain the upsampled image of the second frame. The upsampled image of the second frame is added to the reference upsampled image to obtain the high-resolution image of the second branch. For the third branch, the low-resolution image of the third frame is input into the third encoder to obtain the coded features of the third frame. The coded features of the third frame are aligned with the coded features of the base frame by the second alignment module to obtain the aligned features of the third frame. The aligned features of the third frame, the coded features of the base frame, and the fused features of the second frame are then processed by the third fusion module to obtain the fused features of the third frame. The fused features of the third frame are processed by the third upsampling module to obtain the upsampled image of the third frame. The upsampled image of the third frame is added to the reference upsampled image to obtain the high-resolution image of the third branch; ... And so on, until a high-resolution image of the Nth branch is obtained; Recurrent neural networks, during training, are based on the loss function Modify network parameters: In the formula For the high-resolution image of the Nth branch, y HR For true high-resolution images, For the i-th branch, the high-resolution image is represented by RS, which indicates random sampling. After the above training process reaches the preset termination condition, a multi-frame image super-resolution network is obtained. A multi-frame image super-resolution network is used to super-resolution the M low-resolution images to be restored, where M is a positive integer. If M≤N, the restored high-resolution image is directly output on the Mth branch; if M>N, the Nth branch is used as a loop branch until the loop outputs the restored high-resolution image on the Mth branch.

2. The multi-frame image super-resolution method based on recurrent neural networks according to claim 1, characterized in that, All alignment modules have the same structure; The alignment module includes an optical flow estimation network, an offset estimation convolutional layer, and a deformable convolutional layer. The optical flow estimation network is used to calculate the optical flow information of the coding features of the i-th frame and the coding features of the base frame, and to perform a distortion operation on the coding features of the i-th frame according to the optical flow information to obtain the coarsely aligned features of the i-th frame. Then, the offset estimation convolutional layer is used to calculate the offset between the coarsely aligned features of the i-th frame and the coded features of the base frame. The deformable convolutional layer uses the offset and optical flow information to finely align the coded features of the i-th frame to the coded features of the base frame, thus obtaining the aligned features of the i-th frame.

3. The multi-frame image super-resolution method based on recurrent neural networks according to claim 2, characterized in that, All fusion modules have the same structure; The fusion module consists of convolutional layers, activation layers, and F residual blocks, where F is a positive integer; After the i-th frame is aligned, the base frame encoded features and the (i-1)-th frame fused features are concatenated along the channel dimension. Then, the number of channels is reduced by sequentially passing through convolutional and activation layers. Finally, the features obtained by sequentially passing through F residual blocks are added to the (i-1)-th frame fused features to obtain the i-th frame fused features.

4. The multi-frame image super-resolution method based on recurrent neural networks according to claim 3, characterized in that, The alignment module aligns the coded features of the i-th frame to the coded features of the base frame using optical flow guidance, resulting in the aligned features of the i-th frame, represented as follows: f i = ε(x i ), f base =ε(x1), In the formula This represents the features after alignment in the i-th frame. Indicates the alignment operation, f i Let x represent the coding features of the i-th frame, ε represent the coding operation, and x represent the coding features of the i-th frame. i f represents the low-resolution image of the i-th frame. base This represents the basic frame coding features, where x1 represents the low-resolution image of the first frame. The fused feature h of the i-th frame i for: In the formula Indicates the fusion operation, h i-1 This represents the fused features of the (i-1)th frame.

5. The multi-frame image super-resolution method based on recurrent neural networks according to claim 4, characterized in that, The recurrent neural network is trained end-to-end using the AdamW optimization algorithm.

6. The multi-frame image super-resolution method based on recurrent neural networks according to claim 5, characterized in that, All encoders have the same structure; The encoder sequentially includes a first encoding convolutional layer, a first encoding activation layer, a first encoding residual block, a second encoding residual block, a third encoding residual block, a fourth encoding residual block, and a fifth encoding residual block. After the low-resolution image enters the first encoding convolutional layer, the encoded features are output by the fifth encoding residual block.

7. The multi-frame image super-resolution method based on recurrent neural networks according to claim 6, characterized in that, The offset estimation convolutional layers of the alignment module include offset first convolutional layer, offset first activation layer, offset second convolutional layer, offset second activation layer, offset third convolutional layer, offset third activation layer and offset fourth convolutional layer in sequence. The features after coarse alignment of the i-th frame and the encoded features of the base frame are input into the offset first convolutional layer and the offset fourth convolutional layer outputs the offset amount. The fusion module consists of 40 residual blocks.

8. The multi-frame image super-resolution method based on recurrent neural networks according to claim 7, characterized in that, All upsampling modules have the same structure; The upsampling module includes an upsampling first convolutional layer, an upsampling first activation layer, an upsampling first residual block, an upsampling second residual block, an upsampling third residual block, an upsampling fourth residual block, an upsampling fifth residual block, a first upsampling layer, a second upsampling layer, a third upsampling layer, an upsampling second convolutional layer, an upsampling second activation layer, and an upsampling third convolutional layer.

9. The multi-frame image super-resolution method based on recurrent neural networks according to claim 8, characterized in that, The activation function for all activation layers is the LeakyReLU function; All upsampling operations for all upsampling layers use the PixelShuffle function; Each residual block performs a first convolution operation, a first ReLU activation operation, and a second convolution operation on the input features at its input end, and then adds the output features of the residual block to the input features at its input end.

10. The multi-frame image super-resolution method based on recurrent neural networks according to claim 9, characterized in that, The first and second convolution operations of the residual block both consist of 64 3×3 convolution kernels, with a stride of 1 and padding of 1. The encoder's first convolutional layer consists of four 3×3 convolutional kernels with a stride of 1 and padding of 1. The offset estimation of the alignment module is as follows: The first convolutional layer consists of 130 3×3 convolutional kernels with a stride of 1 and padding of 1; the second to fourth offset convolutional layers each consist of 64 3×3 convolutional kernels with a stride of 1 and padding of 1. The deformable convolutional layer of the alignment module consists of 64 3×3 deformable convolutional kernels grouped into 8s; The convolutional layers of the fusion module consist of 192 3×3 convolutional kernels with a stride of 1 and padding of 1. The upsampling module's first upsampling convolutional layer consists of 128 3×3 convolutional kernels with a stride of 1 and padding of 1; the second and third upsampling convolutional layers each consist of 64 3×3 convolutional kernels with a stride of 1 and padding of 1.

Images