Structure-detail separation based bi-directional recurrent neural network super-resolution method

Abstract

The application discloses a kind of based on structure-detail separation bidirectional recurrent neural network super-resolution method, belong to video super-resolution technical field.The application mainly includes the following steps:1, the spatial feature, structure feature and detail feature of each frame picture of low-resolution video are extracted;2, the forward and backward optical flow of each frame picture is calculated;3, the spatial feature, structure / detail feature and forward optical flow of each frame picture are input into network, and the forward structure / detail feature of current frame is obtained;4, the spatial feature, structure / detail feature, backward optical flow and forward structure / detail feature of each frame picture are input into network, and the backward structure / detail feature of current frame is obtained;5, the spatial feature, structure / detail feature and forward and backward structure / detail feature of each frame picture are calculated to obtain reconstruction structure / detail feature, and high-resolution result is obtained by calculating the result with spatial feature.The video super-resolution method based on the application can obtain high-quality output with more details.

Description

Technical Field

[0001] This invention belongs to the field of video super-resolution technology, and specifically designs a bidirectional recurrent neural network super-resolution method based on structure-detail separation. Background Technology

[0002] Video is a common form of multimedia, consisting of a sequence of continuous images. Super-resolution is defined as reconstructing a high-resolution output from a low-resolution input. This is a classic and challenging problem in computer vision and image processing, and it has numerous practical applications, such as medical image reconstruction, satellite image remote sensing, digital high-definition imaging, and microscopic imaging.

[0003] Super-resolution objects can be broadly categorized into image and video: image super-resolution reconstructs a high-resolution image sequence from a single low-resolution image input; video super-resolution reconstructs a high-resolution image from multiple consecutive frames input and utilizes inter-frame relationships. Traditional video super-resolution methods often rely on affine transformations or probabilistic statistical methods for estimation, but these methods struggle to effectively estimate complex motion processes within videos. With the significant success of deep learning across various fields, deep learning-based video super-resolution algorithms have been extensively studied. Currently, deep learning-based super-resolution algorithms mainly include three structures: Convolutional Neural Networks (CNNs), which utilize sliding windows to extract short-range features between adjacent frames and participate in computation; Recurrent Neural Networks (RNNs), which use hidden states to store long-range features from historical or future frames and participate in computation, with features continuously updated during the computation process; and Transformer-based networks, which utilize attention mechanisms to extract long-range or short-range features and participate in computation.

[0004] However, existing video super-resolution methods often input the entire image into the network for computation, ignoring the fact that structural and detail information in each frame has different processing difficulties and different impacts on visual effects. Structural (low-frequency) information constitutes the majority of an image, with minimal changes between adjacent frames and insignificant impact on visual effects; while detail (high-frequency) information constitutes a smaller proportion of an image, with significant changes between adjacent frames and a substantial impact on visual effects. If the entire image is used as training data for the network, the trained network will have a strong ability to reconstruct the structural information that constitutes a large proportion of the image, but a weak ability to reconstruct the detail information that constitutes a small proportion but has a significant impact on visual effects. Therefore, the super-resolved image suffers from poor visual effects due to a lack of detail information or low-quality detail information. Summary of the Invention

[0005] To address the aforementioned technical problems, this invention discloses a bidirectional recurrent neural network super-resolution method based on structure-detail separation. This method separates the structural features and detail features of an image and inputs them into the network. The bidirectional recurrent network is then used to reconstruct the structural and detail information respectively, resulting in super-resolution images with richer detail information and superior visual effects.

[0006] The technical solution adopted by this invention to solve its technical problem is as follows:

[0007] A super-resolution method based on bidirectional recurrent neural networks with structure-detail separation includes the following steps:

[0008] Step (1). Extract low-resolution video data and perform image enhancement;

[0009] Step (2). Extract the spatial features of each frame in the low-resolution video;

[0010] Step (3). Extract the structural information, detail information, spatial structural features, and spatial detail features of each frame in the low-resolution video;

[0011] Step (4). Calculate the forward and backward optical flow of each frame in the low-resolution video;

[0012] Step (5). Input the spatial features, structural information, spatial structural features and forward optical flow of each frame of the image obtained in steps (2)-(4) into the feedforward recurrent neural network to obtain the forward structural features of each frame of the image; input the spatial features, detail information, spatial detail features and forward optical flow of each frame of the image obtained in steps (2)-(4) into the feedforward recurrent neural network to obtain the forward detail features of each frame of the image.

[0013] Step (6). Input the spatial features, structural information, spatial structural features, backward optical flow and forward structural features of each frame of the image obtained in steps (2)-(5) into the feedback recurrent neural network to obtain the backward structural features of each frame of the image; input the spatial features, detail information, spatial detail features, backward optical flow and forward detail features of each frame of the image obtained in steps (2)-(5) into the feedback recurrent neural network to obtain the backward detail features of each frame of the image.

[0014] Step (7). Perform feature fusion and reconstruction on the spatial features, structural information, and forward and backward structural features of each frame obtained in steps (2)-(6) to obtain the reconstructed structural features of each frame; perform feature fusion and reconstruction on the spatial features, detail information, and forward and backward detail features of each frame obtained in steps (2)-(6) to obtain the reconstructed detail features of each frame. Calculate the two to obtain the high-resolution result after super-resolution.

[0015] In steps (1)-(3), the network first performs image enhancement on the input low-resolution video sequence using methods such as rotation and symmetry, and then extracts the spatial features of each frame of the video sequence. Since these features are shallow features, they can preserve the information of the original image relatively completely. These shallow features serve as guiding information in the subsequent calculation of reconstructed features, helping the network to converge quickly and achieve good performance. Then, these shallow features are subjected to structure-detail separation, and the separated structure features and detail features are input into a bidirectional recurrent network for feature reconstruction.

[0016] In step (4), the low-resolution video sequence is input into the optical flow calculation network to calculate the forward and backward optical flow of each frame. Since the network structure used in this paper is a bidirectional recurrent network structure, two alignments are required: aligning the historical frame features and the future frame features to the current frame. The advantage of using a bidirectional recurrent neural network is that any image input to the network at any time can utilize the features of the entire sequence to achieve super-resolution. For example, if the current input to the network is frame t (t∈[1,T]), the historical frame features available in the feedforward recurrent neural network come from frame 1-t-1, and the future frame features available in the feedback recurrent neural network come from frame t+1-T.

[0017] In steps (5)-(6), the spatially separated features and optical flow of the current frame obtained earlier are input into the feedforward neural network. The forward optical flow guides the forward hidden state features of the historical frames to align with the current frame. Then, the aligned features are subjected to structure-detail separation and fused with the structure and detail features of the current frame to reconstruct the features, thereby obtaining the forward structure / detail features of the current frame, and thus the forward hidden state features of the current frame. When calculating the backward structure / detail features of the current frame, the backward optical flow is used to guide the backward hidden state features of future frames to align with the current frame and complete the structure-detail separation. At the same time, the forward structure / detail features are added when performing feature fusion.

[0018] In step (7), the detail-related features and structure-related features of the current frame obtained earlier are fused, reconstructed, and upsampled to obtain the super-resolution high-resolution result.

[0019] The beneficial effects of this invention are:

[0020] This invention designs a complete bidirectional recurrent neural network (RNN) super-resolution architecture based on image structure-detail separation. The architecture includes a data preprocessing module, a spatial feature extraction module, a structure-detail separation module, an optical flow calculation module, a feature processing module, and a super-resolution reconstruction module. It is a multi-layered network architecture that leverages the advantages of RNNs in storing long-range features and utilizing the entire sequence information of any input, preventing low super-resolution quality in the first few frames of a video. Furthermore, by calculating the structural and detail features of the image separately and then fusing them, the detailed information of the image can be effectively reconstructed, resulting in a better visual effect in the super-resolution image. Attached Figure Description

[0021] Figure 1 This is a block diagram of the bidirectional recurrent neural network super-resolution method based on structure-detail separation used in the embodiments of the present invention;

[0022] Figure 2 This is an overall flowchart of an embodiment of the present invention. Detailed Implementation

[0023] The method of the present invention will be further described below with reference to the accompanying drawings.

[0024] The structural block diagram of the bidirectional recurrent neural network super-resolution method based on structure-detail separation of this invention is as follows: Figure 1 As shown, this method incorporates six modules: a data preprocessing module, a spatial feature extraction module, a structure-detail separation module, an optical flow calculation module, a feature processing module, and a super-resolution reconstruction module.

[0025] The data preprocessing module is used to process the input raw video data stream and execute the method in step (1) below.

[0026] Step (1). Obtain the low-resolution video sequence and perform mirror symmetry, horizontal 90° flip, and vertical 90° flip on each frame sequentially to achieve image enhancement. Denote the image-enhanced low-resolution video sequence as... Where T represents the number of video frames. This represents the image at frame t; subsequently, each frame is input into the spatial feature extraction module.

[0027] The spatial feature extraction module is used to extract the spatial features of each frame of the low-resolution video by performing the method in step (2) below.

[0028] Step (2). Take the t-th frame (t∈[1,T]) of the low-resolution video obtained in step (1). The input frame's channel dimension is increased from 3 to 64 using convolutional layers, followed by spatial feature extraction.

[0029] The spatial feature extraction process is expressed as follows:

[0030]

[0031] in, Represents the image of frame t. The spatial features are represented by R(.), which denotes the stacked residual block structure used for feature extraction, Relu(.) represents the activation function, and conv(.) represents the convolutional layer. The spatial resolution remains constant during the spatial feature computation.

[0032] A core design feature of this invention is to separate the structural and detail parts of each frame in a low-resolution video and perform feature reconstruction on them separately. The structure-detail separation module is used to separate the two and perform the method in step (3) below.

[0033] Step (3). Take the t-th frame of the low-resolution video obtained in step (1). and the t-th frame image obtained in step (2) Spatial features Perform structure-detail separation, specifically:

[0034] Step (3.1). Take the t-th frame of the low-resolution video obtained in step (1). Interpolation is performed; in this embodiment, the upsampling rate is 0.5.

[0035]

[0036] in, Represents the image of frame t. The upsampled features have a spatial resolution of the t-th frame image. 0.5 times, Bilinear(.,.) represents the bilinear interpolation method.

[0037] Step (3.2). Take the t-th frame image obtained in step (3.1). Upsampled features Interpolation is performed with an upsampling rate of 2:

[0038]

[0039] in, Represents the image of frame t. Structural information, resolution and the image of frame t Same; Bilinear(.,.) represents the bilinear interpolation method.

[0040] Step (3.3). Take the t-th frame image obtained in step (3.2). Structural information With the t-th frame image Making a difference:

[0041]

[0042] in, Represents the image of frame t. Detailed information.

[0043] Step (3.4). Take the t-th frame of the low-resolution video obtained in step (2). Spatial features Repeat steps (3.1)-(3.3) to obtain the t-th frame image. Spatial structural features and spatial details

[0044] When using features from historical or future frames in video super-resolution, these features need to be aligned to the current frame first. Before the alignment process, the corresponding optical flow needs to be calculated to guide the feature alignment. The optical flow calculation module is used to calculate the forward and backward optical flow of the entire video sequence and executes the method in step (4) below.

[0045] Step (4). Enhance the low-resolution video sequence obtained in step (1). The forward and backward optical flow calculations are performed as follows:

[0046] Step (4.1). Extract the t-th frame image from the low-resolution video. And the image of frame t-1 Input network to calculate forward optical flow:

[0047]

[0048] in, This represents the image of frame t-1. To the t-th frame image Aligned forward optical flow, SpyNet(.,.) represents the optical flow computation network.

[0049] Step (4.2). Extract the t-th frame image from the low-resolution video. And the image of frame t+1 Input network to calculate backward optical flow:

[0050]

[0051] in, This represents the image at frame t+1. To the t-th frame image Aligned backward optical flow, SpyNet(.,.) represents the optical flow computation network.

[0052] After the structural and detail features of each frame of the low-resolution video are separated, the feature processing module is used to perform feature alignment, feature fusion and feature extraction to obtain the forward / backward structural features and forward / backward detail features of the current frame, and the methods in steps (5)-(6) below are executed.

[0053] Step (5). Based on the spatial features of the low-resolution video sequence obtained in step (2). The structural information of the low-resolution video sequence obtained in step (3) Detailed information Spatial structural features Spatial details and the forward optical flow of the low-resolution video sequence obtained in step (4) Calculate the forward structural features and forward detail features of the current frame, specifically as follows:

[0054] Step (5.1). Use the t-th frame image of the low-resolution video obtained in step (4.1). Forward optical flow Guide the image of frame t-1 Forward hidden state features To the t-th frame image Alignment:

[0055]

[0056] in, This represents the image of frame t-1. Forward hidden state features To the t-th frame image The aligned features are represented by Warp(.,.).

[0057] Step (5.2). Use the t-th frame of the low-resolution video obtained in step (4.1). Forward optical flow Guide the image of frame t-1 Forward optical flow Alignment:

[0058]

[0059] in, Indicates the image used to guide the (t-2)th frame. To the t-th frame image Aligned forward optical flow, Warp(.,.) represents the alignment operation.

[0060] Then use the obtained forward optical flow Guide the image of frame t-2 Forward hidden state features To the t-th frame image Alignment:

[0061]

[0062] in, This represents the image of frame t-2. Forward hidden state features To the t-th frame image The aligned features are represented by Warp(.,.).

[0063] Step (5.3). Take the (t-1)th frame image obtained in steps (5.1)-(5.2). Features after forward hidden state alignment and the image of frame t-2 Features after forward hidden state alignment Repeat step (3) to obtain the (t-1)th frame image. Post-alignment structural features of the forward hidden state and detailed features and the image of frame t-2 Post-alignment structural features of the forward hidden state and detailed features

[0064] Step (5.4). Take the historical frame images obtained in step (5.3). Post-alignment structural features of the forward hidden state and Feature extraction was performed to obtain the structural features of historical frames:

[0065]

[0066] in, Represents the image of frame t. The historical frame structure features are defined by Relu(.), which represents the activation function, R(.), which represents the stacked residual block structure used to extract features, and c(.,.), which represents concatenating the features along the channel dimension.

[0067] Step (5.5). Take the t-th frame of the low-resolution video obtained in step (3.2). Structural information The t-th frame image obtained in step (3.4) Spatial structural features and the t-th frame image obtained in step (5.4) Historical frame structure features Feature extraction is performed to obtain the forward structure features:

[0068]

[0069] in, Represents the image of frame t. The forward structural features are defined by R(.), which represents the stacked residual block structure used to extract features, and c(.,.,.) represents the concatenation of features along the channel dimension.

[0070] Step (5.6). Take the historical frame images obtained in step (5.3). Post-aligned details of the forward hidden state and Repeat step (5.4) to obtain the t-th frame image. Historical frame details

[0071] Step (5.7). Take the t-th frame of the low-resolution video obtained in step (3.3). Detailed information The t-th frame image obtained in step (3.4) Spatial details and the t-th frame image obtained in step (5.6) Historical frame details Repeat step (5.5) to obtain the t-th frame image. Forward detail features

[0072] Step (5.8). Take the t-th frame of the low-resolution video obtained in step (2). Spatial features The t-th frame image obtained in step (5.5) Forward structural features and the t-th frame image obtained in step (5.7) Forward detail features Feature extraction is performed to obtain the forward hidden state features:

[0073]

[0074] in, Represents the image of frame t. The forward hidden state features are represented by Relu(.), which represents the activation function, and SD(.,.) represents the stacked structure-detail residual block structure.

[0075] Step (6). Based on the spatial features of the low-resolution video sequence obtained in step (2). The structural information of the low-resolution video sequence obtained in step (3) Detailed information Spatial structural features Spatial details Backward optical flow of the low-resolution video sequence obtained in step (4) The forward structure features of the low-resolution video sequence obtained in step (5) and forward detail features Calculate the backward structural features and backward detail features of the current frame, specifically as follows:

[0076] Step (6.1). Use the t-th frame of the low-resolution video obtained in step (4.2). Backflow of light Guide the image of frame t+1 Backward hidden state features To the t-th frame image Alignment:

[0077]

[0078] in, This represents the image at frame t+1. Backward hidden state features To the t-th frame image The aligned features are represented by Warp(.,.).

[0079] Step (6.2). Use the t-th frame of the low-resolution video obtained in step (4.2). Backflow of light Guide the image of frame t+1 Backflow of light Alignment:

[0080]

[0081] in, This indicates the image used to guide the (t+2)th frame. To the t-th frame image Aligned backward optical flow, Warp(.,.) represents the alignment operation.

[0082] Then use the obtained backward optical flow The (t+2)th frame image Backward hidden state features To the t-th frame image Alignment:

[0083]

[0084] in, This represents the image at frame t+2. Backward hidden state features To the t-th frame image The aligned features are represented by Warp(.,.).

[0085] Step (6.3). Take the (t+1)th frame image obtained in steps (6.1)-(6.2). Features after back-hidden state alignment and the image of frame t+2 Features after back-hidden state alignment Repeat step (3) to obtain the (t+1)th frame image. Backward hidden state aligned structural features and detailed features and the image of frame t+2 Backward hidden state aligned structural features and detailed features

[0086] Step (6.4). Take the future frame image obtained in step (6.3). Backward hidden state aligned structural features and Feature extraction is performed to obtain the structural features of future frames:

[0087]

[0088] in, Represents the image of frame t. The future frame structure features are defined by Relu(.), which represents the activation function, R(.), which represents the stacked residual block structure used to extract features, and c(.,.), which represents concatenating the features along the channel dimension.

[0089] Step (6.5). Take the t-th frame of the low-resolution video obtained in step (3.2). Structural information The t-th frame image obtained in step (3.4) Spatial structural features The t-th frame image obtained in step (5.5) Forward structural features and the t-th frame image obtained in step (6.4) Future frame structure features Feature extraction is performed to obtain backward structural features:

[0090]

[0091] in, Represents the image of frame t. The backward structural features are represented by R(.), which represents the stacked residual block structure used to extract features, and c(.,.,.,.) which represents concatenating the features along the channel dimension.

[0092] Step (6.6). Take the future frame image obtained in step (6.3). Backward hidden state aligned detail features and Repeat step (6.4) to obtain the t-th frame image. Future frame detail features

[0093] Step (6.7). Take the t-th frame of the low-resolution video obtained in step (3.3). Detailed information The t-th frame image obtained in step (3.4) Spatial details The t-th frame image obtained in step (5.7) Forward detail features and the t-th frame image obtained in step (6.6) Future frame detail features Repeat step (6.5) to obtain the t-th frame image. Backward detail features

[0094] Step (6.8). Take the t-th frame of the low-resolution video obtained in step (2). Spatial features The t-th frame image obtained in step (6.5) Backward structural features and the t-th frame image obtained in step (6.7) Backward detail features Feature extraction is performed to obtain the backward hidden state features:

[0095]

[0096] in, Represents the image of frame t. The backward hidden state features are represented by Relu(.), which represents the activation function, and SD(.,.) represents the stacked structure-detail residual block structure.

[0097] The super-resolution reconstruction module is used to fuse and reconstruct the features obtained above. By rearranging the features, the channel dimension information of the features is extended to the spatial dimension, thereby completing the upsampling operation of the low-resolution features. Perform the method in step (7) below.

[0098] Step (7). Spatial features of the low-resolution video sequence obtained in step (2). The structural information of the low-resolution video sequence obtained in step (3) Detailed information Spatial structural features Spatial details The forward structure features of the low-resolution video sequence obtained in step (5) and forward detail features and the backward structural features of the low-resolution video sequence obtained in step (6). and backward detail features Feature fusion and feature extraction are performed, specifically as follows:

[0099] Step (7.1). Take the t-th frame of the low-resolution video obtained in step (3.2). Structural information The t-th frame image obtained in step (3.4) Spatial structural features The t-th frame image obtained in step (5.5) Forward structural features and the t-th frame image obtained in step (6.5) Backward structural features Feature extraction and fusion are performed to obtain the reconstructed structural features:

[0100]

[0101] in, Represents the image of frame t. The reconstructed structural features are represented by R(.), which denotes the stacked residual block structure used for feature extraction, and c(.,.,.) represents the concatenation of features along the channel dimension.

[0102] Step (7.2). Take the t-th frame of the low-resolution video obtained in step (3.3). Detailed information The t-th frame image obtained in step (3.4) Spatial details The t-th frame image obtained in step (5.7) Forward detail features and the t-th frame image obtained in step (6.7) Backward detail features Repeat step (7.1) to obtain the t-th frame image. Reconstruction details

[0103] Step (7.3). Take the t-th frame image obtained in steps (7.1)-(7.2). Reconstructed structural features and reconstruction details Feature fusion and extraction are performed to obtain reconstructed spatial features:

[0104]

[0105] in, Represents the image of frame t. The reconstruction space features are represented by Relu(.), which represents the activation function, and SD(.,.) represents the stacked structure-detail residual block structure.

[0106] Step (7.4). Take the t-th frame of the low-resolution video obtained in step (2). Spatial features and the t-th frame image obtained in step (7.3) Reconstructed spatial features Feature fusion and extraction are performed to obtain reconstructed features:

[0107]

[0108] in, Represents the image of frame t. The reconstructed features are represented by R(.), which denotes the stacked residual block structure used to extract features, and c(.,.) denotes the concatenation of features along the channel dimension.

[0109] Step (7.5). Take the t-th frame image obtained in step (7.4). Reconstruction features Upsampling yields high-resolution features:

[0110]

[0111] in, Represents the image of frame t. High-resolution reconstruction features with a resolution of 4 times; PixelShuffle(.) represents subpixel convolution.

[0112] Step (7.6). Take the t-th frame image obtained in step (7.5). High-resolution reconstruction features Feature reconstruction is performed to obtain the super-resolution result:

[0113]

[0114] in, Represents the image of frame t. The high-resolution image after super-resolution; conv(.) represents a series of convolution operations, converting the image of frame t... High-resolution reconstruction features The channel dimension changed from 64 to 3.

[0115] Final super-resolution result This refers to the high-resolution video output from a low-resolution video using the bidirectional recurrent neural network super-resolution method based on structure-detail separation proposed in this invention.

[0116] The above description is merely a specific embodiment of this application and an explanation of the technical principles employed. Those skilled in the art should understand that the scope of the invention involved in this application is not limited to technical solutions formed by specific combinations of the above-described technical features, but should also cover other technical solutions formed by arbitrary combinations of the above-described technical features or their equivalents without departing from the concept of this application. For example, technical solutions formed by substituting the above features with (but not limited to) technical features with similar functions disclosed in this application.

Claims

1. A super-resolution method based on bidirectional recurrent neural networks using structure-detail separation, characterized in that, Includes the following steps: Step (1). Acquire low-resolution video and perform image enhancement frame by frame; Step (2). Extract the spatial features of each frame of the image; Step (3). Perform structure-detail separation on each frame image and its spatial features to extract the structural information, detail information, spatial structural features, and spatial detail features of each frame image. Step (3) includes: Step (3.1). The low-resolution video after image enhancement obtained in step (1) is then processed. Frame Images Interpolation is performed with an upsampling rate of 0.5 to obtain the upsampled features: Step (3.2). Interpolate the upsampled features obtained in step (3.1) with an upsampling rate of 2 to obtain the first... Frame Images Structural information: Step (3.3). The first step obtained in step (3.2) is... Frame Images Structural information and the first Frame Images By taking the difference, we get the first... Frame Images Detailed information; Step (3.4). The low-resolution video after image enhancement obtained in step (2) is then processed... Frame Images The spatial features are repeated twice in steps (3.1)-(3.2) to obtain the first interpolation. Frame Images Spatial structural characteristics; the first Frame Images Subtracting the spatial structural features from the spatial features yields the first... Frame Images Spatial detail features; Step (4). Calculate the forward and backward optical flow for each frame; Step (5). Input the spatial features, structural information, spatial structural features and forward optical flow of each frame of the image obtained in steps (2)-(4) into the feedforward recurrent neural network to obtain the forward structural features of each frame of the image; input the spatial features, detail information, spatial detail features and forward optical flow of each frame of the image obtained in steps (2)-(4) into the feedforward recurrent neural network to obtain the forward detail features of each frame of the image. Step (6). Input the spatial features, structural information, spatial structural features, backward optical flow and forward structural features of each frame obtained in steps (2)-(5) into the feedback recurrent neural network to obtain the backward structural features of each frame; input the spatial features, detail information, spatial detail features, backward optical flow and forward detail features of each frame obtained in steps (2)-(5) into the feedback recurrent neural network to obtain the backward detail features of each frame. Step (7) includes: Step (7.1). Splicing the first... Frame Images The structural information, spatial structural features, forward structural features, and backward structural features are collected and extracted. Then, the first... Frame Images The structural information is added to the extracted features to obtain the first... Frame Images The reconstructed structural features; Step (7.2). Splicing the first... Frame Images Detailed information, spatial detail features, forward detail features, and backward detail features are collected and extracted. Then, the first... Frame Images The detailed information is added to the extracted features to obtain the first... Frame Images Reconstruction details; Step (7.3). Splicing the first... Frame Images The reconstructed structural features and detailed features are then extracted to obtain the first... Frame Images The reconstructed spatial characteristics; Step (7.4). Splicing the first... Frame Images The spatial features are obtained by reconstructing spatial features and extracting features to get the first spatial feature. Frame Images Reconstruction features; Step (7.5). For the first Frame Images Interpolate the reconstructed features to obtain the first... Frame Images High-resolution reconstruction features after super-resolution; Step (7.6). The first... Frame Images The channel dimension of the high-resolution reconstructed features is converted to 3, resulting in the third... Frame Images The super-resolution high-resolution image is obtained by iterating through all frames.

2. The super-resolution method based on bidirectional recurrent neural networks according to claim 1, characterized in that, In step (1), image enhancement includes mirror symmetry, horizontal 90° flip, and vertical 90° flip.

3. The super-resolution method based on bidirectional recurrent neural networks according to claim 1, characterized in that, In step (2), the channel dimension of each frame of the low-resolution video after image enhancement is expanded from 3 to 64 while keeping the resolution unchanged, so as to obtain the spatial features of each frame.

4. The bidirectional recurrent neural network super-resolution method based on structure-detail separation according to claim 1, characterized in that, Step (4) includes: Step (4.1). Based on the image enhancement of the low-resolution video, the first... Frame Images Passing the exam Frame Images , obtained the Frame Images To the Frame Images Aligned forward optical flow; Step (4.2). Based on the image enhancement of the low-resolution video, the first... Frame Images Passing the exam Frame Images , obtained the Frame Images To the Frame Images Aligned backward optical flow.

5. The super-resolution method based on bidirectional recurrent neural networks according to claim 1, characterized in that, Step (5) includes: Step (5.1). Use the first Forward optical flow guidance of frame images Frame Images Forward hidden state features to the first Frame Images Align to obtain the first Frame Images Features aligned with the forward hidden state; Step (5.2). Use the first Frame Images Forward optical flow guidance Frame Images The forward optical flow is obtained to guide the first Frame Images To the Frame Images Align the forward optical flow and guide the first Frame Images Forward hidden state features to the first Frame Images Align to obtain the first Frame Images Features aligned with the forward hidden state; Step (5.3). The first... frame Features after forward hidden state alignment and the first frame After alignment of the forward hidden state, the features are subjected to structure-detail separation using the method in step (3) to obtain the first... Frame Images After forward hidden state alignment, structural features and detail features, and the first Frame Images The structural and detail features after alignment of the forward hidden state; Step (5.4). Stitch together historical frame images. , After aligning the structural features with the forward hidden state, feature extraction is performed to obtain the first... Frame Images Historical frame structure characteristics; Step (5.5). Splicing the first... Frame Images The structural information, spatial structural features, and historical frame structural features are analyzed and feature extraction is performed. Then, the spatial structural features are added to the extracted features to obtain the first... Frame Images Forward structural features; Step (5.6). Stitch together historical frame images. , After aligning the forward hidden state with detailed features, feature extraction is performed to obtain the first... Frame Images Historical frame details; Step (5.7). Splicing the first... Frame Images The detailed information, spatial detail features, and historical frame detail features are collected and extracted. Then, the spatial detail features are added to the extracted features to obtain the first... Frame Images Forward detail features; Step (5.8). Splicing the first... Frame Images The forward structural features and forward detail features are extracted, and then the first feature is... Frame Images The spatial features are added to the extracted features to obtain the first... Frame Images The forward hidden state features.

6. The bidirectional recurrent neural network super-resolution method based on structure-detail separation according to claim 1, characterized in that, Step (6) includes: Step (6.1). Use the first Frame Images Backward optical flow guidance Frame Images The backward hidden state features to the first Frame Images Align to obtain the first Frame Images Features aligned with the backward hidden state; Step (6.2). Use the first Frame Images Backward optical flow guidance Frame Images The backward optical flow alignment is used to guide the first Frame Images To the Frame Images Aligned backward optical flow and guide the first Frame Images The backward hidden state features to the first Frame Images Align to obtain the first Frame Images Features aligned with the backward hidden state; Step (6.3). The first... frame The features after back-hidden state alignment and the first Frame Images After the backward hidden state is aligned, the features are subjected to structure-detail separation using the method in step (3) to obtain the first... Frame Images The structural and detail features after back-hidden state alignment, and the first Frame Images The structural and detail features after alignment in the backward hidden state; Step (6.4). Stitching together future frame images , After aligning the backward hidden state with structural features and extracting features, the first... Frame Images Future frame structure features; Step (6.5). Splicing the first... Frame Images The structural information, spatial structural features, forward structural features, and future frame structural features are collected and feature extraction is performed to obtain the first... Frame Images The backward structural features; Step (6.6). Stitching together future frame images , After aligning the back hidden state with detailed features, feature extraction is performed to obtain the first... Frame Images Future frame detail features; Step (6.7). Splicing the first... Frame Images The detailed information, spatial detail features, forward detail features, and future frame detail features are collected and extracted. Then, the spatial detail features are added to the extracted features to obtain the first detail feature. Frame Images Backward detail features ; Step (6.8). Splicing the first... Frame Images The backward structural features and backward detail features are extracted, and then the 1st... Frame Images The spatial features are added to the extracted features to obtain the first... Frame Images The back hidden state features.

Images