Video super-resolution system and video super-resolution calculation method
By employing a lightweight architecture and a recursive approach, this video super-resolution system utilizes information from the previous frame for optical flow calculation and deep learning, solving the problems of real-time image latency and processing of extremely long video streams in existing technologies, and achieving low-power and high-efficiency video super-resolution processing.
Patent Information
- Application Number
- CN202410797129.3
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-06-20
- Publication Date
- 2025-12-23
AI Technical Summary
Existing video super-resolution technologies cannot process images in real time, resulting in delays and an inability to handle extremely long video streams, and they are also ineffective at handling noise and compression.
The video super-resolution system with a lightweight architecture uses motion estimation, mapping and neural network super-resolution devices to perform optical flow calculation and mapping using information from the previous frame, and combines deep learning to process video frames, and uses a recursive approach to predict the current frame.
It enables real-time processing of high-resolution images with low power consumption, avoiding latency, and can handle extremely long video streams and noisy and poorly compressed video streams.
Smart Images
Figure CN121190302A_ABST
Abstract
Description
Technical Field
[0001] This application relates to video super-resolution systems and methods for calculating video super-resolution, and more particularly to video super-resolution systems and methods that only require information from the previous frame and employ a recursive approach to process extremely long video streams. Background Technology
[0002] Consumers' demands for image quality are constantly increasing, leading to the development of video super-resolution (VSR). VSR can significantly improve video clarity; however, it has limitations and cannot be applied to real-time video. Using VSR in real-time video would require enormous resources and consume excessive power. Therefore, current hardware cannot perform VSR in real-time video processing.
[0003] Furthermore, existing video super-resolution methods require information from several future frames and several past frames to perform calculations, resulting in latency in real-time video (such as the footage from real-time online games). Moreover, existing video super-resolution methods cannot handle extremely long video streams. Additionally, existing video super-resolution methods are ineffective at handling noise and compression. Summary of the Invention
[0004] In view of the shortcomings of the prior art, one of the objectives of this application is (but not limited to) to provide a video super-resolution system and a video super-resolution calculation method to improve the shortcomings of the prior art.
[0005] In some embodiments, the video super-resolution system includes a motion estimation device, a mapping device, and a neural network super-resolution device. The motion estimation device calculates optical flow based on the current frame and a previous frame received from a memory. The mapping device maps the previous frame and a previous output received from the memory based on the optical flow to generate a mapped frame and a mapped output, respectively. The neural network super-resolution device extracts features from the current frame, the mapped frame, the mapped output, and a count value to generate at least one feature; performs deep learning on the at least one feature and a previous hidden state of the previous output to generate a current hidden state and a deep learning result; and extracts features from the deep learning result to generate a current output. The neural network super-resolution device stores the current frame, the current hidden state, and the current output in a memory.
[0006] In some embodiments, a processor reads at least one instruction stored in a memory to execute a video super-resolution calculation method, the video super-resolution calculation method comprising the following steps: calculating optical flow based on the current frame and a previous frame received from the memory; mapping the previous frame received from the memory and a previous output based on the optical flow to generate a mapped frame and a mapped output, respectively; extracting features from the current frame, the mapped frame, the mapped output, and a count value to generate at least one feature; performing deep learning on the at least one feature and a previous hidden state of the previous output to generate a current hidden state and a deep learning result; extracting features from the deep learning result to generate a current output; and storing the current frame, the current hidden state, and the current output in the memory.
[0007] The technical means embodied in the embodiments of this application can improve at least one of the shortcomings of the prior art. The video super-resolution system and video super-resolution calculation method of this application adopt a lightweight architecture and quantize relevant information, thus requiring low power consumption and can be applied to real-time video (e.g., 4K resolution video with a refresh rate of 120Hz). This application only needs information from the previous frame to predict the current frame; since it does not need information from several future frames, there is no video delay. Furthermore, this application uses a recursive approach to predict and process images, therefore, it can handle extremely long video streams. Moreover, this application can handle video streams with noise and poor compression.
[0008] The features, functions, and effects of this application will be described in detail below with reference to the accompanying drawings, and preferred embodiments will be provided. Attached Figure Description
[0009] Figure 1 This is a schematic diagram of a video super-resolution system and a memory according to some embodiments of this application;
[0010] Figure 2 This is a flowchart illustrating a video super-resolution calculation method based on some embodiments of this application;
[0011] Figure 3 This is a schematic diagram of a motion estimation device according to some embodiments of this application;
[0012] Figure 4 This is a schematic diagram illustrating the operation of a mapping device according to some embodiments of this application;
[0013] Figure 5 A schematic diagram of a neural network super-resolution device according to some embodiments of this application; and
[0014] Figure 6 This is a schematic diagram of a video super-resolution system and a memory according to some embodiments of this application. Detailed Implementation
[0015] All terms used herein have their ordinary meanings. The terms are defined in commonly used dictionaries, and any example use of any term discussed herein is merely illustrative and should not be construed as limiting the scope or meaning of this application. Similarly, this application is not limited to the various embodiments shown in this specification.
[0016] To address the issues of video super-resolution not being applicable to real-time video, the delay caused by video super-resolution in real-time video, and the inability of video super-resolution to handle extremely long video streams, this application proposes a video super-resolution system and a video super-resolution calculation method, detailed below.
[0017] Figure 1 This is a schematic diagram of a video super-resolution system 100 and a memory 900 according to some embodiments of this application. As shown, the video super-resolution system 100 includes a motion estimation device 110, a mapping device 120, a neural network super-resolution device 130, and a counter 140. In some embodiments, the memory 900 may be a Double Data Rate Synchronous Dynamic Random Access Memory (DDRSDRAM).
[0018] To make the operation of the Video Super-Resolution System 100 easier to understand, please refer to the following: Figure 2 , Figure 2 This is a flowchart illustrating a video super-resolution calculation method 200 based on some embodiments of this application.
[0019] Please refer to Figure 1 and Figure 2 In step 210, optical flow is calculated based on the current frame and the previous frame received from the memory. For example, motion estimation device 110 can calculate optical flow MV based on the current frame Ft and the previous frame Ft-1 received from the memory 900. This application only needs information from the previous frame to predict the current frame. Since it does not need information from the next few frames, this application does not have image delay.
[0020] For further explanation of step 210, please refer to Figure 3 , Figure 3 This is a schematic diagram of a motion estimation device 110 according to some embodiments of this application. As shown, the motion estimation device 110 includes a feature extractor 111, a correlation matcher 113, an optical flow calculator 115, and an upsampler 117.
[0021] In some embodiments, feature extractor 111 is used to extract features and scale down the current frame and the previous frame to generate multiple higher-order features. For example, feature extractor 111 extracts features and scales down the current frame Ft and the previous frame Ft-1 to generate higher-order features (such as higher-order features f1 and f2). These higher-order features can be building features, environmental features, or facial features, for subsequent tracking of the optical flow of the target between different frames. Furthermore, correlation matcher 113 is used to perform correlation matching on the multiple higher-order features to generate multiple related features fr. For example, correlation matcher 113 performs correlation matching on higher-order features (such as higher-order features f1 and f2), and the one with the highest probability is the same point in different frames (such as the current frame Ft and the previous frame Ft-1).
[0022] In some embodiments, the optical flow calculator 115 is used to calculate multiple relevant features fr to generate an optical flow MV. For example, the correlation matcher 113 can calculate the same point on different frames (such as the current frame Ft and the previous frame Ft-1), and the optical flow calculator 115 can calculate the corresponding optical flow MV based on the above information. The optical flow MV may include optical flow in the X direction (x flow) and optical flow in the Y direction (y flow).
[0023] In some embodiments, upsampler 117 is used to upsample the optical flow MV to generate an optical flow MV with the same image size as the current frame Ft. For example, since feature extractor 111 scales down the current frame Ft and the previous frame Ft-1, upsampler 117 needs to upsample the optical flow MV to generate an optical flow MV with the same image size as the current frame Ft. In some embodiments, upsampler 117 may also optimize the optical flow MV. In some embodiments, upsampler 117 may be an upsampling module. In some embodiments, upsampler 117 may include a convolutional layer and a scale-up module.
[0024] In step 220, the previous frame and the previous output received from the self-memory are mapped according to the optical flow to generate a mapped frame and a mapped output, respectively. For example, the mapping device 120 can map the previous frame Ft-1 and the previous output Ot-1 received from the self-memory 900 according to the optical flow MV to generate a mapped frame F't-1 and a mapped output O't-1, respectively.
[0025] For further explanation of step 220, please refer to Figure 4 , Figure 4This is a schematic diagram illustrating the operation of a mapping device 120 according to some embodiments of this application. As shown, the mapping device 120 obtains multiple candidate values from the previous frame Ft-1 based on the optical flow MV, and performs interpolation on the multiple candidate values to generate a mapped frame F't-1. For example, the mapping device 120 selects four candidate values from the previous frame Ft-1 based on the position information provided by the optical flow MV, and calculates the mapped frame F't-1 using bilinear interpolation.
[0026] In some embodiments, the mapping device 120 obtains multiple candidate values from the previous output Ot-1 based on the optical flow MV, and performs interpolation on the multiple candidate values to generate the mapped output O't-1. For example, the mapping device 120 selects four candidate values from the previous output Ot-1 based on the position information provided by the optical flow MV, and calculates the mapped output O't-1 using bilinear interpolation.
[0027] In step 230, feature extraction is performed on the current frame, the mapped frame, the mapped output, and the count value to generate at least one feature. For example, the neural network super-resolution device 130 may perform feature extraction on the current frame Ft, the mapped frame F't-1, the mapped output O't-1, and the count value t to generate at least one feature.
[0028] In some embodiments, counter 140 can generate a count value t and provide the count value t to the neural network super-resolution device 130. The neural network super-resolution device 130 can know the processing progress through the count value t and perform adaptive processing in different processing stages. For example, in the early processing stages, noise processing is required; however, in the later processing stages, since the noise is smaller, noise processing is not required. Therefore, noise processing is not performed in the later processing stages. In this way, since the neural network super-resolution device 130 can perform adaptive processing through the count value t, this application can process video streams with noise and poor compression.
[0029] For further explanation of step 230, please refer to Figure 5 , Figure 5 This is a schematic diagram of a neural network super-resolution device 130 according to some embodiments of this application. As shown, the neural network super-resolution device 130 includes a fusion unit 131, a feature extractor 132, a memory unit 133, a feature extractor 134, a resolution upscalorer 135, and a downscaling unit 136.
[0030] In some embodiments, the fusion unit 131 performs a fusion calculation on the current frame Ft, the mapped frame F't-1, the mapped output O't-1, and the count value t, combining multiple pieces of information to produce a fusion result. The feature extractor 132 extracts features from the fusion result to generate at least one feature. In some embodiments, the fusion unit 131 may be a fusion unit. In some embodiments, the fusion unit 131 may include convolutional layers, concatenated modules, and fully connected layers. In some embodiments, the feature extractor 132 may be a residual block.
[0031] In step 240, deep learning is performed on at least one feature and the previous hidden state of the previous output to generate the current hidden state and the deep learning result, and the current hidden state (i.e., important information of the current frame) is stored for use in the next frame. For example, the neural network super-resolution device 130 can perform deep learning on at least one feature and the previous hidden state of the previous output Ot-1 to generate the current hidden state Ht and the deep learning result, and store the current hidden state (i.e., important information of the current frame) for use in the next frame.
[0032] In some embodiments, memory unit 133 is used to perform deep learning on at least one feature and the previous hidden state to generate the current hidden state Ht and the deep learning result, and to store the current hidden state Ht in memory 900 for use in the next frame. In some embodiments, memory unit 133 may include a convolutional long short-term memory network (Conv-LSTM) and a convolutional gated recurrent unit (Conv-GRU).
[0033] In step 250, feature extraction is performed on the deep learning results to produce the current output. For example, the neural network super-resolution device 130 can perform feature extraction on the deep learning results to produce the current output Ot.
[0034] In some embodiments, feature extractor 134 is used to extract features from the deep learning results to generate multiple deep learning features. Resolution upscalorer 135 is used to upscale the multiple deep learning features to generate the current output Ot. In some embodiments, feature extractor 134 may be a residual block. In some embodiments, resolution upscalorer 135 may be an upscale unit. In some embodiments, resolution upscalorer 135 may include a pixel-shuffle module, or resolution upscalorer 135 may include a convolutional layer and a size upscaling module.
[0035] In step 260, the current frame, the current hidden state, and the current output are stored in memory. For example, the neural network super-resolution device 130 can store the current frame Ft, the current hidden state Ht, and the current output Ot in memory 900. This application can store the current frame Ft, the current hidden state Ht, and the current output Ot in memory 900 for use in the next frame. In other words, this application uses a recursive approach to predict and process images; therefore, this application can handle extremely long video streams (e.g., video streams with more than 1000 frames).
[0036] In some embodiments, the reduction unit 136 is used to rearrange the pixels of the current output Ot to produce a current output Ot with the same aspect ratio as the current frame. The reduction unit 136 can perform the reduction without losing resolution information. For example, the reduction unit 136 may be, but is not limited to, a de-arrangement unit, which can be used to reduce the previous output Ot to its original aspect ratio and store the current output Ot in the memory 900 for use in the next frame, and can perform the reduction without losing resolution information.
[0037] Figure 6 This is a schematic diagram of a video super-resolution system 100 and a memory 900 according to some embodiments of this application. As shown, this application can be implemented by executing at least one instruction via a processor 150. Figure 2 A video super-resolution calculation method 200. For example, this application can execute at least one instruction in memory 900 via processor 150 to perform related control operations to control... Figure 1 To perform various devices / components Figure 2 A method for calculating video super-resolution 200.
[0038] It should be noted that this application is not intended to... Figures 1 to 6 The embodiments shown are limited and are merely illustrative of one implementation of this application to facilitate understanding of the technology. The scope of this application is determined by the claims. Modifications and alterations made by those skilled in the art to the embodiments of this application without departing from the spirit of this application still fall within the scope of the claims.
[0039] In summary, the video super-resolution system 100 and video super-resolution calculation method 200 of this application employ a lightweight architecture and quantize relevant information, thus requiring low power consumption and can be applied to real-time video (e.g., 4K resolution video with a refresh rate of 120Hz). This application only needs information from the previous frame to predict the current frame; since it does not require information from several future frames, there is no video delay. Furthermore, this application uses a recursive approach to predict and process images, therefore, it can handle extremely long video streams. In addition, this application can handle video streams with noise and poor compression.
[0040] Although the embodiments of this application are described above, these embodiments are not intended to limit this application. Those skilled in the art can make changes to the technical features of this application based on the express or implied content of this application. All such changes may fall within the scope of patent protection claimed by this application. In other words, the scope of patent protection of this application shall be determined by the content defined in the claims.
[0041] Explanation of reference numerals in the attached figures
[0042] 100: Video Super-Resolution System
[0043] 110: Motion Estimation Device
[0044] 111: Feature Extractor
[0045] 113: Relevant Matcher
[0046] 115: Optical Flow Calculator
[0047] 117: Upsampler
[0048] 120: Mapping device
[0049] 130: Neural Network Super-Resolution Device
[0050] 131: Fusioner
[0051] 132: Feature Extractor
[0052] 133: Memory unit
[0053] 134: Feature Extractor
[0054] 135: Resolution Booster
[0055] 136: Reduction Unit
[0056] 140: Counter
[0057] 150: Processor
[0058] 200: Method
[0059] 210-260: Steps
[0060] 900: Memory
[0061] Ft: Current frame
[0062] Ft-1: Previous frame
[0063] F't-1: Mapped Frame
[0064] f1, f2: Higher-order features
[0065] fr: related features
[0066] Ht: Current hidden state
[0067] MV: Optical Flow
[0068] Ot: Current output
[0069] Ot-1: Previous output
[0070] O't-1: Mapping output
[0071] t: count value
Claims
1. A video super-resolution system, comprising: A motion estimation device is used to calculate optical flow based on the current frame and the previous frame received from memory; A mapping device is used to map the previous frame and the previous output received from the memory according to the optical flow, so as to generate a mapped frame and a mapped output respectively. as well as A neural network super-resolution device is used to extract features from the current frame, the mapped frame, the mapped output, and the count value to generate at least one feature, perform deep learning on the at least one feature and the previous hidden state of the previous output to generate the current hidden state and the deep learning result, and extract features from the deep learning result to generate the current output. The neural network super-resolution device stores the current frame, the current hidden state, and the current output in the memory.
2. The video super-resolution system according to claim 1, wherein the motion estimation device comprises: A feature extractor is used to extract and reduce the size of features from the current frame and the previous frame to generate multiple high-order features. as well as A correlation matcher is used to perform correlation matching on the multiple higher-order features to generate multiple correlation features.
3. The video super-resolution system according to claim 2, wherein the motion estimation device further comprises: An optical flow calculator is used to calculate the optical flow based on the plurality of related features.
4. The video super-resolution system according to claim 3, wherein the motion estimation device further comprises: An upsampler is used to upsample the optical flow to produce an optical flow of the same size as the image of the current frame.
5. The video super-resolution system according to claim 1, wherein the mapping device obtains a plurality of candidate values from the previous frame based on the optical flow, and performs interpolation on the plurality of candidate values to generate the mapped frame.
6. The video super-resolution system according to claim 1, wherein the mapping device obtains a plurality of candidate values from the previous output based on the optical flow, and performs interpolation on the plurality of candidate values to generate the mapping output.
7. The video super-resolution system according to claim 1, wherein the neural network super-resolution device comprises: A fusion processor is used to perform fusion calculations on the current frame, the mapped frame, the mapped output, and the count value to generate a fusion result; as well as A feature extractor is used to extract features from the fusion result to generate the at least one feature.
8. The video super-resolution system according to claim 1, wherein the neural network super-resolution device comprises: A memory unit is configured to perform deep learning on the at least one feature and the previous hidden state to generate the current hidden state and the deep learning result, and to store the current hidden state in the memory.
9. The video super-resolution system according to claim 1, wherein the neural network super-resolution device comprises: A feature extractor is used to extract features from the deep learning results to generate multiple deep learning features; as well as A resolution booster is used to boost the resolution of the plurality of deep learning features to produce the current output.
10. A video super-resolution calculation method, wherein a processor reads at least one instruction stored in a memory to execute the video super-resolution calculation method, comprising: The optical flow is calculated based on the current frame and the previous frame received from the memory. The optical flow is used to map the previous frame and the previous output received from the memory to generate a mapped frame and a mapped output, respectively. Feature extraction is performed on the current frame, the mapped frame, the mapped output, and the count value to generate at least one feature; Deep learning is performed on the at least one feature and the previous hidden state of the previous output to generate the current hidden state and the deep learning result; Feature extraction is performed on the deep learning results to generate the current output; as well as The current frame, the current hidden state, and the current output are stored in the memory.