Real-time 8K video sensing super-resolution reconstruction method and system

Abstract

The invention discloses a real-time 8K video sensing super-resolution reconstruction method and system, and the method comprises the steps: S1, carrying out the serialized analysis of an input low-resolution video stream through a space-time window mechanism, constructing a space-time context window containing a target frame and an adjacent frame, and obtaining a low-resolution frame sequence set; s2, performing multi-scale spatio-temporal feature extraction on the low-resolution frame sequence set by adopting a shallow 3D convolutional neural network to obtain a spatio-temporal feature map; s3, processing the spatial-temporal feature map through two parallel modules of motion compensation analysis and texture complexity analysis, and generating a perception attention map reflecting visual perception importance; and S4, performing super-resolution reconstruction on the spatial-temporal feature map by using a depth residual generation network guided by a perceptual attention map, and outputting a high-resolution 8K video frame. According to the 8K super-resolution reconstruction method, high-quality 8K super-resolution reconstruction of a low-resolution video can be realized, the processing efficiency is remarkably improved while the reconstruction quality is ensured, and the application requirement of real-time 8K video processing is met.

Description

Technical Field

[0001] This invention relates to the field of video data processing technology, and in particular to a real-time 8K video perception super-resolution reconstruction method and system. Background Technology

[0002] With the rapid development of digital media technology and the widespread application of 5G communication technology, ultra-high-definition video content has become an important carrier of modern information dissemination. 8K ultra-high-definition video, with its ultra-high resolution and rich visual details, has shown enormous application potential in fields such as broadcasting, video surveillance, medical imaging, and virtual reality. However, limited by factors such as network bandwidth, storage costs, and transmission efficiency, a large amount of video content in practical applications is still acquired, transmitted, and stored at 4K or even lower resolutions, creating a significant contradiction with the ever-increasing demand for 8K displays. Video super-resolution reconstruction technology, as a key technology to resolve this contradiction, can use algorithms to upscale low-resolution video content to 8K resolution, thereby meeting the demands of ultra-high-definition displays without increasing transmission and storage costs.

[0003] Current video super-resolution reconstruction techniques are mainly divided into two categories: traditional interpolation-based methods and modern deep learning-based methods. Traditional interpolation methods, such as bicubic interpolation and Lanczos interpolation, are computationally simple, but the generated images often suffer from blurred edges and loss of detail, making it difficult to meet the high-quality requirements of 8K video. While deep learning-based methods have significantly improved reconstruction quality, most existing techniques still have many shortcomings.

[0004] Existing technology CN202210719343.8 discloses an 8K video super-resolution processing and display platform, which mainly focuses on the construction of the hardware platform, deploying the super-resolution algorithm through an FPGA chip. However, this technology has obvious limitations. First, it adopts a traditional hardware acceleration scheme, lacking specific optimization for the temporal characteristics of video. It can only process single-frame image information and cannot effectively utilize the temporal correlation in the video sequence, leading to temporal inconsistencies and flickering in the reconstruction results in moving areas. Second, it does not consider the characteristics of human visual perception, applying the same processing strategy to all pixel areas, failing to achieve optimal allocation of computing resources, and easily resulting in low reconstruction efficiency when processing complex scenes. In addition, the algorithm implementation of this technology mainly relies on the FPGA hardware platform, lacking flexibility and scalability, and is difficult to adapt to the processing needs of different types of video content.

[0005] Besides the aforementioned patented technologies, other existing video super-resolution methods also generally suffer from some common problems. Most existing methods face enormous computational complexity challenges when processing 8K video, making it difficult to meet the requirements of real-time processing. Traditional single-frame-based super-resolution methods ignore the temporal correlation between frames in a video sequence, failing to fully utilize motion information to guide the reconstruction process. While existing video super-resolution methods consider temporal information, most employ complex spatiotemporal network architectures, resulting in huge computational overhead and making it difficult to achieve real-time 8K video processing with limited hardware resources.

[0006] More importantly, existing technologies generally lack in-depth consideration of visual perception characteristics. The human visual system exhibits significant differences in sensitivity to different regions; moving areas and areas with rich textures tend to attract more attention, while background areas and smooth areas are relatively less important. However, most existing super-resolution methods employ a uniform processing strategy for all pixel regions, failing to differentiate processing based on visual importance, resulting in wasted computational resources and low reconstruction efficiency.

[0007] To address the specific needs of real-time 8K video processing, existing technologies still face challenges such as insufficient spatiotemporal feature extraction, inadequate attention mechanism design, and insufficient network architecture optimization. Regarding spatiotemporal feature extraction, existing methods either consider only spatial information while neglecting temporal information, or employ overly complex 3D convolutional networks, resulting in excessive computational overhead. In terms of attention mechanism design, most existing methods use a single attention model, failing to simultaneously consider the importance of motion and texture information. Regarding network architecture optimization, existing methods often prioritize high accuracy while neglecting real-time requirements, making it difficult to achieve a good balance between accuracy and efficiency.

[0008] Therefore, there is an urgent need to develop a new real-time 8K video perceptual super-resolution reconstruction method. This method should be able to fully utilize the spatiotemporal characteristics of video sequences and combine them with human visual perception mechanisms to achieve high-quality and high-efficiency 8K video super-resolution reconstruction, so as to meet the actual needs of modern ultra-high-definition video applications. Summary of the Invention

[0009] In view of this, the present invention provides a real-time 8K video perceptual super-resolution reconstruction method. The purpose is to construct a multi-scale feature extraction network based on a spatiotemporal window mechanism and combine a perceptual attention mechanism based on motion compensation analysis and texture complexity analysis to achieve high-quality 8K super-resolution reconstruction of low-resolution videos. This method significantly improves processing efficiency while ensuring reconstruction quality, thus meeting the application requirements of real-time 8K video processing.

[0010] To achieve the above objectives, the present invention provides a real-time 8K video perceptual super-resolution reconstruction method, comprising the following steps: S1: The input low-resolution video stream is serialized and parsed using a spatiotemporal window mechanism to construct a spatiotemporal context window containing the target frame and its adjacent frames, thereby obtaining a set of low-resolution frame sequences. S2: A trained shallow 3D convolutional neural network is used to extract multi-scale spatiotemporal features from a set of low-resolution frame sequences to obtain a spatiotemporal feature map; S3: Spatiotemporal feature maps are processed through two parallel modules, motion compensation analysis and texture complexity analysis, to generate a perceptual attention map that reflects the importance of visual perception; S4: A deep residual generation network guided by a perceptual attention map is used to perform super-resolution reconstruction of the spatiotemporal feature map and output high-resolution 8K video frames.

[0011] As a further improvement of the present invention: Optionally, in step S1, the input low-resolution video stream is serialized and parsed using a spatiotemporal window mechanism to construct a spatiotemporal context window containing the target frame and its adjacent frames, thereby obtaining a set of low-resolution frame sequences, including: The continuous low-resolution video frame stream obtained after network transmission and decoding is parsed and decomposed into independent video frame units; in order to effectively utilize the temporal continuity information of the video sequence, the target frame that needs to be reconstructed is selected. Adjacent to it Frames are combined to form a short-time frame sequence, wherein the adjacent frames include the target frame. Previous frame sequence and target frame Subsequent frame sequences ; target frame and its adjacent The frames are arranged in chronological order to form a set of low-resolution frame sequences. Specifically: in, Indicates time The target reconstructed frame, Indicates the first frame before the target frame Low-resolution images of frames, Indicates the number of frames after the target frame Low-resolution images of frames, .

[0012] Optionally, in step S2, a trained shallow 3D convolutional neural network is used to perform multi-scale spatiotemporal feature extraction on the low-resolution frame sequence set to obtain a spatiotemporal feature map, including: low-resolution frame sequence set After upsampling, the input is fed into a trained shallow 3D convolutional neural network for feature extraction. The trained shallow 3D convolutional neural network includes three consecutive 3D convolutional layers, each of which uses a convolutional kernel of a different size to extract multi-scale feature information. The first 3D convolutional layer uses a 3×3×3 convolutional kernel for fine-grained feature extraction, which is used to capture local texture details and subtle motion information; the second 3D convolutional layer uses a 5×5×3 convolutional kernel for medium-scale feature extraction, which is used to capture medium-range spatial structure and motion patterns; the third 3D convolutional layer uses a 7×7×3 convolutional kernel for large-scale feature extraction, which is used to capture global spatial layout and long-range motion trajectories. The 3D convolution operation is as follows: in, Indicates the first Output feature map of 3D convolution layer Indicates the first The 3D convolution kernel weight matrix of the layer, Indicates the first The input feature map of the layer, when hour That is, the set of upsampled low-resolution frame sequences of the input. , Indicates the first The layer's bias vector, This represents the activation function. 3D convolution operators ;when When, define Spatiotemporal feature map ;when When, define For the predicted high-resolution video frames; The training process of a shallow 3D convolutional neural network includes: constructing a training dataset, generating corresponding low-resolution frame sequences from high-resolution video frames through bicubic interpolation downsampling, forming input-output training pairs; defining the training loss function as reconstruction loss, which calculates the mean square error between the high-resolution video frames predicted by the shallow 3D convolutional neural network and the real high-resolution video frames, wherein the high-resolution video frames are video frames with a resolution higher than that of the low-resolution video frames. The network parameters were updated using a stochastic gradient descent optimizer. During training, the learning rate was set to 0.001, the batch size was 16 samples, and the number of training iterations was 20,000. The feature extraction performance was evaluated on an independent validation dataset every 1,000 iterations. The generalization ability of the model was judged by calculating the reconstruction loss on the validation dataset. Training was stopped early when the reconstruction loss on the validation dataset no longer decreased after 5 consecutive iterations.

[0013] The advantage of using a shallow 3D convolutional neural network for multi-scale spatiotemporal feature extraction in this step lies in its ability to efficiently capture spatiotemporal correlation information in video sequences. Compared to traditional 2D convolutional methods that can only process spatial information of a single frame, 3D convolution can learn features simultaneously in both spatial and temporal dimensions, thereby better understanding the motion relationships and temporal dependencies between video frames. This step achieves multi-scale feature extraction by employing three convolutional kernels of different sizes. This design allows for the simultaneous capture of visual features at different levels within the same network architecture. The small-sized convolutional kernel in the first layer focuses on extracting local texture details and subtle motion changes; the medium-sized convolutional kernel in the second layer is responsible for capturing medium-range spatial structures and motion patterns; and the large-sized convolutional kernel in the third layer is used to understand the global spatial layout and long-range motion trajectories. This multi-scale feature extraction strategy ensures that the network can obtain rich feature representations within different receptive fields, providing a more comprehensive and accurate feature foundation for subsequent super-resolution reconstruction.

[0014] Optionally, in step S3, the spatiotemporal feature map is processed through two parallel modules: motion compensation analysis and texture complexity analysis, to generate a perceptual attention map reflecting the importance of visual perception, including: The motion compensation analysis module analyzes spatiotemporal feature maps. The time dimension information is used to calculate pixel-level motion intensity and direction; the motion compensation analysis module first calculates the optical flow field between adjacent frames, and then calculates the motion intensity map based on the optical flow amplitude. Specifically: in, Indicates coordinate position The intensity of the movement at that location Indicates coordinate position The horizontal optical flow component, Indicates coordinate position The optical flow component in the vertical direction; The texture complexity analysis module analyzes the spatiotemporal feature map. The spatial dimension information in the image is used to detect edges, corners, and high-frequency texture features; the texture complexity analysis module uses the Sobel operator to calculate the texture complexity map. Specifically: in, Indicates coordinate position Texture complexity value at that location, Indicates coordinate position The gradient component in the horizontal direction, Indicates coordinate position The gradient component in the vertical direction; Finally, the exercise intensity graph and texture complexity graph Weighted fusion is performed to generate the final perceptual attention map. Specifically: in, Indicates coordinate position Perceptual attention weights at the location, The fusion weighting coefficient represents the intensity of the exercise. The fusion weight coefficients represent the texture complexity and satisfy the following conditions: .

[0015] The advantage of this step, which uses two parallel modules—motion compensation analysis and texture complexity analysis—to generate a perceptual attention map is its ability to accurately capture the focus of human visual perception on video content. This step, through the parallel processing of motion and texture information, fully leverages the spatiotemporal characteristics of video sequences. The motion compensation analysis module focuses on mining information in the temporal dimension, accurately identifying motion regions and motion intensity in the video through optical flow calculations. These regions are often the focus of viewer attention and should be given higher priority in super-resolution reconstruction. The texture complexity analysis module focuses on information in the spatial dimension, identifying edges, corners, and high-frequency texture features in the image through gradient calculations. These features have a significant impact on visual perception quality and are details that need to be carefully preserved in super-resolution reconstruction.

[0016] Optionally, in step S4, a deep residual generation network guided by a perceptual attention map is used to perform super-resolution reconstruction of the spatiotemporal feature map, outputting high-resolution 8K video frames, including: A deep residual generation network containing multiple attention-guided residual blocks is constructed as the main architecture for super-resolution reconstruction. Within each attention-guided residual block, the input feature map of the guiding residual block is first combined with the perceptual attention map. Perform element-wise multiplication to implement attention weights; The attention guidance operation specifically includes: in, Indicates the first The attention-guided feature map in each residual block Indicates the first The input feature map of each residual block, where ⊙ represents the element-wise multiplication operator; The attention-guided feature maps are then optimized using a residual learning mechanism. The formula for calculating the residual block is as follows: in, Indicates the first Output feature maps of each residual block Indicates the first The first convolution operation of each residual block Indicates the first The second convolution operation on each residual block. Indicates the activation function; The deep residual generation network includes The cascaded processing of attention-guided residual blocks, after feature optimization of multiple residual blocks, finally performs upsampling operations through sub-pixel convolutional layers to enlarge the feature map size to the target 8K resolution; the specific upsampling factor of the sub-pixel convolution is: in, Indicates the upsampling factor. This indicates the height in pixels of the target 8K image. This indicates the width in pixels of the target 8K image. This indicates the height (in pixels) of the input low-resolution image. This indicates the width (in pixels) of the input low-resolution image.

[0017] This invention also discloses a real-time 8K video perceptual super-resolution reconstruction system, comprising: Serialization module: It performs serialization parsing on the input low-resolution video stream through a spatiotemporal window mechanism, constructs a spatiotemporal context window containing the target frame and its adjacent frames, and obtains a set of low-resolution frame sequences. Feature extraction module: The trained shallow 3D convolutional neural network is used to extract multi-scale spatiotemporal features from the low-resolution frame sequence set to obtain spatiotemporal feature maps; Attention Module: The spatiotemporal feature map is processed by two parallel modules, motion compensation analysis and texture complexity analysis, to generate a perceptual attention map that reflects the importance of visual perception. Super-resolution reconstruction module: A deep residual generation network guided by a perceptual attention map is used to perform super-resolution reconstruction of the spatiotemporal feature map and output high-resolution 8K video frames.

[0018] Compared with the prior art, the present invention has at least the following beneficial effects: This invention, by employing a spatiotemporal window mechanism combined with a shallow 3D convolutional neural network design, can fully utilize the spatiotemporal correlation information of video sequences with relatively low computational complexity. Compared to traditional single-frame super-resolution methods, this invention can effectively capture motion information and temporal continuity between video frames, significantly improving the temporal consistency and visual quality of the reconstruction results. Simultaneously, the shallow network architecture has higher computational efficiency than existing deep and complex networks, significantly reducing computational overhead and memory usage while ensuring sufficient feature extraction, making real-time 8K video processing possible. Through the design of multi-scale convolutional kernels, the network can simultaneously capture fine-grained texture details, medium-scale spatial structure, and global layout information, providing a rich feature base for subsequent super-resolution reconstruction.

[0019] This invention introduces human visual perception mechanisms into the 8K video super-resolution reconstruction process. It generates a perceptual attention map through two parallel modules: motion compensation analysis and texture complexity analysis, enabling a differentiated processing strategy based on visual importance. This perception-driven method automatically identifies key visual regions in the video, such as moving objects, textured areas, and edge details, allocating more computational resources and reconstruction accuracy to these regions, while employing relatively simplified processing for smooth background areas with lower visual importance. This strategy not only conforms to the laws of human visual cognition and improves subjective visual quality but also achieves optimized allocation of computational resources, significantly improving overall processing efficiency while ensuring the reconstruction quality of key regions. Experimental results show that, compared to traditional uniform processing methods, the perceptual attention mechanism of this invention achieves better visual results with the same computational resources.

[0020] This invention achieves an optimal balance between reconstruction quality and computational efficiency through an innovative architecture design that combines a deep residual generation network with perceptual attention guidance. The attention-guided residual block design dynamically adjusts the feature processing strategy based on perceptual importance, enabling the network to perform refined reconstruction in important regions and fast processing in less important regions. The multi-level upsampling strategy using sub-pixel convolution ensures a smooth transition and preservation of detail from low resolution to 8K resolution. Attached Figure Description

[0021] Figure 1 This is a flowchart illustrating a real-time 8K video perception super-resolution reconstruction method according to an embodiment of the present invention. Figure 2 Weight parameters and Heatmap of the impact on PSNR; Figure 3 This is a schematic diagram illustrating the impact of residual blocks on PSNR. Detailed Implementation

[0022] The present invention will be further described below with reference to the accompanying drawings, but this is not intended to limit the present invention in any way. Any modifications or substitutions made based on the teachings of the present invention shall fall within the protection scope of the present invention.

[0023] Example 1: A real-time 8K video perceptual super-resolution reconstruction method, such as... Figure 1 As shown, it includes the following steps: S1: The input low-resolution video stream is serialized and parsed using a spatiotemporal window mechanism to construct a spatiotemporal context window containing the target frame and its adjacent frames, thereby obtaining a set of low-resolution frame sequences. The continuous low-resolution video frame stream obtained after network transmission and decoding is parsed and decomposed into independent video frame units; in order to effectively utilize the temporal continuity information of the video sequence, the target frame that needs to be reconstructed is selected. Adjacent to it Frames are combined to form a short-time frame sequence, as described in this embodiment. The adjacent frames include the target frame. Previous frame sequence and target frame Subsequent frame sequences Optionally, when the video content is a static scene or a low-motion scene, a content-adaptive spatiotemporal window optimization strategy is adopted. The scene motion intensity is determined by calculating the structural similarity index (SSIM) between consecutive frames. When the SSIM value is greater than 0.95, it is considered a static scene, and the spatiotemporal window size is adjusted accordingly. Dynamically adjusted to 4 to reduce computational overhead; target frame and its adjacent The frames are arranged in chronological order to form a set of low-resolution frame sequences. Specifically: in, Indicates time The target reconstructed frame, Indicates the first frame before the target frame Low-resolution images of frames, Indicates the number of frames after the target frame Low-resolution images of frames, .

[0024] S2: A trained shallow 3D convolutional neural network is used to extract multi-scale spatiotemporal features from a set of low-resolution frame sequences to obtain spatiotemporal feature maps. low-resolution frame sequence set After upsampling, the data is input into a trained shallow 3D convolutional neural network for feature extraction. The trained 3D convolutional neural network includes three consecutive 3D convolutional layers, each of which uses a convolutional kernel of a different size to extract multi-scale feature information. The first 3D convolutional layer uses a 3×3×3 kernel for fine-grained feature extraction, capturing local texture details and subtle motion information. The second 3D convolutional layer uses a 5×5×3 kernel for medium-scale feature extraction, capturing medium-range spatial structures and motion patterns. The third 3D convolutional layer uses a 7×7×3 kernel for large-scale feature extraction, capturing global spatial layout and long-range motion trajectories. In this embodiment, the specific architecture configuration of the shallow 3D convolutional neural network is as follows: the first 3D convolutional layer has 3 input channels, 32 output channels, a kernel stride of [1, 1, 1], and same padding. The second 3D convolutional layer has 32 input channels, 64 output channels, a kernel stride of [1, 1, 1], and same padding. The third 3D convolutional layer has 64 input channels, 128 output channels, a kernel stride of [1, 1, 1], and same padding. The 3D convolution operation specifically refers to: in, Indicates the first Output feature map of 3D convolution layer Indicates the first The 3D convolution kernel weight matrix of the layer, Indicates the first The input feature map of the layer, when hour That is, the set of upsampled low-resolution frame sequences of the input. , Indicates the first The layer's bias vector, This represents the activation function; in this embodiment, it is the ReLU activation function. 3D convolution operators ;when When, define Spatiotemporal feature map ;when When, define For the predicted high-resolution video frames; The training process of a shallow 3D convolutional neural network includes: constructing a training dataset, generating corresponding low-resolution frame sequences from high-resolution video frames through bicubic interpolation downsampling, forming input-output training pairs; defining the network training loss function as reconstruction loss, which calculates the mean square error between the high-resolution video frames predicted by the shallow 3D convolutional neural network and the real high-resolution video frames, wherein the high-resolution video frames are video frames with a resolution higher than that of the low-resolution video frames. The network parameters are updated using a stochastic gradient descent optimizer. During training, the learning rate is set to 0.001, the batch size is 16 samples, and the number of training iterations is 20,000. The feature extraction performance is evaluated on an independent validation dataset every 1,000 iterations. The generalization ability of the model is judged by calculating the reconstruction loss on the validation dataset. Training is stopped early when the reconstruction loss on the validation dataset no longer decreases for 5 consecutive iterations.

[0025] S3: Spatiotemporal feature maps are processed through two parallel modules: motion compensation analysis and texture complexity analysis, to generate a perceptual attention map that reflects the importance of visual perception. The motion compensation analysis module analyzes spatiotemporal feature maps. The time dimension information is used to calculate pixel-level motion intensity and direction; the motion compensation analysis module first calculates the optical flow field between adjacent frames, and then calculates the motion intensity map based on the optical flow amplitude. Specifically: in, Indicates coordinate position The intensity of the movement at that location Indicates coordinate position The horizontal optical flow component, Indicates coordinate position The optical flow component in the vertical direction; The texture complexity analysis module analyzes the spatiotemporal feature map. The spatial dimension information in the image is used to detect edges, corners, and high-frequency texture features. In this embodiment, the texture complexity analysis uses the Sobel operator to calculate the image gradient. The horizontal convolution kernel of the Sobel operator is: [[-1,0,1], [-2,0,2], [-1,0,1]], and the vertical convolution kernel is: [[-1,-2,-1], [0,0,0], [1,2,1]]. This module uses the Sobel operator to calculate the texture complexity map. Specifically: in, Indicates coordinate position Texture complexity value at that location, Indicates coordinate position The gradient component in the horizontal direction, Indicates coordinate position The gradient component in the vertical direction; when processing videos containing complex texture patterns or artificially synthesized content, the texture complexity analysis module can choose to use a texture description method based on Local Binary Pattern (LBP), which captures texture information by calculating the LBP encoding of each pixel's neighborhood. Finally, the exercise intensity graph and texture complexity graph Weighted fusion is performed to generate the final perceptual attention map. Specifically: in, Indicates coordinate position Perceptual attention weights at the location, The fusion weighting coefficient represents the intensity of the exercise. The fusion weight coefficients represent the texture complexity and satisfy the following conditions: In this embodiment, , ,like Figure 2 As shown, Figure 2 Displays the perceptual attention weight parameters and A heatmap showing the impact on reconstruction quality; before weighted fusion, the motion intensity map and texture complexity map are normalized.

[0026] S4: A deep residual generation network guided by a perceptual attention map is used to perform super-resolution reconstruction of the spatiotemporal feature map, outputting high-resolution 8K video frames: A deep residual generation network containing multiple attention-guided residual blocks is constructed as the main architecture for super-resolution reconstruction; in this embodiment, the deep residual generation network contains 16 attention-guided residual blocks, such as... Figure 3 As shown, Figure 3 The improvement in reconstruction accuracy of deep residual networks is demonstrated. Experimental results show that the accuracy improvement tends to saturate when the number of residual blocks exceeds 16. Each residual block contains two 3×3 convolutional layers. The first convolutional layer has 128 output channels, and the second convolutional layer has the same number of output channels as the number of input channels, which is 64. Each convolutional layer is followed by batch normalization and ReLU activation function. The attention guidance operation specifically includes: in, Indicates the first The attention-guided feature map in each residual block Indicates the first The input feature map of each residual block, where ⊙ represents the element-wise multiplication operator; The attention-guided feature maps are then optimized using a residual learning mechanism. The formula for calculating the residual block is as follows: in, Indicates the first Output feature maps of each residual block Indicates the first The first convolution operation of each residual block Indicates the first The second convolution operation on each residual block. Indicates the activation function; The deep residual generation network includes The cascaded processing of attention-guided residual blocks, after feature optimization of multiple residual blocks, finally performs upsampling operations through sub-pixel convolutional layers to enlarge the feature map size to the target 8K resolution; the specific upsampling factor of the sub-pixel convolution is: in, Indicates the upsampling factor. This indicates the height in pixels of the target 8K image. This indicates the width in pixels of the target 8K image. This indicates the height (in pixels) of the input low-resolution image. This indicates the width (in pixels) of the input low-resolution image.

[0027] Example 2: This invention also discloses a real-time 8K video perceptual super-resolution reconstruction system, comprising the following five modules: Serialization module: It performs serialization parsing on the input low-resolution video stream through a spatiotemporal window mechanism, constructs a spatiotemporal context window containing the target frame and its adjacent frames, and obtains a set of low-resolution frame sequences. Feature extraction module: The trained shallow 3D convolutional neural network is used to extract multi-scale spatiotemporal features from the low-resolution frame sequence set to obtain spatiotemporal feature maps; Attention Module: The spatiotemporal feature map is processed by two parallel modules, motion compensation analysis and texture complexity analysis, to generate a perceptual attention map that reflects the importance of visual perception. Super-resolution reconstruction module: A deep residual generation network guided by a perceptual attention map is used to perform super-resolution reconstruction of the spatiotemporal feature map and output high-resolution 8K video frames.

[0028] It should be noted that the sequence numbers of the above embodiments of the present invention are merely for descriptive purposes and do not represent the superiority or inferiority of the embodiments. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, apparatus, article, or method that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, apparatus, article, or method. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, apparatus, article, or method that includes that element.

[0029] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) as described above, and includes several instructions to cause a terminal device (which may be a mobile phone, computer, server, or network device, etc.) to execute the methods described in the various embodiments of the present invention.

[0030] The above are merely preferred embodiments of the present invention and do not limit the scope of the patent. Any equivalent structural or procedural transformations made based on the description and drawings of the present invention, or direct or indirect applications in other related technical fields, are similarly included within the scope of patent protection of the present invention.

Claims

1. A real-time 8K video perceptual super-resolution reconstruction method, characterized in that, Includes the following steps: S1: The input low-resolution video stream is serialized and parsed using a spatiotemporal window mechanism to construct a spatiotemporal context window containing the target frame and its adjacent frames, thereby obtaining a set of low-resolution frame sequences. S2: A trained shallow 3D convolutional neural network is used to extract multi-scale spatiotemporal features from a set of low-resolution frame sequences to obtain a spatiotemporal feature map; S3: Spatiotemporal feature maps are processed through two parallel modules, motion compensation analysis and texture complexity analysis, to generate a perceptual attention map that reflects the importance of visual perception; S4: A deep residual generation network guided by a perceptual attention map is used to perform super-resolution reconstruction of the spatiotemporal feature map and output high-resolution 8K video frames.

2. The real-time 8K video perceptual super-resolution reconstruction method according to claim 1, characterized in that, Step S1 includes: The continuous low-resolution video frame stream obtained after network transmission and decoding is parsed and decomposed into independent video frame units; in order to effectively utilize the temporal continuity information of the video sequence, the target frame that needs to be reconstructed is selected. Adjacent to it Frames are combined to form a short-time frame sequence, with adjacent frames including the target frame. Previous frame sequence and target frame Subsequent frame sequences ; target frame and its adjacent The frames are arranged in chronological order to form a set of low-resolution frame sequences. Specifically: in, Indicates time The target reconstructed frame, Indicates the first frame before the target frame Low-resolution images of frames, Indicates the number of frames after the target frame Low-resolution images of frames, .

3. The real-time 8K video perceptual super-resolution reconstruction method according to claim 2, characterized in that, Step S2 includes: low-resolution frame sequence set After upsampling, the input is fed into a trained shallow 3D convolutional neural network for feature extraction. The trained shallow 3D convolutional neural network includes three consecutive 3D convolutional layers, each of which uses a convolutional kernel of a different size to extract multi-scale feature information. The first 3D convolutional layer uses a 3×3×3 convolutional kernel for fine-grained feature extraction, which is used to capture local texture details and subtle motion information; the second 3D convolutional layer uses a 5×5×3 convolutional kernel for medium-scale feature extraction, which is used to capture medium-range spatial structure and motion patterns; the third 3D convolutional layer uses a 7×7×3 convolutional kernel for large-scale feature extraction, which is used to capture global spatial layout and long-range motion trajectories. The 3D convolution operation is as follows: in, Indicates the first Output feature map of 3D convolution layer Indicates the first The 3D convolution kernel weight matrix of the layer, Indicates the first The input feature map of the layer, when hour That is, the set of upsampled low-resolution frame sequences of the input. , Indicates the first The layer's bias vector, This represents the activation function. 3D convolution operators ;when When, define Spatiotemporal feature map ;when When, define For the predicted high-resolution video frames; The training process of a shallow 3D convolutional neural network includes: constructing a training dataset, generating corresponding low-resolution frame sequences from high-resolution video frames through bicubic interpolation downsampling, forming input-output training pairs; defining the training loss function as reconstruction loss, which calculates the mean square error between the high-resolution video frames predicted by the shallow 3D convolutional neural network and the real high-resolution video frames, wherein the high-resolution video frames are video frames with a resolution higher than that of the low-resolution video frames. The network parameters are updated using a stochastic gradient descent optimizer. During training, the learning rate is set to 0.001, the batch size is 16 samples, and the number of training iterations is 20,000. The feature extraction performance is evaluated on an independent validation dataset every 1,000 iterations. The generalization ability of the model is judged by calculating the reconstruction loss on the validation dataset. Training is stopped early when the reconstruction loss on the validation dataset no longer decreases for 5 consecutive iterations.

4. The real-time 8K video perceptual super-resolution reconstruction method according to claim 3, characterized in that, Step S3 includes: The motion compensation analysis module analyzes spatiotemporal feature maps. The time dimension information is used to calculate pixel-level motion intensity and direction; the motion compensation analysis module first calculates the optical flow field between adjacent frames, and then calculates the motion intensity map based on the optical flow amplitude. Specifically: in, Indicates coordinate position The intensity of the movement at that location Indicates coordinate position The horizontal optical flow component, Indicates coordinate position The optical flow component in the vertical direction; The texture complexity analysis module analyzes the spatiotemporal feature map. The spatial dimension information in the image is used to detect edges, corners, and high-frequency texture features; the texture complexity analysis module uses the Sobel operator to calculate the texture complexity map. Specifically: in, Indicates coordinate position Texture complexity value at that location, Indicates coordinate position The gradient component in the horizontal direction, Indicates coordinate position The gradient component in the vertical direction; Finally, the exercise intensity graph and texture complexity graph Weighted fusion is performed to generate the final perceptual attention map. .

5. The real-time 8K video perceptual super-resolution reconstruction method according to claim 4, characterized in that, The weighted fusion includes: in, Indicates coordinate position Perceptual attention weights at the location, The fusion weighting coefficient represents the intensity of the exercise. The fusion weight coefficients represent the texture complexity and satisfy the following conditions: .

6. The real-time 8K video perceptual super-resolution reconstruction method according to claim 4, characterized in that, Step S4 includes: A deep residual generation network containing multiple attention-guided residual blocks is constructed as the main architecture for super-resolution reconstruction. Within each attention-guided residual block, the input feature map of the guiding residual block is first combined with the perceptual attention map. Perform element-wise multiplication to implement attention weights; The attention guidance operation specifically includes: in, Indicates the first The attention-guided feature map in each residual block Indicates the first The input feature map of each residual block, where ⊙ represents the element-wise multiplication operator; The attention-guided feature maps are then optimized using a residual learning mechanism. The formula for calculating the residual block is as follows: in, Indicates the first Output feature maps of each residual block Indicates the first The first convolution operation of each residual block Indicates the first The second convolution operation on each residual block. Indicates the activation function; The deep residual generation network includes The cascaded processing of attention-guided residual blocks, after feature optimization of multiple residual blocks, finally performs upsampling operations through sub-pixel convolutional layers to enlarge the feature map size to the target 8K resolution; the specific upsampling factor of the sub-pixel convolution is: in, Indicates the upsampling factor. This indicates the height in pixels of the target 8K image. This indicates the width in pixels of the target 8K image. This indicates the height (in pixels) of the input low-resolution image. This indicates the width (in pixels) of the input low-resolution image.

7. A real-time 8K video perception super-resolution reconstruction system, characterized in that, include: Serialization module: It performs serialization parsing on the input low-resolution video stream through a spatiotemporal window mechanism, constructs a spatiotemporal context window containing the target frame and its adjacent frames, and obtains a set of low-resolution frame sequences. Feature extraction module: The trained shallow 3D convolutional neural network is used to extract multi-scale spatiotemporal features from the low-resolution frame sequence set to obtain spatiotemporal feature maps; Attention Module: The spatiotemporal feature map is processed by two parallel modules, motion compensation analysis and texture complexity analysis, to generate a perceptual attention map that reflects the importance of visual perception. Super-resolution reconstruction module: A deep residual generation network guided by a perceptual attention map is used to perform super-resolution reconstruction of spatiotemporal feature maps and output high-resolution 8K video frames; To achieve a real-time 8K video perception super-resolution reconstruction method as described in any one of claims 1-6.

Images