A deep learning-based live video quality enhancement method

Abstract

The present application relates to the technical field of image data processing, and more particularly to a live video quality enhancement method based on deep learning. Including: acquiring multiple images, inputting the multiple images into a pseudo flow field estimation subnetwork to obtain a motion vector field; based on the motion vector field, performing registration processing on the multiple images to obtain aligned multiple images, performing feature extraction and fusion processing on the aligned multiple images to obtain multiple image feature data; using an adaptive spatiotemporal feature adjustment and reconstruction enhancement algorithm to perform enhancement processing on the multiple image feature data to obtain enhanced image feature data; based on the enhanced image feature data, generating enhanced video frames. The technical problems of inaccurate processing of live video corresponding images and poor adaptability of the quality enhancement processing are solved.

Description

Technical Field

[0001] This invention relates to the field of image data processing technology, and in particular to a method for enhancing the image quality of live video based on deep learning. Background Technology

[0002] Live video quality enhancement technology is an important research direction in the current multimedia field. Especially in dynamic scenes, improving video quality while maintaining real-time performance remains a challenge. Existing image and video enhancement methods typically rely on traditional denoising, super-resolution reconstruction, or enhancement algorithms. However, these methods often face problems such as temporal consistency and dynamic scene detail recovery when processing live video streams. Traditional enhancement methods are often not refined enough in handling temporally changing dynamic scenes, easily leading to detail loss or artifact generation. Especially in high-speed motion or complex backgrounds, the image quality of live video often cannot be fully improved.

[0003] With the development of deep learning technology, novel techniques such as Convolutional Neural Networks (CNNs) and Generative Adversarial Networks (GANs) have been widely applied in the field of video processing. Deep learning methods can learn from large-scale data, effectively improving the ability to restore image details and training enhancement strategies more suitable for dynamic scenes. However, existing deep learning methods often fail to effectively consider the alignment issues between video frames and the temporal consistency of images when dealing with the spatiotemporal relationships between video frames. This often affects the video enhancement effect when processing high-speed motion or complex dynamic scenes.

[0004] Meanwhile, the existing technologies mentioned above also have technical problems such as inaccurate image processing of live video and poor adaptability in the image enhancement process. Summary of the Invention

[0005] This invention provides a deep learning-based method for enhancing the image quality of live videos, in order to solve the technical problems of inaccurate image processing of live videos and poor adaptability in the image quality enhancement process.

[0006] The present invention provides a method for enhancing the image quality of live video based on deep learning, which specifically includes the following technical solutions: A deep learning-based method for enhancing the image quality of live video includes the following steps: S1. Acquire multiple frames of images and input them into the pseudo flow field estimation sub-network to obtain the motion vector field; based on the motion vector field, perform registration processing on the multiple frames of images to obtain aligned multiple frames of images; perform feature extraction and fusion processing on the aligned multiple frames of images to obtain feature data of the multiple frames of images. S2. The adaptive spatiotemporal feature adjustment and reconstruction enhancement algorithm is used to enhance the feature data of multiple frames of images to obtain enhanced image feature data; based on the enhanced image feature data, enhanced video frames are generated.

[0007] Preferably, S1 specifically includes: A pseudo-flow field estimation subnetwork is introduced to adjust the image position by estimating the motion vector field between consecutive frames. The pseudo-flow field estimation subnetwork structure includes an optical flow estimation layer and a residual network layer.

[0008] Preferably, S1 specifically includes: The optical flow estimation layer generates a motion vector field by calculating pixel-level brightness and gradient information between different frames. This motion vector field is the one from the previous frame to the current frame, and the one from the current frame to the next frame. The motion vector field; the residual network layer is used to refine the motion vector field.

[0009] Preferably, S2 specifically includes: The adaptive spatiotemporal feature adjustment and reconstruction enhancement algorithm enhances multi-frame image feature data by combining spatiotemporal feature adjustment, dynamic fuzzy kernel processing, and nonlinear reconstruction.

[0010] Preferably, S2 specifically includes: In the implementation of the adaptive spatiotemporal feature adjustment and reconstruction enhancement algorithm, the feature data of multiple frames of images are preprocessed by normalization to obtain the normalized feature data of multiple frames of images; based on the normalized feature data of multiple frames of images, the spatiotemporal gradient difference is calculated to measure the changes of the image in time and space.

[0011] Preferably, S2 specifically includes: In the implementation of the adaptive spatiotemporal feature adjustment and reconstruction enhancement algorithm, weights are dynamically assigned to each frame of the image, and adaptive weights are dynamically calculated based on the spatiotemporal gradient differences. Based on the adaptive weights, the normalized multi-frame image feature data are weighted to obtain the weighted multi-frame image feature data.

[0012] Preferably, S2 specifically includes: In the implementation of the adaptive spatiotemporal feature adjustment and reconstruction enhancement algorithm, an adaptive fuzzy kernel is introduced to smooth the weighted multi-frame image feature data. During the smoothing process, the L2 norm of the weighted multi-frame image feature data is calculated to obtain the dynamic intensity value of the weighted multi-frame image feature data.

[0013] Preferably, S2 specifically includes: In the implementation of the adaptive spatiotemporal feature adjustment and reconstruction enhancement algorithm, based on the dynamic intensity value, the fuzzy kernel of the weighted multi-frame image feature data is calculated and a fuzzy kernel set is formed; by performing a convolution operation between the weighted multi-frame image feature data and the fuzzy kernel set, smoothed multi-frame image feature data is obtained.

[0014] Preferably, S2 specifically includes: In the implementation of the adaptive spatiotemporal feature adjustment and reconstruction enhancement algorithm, a nonlinear reconstruction function is introduced to reconstruct the smoothed multi-frame image feature data to obtain the final enhanced image feature data.

[0015] The beneficial effects of the technical solution of the present invention are: 1. By performing spatiotemporal feature adjustment, dynamic blur kernel processing, and nonlinear adaptive reconstruction on multiple frames of images, image details are effectively restored, especially in dynamic scenes. By enhancing image details through spatiotemporal weighting and nonlinear activation functions, the texture, edges, and other high-frequency information of the image can be finely improved, avoiding blurring and distortion in traditional methods.

[0016] 2. In the process of processing multi-frame images, the multi-frame images are input into the pseudo flow field estimation sub-network to obtain the motion vector field. Based on the motion vector field, the existing optical flow interpolation technology is used to register the multi-frame images, which ensures the temporal consistency of the images, effectively captures the motion information between the multi-frame images, avoids the image inconsistency caused by shooting angle, motion blur and dynamic scenes, and ensures the smooth and natural transition of the video during continuous playback. Attached Figure Description

[0017] Figure 1 This is a flowchart of a live video quality enhancement method based on deep learning, as described in this invention. Detailed Implementation

[0018] To further illustrate the technical means and effects adopted by the present invention to achieve its intended purpose, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0019] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains.

[0020] The following description, in conjunction with the accompanying drawings, details a specific scheme for a deep learning-based live video quality enhancement method provided by this invention.

[0021] See attached document Figure 1 The diagram illustrates a flowchart of a deep learning-based live video quality enhancement method according to an embodiment of the present invention. The method includes the following steps: S1. Acquire multiple frames of images, input the multiple frames of images into the pseudo flow field estimation sub-network to obtain the motion vector field; based on the motion vector field, perform registration processing on the multiple frames of images to obtain aligned multiple frames of images, and perform feature extraction and fusion processing on the aligned multiple frames of images to obtain multi-frame image feature data. The video capture device acquires three consecutive frames of live video footage, i.e., multiple frames, including the current frame. Previous frame image The next frame image The current frame is used as a reference frame, serving as the basis for registration of the remaining frames; the previous frame is the frame with an earlier timestamp; and the next frame is the frame with a later timestamp. The three consecutive frames of live video footage will differ spatially due to shooting angle, motion blur, and dynamic scenes. A pseudo-flow field estimation subnetwork is introduced to estimate the motion vector field between consecutive frames, adjusting the image position to ensure accurate alignment of images at different time points, thus providing a consistent foundation for subsequent feature extraction and fusion. The pseudo-flow field estimation subnetwork is based on a convolutional neural network and consists of multiple convolutional layers. Specifically, the pseudo-flow field estimation subnetwork structure includes an optical flow estimation layer and a residual network layer. The optical flow estimation layer generates a motion vector field by calculating the brightness difference between adjacent frames. The residual network layer further refines the motion vector field to reduce errors and improve the accuracy of motion estimation. The specific processing includes: Multiple frames of images The input is fed into the pseudo-flow field estimation subnetwork. In the optical flow estimation layer, pixel-level brightness and gradient information between different frames is calculated to generate a motion vector field, i.e., from the previous frame image. Up to the current frame image motion vector field and from the current frame image To the next frame motion vector field These two motion vector fields describe the motion from the previous frame image. Up to the current frame image and from the current frame image To the next frame Spatial displacement between frames. Using the above method, the pseudo-flow field estimation subnetwork can capture motion information between multiple frames.

[0022] Furthermore, based on the estimated motion vector field and Multiple frames of images are registered using existing optical flow interpolation techniques (such as bilinear interpolation). Specifically, this is achieved through motion vector fields. It can display the previous frame image. The pixels in the image are adjusted to the current frame based on the displacement indicated by the motion vector field. The corresponding position. Similarly, using the motion vector field. The current frame image Adjust the pixels in the image to the next frame. The corresponding position. Thus, through two optical flow interpolations, multiple frames of images... All images are aligned to have the same spatial structure, resulting in aligned multi-frame images. The aligned multi-frame images have the same spatial coordinate system. After registration, they can proceed to the subsequent feature extraction and fusion processing.

[0023] Furthermore, for the registered and aligned multi-frame images Feature extraction and fusion are performed. Specifically, a convolutional neural network (CNN) with shared weights is used to extract features from the aligned multi-frame images, resulting in multi-frame image feature data. .

[0024] S2. The adaptive spatiotemporal feature adjustment and reconstruction enhancement algorithm is used to enhance the feature data of multiple frames of images to obtain enhanced image feature data; based on the enhanced image feature data, enhanced video frames are generated.

[0025] An adaptive spatiotemporal feature adjustment and reconstruction enhancement algorithm is used to enhance the feature data of multi-frame images. This algorithm combines spatiotemporal feature adjustment, dynamic blur kernel processing, and nonlinear reconstruction to appropriately enhance the features of the image at different time points and spatial regions, while ensuring temporal consistency and detail recovery in dynamic scenes. The specific implementation process is as follows: Multi-frame image feature data Normalization preprocessing is performed to obtain normalized multi-frame image feature data. , , To measure the temporal and spatial changes of an image, spatiotemporal gradient differences are calculated, thereby dynamically assigning weights to each frame to enhance the spatiotemporal consistency and detail representation of the image. Adaptive weights for each frame are dynamically calculated based on these spatiotemporal gradient differences. Taking time as an example, the specific formula is as follows: , , in, It is the difference in spatiotemporal gradient, representing Differences in the temporal and spatial dimensions of a moment image; yes The gradient of the image at any given time represents the difference between the current frame and the frames before and after it in the time dimension, and is obtained through a time difference operation. yes The gradient of the image at any given time in the spatial direction represents the local feature changes of the current frame image and the previous and next frames in the spatial dimension. The spatial gradient reflects the changes in texture, edge, shape and other information of local regions in the image. It is calculated using convolution or discrete difference methods. These are adaptive weights, representing the adaptive importance weights of the current frame image during the image enhancement process; This is an adjustment factor that controls the spatiotemporal weighting sensitivity, used to control the degree of influence of spatiotemporal gradient differences on adaptive weight calculation. It is determined based on expert experience, with a reference value of [value missing]. ; It is a time index of multiple frames of images; Based on adaptive weights, the normalized multi-frame image feature data is weighted to obtain weighted multi-frame image feature data. ,Right now , , .

[0026] Furthermore, to reduce background noise and enhance details in dynamic regions, thereby avoiding excessive blurring and loss of detail and maintaining image sharpness, an adaptive blur kernel is introduced to smooth the weighted multi-frame image feature data. Specifically, this involves: Taking time as an example, the L2 norm is calculated on the weighted multi-frame image feature data to obtain... Time of the first Weighted image feature data dynamic intensity value ; , in, yes Time of the first Weighted image feature data; yes Time of the first Weighted image feature data; yes Time of the first Weighted image feature data; Based on the dynamic intensity value, calculate Time of the first Weighted image feature data Fuzzy kernel and constitute Fuzzy kernel set of time-weighted image feature data ,based on The formula for calculating the fuzzy kernel using the function is: , in, This is the blur kernel control factor, used to adjust the relationship between the blur kernel and the dynamic intensity value. It controls the degree to which changes in the dynamic intensity of the image region affect the blur kernel. It is obtained through experimental methods such as Bayesian optimization, with a reference range of [missing information]. ; This is the dynamic intensity threshold, used to control the dynamic intensity value at a reference point when determining the size of the fuzzy kernel. It is determined based on expert experience, with a reference value of [value missing]. .

[0027] Furthermore, by using weighted image feature data With fuzzy kernel set Perform convolution to obtain smoothed image feature data. .

[0028] Similarly, the smoothed multi-frame image feature data is obtained after the above processing. , , .

[0029] Furthermore, a nonlinear reconstruction function is introduced to reconstruct the smoothed multi-frame image feature data, which can more finely restore image details and suppress background noise, and serve as the final enhanced image feature data. The nonlinear reconstruction function is obtained based on feature fusion and nonlinear activation techniques in deep learning, and the specific formula is as follows: , in, It is the enhanced image feature data, representing the image feature data obtained through a nonlinear reconstruction process; This is a global adjustment factor used to adjust the intensity of image detail enhancement. It determines the strength of the final enhanced image feature data, ensuring that details are not over-enhanced during the restoration process, thereby avoiding artifacts or distortion. It should be set according to actual application requirements; a reference value is [value to be filled in]. ; It is a non-linear activation function used to enhance image details; It is an adaptive weight; yes Product, representing element-wise multiplication; It is an activation function used to normalize the L2 norm of smoothed multi-frame image feature data; yes Adjustment factor, used to control The function's response speed is determined based on expert experience, with a reference value of [value missing]. ; Through regulatory factors Dynamically adjusting the overall intensity of the image allows the feature recovery process to consider both the overall intensity of the image and avoid over-enhancing or losing important details. It performs nonlinear normalization on the intensity of image features; By smoothing image features, image details are restored while noise is suppressed; This indicates a weighted summation of smoothed image feature data from the current frame and the frames before and after it. This represents the result of weighted and nonlinear processing of the smoothed image feature data of the current frame and the frames before and after it.

[0030] Ultimately, based on the enhanced image feature data, enhanced video frames are generated using existing techniques such as generative adversarial networks or convolutional neural networks to achieve image quality enhancement.

[0031] In summary, a deep learning-based method for enhancing the image quality of live video has been developed.

[0032] The order of the embodiments is for illustrative purposes only and does not represent the superiority or inferiority of the embodiments. The processes depicted in the drawings do not necessarily require a specific or sequential order to achieve the desired result. In some embodiments, multitasking and parallel processing are possible or may be advantageous.

[0033] The various embodiments in this specification are described in a progressive manner. The same or similar parts between the various embodiments can be referred to each other. Each embodiment focuses on describing the differences from other embodiments.

[0034] The above embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention, and should all be included within the protection scope of the present invention.

Claims

1. A method for enhancing the image quality of live video based on deep learning, characterized in that, Includes the following steps: S1. Acquire three consecutive frames of live video from the video acquisition terminal as multi-frame images, including the current frame image, the previous frame image, and the next frame image. The current frame is used as a reference frame and serves as the registration basis for the other frames. Input the multi-frame images into the pseudo flow field estimation sub-network and adjust the image position by estimating the motion vector field between the previous and next frames. The pseudo-flow field estimation sub-network structure includes an optical flow estimation layer and a residual network layer. The optical flow estimation layer generates a motion vector field by calculating pixel-level brightness and gradient information between different frame images, namely, the motion vector field from the previous frame image to the current frame image, and the motion vector field from the current frame image to the next frame image. The motion vector field; the residual network layer is used to refine the motion vector field; based on the motion vector field, the multi-frame images are registered to obtain aligned multi-frame images, and the aligned multi-frame images are subjected to feature extraction and fusion processing to obtain multi-frame image feature data; S2. An adaptive spatiotemporal feature adjustment and reconstruction enhancement algorithm is used to enhance the feature data of multiple frames of images, resulting in enhanced image feature data. Based on the enhanced image feature data, enhanced video frames are generated. The adaptive spatiotemporal feature adjustment and reconstruction enhancement algorithm enhances the feature data of multiple frames of images by combining spatiotemporal feature adjustment, dynamic blur kernel processing, and nonlinear reconstruction. The specific implementation process includes: Multi-frame image feature data Normalization preprocessing was performed separately to obtain normalized multi-frame image feature data. , , Based on the normalized multi-frame image feature data, the spatiotemporal gradient differences are calculated respectively; The formula for calculating the spatiotemporal gradient difference at time t is as follows: It is the difference in spatiotemporal gradient, representing Differences in the temporal and spatial dimensions of a moment image; yes The gradient of the image at any given time represents the difference between the current frame and the frames before and after it in the time dimension, and is obtained through a time difference operation. yes The gradient of the image at any given time in the spatial direction represents the local feature changes of the current frame image compared to the previous and subsequent frames in the spatial dimension, and is calculated using convolution or discrete difference methods. The adaptive weights for each frame image are dynamically calculated based on the spatiotemporal gradient differences, as shown in the following formula: It is an adaptive weight; It is a regulating factor that controls the spatiotemporal weighted sensitivity; It is a time index of multiple frames of images; based on adaptive weights, the normalized feature data of multiple frames of images are weighted separately to obtain weighted feature data of multiple frames of images. ,in , , ; An adaptive blur kernel is introduced to smooth the weighted multi-frame image feature data. During the smoothing process, the L2 norm of the weighted multi-frame image feature data is calculated to obtain the dynamic intensity value of the weighted image feature data, as shown in the following formula: in, yes Time of the first Weighted image feature data The dynamic intensity value; yes Time of the first Weighted image feature data; yes Time of the first Weighted image feature data; yes Time of the first Weighted image feature data; Based on the dynamic intensity value, the blur kernel of the weighted multi-frame image feature data is calculated using the Sigmoid function. , , And respectively constitute the fuzzy kernel set of the weighted multi-frame image feature data. , , The formula for calculating the fuzzy kernel is: in, It is a fuzzy kernel control factor; yes Time of the first The dynamic intensity value of each weighted image feature data; It is the dynamic intensity threshold; By convolving the weighted multi-frame image feature data with their corresponding fuzzy kernel sets, smoothed multi-frame image feature data is obtained. , , A nonlinear reconstruction function is introduced to reconstruct the smoothed multi-frame image feature data, resulting in enhanced image feature data. The specific formula for the nonlinear reconstruction function is as follows: in, It is enhanced image feature data; It is a global regulating factor; It is a non-linear activation function; It is smoothed image feature data; yes product; It is an activation function; yes Regulatory factors.

Images