A real-time video quality optimization and enhancement method based on deep learning

Abstract

The present invention discloses a real-time video quality optimization and enhancement method based on deep learning, comprising the following steps: S1, collecting video streams and extracting continuous frames; S2, extracting motion region features using an optical flow estimation algorithm and generating a multimodal feature matrix; S3, performing normalization processing; S4, outputting denoising optimization frames using an adaptive multidimensional residual network and frequency domain analysis technology; S5, generating detail enhancement frames using a progressive generative adversarial network; S6, constructing an inter-frame relationship graph through local sensitive hashing, correcting frames using a dual-stream time-aware network, and optimizing frame sequences based on a dynamic time warping method, and outputting corrected frame sequences; S7, inputting the corrected frame sequence into a dynamic feedback optimization module to generate an optimized frame sequence; S8, recombining the optimized frame sequence into a continuous video and transmitting it to a target platform. The present invention comprehensively optimizes video clarity and temporal consistency through methods such as multimodal feature fusion, dynamic optimization, and progressive generative networks.

Description

Technical Field

[0001] The present invention relates to the field of video processing technology, and in particular to a real-time video quality optimization and enhancement method based on deep learning. Background Art

[0002] With the rapid development of video processing technology, video quality optimization and enhancement play an important role in many application scenarios, including but not limited to live video, intelligent monitoring, online conferencing, and virtual reality. However, traditional video quality optimization methods still have many shortcomings in practical applications and cannot meet the requirements of real-time and high-quality output in complex scenarios.

[0003] Existing methods for optimizing video quality mainly rely on classic image processing algorithms, such as filtering-based denoising methods, interpolation-based resolution enhancement techniques, and dynamic adjustment methods based on specific rules. These methods are effective in processing static scenes or simple dynamic features, but their performance often drops significantly when processing video streams containing complex textures, high dynamic changes, and multi-scene switching. Traditional filtering algorithms are usually unable to effectively distinguish between high-frequency texture details and high-frequency noise in video frames, resulting in loss of details or residual noise. In addition, these algorithms usually process according to fixed rules, lack adaptive capabilities, and are difficult to cope with dynamic changes in different scenes.

[0004] In recent years, with the rise of deep learning technology, image and video enhancement methods based on convolutional neural networks have made significant progress. For example, super-resolution networks are used to improve video resolution, and generative adversarial networks are used to generate more realistic video frames. These methods have excellent visual effects, but most methods only target specific tasks, such as denoising or resolution enhancement, and it is difficult to comprehensively handle multiple quality issues in videos. In addition, deep learning methods usually require large-scale computing resources, especially in real-time scenarios, where algorithm latency and computing efficiency issues have become important bottlenecks.

[0005] Inter-frame dynamic consistency optimization is another technical challenge that has attracted much attention in video processing. Existing methods usually rely on fixed time series modeling techniques to align video frames, but these methods are prone to frame skipping, artifacts and other problems when dealing with high-speed motion or complex scene switching, resulting in reduced smoothness of the output video. In addition, inter-frame consistency correction methods usually find it difficult to balance global dynamic consistency and smoothness of local details, further affecting the user experience.

[0006] In addition, the existing technology generally lacks the comprehensive utilization of multimodal features. Most methods only optimize videos based on a single mode, such as image texture or optical flow features, ignoring the rich semantic information and spatiotemporal characteristics in the video. This limitation makes it difficult for traditional methods to achieve efficient feature extraction and fusion when processing videos with multiple scenes, multiple targets or complex motion features, further limiting the effect of video quality optimization.

[0007] In existing dynamic optimization methods, parameter adjustment and model update are usually based on predefined rules or static strategies, which makes it difficult to adapt to changes in different scenarios in real time. User feedback and real-time video quality assessment are often not directly integrated into the optimization process, resulting in insufficient system responsiveness to dynamic environments. Although technologies such as reinforcement learning and incremental learning have been proposed, their application in video quality optimization is still in its early stages, and the optimization objectives are usually not diverse enough and have poor adaptability to complex scenarios.

[0008] Therefore, how to provide a real-time video quality optimization and enhancement method based on deep learning is an urgent problem that technicians in this field need to solve. Summary of the invention

[0009] One purpose of the present invention is to propose a real-time video quality optimization and enhancement method based on deep learning. The present invention comprehensively optimizes video clarity, detail fidelity and time consistency through multimodal feature fusion, adaptive dynamic network optimization, progressive generative adversarial network and reinforcement learning feedback mechanism. The optical flow feature and semantic information extraction, frequency domain analysis and dual-stream time perception network are used to significantly improve the visual performance in complex scenes. At the same time, the dynamic feedback optimization module is used to achieve real-time adaptation to dynamic changes in multiple scenes. It has the advantages of strong real-time performance, high optimization efficiency, accurate semantic expression and wide applicability in multiple scenes, providing reliable technical support for high-quality video processing.

[0010] A real-time video quality optimization and enhancement method based on deep learning according to an embodiment of the present invention comprises the following steps:

[0011] S1, receiving video streams in various input formats through a video acquisition device, extracting continuous frames from the video streams, and converting the color space of the frames to generate standardized video frames;

[0012] S2, inputting the standardized video frame into an optical flow estimation algorithm to extract motion area features, and performing multimodal feature analysis on the standardized video frame in combination with a semantic enhancement convolutional network to generate a multimodal feature matrix;

[0013] S3, normalizing the standardized video frame, adjusting the pixel value of the frame, and combining the multimodal feature matrix to generate a normalized video frame and optimize input data;

[0014] S4, inputting the normalized video frame into an adaptive multidimensional residual network, adjusting the network depth according to the optimized input data, and using frequency domain analysis technology to separate and denoise the high-frequency noise and low-frequency noise of the video frame, and outputting a denoised optimized frame;

[0015] S5, inputting the denoised optimized frame into a progressive generative adversarial network, optimizing the low-level texture features and high-level semantic features of the denoised optimized frame layer by layer, and outputting a detail enhanced frame;

[0016] S6. Input the detail enhanced frame into the inter-frame consistency correction module, construct an inter-frame relationship graph through local sensitive hashing, use a dual-stream time-aware network to correct the global dynamic consistency and local detail continuity between frames, and optimize the temporal consistency of the frame sequence based on the dynamic time warping method, and output the corrected frame sequence;

[0017] S7, inputting the corrected frame sequence into a dynamic feedback optimization module, dynamically adjusting the parameters of the generated network and optimizing the network weights according to the video quality evaluation index and user feedback information, and generating an optimized frame sequence;

[0018] S8. Recombining the optimized frame sequence into a continuous video, compressing the video through video encoding technology, and transmitting the video to the target platform in combination with a real-time streaming protocol.

[0019] Optionally, the S2 specifically includes:

[0020] S21, inputting the standardized video frame into the optical flow estimation algorithm, and Processing is performed, where (x, y) represents the pixel position coordinates, t represents the timestamp, and the motion information of the pixels between frames is calculated based on the optical flow constraint equation and Taylor series expansion. The constraint equation is defined as:

[0021] ;

[0022] Among them, u represents the pixel displacement of the optical flow in the horizontal direction, and v represents the pixel displacement of the optical flow in the vertical direction;

[0023] Estimate the motion of each pixel between two frames by solving constraint equations;

[0024] S22. Based on the optical flow constraint equation, the sparse optical flow tracking algorithm is used to calculate the optical flow field of the pixel points, and the optical flow feature matrix of the motion area is extracted:

[0025] ;

[0026] Among them, E(u,v) represents the energy function of the optical flow field, represents the pixel area to be processed, It represents the pixel value corresponding to the pixel displacement u and v at time t+1. represents the regularization parameter, Represents the gradient of optical flow in the horizontal direction. Represents the gradient of optical flow in the vertical direction;

[0027] S23, extracting low-level spatial features from the standardized video frame through a semantic enhancement convolutional network, taking the standardized video frame as input, and generating a low-level feature mapping matrix using a convolution operation;

[0028] S24, using a pre-trained deep convolutional network to perform high-level feature extraction on the low-level feature mapping matrix to generate a semantic feature matrix;

[0029] S25, performing regional weighting on the semantic feature matrix, assigning different weights based on the priorities of key regions, and using an adaptive mechanism to generate a regional enhancement feature matrix;

[0030] S26. Combining the optical flow feature matrix and the regional enhancement feature matrix, generating a multimodal feature matrix through a channel dimension splicing operation.

[0031] Optionally, the S4 specifically includes:

[0032] S41, inputting the normalized video frame into an adaptive multi-dimensional residual network, and dynamically adjusting the number of residual networks to control the network depth:

[0033] ;

[0034] in, represents the output features of the residual network, represents the input features of the residual network, Represents the residual mapping function:

[0035] ;

[0036] in, and Represents the weight matrix of the convolution kernel, ReLU represents the activation function, and represents the bias term;

[0037] S42, by counting the complexity features of the motion region within the frame, the complexity features include the number of edge textures, color gradient distribution and motion direction components, and calculating the feature complexity index:

[0038] ;

[0039] Among them, C represents the feature complexity index, Indicates the frame at pixel position The gradient of Represents the second-order gradient of the image in the horizontal direction, Represents the second-order gradient of the image in the vertical direction, Represents the first-order gradient of the image in the horizontal direction, represents the first-order gradient of the image in the vertical direction, Q represents the pixel area of ​​the frame; the network depth is controlled according to the size of the feature complexity index C;

[0040] S43, performing frequency domain analysis on the normalized video frame, using discrete Fourier transform to decompose the spatial domain features of the frame into frequency domain features, and calculating the frequency domain distribution of the frame:

[0041] ;

[0042] Where f(x,y) represents the pixel value of the input frame, F(u,v) represents the frequency component in the frequency domain, M represents the width of the frame, N represents the height of the frame, and a and b represent the frequency index;

[0043] S44, according to the frequency domain analysis result, the frequency component is divided into a high-frequency part and a low-frequency part, wherein the high-frequency part represents the texture details and edge information in the frame, and the low-frequency part represents the illumination and color components of the frame, and the boundary frequencies of high and low frequencies are set by a threshold segmentation method to extract high-frequency region features and low-frequency region features respectively;

[0044] S45, the high-frequency part and the low-frequency part are processed using independent denoising paths, the high-frequency part uses an edge enhancement filter to retain edge and texture features and suppress high-frequency noise; the low-frequency part uses a smoothing filter to reduce color distortion and optimize lighting information, and generates a preliminary denoised frame ;

[0045] S46, the output features of the residual network and preliminary denoised frames Perform pixel-by-pixel fusion and generate the final denoised and optimized frame in a weighted manner.

[0046] Optionally, the S5 specifically includes:

[0047] S51, inputting the denoised optimized frame into a progressive generative adversarial network, wherein the progressive generative adversarial network includes a generator and a discriminator, wherein the generator performs feature optimization on the input frame layer by layer to preliminarily generate an optimized frame;

[0048] S52, in the first stage of the generator, the low-level texture features of the denoising optimization frame are optimized, and the clarity of the local details in the frame is improved by enhancing the edge contrast and texture consistency in the frame. At the same time, the generator uses a convolutional neural network to extract and strengthen the boundary area of ​​the input frame, enhance the sharpness of the blurred edge area, and suppress the random noise in the low-level texture;

[0049] S53, in the second stage of the generator, performing intermediate feature association processing on the texture-optimized frame, optimizing the smooth transition between the texture and the spatial structure through local correlation analysis, and adjusting the uniformity of the local color distribution;

[0050] S54. In the third stage of the generator, based on the global perception mechanism, the high-level semantic features are optimized. The generator combines the semantic distribution of the whole frame, strengthens the features of important areas, highlights the key content through semantic weight distribution, optimizes the smoothness and coordination of the background area, enhances the coherence of the global semantic scene, and optimizes the feature contrast between the moving target area and the static background area through the dynamic scene adjustment strategy;

[0051] S55. Multi-scale feature evaluation is introduced in the discriminator to evaluate the generated frames layer by layer: in the first stage, the edge consistency and texture sharpness of low-level texture features are detected; in the second stage, the integrity of the mid-level feature structure is analyzed; in the third stage, the authenticity and global coordination of high-level semantic features are evaluated;

[0052] S56. According to the evaluation result of the discriminator, the generator is dynamically adjusted to further optimize the feature representation of the frame and finally output a detail enhanced frame.

[0053] Optionally, the S6 specifically includes:

[0054] S61, inputting the detail enhanced frame into an inter-frame consistency correction module, wherein the inter-frame consistency correction module includes an inter-frame relationship construction unit, a dynamic feature optimization unit and a time consistency correction unit;

[0055] S62, the inter-frame relationship building unit uses local sensitive hashing technology to perform similarity calculation on features of the video frames;

[0056] The feature vector of each frame is mapped to a low-dimensional hash space through a hash function family to generate a hash signature;

[0057] According to the hash signature of the frame, the feature similarity between frames is calculated and an inter-frame relationship graph G (V, E) is constructed, where V represents a set of frame nodes and E represents a set of similarity edges between frames. The inter-frame relationship graph is used to characterize the dynamic feature correlation between frames in a frame sequence;

[0058] S63, inputting the inter-frame relationship graph into a dual-stream time-aware network, wherein the dual-stream time-aware network is composed of a global stream and a local stream, wherein the global stream processes the long-distance dependencies in the frame sequence through a multi-head attention mechanism to generate a global dynamic feature optimization graph; and the local stream uses a convolution operation to capture the short-distance correlation between adjacent frames and correct the local jump and artifact problems of the frame sequence;

[0059] S64. In the global flow, based on the node weight distribution of the inter-frame relationship graph, the dynamic change characteristics of each frame are analyzed and the temporal correlation is optimized. The local flow uses feature alignment technology to enhance the local temporal continuity based on the short-distance feature differences between frames.

[0060] S65. Perform consistency evaluation on the frame sequence based on the dynamic time warping method, input the frame sequence and the target frame sequence, and quantify the temporal consistency by calculating the minimum cost of the dynamic matching path:

[0061] ;

[0062] Where D(S,T) represents the target function, S represents the frame sequence, and T represents the target frame sequence. Representation frame and frame The similarity cost of , K represents the dynamic path length;

[0063] S66. According to the dynamic time warping evaluation results, the temporal consistency of the frame sequence is corrected. The global flow adjusts the dynamic feature distribution of the frame to smooth long-distance changes. The local flow corrects the short-distance frame skipping problem, eliminates artifacts through feature alignment, and outputs the corrected frame sequence.

[0064] Optionally, the S7 specifically includes:

[0065] S71, inputting the corrected frame sequence into a dynamic feedback optimization module, and performing quality assessment on the corrected frame sequence, using peak signal-to-noise ratio and structural similarity indicators to assess the visual quality of the frame sequence, and analyzing inter-frame dynamic correlation characteristics and temporal consistency characteristics to generate an optimization target value;

[0066] S72. Construct a generative network parameter optimization model based on reinforcement learning, including state space, action space, reward function and strategy network;

[0067] S73, training the policy network based on the improved proximal policy optimization of the adaptive advantage function, and stabilizing the optimization process by limiting the amplitude of the policy update:

[0068] ;

[0069] in, represents the policy optimization objective function, Indicates that at time step The expected value of Indicates the current training step number, represents the strategy ratio, represents the adaptive advantage function, clip represents the clipping operation, Indicates the time adaptive clipping range, represents the entropy regularization weight, represents the policy entropy, Indicates the current strategy;

[0070] S74, dynamically adjusting the generation network parameters according to the output of the policy network, including adjusting the convolution kernel size, activation function threshold and learning rate distribution of different regions;

[0071] S75. In the optimized generation network, the feature representation of the user's attention area is enhanced, and finally an optimized frame sequence is generated.

[0072] The beneficial effects of the present invention are:

[0073] First, the present invention comprehensively extracts the optical flow features, low-level spatial features, and high-level semantic features of video frames through the combination of multimodal feature analysis and semantic enhancement convolutional networks, and effectively improves the feature expression of key areas through a regional weighting mechanism. The multimodal feature matrix generated by this method not only contains the motion information and texture details of the video frame, but also integrates the global scene characteristics of the semantic level, providing a richer and more accurate feature input for the subsequent optimization process, fundamentally making up for the problem of insufficient feature extraction in a single mode in existing methods.

[0074] Secondly, the present invention effectively solves the denoising and detail fidelity problems of traditional methods in complex scenes through an adaptive multi-dimensional residual network. The mechanism of adaptively adjusting the network depth ensures that the residual network can dynamically allocate computing resources when processing scenes of different complexities, which not only ensures processing efficiency but also enhances denoising optimization capabilities. At the same time, the high-frequency texture and low-frequency illumination in the video frame are separated through frequency domain analysis, and targeted denoising path processing is used respectively, which realizes the clear retention of texture features and the optimization of illumination information, making the output frame more refined in detail performance.

[0075] In addition, the progressive generative adversarial network optimizes the low-level texture features, mid-level correlation features, and high-level semantic features of the denoising optimized frame layer by layer, forming a multi-level optimization system from texture to semantics. By refining the target tasks of each layer of the generator and introducing a multi-scale discriminator for evaluation, the network can achieve comprehensive improvements in the texture consistency, spatial structure integrity, and semantic coherence of the video frame, thereby outputting detail-optimized frames with significantly enhanced visual effects.

[0076] The inter-frame consistency correction module of the present invention aims to solve the problem of poor correlation between dynamic features between frames. It proposes a method for constructing an inter-frame relationship graph based on local sensitive hashing, and combines a dual-stream time-aware network to optimize global dynamic consistency and local detail smoothness. The introduction of the dynamic time warping method further improves the temporal consistency of the frame sequence, providing a smoother and more realistic visual performance for complex motion scenes.

[0077] The dynamic feedback optimization module uses the reinforcement learning model to adjust the parameters of the generated network in real time by integrating video quality evaluation indicators and user feedback information, thereby achieving efficient adaptation to dynamic changes in multiple scenes. The reinforcement learning algorithm is based on an improved proximal policy optimization method, combined with time-adaptive clipping and entropy regularization strategies, which improves the stability of the strategy while enhancing the network's ability to handle complex scenes. In addition, the incremental learning mechanism ensures that the generated network can be adaptively optimized as the video scene changes, further improving the real-time and scalability of the algorithm. BRIEF DESCRIPTION OF THE DRAWINGS

[0078] The accompanying drawings are used to provide a further understanding of the present invention and constitute a part of the specification. Together with the embodiments of the present invention, they are used to explain the present invention and do not constitute a limitation of the present invention. In the accompanying drawings:

[0079] Figure 1 This is an overall flow chart of a real-time video quality optimization and enhancement method based on deep learning proposed by the present invention;

[0080] Figure 2 A schematic diagram of the progressive generative adversarial network structure of a deep learning-based real-time video quality optimization and enhancement method proposed in the present invention. DETAILED DESCRIPTION

[0081] The present invention will now be described in further detail with reference to the accompanying drawings. These drawings are simplified schematic diagrams, which only illustrate the basic structure of the present invention in a schematic manner, and therefore only show the components related to the present invention.

[0082] refer to Figure 1 and Figure 2 , a real-time video quality optimization and enhancement method based on deep learning, comprising the following steps:

[0083] S1, receiving video streams in various input formats through a video acquisition device, extracting continuous frames from the video streams, and converting the color space of the frames to generate standardized video frames;

[0084] S2, inputting the standardized video frame into an optical flow estimation algorithm to extract motion area features, and performing multimodal feature analysis on the standardized video frame in combination with a semantic enhancement convolutional network to generate a multimodal feature matrix;

[0085] S3, normalizing the standardized video frame, adjusting the pixel value of the frame, and combining the multimodal feature matrix to generate a normalized video frame and optimize input data;

[0086] S4, inputting the normalized video frame into an adaptive multidimensional residual network, adjusting the network depth according to the optimized input data, and using frequency domain analysis technology to separate and denoise the high-frequency noise and low-frequency noise of the video frame, and outputting a denoised optimized frame;

[0087] S5, inputting the denoised optimized frame into a progressive generative adversarial network, optimizing the low-level texture features and high-level semantic features of the denoised optimized frame layer by layer, and outputting a detail enhanced frame;

[0088] S6. Input the detail enhanced frame into the inter-frame consistency correction module, construct an inter-frame relationship graph through local sensitive hashing, use a dual-stream time-aware network to correct the global dynamic consistency and local detail continuity between frames, and optimize the temporal consistency of the frame sequence based on the dynamic time warping method, and output the corrected frame sequence;

[0089] S7, inputting the corrected frame sequence into a dynamic feedback optimization module, dynamically adjusting the parameters of the generated network and optimizing the network weights according to the video quality evaluation index and user feedback information, and generating an optimized frame sequence;

[0090] S8. Recombining the optimized frame sequence into a continuous video, compressing the video through video encoding technology, and transmitting the video to the target platform in combination with a real-time streaming protocol.

[0091] In this implementation, S2 specifically includes:

[0092] S21, inputting the standardized video frame into the optical flow estimation algorithm, and Processing is performed, where (x, y) represents the pixel position coordinates, t represents the timestamp, and the motion information of the pixels between frames is calculated based on the optical flow constraint equation and Taylor series expansion. The constraint equation is defined as:

[0093] ;

[0094] Estimate the motion of each pixel between two frames by solving constraint equations;

[0095] S22. Based on the optical flow constraint equation, the sparse optical flow tracking algorithm is used to calculate the optical flow field of the pixel points, and the optical flow feature matrix of the motion area is extracted:

[0096] ;

[0097] Among them, E(u,v) represents the energy function of the optical flow field, represents the pixel area to be processed, It represents the pixel value corresponding to the pixel displacement u and v at time t+1. represents the regularization parameter, Represents the gradient of optical flow in the horizontal direction. Represents the gradient of optical flow in the vertical direction;

[0098] S23, extracting low-level spatial features from the standardized video frame through a semantic enhancement convolutional network, taking the standardized video frame as input, and generating a low-level feature mapping matrix using a convolution operation;

[0099] S24, using a pre-trained deep convolutional network to perform high-level feature extraction on the low-level feature mapping matrix to generate a semantic feature matrix;

[0100] S25, performing regional weighting on the semantic feature matrix, assigning different weights based on the priorities of key regions, and using an adaptive mechanism to generate a regional enhancement feature matrix;

[0101] S26. Combining the optical flow feature matrix and the regional enhancement feature matrix, generating a multimodal feature matrix through a channel dimension splicing operation.

[0102] In this implementation, S4 specifically includes:

[0103] S41, inputting the normalized video frame into an adaptive multi-dimensional residual network, and dynamically adjusting the number of residual networks to control the network depth:

[0104] ;

[0105] in, represents the output features of the residual network, represents the input features of the residual network, Represents the residual mapping function:

[0106] ;

[0107] in, and Represents the weight matrix of the convolution kernel, ReLU represents the activation function, and represents the bias term;

[0108] S42, by counting the complexity features of the motion region within the frame, the complexity features include the number of edge textures, color gradient distribution and motion direction components, and calculating the feature complexity index:

[0109] ;

[0110] Among them, C represents the feature complexity index, Indicates the frame at pixel position The gradient of Represents the second-order gradient of the image in the horizontal direction, Represents the second-order gradient of the image in the vertical direction, Represents the first-order gradient of the image in the horizontal direction, represents the first-order gradient of the image in the vertical direction, Q represents the pixel area of ​​the frame; the network depth is controlled according to the size of the feature complexity index C;

[0111] S43, performing frequency domain analysis on the normalized video frame, using discrete Fourier transform to decompose the spatial domain features of the frame into frequency domain features, and calculating the frequency domain distribution of the frame:

[0112] ;

[0113] Where f(x,y) represents the pixel value of the input frame, F(u,v) represents the frequency component in the frequency domain, M represents the width of the frame, N represents the height of the frame, and a and b represent the frequency index;

[0114] S44, according to the frequency domain analysis result, the frequency component is divided into a high-frequency part and a low-frequency part, wherein the high-frequency part represents the texture details and edge information in the frame, and the low-frequency part represents the illumination and color components of the frame, and the boundary frequencies of high and low frequencies are set by a threshold segmentation method to extract high-frequency region features and low-frequency region features respectively;

[0115] S45, the high-frequency part and the low-frequency part are processed using independent denoising paths, the high-frequency part uses an edge enhancement filter to retain edge and texture features and suppress high-frequency noise; the low-frequency part uses a smoothing filter to reduce color distortion and optimize lighting information, and generates a preliminary denoised frame ;

[0116] S46, the output features of the residual network and preliminary denoised frames Perform pixel-by-pixel fusion and generate the final denoised and optimized frame in a weighted manner.

[0117] In this implementation manner, S5 specifically includes:

[0118] S51, inputting the denoised optimized frame into a progressive generative adversarial network, wherein the progressive generative adversarial network includes a generator and a discriminator, wherein the generator performs feature optimization on the input frame layer by layer to preliminarily generate an optimized frame;

[0119] S52, in the first stage of the generator, the low-level texture features of the denoising optimization frame are optimized, and the clarity of the local details in the frame is improved by enhancing the edge contrast and texture consistency in the frame. At the same time, the generator uses a convolutional neural network to extract and strengthen the boundary area of ​​the input frame, enhance the sharpness of the blurred edge area, and suppress the random noise in the low-level texture;

[0120] S53, in the second stage of the generator, performing intermediate feature association processing on the texture-optimized frame, optimizing the smooth transition between the texture and the spatial structure through local correlation analysis, and adjusting the uniformity of the local color distribution;

[0121] S54. In the third stage of the generator, based on the global perception mechanism, the high-level semantic features are optimized. The generator combines the semantic distribution of the whole frame, strengthens the features of important areas, highlights the key content through semantic weight distribution, optimizes the smoothness and coordination of the background area, enhances the coherence of the global semantic scene, and optimizes the feature contrast between the moving target area and the static background area through the dynamic scene adjustment strategy;

[0122] S55. Multi-scale feature evaluation is introduced in the discriminator to evaluate the generated frames layer by layer: in the first stage, the edge consistency and texture sharpness of low-level texture features are detected; in the second stage, the integrity of the mid-level feature structure is analyzed; in the third stage, the authenticity and global coordination of high-level semantic features are evaluated;

[0123] S56. According to the evaluation result of the discriminator, the generator is dynamically adjusted to further optimize the feature representation of the frame and finally output a detail enhanced frame.

[0124] In this implementation manner, S6 specifically includes:

[0125] S61, inputting the detail enhanced frame into an inter-frame consistency correction module, wherein the inter-frame consistency correction module includes an inter-frame relationship construction unit, a dynamic feature optimization unit and a time consistency correction unit;

[0126] S62, the inter-frame relationship building unit uses local sensitive hashing technology to perform similarity calculation on features of the video frames;

[0127] The feature vector of each frame is mapped to a low-dimensional hash space through a hash function family to generate a hash signature;

[0128] According to the hash signature of the frame, the feature similarity between frames is calculated and an inter-frame relationship graph G(V,E) is constructed, where V represents a set of frame nodes and E represents a set of similarity edges between frames. The inter-frame relationship graph is used to characterize the dynamic feature correlation between frames in a frame sequence;

[0129] S63, inputting the inter-frame relationship graph into a dual-stream time-aware network, wherein the dual-stream time-aware network is composed of a global stream and a local stream, wherein the global stream processes the long-distance dependencies in the frame sequence through a multi-head attention mechanism to generate a global dynamic feature optimization graph; and the local stream uses a convolution operation to capture the short-distance correlation between adjacent frames and correct the local jump and artifact problems of the frame sequence;

[0130] S64. In the global flow, based on the node weight distribution of the inter-frame relationship graph, the dynamic change characteristics of each frame are analyzed and the temporal correlation is optimized. The local flow uses feature alignment technology to enhance the local temporal continuity based on the short-distance feature differences between frames.

[0131] S65. Perform consistency evaluation on the frame sequence based on the dynamic time warping method, input the frame sequence and the target frame sequence, and quantify the temporal consistency by calculating the minimum cost of the dynamic matching path:

[0132] ;

[0133] Where D(S,T) represents the target function, S represents the frame sequence, and T represents the target frame sequence. Representation frame and frame The similarity cost of , K represents the dynamic path length;

[0134] S66. According to the dynamic time warping evaluation results, the temporal consistency of the frame sequence is corrected. The global flow adjusts the dynamic feature distribution of the frame to smooth long-distance changes. The local flow corrects the short-distance frame skipping problem, eliminates artifacts through feature alignment, and outputs the corrected frame sequence.

[0135] In this implementation manner, the S7 specifically includes:

[0136] S71, inputting the corrected frame sequence into a dynamic feedback optimization module, and performing quality assessment on the corrected frame sequence, using peak signal-to-noise ratio and structural similarity indicators to assess the visual quality of the frame sequence, and analyzing inter-frame dynamic correlation characteristics and temporal consistency characteristics to generate an optimization target value;

[0137] S72. Construct a generative network parameter optimization model based on reinforcement learning, including state space, action space, reward function and strategy network;

[0138] S73, training the policy network based on the improved proximal policy optimization of the adaptive advantage function, and stabilizing the optimization process by limiting the amplitude of the policy update:

[0139] ;

[0140] in, represents the policy optimization objective function, Indicates that at time step The expected value of Indicates the current training step number, represents the strategy ratio, represents the adaptive advantage function, clip represents the clipping operation, Indicates the time adaptive clipping range, represents the entropy regularization weight, represents the policy entropy, Indicates the current strategy;

[0141] S74, dynamically adjusting the generation network parameters according to the output of the policy network, including adjusting the convolution kernel size, activation function threshold and learning rate distribution of different regions;

[0142] S75. In the optimized generation network, the feature representation of the user's attention area is enhanced, and finally an optimized frame sequence is generated.

[0143] Example 1

[0144] In order to verify the feasibility of the present invention in implementation, the present invention is applied to a traffic monitoring system in a certain city. The monitoring camera is set at the intersection of the highway entrance and the main road. The video resolution collected is 1080p, the frame rate is 30 fps, and the monitoring time is the peak period of the day (7:00 to 9:00 am, 5:00 to 7:00 pm). The video data contains a variety of complex scenes, such as fast vehicle movement, low light conditions, and picture interference in rainy weather.

[0145] First, continuous video frames are extracted from the surveillance camera in real time, and the vehicle motion area is identified through the optical flow analysis algorithm. The vehicle and background areas are segmented and prioritized in combination with the semantic enhancement convolutional network. The extracted multimodal feature matrix contains the vehicle motion trajectory, key point information of the license plate area, and the semantic features of the background scene, providing rich input data for subsequent optimization steps.

[0146] Next, the normalized video frames are fed into an adaptive multidimensional residual network to separate the noise components in the video. Through frequency domain analysis technology, high-frequency noise at the edges of the vehicle is effectively removed while maintaining the vehicle texture details and the smoothness of the background lighting. The mechanism of dynamically adjusting the depth of the residual network ensures the processing accuracy of complex scenes and optimizes computational efficiency.

[0147] The denoised and optimized frames are then fed into a progressive generative adversarial network for detail enhancement. The network optimizes the edge sharpness, texture details, and semantic expression of the vehicle layer by layer. The generator significantly improves the overall quality of the image by combining low-level feature optimization (such as edge enhancement) and high-level feature enhancement (such as improving the clarity of license plate characters).

[0148] In addition, in order to solve the problem of frame skipping caused by high-speed vehicle motion, the corrected frame sequence is optimized by the inter-frame consistency correction module, which significantly improves the smoothness of the video. The dual-stream time-aware network achieves a balance between global dynamic changes and local detail continuity, ensuring the smoothness of the vehicle's motion trajectory while removing artifacts.

[0149] Finally, through the dynamic feedback optimization module, combined with real-time video quality evaluation indicators (PSNR and SSIM) and manual scoring feedback, the generated network parameters are dynamically adjusted to further optimize the video clarity and time consistency in complex scenes. The optimized video images are re-encoded and pushed to the monitoring center for real-time viewing by managers.

[0150] To verify the superiority of the present invention, we conducted a detailed data analysis of the videos before and after optimization, focusing on comparative tests from four dimensions: clarity index (PSNR), structural similarity index (SSIM), inter-frame temporal consistency score, and manual score. The following table shows the test results.

[0151] Table 1 Comparison of video clarity before and after optimization

[0152] ;

[0153] Table 2 Comparison of video structure similarity before and after optimization

[0154] ;

[0155] Table 3 Comparison of temporal consistency scores between frames

[0156] ;

[0157] Table 4 Comparison of manual scoring

[0158] ;

[0159] By analyzing the video quality data before and after the above optimization, it can be intuitively seen that the present invention has a significant effect on improving video quality in different scenarios.

[0160] First, from the comparison results of video clarity (PSNR) in Table 1 above, the present invention has achieved significant clarity improvement in all test scenarios. Among them, the PSNR of the high-speed vehicle motion scene increased from 28.3 dB to 36.7 dB, an increase of 29.7%; the PSNR of the night low-light scene increased from 24.8 dB to 33.1 dB, an increase of 33.5%; the PSNR of the rainy weather scene increased from 25.2 dB to 34.3 dB, an increase of 36.1%. The average PSNR of the comprehensive scene increased by 32.9%, which shows that the method of the present invention has strong adaptability to different complex scenes, especially in low light and bad weather conditions, the effect of denoising and detail optimization is particularly significant.

[0161] Secondly, from the comparison results of the video structural similarity (SSIM) in Table 2 above, it can be seen that the optimized video has been significantly improved in terms of intra-frame structural fidelity. The SSIM of the high-speed vehicle motion scene increased from 0.821 to 0.946, the SSIM of the night low-light scene increased from 0.751 to 0.923, and the SSIM of the rainy weather scene increased from 0.765 to 0.931. The average SSIM of the comprehensive scene increased from 0.779 to 0.933, an increase of nearly 20%. This result shows that the method of the present invention can effectively retain the structural information of the video frame and avoid the problem of detail loss caused by denoising and resolution improvement in traditional optimization methods.

[0162] Temporal consistency between frames is an important indicator that affects the smoothness of the video. From the temporal consistency scores between frames in Table 3 above, it can be seen that the temporal consistency of the optimized video has been greatly improved. The score of the high-speed vehicle motion scene increased from 7.1 to 9.4, an increase of 32.4%; the night low-light scene increased from 6.3 to 8.7, an increase of 38.1%; the rainy weather scene increased from 6.5 to 8.9, an increase of 36.9%. The temporal consistency score of the comprehensive scene increased by 36.4%, which shows that the inter-frame consistency correction module of the present invention can effectively solve the problems of frame skipping and artifacts, and significantly improve the dynamic smoothness of the video.

[0163] Finally, it can be seen from the manual scoring results in Table 4 that the improvement of the visual sensory effect of the present invention has been highly recognized by experts. The manual score of the high-speed vehicle motion scene increased from 7.2 points to 9.5 points, the score of the night low-light scene increased from 6.8 points to 9.3 points, the score of the rainy weather scene increased from 6.9 points to 9.2 points, and the score of the comprehensive scene increased from 7.0 points to 9.3 points. This shows that the present invention has achieved high-quality optimization effects in clarity, detail expression and overall visual perception.

[0164] It can be seen that the optimization method of the present invention performs well in multiple indicators such as clarity, structural fidelity, temporal consistency and visual perception, and can significantly improve the video quality in complex scenes, meeting the needs of high-demand application scenarios such as real-time monitoring and live video. These data fully demonstrate the significant advantages and wide applicability of the present invention in practical applications.

[0165] The above description is only a preferred specific implementation manner of the present invention, but the protection scope of the present invention is not limited thereto. Any technician familiar with the technical field can make equivalent replacements or changes according to the technical scheme and inventive concept of the present invention within the technical scope disclosed by the present invention, which should be covered by the protection scope of the present invention.

Claims

1. A real-time video quality optimization and enhancement method based on deep learning, characterized in that: The steps include: S1, receiving video streams in various input formats through a video acquisition device, extracting continuous frames from the video streams, and converting the color space of the frames to generate standardized video frames; S2, inputting the standardized video frame into an optical flow estimation algorithm to extract motion area features, and performing multimodal feature analysis on the standardized video frame in combination with a semantic enhancement convolutional network to generate a multimodal feature matrix; S3, normalizing the standardized video frame, adjusting the pixel value of the frame, and combining the multimodal feature matrix to generate a normalized video frame and optimize input data; S4, inputting the normalized video frame into an adaptive multidimensional residual network, adjusting the network depth according to the optimized input data, and using frequency domain analysis technology to separate and denoise the high-frequency noise and low-frequency noise of the video frame, and outputting a denoised optimized frame; S5, inputting the denoised optimized frame into a progressive generative adversarial network, optimizing the low-level texture features and high-level semantic features of the denoised optimized frame layer by layer, and outputting a detail enhanced frame; S6. Input the detail enhanced frame into the inter-frame consistency correction module, construct an inter-frame relationship graph through local sensitive hashing, use a dual-stream time-aware network to correct the global dynamic consistency and local detail continuity between frames, and optimize the temporal consistency of the frame sequence based on the dynamic time warping method, and output the corrected frame sequence; S7, inputting the corrected frame sequence into a dynamic feedback optimization module, dynamically adjusting the parameters of the generated network and optimizing the network weights according to the video quality evaluation index and user feedback information, and generating an optimized frame sequence; S8. Recombining the optimized frame sequence into a continuous video, compressing the video through video encoding technology, and transmitting the video to the target platform in combination with a real-time streaming protocol.

2. According to claim 1, a real-time video quality optimization and enhancement method based on deep learning is characterized in that: The S2 specifically includes: S21, inputting the standardized video frame into the optical flow estimation algorithm, and Processing is performed, where (x, y) represents the pixel position coordinates, t represents the timestamp, and the motion information of the pixels between frames is calculated based on the optical flow constraint equation and Taylor series expansion. The constraint equation is defined as: ； Among them, u represents the pixel displacement of the optical flow in the horizontal direction, and v represents the pixel displacement of the optical flow in the vertical direction; Estimate the motion of each pixel between two frames by solving constraint equations; S22. Based on the optical flow constraint equation, the sparse optical flow tracking algorithm is used to calculate the optical flow field of the pixel points, and the optical flow feature matrix of the motion area is extracted: ; Among them, E(u,v) represents the energy function of the optical flow field, represents the pixel area to be processed, It represents the pixel value corresponding to the pixel displacement u and v at time t+1. represents the regularization parameter, Represents the gradient of optical flow in the horizontal direction. Represents the gradient of optical flow in the vertical direction; S23, extracting low-level spatial features from the standardized video frame through a semantic enhancement convolutional network, taking the standardized video frame as input, and generating a low-level feature mapping matrix using a convolution operation; S24, using a pre-trained deep convolutional network to perform high-level feature extraction on the low-level feature mapping matrix to generate a semantic feature matrix; S25, performing regional weighting on the semantic feature matrix, assigning different weights based on the priorities of key regions, and using an adaptive mechanism to generate a regional enhancement feature matrix; S26. Combining the optical flow feature matrix and the regional enhancement feature matrix, generating a multimodal feature matrix through a channel dimension splicing operation.

3. The method for optimizing and enhancing real-time video quality based on deep learning according to claim 1, characterized in that: The S4 specifically includes: S41, inputting the normalized video frame into an adaptive multi-dimensional residual network, and dynamically adjusting the number of residual networks to control the network depth: ; in, represents the output features of the residual network, represents the input features of the residual network, Represents the residual mapping function: ; in, and Represents the weight matrix of the convolution kernel, ReLU represents the activation function, and represents the bias term; S42, by counting the complexity features of the motion region within the frame, the complexity features include the number of edge textures, color gradient distribution and motion direction components, and calculating the feature complexity index: ; Among them, C represents the feature complexity index, Indicates the frame at pixel position The gradient of Represents the second-order gradient of the image in the horizontal direction, Represents the second-order gradient of the image in the vertical direction, Represents the first-order gradient of the image in the horizontal direction, represents the first-order gradient of the image in the vertical direction, Q represents the pixel area of ​​the frame; the network depth is controlled according to the size of the feature complexity index C; S43, performing frequency domain analysis on the normalized video frame, using discrete Fourier transform to decompose the spatial domain features of the frame into frequency domain features, and calculating the frequency domain distribution of the frame: ; Where f(x,y) represents the pixel value of the input frame, F(u,v) represents the frequency component in the frequency domain, M represents the width of the frame, N represents the height of the frame, and a and b represent the frequency index; S44, according to the frequency domain analysis result, the frequency component is divided into a high-frequency part and a low-frequency part, wherein the high-frequency part represents the texture details and edge information in the frame, and the low-frequency part represents the illumination and color components of the frame, and the boundary frequencies of high and low frequencies are set by a threshold segmentation method to extract high-frequency region features and low-frequency region features respectively; S45, the high-frequency part and the low-frequency part are processed using independent denoising paths, the high-frequency part uses an edge enhancement filter to retain edge and texture features and suppress high-frequency noise; the low-frequency part uses a smoothing filter to reduce color distortion and optimize lighting information, and generates a preliminary denoised frame ; S46, the output features of the residual network and preliminary denoised frames Perform pixel-by-pixel fusion and generate the final denoised and optimized frame in a weighted manner.

4. The method for optimizing and enhancing real-time video quality based on deep learning according to claim 1, characterized in that: The S5 specifically includes: S51, inputting the denoised optimized frame into a progressive generative adversarial network, wherein the progressive generative adversarial network includes a generator and a discriminator, wherein the generator performs feature optimization on the input frame layer by layer to preliminarily generate an optimized frame; S52, in the first stage of the generator, the low-level texture features of the denoising optimization frame are optimized, and the clarity of the local details in the frame is improved by enhancing the edge contrast and texture consistency in the frame. At the same time, the generator uses a convolutional neural network to extract and strengthen the boundary area of ​​the input frame, enhance the sharpness of the blurred edge area, and suppress the random noise in the low-level texture; S53, in the second stage of the generator, performing intermediate feature association processing on the texture-optimized frame, optimizing the smooth transition between the texture and the spatial structure through local correlation analysis, and adjusting the uniformity of the local color distribution; S54. In the third stage of the generator, based on the global perception mechanism, the high-level semantic features are optimized. The generator combines the semantic distribution of the whole frame, strengthens the features of important areas, highlights the key content through semantic weight distribution, optimizes the smoothness and coordination of the background area, enhances the coherence of the global semantic scene, and optimizes the feature contrast between the moving target area and the static background area through the dynamic scene adjustment strategy; S55. Multi-scale feature evaluation is introduced in the discriminator to evaluate the generated frames layer by layer: in the first stage, the edge consistency and texture sharpness of low-level texture features are detected; in the second stage, the integrity of the mid-level feature structure is analyzed; in the third stage, the authenticity and global coordination of high-level semantic features are evaluated; S56. According to the evaluation result of the discriminator, the generator is dynamically adjusted to further optimize the feature representation of the frame and finally output a detail enhanced frame.

5. The method for optimizing and enhancing real-time video quality based on deep learning according to claim 1, characterized in that: The S6 specifically includes: S61, inputting the detail enhanced frame into an inter-frame consistency correction module, wherein the inter-frame consistency correction module includes an inter-frame relationship construction unit, a dynamic feature optimization unit and a time consistency correction unit; S62, the inter-frame relationship building unit uses local sensitive hashing technology to perform similarity calculation on features of the video frames; The feature vector of each frame is mapped to a low-dimensional hash space through a hash function family to generate a hash signature; According to the hash signature of the frame, the feature similarity between frames is calculated and an inter-frame relationship graph G (V, E) is constructed, where V represents a set of frame nodes and E represents a set of similarity edges between frames. The inter-frame relationship graph is used to characterize the dynamic feature correlation between frames in a frame sequence; S63, inputting the inter-frame relationship graph into a dual-stream time-aware network, wherein the dual-stream time-aware network is composed of a global stream and a local stream, wherein the global stream processes the long-distance dependencies in the frame sequence through a multi-head attention mechanism to generate a global dynamic feature optimization graph; and the local stream uses a convolution operation to capture the short-distance correlation between adjacent frames and correct the local jump and artifact problems of the frame sequence; S64. In the global flow, based on the node weight distribution of the inter-frame relationship graph, the dynamic change characteristics of each frame are analyzed and the temporal correlation is optimized. The local flow uses feature alignment technology to enhance the local temporal continuity based on the short-distance feature differences between frames. S65. Perform consistency evaluation on the frame sequence based on the dynamic time warping method, input the frame sequence and the target frame sequence, and quantify the temporal consistency by calculating the minimum cost of the dynamic matching path: ； Where D(S,T) represents the target function, S represents the frame sequence, and T represents the target frame sequence. Representation frame and frame The similarity cost of , K represents the dynamic path length; S66. According to the dynamic time warping evaluation results, the temporal consistency of the frame sequence is corrected. The global flow adjusts the dynamic feature distribution of the frame to smooth long-distance changes. The local flow corrects the short-distance frame skipping problem, eliminates artifacts through feature alignment, and outputs the corrected frame sequence.

6. The method for optimizing and enhancing real-time video quality based on deep learning according to claim 1, characterized in that: The S7 specifically includes: S71, inputting the corrected frame sequence into a dynamic feedback optimization module, and performing quality assessment on the corrected frame sequence, using peak signal-to-noise ratio and structural similarity indicators to assess the visual quality of the frame sequence, and analyzing inter-frame dynamic correlation characteristics and temporal consistency characteristics to generate an optimization target value; S72. Construct a generative network parameter optimization model based on reinforcement learning, including state space, action space, reward function and strategy network; S73, training the policy network based on the improved proximal policy optimization of the adaptive advantage function, and stabilizing the optimization process by limiting the amplitude of the policy update: ； in, represents the policy optimization objective function, Indicates that at time step The expected value of Indicates the current training step number, represents the strategy ratio, represents the adaptive advantage function, clip represents the clipping operation, Indicates the time adaptive clipping range, represents the entropy regularization weight, represents the policy entropy, Indicates the current strategy; S74, dynamically adjusting the generation network parameters according to the output of the policy network, including adjusting the convolution kernel size, activation function threshold and learning rate distribution of different regions; S75. In the optimized generation network, the feature representation of the user's attention area is enhanced, and finally an optimized frame sequence is generated.

Images