Method for optimizing quality of playing code stream in real time

Abstract

The invention discloses a method for optimizing the quality of a playing code stream in real time, and relates to the technical field of multimedia processing and streaming media transmission. The method comprises the following steps: in a code stream playing process, a preprocessing module decodes a real-time video frame, obtains noise intensity through a frequency domain analysis method, and performs noise grade division on the video frame; the de-noising module is used for de-noising the video frame through a lightweight de-noising method; the color evaluation module performs color richness grade division on the video frame through a color richness evaluation method; the super-resolution module carries out super-resolution processing on the video frames with different color richness levels through different super-resolution technologies; and the display module displays the video frame subjected to super-resolution processing on a user interface. According to the playing code stream quality real-time optimization method, the noise problem can be adaptively recognized, the resolution of the video is improved, the quality of the live video and the user experience are remarkably improved, meanwhile, low calculation complexity is kept, and the real-time processing capacity is ensured.

Description

Technical Field

[0001] The invention belongs to the technical field of multimedia processing and streaming media transmission, and relates to a method for real-time optimization of playing code stream quality. Background Art

[0002] With the rapid development of Internet technology and multimedia applications, video content has become an indispensable part of people's daily lives. However, during the video streaming transmission process, due to factors such as network bandwidth limitations, data compression algorithm losses, and differences in terminal device processing capabilities, users often encounter poor image quality when watching videos.

[0003] The problems in the prior art are mainly reflected in the following aspects: (1) Video compression and encoding: In order to adapt to different network environments and save bandwidth resources, videos usually need to be compressed to varying degrees. Currently widely used video compression standards such as H.264 / AVC and H.265 / HEVC reduce the amount of video data through a series of technologies such as predictive coding, transform coding, quantization, and entropy coding. Although these methods can effectively reduce bandwidth requirements, excessive compression often leads to visual distortion phenomena such as artifacts or block effects, ringing effects, and mosquito noise in the video signal, which seriously affects the viewing experience.

[0004] (2) Color information encoding: Some video encoding standards have limited support for color information. Especially at low bit rates, the color hierarchy in the video may become monotonous and poor, and the color details of the original scene cannot be truly reproduced. This not only reduces the overall viewing experience of the video, but also affects the understanding and appreciation of specific content. For example, in some professional fields such as art display and medical image analysis, color accuracy and richness are crucial.

[0005] (3) De-noising algorithm: In order to solve the image quality problems caused by compression or signal interference, the industry has developed a variety of denoising algorithms, including Wiener filtering, wavelet transform, and deep learning-based methods. These algorithms can remove noise elements in the picture to a certain extent. However, traditional denoising algorithms may destroy useful image features while removing noise, resulting in blurred edges or loss of other visual effects. In addition, complex denoising processes may also bring high computational costs.

[0006] (4) Super-resolution technology: Super-resolution technology aims to enhance the resolution of video frames through algorithms to provide clearer and more detailed images. Existing super-resolution methods range from traditional interpolation algorithms to the latest deep learning-based technologies. Although these technologies can significantly improve image clarity, they usually require strong computing power and long processing time, especially when running on mobile devices, which may cause playback delays or increase energy consumption.

[0007] In summary, although the existing technology has solved the problems of video quality and user experience to a certain extent, it still has some shortcomings. Therefore, it is particularly necessary to provide a technical solution that can effectively solve the existing video quality problems and flexibly adapt to the needs of various application scenarios. It is based on this background that the present invention proposes an innovative real-time optimization method for video playback bitstream quality, which provides users with better audio-visual enjoyment by comprehensively using multiple technologies such as frequency domain analysis, adaptive denoising, and adaptive super-resolution. Summary of the invention

[0008] In order to make up for the deficiencies of the prior art, the present invention provides a method for real-time optimization of playback stream quality. The method improves the image quality and user experience during video playback by preprocessing, noise removal, color evaluation and super-resolution processing of video frames. The method can effectively deal with problems including noise / compression distortion, insufficient color richness and poor visual experience. The present invention is implemented through the following technical solutions: The present invention provides a method for real-time optimization of playback stream quality, including the following modules, During the bitstream playback process, the preprocessing module decodes the real-time video frames, obtains the noise intensity through the frequency domain analysis method, divides the video frames into noise levels, and sends them to the denoising module; The denoising module denoises the video frames and sends them to the color evaluation module; The color assessment module assesses the color richness of the video and sends it to the super-resolution module; The super-resolution module performs adaptive super-resolution processing based on the color richness evaluation results of the video frame and sends it to the display module; The display module displays the video frames on the user interface.

[0009] Preferably, the preprocessing module performs a two-dimensional Fourier transform on the video frame to obtain a frequency domain diagram, and uses a frequency domain analysis method to extract the high-frequency components of the video frame; calculates the energy value of the high-frequency components of the video frame; if the energy value of the high-frequency part is higher than a preset threshold, it is determined that the current video frame has obvious noise; if the energy value of the high-frequency part is lower than a preset threshold, it is determined that the current video frame does not have obvious noise.

[0010] Preferably, the denoising module uses a denoising algorithm lightweight PMRID network model to denoise video frames with obvious noise; and does not process video frames without obvious noise.

[0011] Preferably, based on color entropy to judge color richness, the color evaluation module divides the video frame into multiple sub-blocks, each sub-block is set to a fixed size; for each divided sub-block, the color entropy value is calculated; if the color entropy value is higher than a preset threshold, the current sub-block is determined to be a high color entropy sub-block; if the color entropy value is lower than the preset threshold, the current sub-block is determined to be a low color entropy sub-block.

[0012] Preferably, the super-resolution module performs adaptive super-resolution processing on the sub-blocks of the video frame; for high color entropy sub-blocks, super-resolution processing is performed through the lightweight super-resolution model ESPCN network model method; for low color entropy sub-blocks, traditional super-resolution technology bicubic interpolation is performed for super-resolution processing; after processing, different sub-blocks are mapped and spliced ​​according to their original positions and sent to the display module.

[0013] The beneficial effects of the present invention are: The method of the present invention improves image quality and user experience by performing a series of adaptive processing on video frames during video playback, including but not limited to preprocessing, noise detection and denoising, color richness assessment, and adaptive super-resolution processing. It is suitable for online video live broadcast, on-demand services, and other application scenarios that require high-quality video transmission. By combining the user feedback mechanism and the dynamic adjustment strategy of the system resource status, the present invention can effectively balance the relationship between video quality improvement and resource consumption, and provide better visual enjoyment while ensuring smooth playback.

[0014] The beneficial effect of the present invention is that it combines advanced image processing technology and adaptive strategies, and can significantly improve the quality of video playback while maintaining efficient operation. In addition, the personalized service level is further improved by introducing user interaction, making the entire solution more in line with the needs of actual application scenarios. It is suitable for various multimedia transmission occasions such as online live broadcast and on-demand services. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] The present invention will be further described below in conjunction with the accompanying drawings; Figure 1 The figure is a flow chart of the method of the present invention. DETAILED DESCRIPTION

[0016] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the specific embodiments of the present invention are described in detail below in conjunction with the accompanying drawings. In the following description, many specific details are set forth to facilitate a full understanding of the present invention. However, the present invention can be implemented in many other ways different from those described herein, and those skilled in the art can make similar improvements without violating the connotation of the present invention, so the present invention is not limited by the specific embodiments disclosed below.

[0017] The accompanying drawings are a specific embodiment of a method for real-time optimization of playback stream quality of the present invention. The embodiment includes the following modules: During the video stream playback, the preprocessing module decodes the real-time video frames, obtains the noise intensity through the frequency domain analysis method, divides the video frames into noise levels, and sends them to the denoising module.

[0018] The preprocessing module performs a two-dimensional Fourier transform on the video frame to obtain a frequency domain image, and uses frequency domain analysis methods, such as a high-pass filter, to extract the high-frequency components of the video frame; calculates the energy value of the high-frequency components of the video frame; if the energy value of the high-frequency part is higher than the preset threshold (the empirical value is 30dB), the current video frame is determined to have obvious noise; if the energy value of the high-frequency part is lower than the preset threshold (the empirical value is 30dB), the current video frame is determined to have no obvious noise. This information will be sent to the denoising module.

[0019] The denoising module denoises the video frames and sends them to the color evaluation module.

[0020] After receiving the video frames, the denoising module uses the denoising algorithm lightweight PMRID network model to denoise the video frames with obvious noise; for video frames without obvious noise, no additional processing is performed. The processed video frames will be passed to the color evaluation module.

[0021] The color assessment module evaluates the color richness of the video and sends it to the super-resolution module.

[0022] Based on the color entropy to judge the color richness, the color evaluation module divides the video frame into multiple sub-blocks, each of which is set to a fixed size (the empirical value is 16×16); for each divided sub-block, the color entropy value is calculated; if the color entropy value is higher than the preset threshold (the empirical value is 3), the current sub-block is determined to be a high color entropy sub-block; if the color entropy value is lower than the preset threshold (the empirical value is 3), the current sub-block is determined to be a low color entropy sub-block. This evaluation result is then sent to the super-resolution module.

[0023] The super-resolution module performs adaptive super-resolution processing based on the color richness evaluation results of the video frame and sends it to the display module.

[0024] The super-resolution module performs adaptive super-resolution processing on the sub-blocks of the video frame according to the results of color evaluation; for high color entropy sub-blocks, super-resolution enhancement processing is performed through the lightweight super-resolution model ESPCN network model method; for low color entropy sub-blocks, super-resolution processing is performed using the traditional super-resolution technology of bicubic interpolation; after processing, different sub-blocks are mapped and spliced ​​according to their original positions and sent to the display module.

[0025] The display module displays the video frames on the user interface.

[0026] The method of the present invention also includes a user feedback mechanism. The system has a built-in user feedback collection function, allowing users to report problems encountered during viewing. The mechanism can collect user visual experience feedback, and adjust the noise threshold and color entropy threshold in the quality optimization process according to the feedback, and choose whether to turn on the denoising process and super-resolution process to achieve a better user experience.

[0027] The method of the present invention also includes resource-aware process management, that is, dynamically adjusting the processing flow. When it is detected that system resources are tight, such as the device temperature is too high or the CPU occupancy rate is too high, or the processing delay is long, the processing flow of the current video frame will be automatically simplified, and traditional super-resolution technology will be used for super-resolution processing to ensure smooth playback while minimizing performance overhead to reduce resource usage.

[0028] The system has a built-in user feedback collection function, allowing users to report viewing problems. Based on user feedback, the noise threshold and color entropy threshold can be dynamically adjusted, and certain processing steps can be selectively turned on or off to optimize the final effect.

[0029] The technical solution of the present invention is further described below in conjunction with FIG1 and embodiments.

[0030] In this embodiment, a method for real-time optimization of playback stream quality is provided, which improves the image quality and user experience during video playback through real-time monitoring and adaptive processing technology. The specific steps are as follows: Step 1: Get the video frames of the bitstream in real time. During the video bitstream playback, the system decodes the bitstream in the corresponding format and extracts the video frames to save them in the local buffer.

[0031] Step 2: The preprocessing module processes the video in the local buffer sequentially and uses the frequency domain analysis method to obtain the high-frequency energy value of the video frame to determine whether the current video frame has obvious noise.

[0032] The frequency domain analysis method steps include: Frequency domain conversion: First, the image is converted into a grayscale image; then, the grayscale image is converted from the spatial domain to the frequency domain through the fast Fourier transform method to obtain a two-dimensional frequency domain representation of the current video frame.

[0033] Obtaining the energy value of the high-frequency area: First, set the high-frequency mask. Considering that the video frame is usually rectangular in shape, a rectangular mask is used here to extract the high-frequency area. Assuming the image size is M × N, the upper left corner coordinates of the rectangular mask are set to (M / 4, N / 4), and the lower right corner coordinates are set to (3M / 4, 3N / 4). The high-frequency area is the part outside the mask; then, calculate the total energy of the high-frequency area, expressed in dB.

[0034] When the total energy in the high-frequency area is higher than the preset threshold (initially set to 30dB, which can be dynamically configured by the user), the current video frame is judged to have obvious noise and will be sent to the denoising module for denoising; when the total energy in the high-frequency area is lower than the preset threshold, the current video frame is judged to have no obvious noise.

[0035] Step 3: The denoising module uses a lightweight PMRID network model with denoising function to denoise video frames with obvious noise.

[0036] Calculate the total processing delay from step 1 to step 3. When the delay is lower than the preset threshold (the empirical value is 25ms), the color evaluation module will implement color evaluation and hand it over to the super-resolution module for super-resolution processing. When the delay is higher than the preset threshold (the empirical value is 25ms), the super-resolution module will go directly to step 6-2 and use the interpolation method to implement super-resolution processing.

[0037] Step 4: The color evaluation module divides the denoised video frame into sub-blocks, sets the fixed sub-block size to 16×16, and implements the color entropy calculation for each sub-block. The specific formula is as follows: Wherein, n represents the number of divided colors. In this embodiment, the grayscale value between 0 and 255 is equally divided into 16 segments, and each segment corresponds to a color. Represents the statistical probability of the i-th color appearing.

[0038] When the color entropy value is higher than a preset threshold (initially set to 3, which can be dynamically configured by the user), it is determined as a high color entropy sub-block; when the color entropy value is higher than the preset threshold, it is determined as a low color entropy sub-block.

[0039] Step 5: The super-resolution module traverses the video frame sub-blocks one by one. For high color entropy sub-blocks, execute step 6-1 and use the lightweight super-resolution model ESPCN to implement super-resolution processing; for low color entropy sub-blocks, execute step 6-2 and use the traditional interpolation method to implement super-resolution processing.

[0040] Step 7: Combine and splice the super-resolution sub-blocks according to their original positions.

[0041] Step 8: Display in the user interface.

Claims

1. A method for real-time optimization of playback stream quality, characterized in that: Includes the following modules, During the bitstream playback process, the preprocessing module decodes the real-time video frames, obtains the noise intensity through the frequency domain analysis method, divides the video frames into noise levels, and sends them to the denoising module; The denoising module denoises the video frames and sends them to the color evaluation module; The color assessment module assesses the color richness of the video and sends it to the super-resolution module; The super-resolution module performs adaptive super-resolution processing based on the color richness evaluation results of the video frame and sends it to the display module; The display module displays the video frames on the user interface.

2. The method for real-time optimization of playback stream quality according to claim 1, characterized in that: The preprocessing module performs a two-dimensional Fourier transform on the video frame to obtain a frequency domain diagram, and uses a frequency domain analysis method to extract the high-frequency components of the video frame; calculates the energy value of the high-frequency components of the video frame; if the energy value of the high-frequency part is higher than a preset threshold, it is determined that the current video frame has obvious noise; if the energy value of the high-frequency part is lower than the preset threshold, it is determined that the current video frame does not have obvious noise.

3. The method for real-time optimization of playback stream quality according to claim 1, characterized in that: The denoising module uses a denoising algorithm and a lightweight PMRID network model to denoise video frames with obvious noise; for video frames without obvious noise, no processing is performed.

4. The method for real-time optimization of playback stream quality according to claim 1, characterized in that: Based on the color entropy to judge the color richness, the color evaluation module divides the video frame into multiple sub-blocks, and each sub-block is set to a fixed size; for each divided sub-block, the color entropy value is calculated; if the color entropy value is higher than the preset threshold, the current sub-block is determined to be a high color entropy sub-block; if the color entropy value is lower than the preset threshold, the current sub-block is determined to be a low color entropy sub-block.

5. The method for real-time optimization of playback stream quality according to claim 1, characterized in that: The super-resolution module performs adaptive super-resolution processing on the sub-blocks of the video frame. For high color entropy sub-blocks, super-resolution processing is performed using the lightweight super-resolution model ESPCN network model method. For low color entropy sub-blocks, traditional super-resolution technology bicubic interpolation is used for super-resolution processing. After processing, different sub-blocks are mapped and spliced ​​according to their original positions and sent to the display module.

6. The method for real-time optimization of playback stream quality according to any one of claims 1 to 5, characterized in that: The method also includes a user feedback mechanism, which can collect user visual experience feedback, and adjust the noise threshold and color entropy threshold in the quality optimization process according to the feedback, and choose whether to turn on the denoising process and super-resolution process to achieve a better user experience.

7. The method for real-time optimization of playback stream quality according to any one of claims 1 to 5, characterized in that: The method also includes dynamically adjusting the processing flow. When it is detected that system resources are tight or the processing delay is long, the processing flow of the current video frame will be simplified and traditional super-resolution technology will be used for super-resolution processing to reduce resource usage.

8. The method for real-time optimization of playback stream quality according to claim 2, characterized in that: The steps of the frequency domain analysis method include: Frequency domain conversion: First, convert the image into a grayscale image; then, use the fast Fourier transform method to convert the grayscale image from the spatial domain to the frequency domain to obtain a two-dimensional frequency domain representation of the current video frame; Obtaining the energy value of the high-frequency area: Set a high-frequency mask and use a rectangular mask to extract the high-frequency area. Assuming the image size is M×N, set the coordinates of the upper left corner of the rectangular mask to (M / 4, N / 4) and the coordinates of the lower right corner to (3M / 4,3N / 4). The high-frequency area is the part outside the mask. Then, calculate the total energy of the high-frequency area and express it in dB. When the total energy in the high-frequency area is higher than the preset threshold, the current video frame is determined to have obvious noise and will be sent to the denoising module for denoising; when the total energy in the high-frequency area is lower than the preset threshold, the current video frame is determined to have no obvious noise.

9. The method for real-time optimization of playback stream quality according to claim 4, characterized in that: The color evaluation module divides the denoised video frame into sub-blocks, sets the fixed sub-block size to 16×16, and performs color entropy calculation for each sub-block. The specific formula is as follows: Where n represents the number of divided colors. Represents the statistical probability of the i-th color appearing; When the color entropy value is higher than the preset threshold, it is determined as a high color entropy sub-block; when the color entropy value is higher than the preset threshold, it is determined as a low color entropy sub-block.

Images