A video super-resolution method and system based on frame type differentiation processing
By employing a master-child dual-process architecture and frame type differentiation processing, combined with heterogeneous queues and luminance statistical regression, the problems of quality and real-time performance, luminance jumps, and deadlocks in real-time video transcoding were solved, achieving high-quality and stable video output.
Patent Information
- Application Number
- CN202611016919.9
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2026-07-09
- Publication Date
- 2026-11-10
AI Technical Summary
Existing technologies struggle to balance video quality and real-time performance in real-time video transcoding. Differentiated processing leads to abrupt changes in screen brightness and deadlocks in cross-process communication. High module coupling and large synchronization overhead prevent effective solutions to issues related to quality, real-time performance, temporal consistency, and version coupling.
It adopts a master-child dual-process architecture, and processes the frames differently according to the frame type. I-frames use deep neural network super-resolution, while P/B frames use fast upsampling. It also eliminates brightness jumps through heterogeneous queue design and brightness statistical regression, and uses a hybrid synchronization mechanism of atomic sequence numbers and named semaphores for cross-process communication.
While ensuring real-time performance, it significantly improves video quality, eliminates screen flicker, avoids deadlock, reduces module coupling, and enhances system stability and throughput.
Smart Images

Figure CN123075205A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of digital video processing technology, specifically to a super-resolution processing method and system for real-time video transcoding or video enhancement. More specifically, this application relates to a method for selecting different super-resolution processing paths based on video frame types (intra-predictive frames, forward-predictive frames, bi-directional predictive frames), and a system that uses luminance statistical regression to eliminate brightness jumps caused by differential processing. This application also relates to a video transcoding system employing a dual-process isolation architecture and a shared memory communication mechanism based on atomic sequence numbers. Background Technology
[0002] Video super-resolution (VSR) technology aims to reconstruct low-resolution videos into high-resolution videos, and is widely used in scenarios such as video transcoding, streaming media enhancement, and restoration of old videos. Based on different processing methods, existing super-resolution technologies can be mainly divided into the following two categories: Methods based on deep neural networks (DNNs) (such as RealESRGAN) can generate high-quality, detailed super-resolution images, but they have high computational complexity and the processing time for a single frame is usually in the hundreds of milliseconds, making it difficult to meet the throughput requirements of real-time transcoding (such as 25fps~50fps).
[0003] Traditional interpolation methods (such as bicubic interpolation) are fast and easy to meet real-time requirements, but the output image details are blurry and the subjective quality is significantly lower than that of DNN methods.
[0004] In real-time video transcoding scenarios, simply applying the two types of techniques mentioned above to video sequences will lead to a series of specific technical problems, for which existing technologies have failed to provide effective solutions. First, quality and real-time performance are difficult to achieve simultaneously. In existing technologies, uniformly applying DNN super-resolution to all frames results in high quality but fails to meet real-time requirements; uniformly using traditional interpolation results in fast speed but poor output quality. Existing technologies lack a differentiated processing scheme that improves the quality of keyframes while ensuring real-time performance. Second, the differentiated processing introduces periodic brightness jumps. When different super-resolution methods are used for I-frames and P / B frames, the DNN alters the image's brightness statistics, while the interpolation method largely maintains the original statistics. This results in brightness differences between I-frames and subsequent P / B frames within each GOP, which appears as periodic screen flickering to the human eye. While existing global brightness alignment schemes can eliminate these jumps, they disrupt the original temporal brightness variations in the video (such as fade-in / fade-out and flash effects), failing to balance "eliminating jumps" with "preserving temporal variations." Finally, regarding audio / video synchronization and deadlock in cross-process communication, in a real-time transcoding system employing a dual-process isolation architecture, the main process passes audio and video frames to child processes via shared memory. Existing technologies suffer from the following drawbacks: Homogeneous buffering strategies: Using the same type of buffer queue for both video and audio fails to consider the fundamental differences in data volume and latency sensitivity between the two. Deadlock caused by super-resolution startup delay: The DNN super-resolution of the first I-frame takes a long time (up to several seconds). If the audio buffer capacity is insufficient, the main process blocks when the audio queue is full, while the child process has not yet started consuming, resulting in a deadlock. Unreliable EOF propagation: The global flag method presents race conditions in multi-threaded cross-process scenarios, potentially leading to data truncation or the inability of child processes to gracefully exit. High synchronization overhead: Shared memory access based on traditional mutexes requires frequent kernel traps, impacting real-time throughput. High module coupling: In a single-process architecture, modules share the same address space, leading to severe resource contention, exception propagation, and version coupling between modules.
[0005] Therefore, existing technologies face multiple contradictions in real-time transcoding systems, including quality, real-time performance, temporal consistency, version coupling, and cross-process communication reliability, and a systematic solution is urgently needed. Summary of the Invention
[0006] The purpose of this invention is to provide a video super-resolution method and system based on frame type differentiation processing, addressing the aforementioned problems. In real-time video transcoding systems, the requirements for computing resources, processing latency, stability, and software stack vary significantly among decoding, super-resolution, encoding, and encapsulation stages. To reduce resource contention, anomaly propagation, and version coupling between modules, this application adopts a master-child dual-process architecture: the master process is responsible for video decoding, differentiated super-resolution processing, and audio preprocessing, while the child process is responsible for video encoding, audio encoding, and output encapsulation. The two processes achieve low-latency, high-throughput audio and video frame transmission through a shared memory ring and named semaphores, and the heterogeneous queue design absorbs the startup latency of the first I-frame DNN super-resolution, fundamentally avoiding cross-process deadlock. This process-level isolation architecture allows the master and child processes to use heterogeneous software stacks and runtime environments, improving the system's modularity and operational stability.
[0007] The technical solution of the present invention is as follows: A video super-resolution method based on frame type differentiation processing includes the following steps: Identify the frame type of the decoded video frames. Frame types include... frame, Frames and frame; If the current frame is Frame, then the The frame performs super-resolution processing based on a deep neural network, and then performs super-resolution... The luminance statistics of the frame regressed to this Brightness statistics of the original input frame; If the current frame is Frame or Frame, then the Frame or The frame undergoes fast upsampling processing, but no luminance statistical alignment is performed.
[0008] By selecting different super-resolution paths based on frame type, DNN super-resolution is used for I-frames to ensure keyframe quality, while fast upsampling is used for P / B frames to ensure processing speed. This significantly improves the overall subjective quality of the output video while meeting real-time transcoding throughput requirements. By performing luminance statistical regression on I-frames and not performing luminance alignment on P / B frames, frames within the entire GOP are placed in a uniform luminance space, eliminating periodic luminance jumps caused by different processing methods, while preserving the original luminance characteristics of P / B frames and avoiding distortion of temporal luminance variations in the video.
[0009] Furthermore, the super-resolution The brightness statistics of the frame are regressed to the brightness statistics of the original input frame, specifically including: Extract raw input Average brightness of the Y channel of a frame and brightness standard deviation ; After extracting the super-resolution Current average brightness of the Y channel of the frame and current brightness standard deviation ; Calculate the brightness scaling factor and brightness offset ; After exceeding the score Y channel pixel value of the frame Perform a linear transformation: The Y channel values after brightness regression were obtained. .
[0010] It can accurately pull the first-order luminance statistics (mean) and second-order luminance statistics (standard deviation) of the super-resolution I-frame back to the level of the original input frame, thereby eliminating the luminance difference between the I-frame and the P / B frame and achieving luminance consistency within the GOP.
[0011] Furthermore, the extraction of the original input Frame luminance mean and brightness standard deviation The process is executed via an asynchronous thread, which runs concurrently before or during the deep neural network super-resolution process. The luminance statistics are cached in a global structure protected by a mutex lock. The structure contains the luminance mean, luminance standard deviation, and validity flag. After the deep neural network super-resolution is completed, the main thread waits for the asynchronous statistics task to complete, reads the statistics from the global structure, and performs luminance regression when the validity flag is true.
[0012] By extracting raw brightness statistics through an asynchronous background thread, statistical calculations and DNN super-resolution inference can be executed concurrently, avoiding blocking the main encoding path and further reducing the additional overhead of I-frame processing, which helps maintain the frame rate stability of real-time transcoding.
[0013] Furthermore, when calculating the brightness scaling factor (scale), if If the value is less than the preset minimum tolerance value, then the minimum tolerance value will be used instead. Perform the calculation.
[0014] By replacing src_std with a preset minimum tolerance value for calculation, the luminance regression process can be stably executed in any texture area, avoiding abnormal pixels in the output frame.
[0015] Furthermore, the aforementioned Frame or The frame undergoes fast upsampling processing, specifically including: directly upscaling the frame resolution to the target resolution in the YUV color space using bicubic interpolation or Lanczos interpolation, and then performing unsharpened mask sharpening on the upscaled Y channel, specifically including: Use Gaussian blur kernel , Blurring the Y channel yields the low-frequency components. ; pass Calculate the sharpened The passage, in which This is the sharpening intensity factor, typically ranging from 0.2 to 0.5.
[0016] Performing bicubic interpolation directly in the YUV space avoids the overhead of color space conversion and ensures extremely low latency in P / B frame processing. Combined with unsharpened mask sharpening in the Y channel, it can compensate for the detail blur introduced by interpolation to a certain extent and improve the subjective sharpness of P / B frames, thereby maximizing output quality while ensuring real-time performance.
[0017] Furthermore, the aforementioned The frame performs super-resolution processing based on a deep neural network, and also includes: [the following is unclear and requires further context: "to..."] After frames are converted from the YUV color space to the RGB color space, they are input into a deep neural network. The high-resolution RGB image output by the deep neural network is then converted back to the YUV color space, resulting in preliminary super-resolution. frame.
[0018] Through the complete conversion link of YUV→RGB→DNN→RGB→YUV, I-frames can be adapted to pre-trained DNN super-resolution models (such as RealESRGAN, which is usually trained in RGB space), thereby obtaining high-quality detail reconstruction results. This ensures that the I-frame, as the reference frame of GOP, has the best visual quality and provides a high-quality reference for the prediction of subsequent P / B frames.
[0019] Furthermore, it also includes the step of using a master-child dual-process architecture for cross-process communication: setting up a circular queue of video frames and a FIFO queue of audio samples on the master process side, and setting up corresponding circular queues of video frames and FIFO queues of audio samples on the child process side; wherein, the capacity of the audio sample FIFO queue is preset to be able to absorb the startup delay required for the child process to perform deep neural network super-resolution processing on the first I-frame; the capacity of the audio sample FIFO queue is set according to the product of the maximum super-resolution time of the first I-frame and the audio sampling rate, and a safety factor is set, specifically: , in For audio sampling rate, For a single The time required for DNN super-resolution of frames. This represents the maximum startup latency that the video SR path may cause; taking into account the first frame. The time consumption of DNN super-resolution of frames is affected by model complexity, input resolution, and GPU computing power. A safety factor should be set. (Typically, 5 to 10 are used), then: .
[0020] The video uses a frame-level circular queue, while the audio uses a sample-level FIFO queue. This matches the essential differences between the two types of data in terms of data volume and latency sensitivity, avoiding blocking or memory waste caused by homogeneous design.
[0021] This application also includes a video super-resolution system based on frame type differentiation processing, comprising: The decoding module is used to decode the input video stream and output the original frame sequence; The frame type identification module is used to identify the frame type of the decoded video frames. Frame types include... frame, Frames and frame; The differentiated super-resolution processing module is used to select the corresponding processing path based on the frame type: if it is If it is a frame, then perform super-resolution processing based on a deep neural network; if it is... Frame or For frames, fast upsampling is then performed; The luminance statistical regression module is used to convert the light after super-resolution by the deep neural network. The luminance statistics of the frame regressed to this Brightness statistics of the original input frame, and for Frames and Frames are not luminance-statistical aligned.
[0022] Furthermore, it includes an inter-process communication module, which adopts a master-child dual-process architecture. The synchronization mechanism is a hybrid strategy combining a slot state machine based on atomic sequence numbers and named semaphore event notification. The master process and the child process transmit frame data through a shared memory ring. Each slot of this shared memory ring contains a named semaphore and an atomic sequence number field seq. The write and read ends use named semaphores to complete blocking wait and wake-up notification, thereby avoiding high CPU usage caused by pure polling. The atomic sequence number serves as a secondary check for data validity, preventing data inconsistency caused by semaphore mis-triggering or memory order issues. The initial value of the sequence number field seq is an even number. The write end executes: waits for seq to be even, writes data, and then increments seq by 1 to make it odd. The read end executes: waits for seq to be odd, reads data, and then increments seq by 1 to make it even. If the shared slot data is not ready, it waits in a timed finite number of polling cycles. After the timeout, it releases the write semaphore and returns an error to avoid indefinite waiting.
[0023] Furthermore, the cross-process communication module also includes an end-of-process propagation unit: when decoding ends, a sentinel frame with a preset flag value in the opaque field is pushed onto the video queue and audio queue respectively. This sentinel frame is transmitted to the child process along with the data stream; after the child process detects the dual-channel sentinel frames for video and audio, it performs a final refresh and exits. The video stream and audio stream each maintain independent EOF flags. If the input file contains only a single stream (such as pure video or pure audio), the EOF flag corresponding to that stream is directly set to true during system initialization, without participating in dual-channel waiting.
[0024] It also includes: caching the video packets and audio packets after differential super-resolution processing and encoding to separate video packet queues and audio packet queues; selecting the next packet to be sent from the two queues according to the display timestamp of each packet; calculating the target transmission time based on the display timestamp, and outputting the packet to the multiplexer when the target transmission time is reached, so that the output rhythm is aligned with the original audio and video timestamps.
[0025] Compared with existing technologies, the advantages of this invention are: 1. To address the issue of the trade-off between quality and real-time performance in existing technologies, this application employs a differentiated frame processing strategy: Deep neural network (DNN) super-resolution is used for I-frames, which serve as GOP reference frames, to ensure the reconstruction quality of keyframes; fast upsampling (such as bicubic interpolation) is used for P-frames and B-frames to significantly reduce computational load. Taking 1080p to 4K super-resolution as an example, the processing time for P / B frames is approximately 1 / 50 to 1 / 100 of that of the DNN method, and the overall processing capability meets the requirements of real-time transcoding. This meets the throughput requirements. Therefore, while ensuring real-time performance, it significantly improves the overall subjective quality of the output video.
[0026] 2. To address the screen flickering issue introduced by differential processing, this application performs luminance statistical regression on the DNN super-resolution output of the I-frame, restoring its mean and standard deviation to the level of the original input I-frame, while the P / B frames retain their original luminance statistics without alignment. As a result, all frames within the entire GOP are in a unified luminance space, eliminating the periodic luminance jumps caused by I-frame luminance shifts at the start of each GOP, and avoiding screen flickering perceptible to the human eye.
[0027] 3. Unlike existing global brightness alignment schemes, this application uses the brightness statistics of the original input I-frame as the reference for brightness regression, rather than a fixed reference frame or the average value of the entire video. Therefore, the original temporal brightness changes of the video (such as fade-in, fade-out, flash effects, and brightness gradations during scene transitions) are completely preserved and will not be compressed or distorted by the brightness alignment operation, ensuring that the artistic intent of the output video is consistent with the original.
[0028] 4. To address the communication shortcomings of dual-process isolation architecture, this application provides a complete solution: Heterogeneous dual-queue design: A frame-level circular queue is used for video, and a sample-level FIFO queue is used for audio. This matches the essential differences between the two types of data in terms of data volume and latency sensitivity, avoiding blocking or memory waste caused by homogeneous design.
[0029] SR-aware audio buffer: The capacity of the audio FIFO is designed to absorb the video path startup delay caused by the first frame DNN super-resolution, fundamentally eliminating the cross-process deadlock caused by the main process being blocked due to the audio buffer being full and the child process not yet consuming the audio.
[0030] Atomic Sequence SHM Protocol: A hybrid synchronization mechanism based on state machine of atomic sequence number and named semaphore event notification: Semaphore is responsible for efficient process blocking / wake-up, and atomic sequence number (parity check) is responsible for observable verification of data state. The separation of responsibilities avoids global lock contention and busy waiting, and ensures the reliability of cross-process data transmission.
[0031] Opaque Sentinel Frame EOF Propagation: Instead of using a global flag, the EOF signal is propagated along with the data stream using a tagged sentinel frame. This eliminates race conditions in multi-threaded, multi-queue, and cross-process scenarios, ensuring that child processes never miss the termination signal under any circumstances and can gracefully exit and output the file completely.
[0032] Packet-level PTS sorting for smooth output: Video and audio packets, after differential super-resolution processing and encoding, are buffered into separate video and audio packet queues, respectively. The next packet to be sent is selected from each queue based on its displayed timestamp. The target transmission time is calculated based on the displayed timestamp, and the packet is output to the multiplexer when the target transmission time is reached, ensuring the output rhythm aligns with the original audio and video timestamps. For SRT and other TSBPD protocols, the multiplexer receives packets at a constant rhythm, allowing the sender to inject packets into the network according to the correct time base. Attached Figure Description
[0033] Figure 1 This is the overall system architecture diagram for this application.
[0034] Figure 2 This is a schematic diagram of the processing flow of the method in this application.
[0035] Figure 3 This is a diagram illustrating the contrast between brightness jumps.
[0036] Figure 4 This is a diagram of a dual-queue architecture for the master and child processes.
[0037] Figure 5 This is a timing diagram of the data flow in a dual-process environment.
[0038] Figure 6 This is a schematic diagram of block-based computation for super-resolution inference.
[0039] Figure 7 A comparison chart of transcoding quality for a football match scene.
[0040] Figure 8 This is a comparison chart of transcoding quality in a news studio setting.
[0041] Figure 9 This is a comparison of the transcoding efficiency of the method in this application and the full-frame DNN super-resolution method.
[0042] Figure 10 This is a comparison chart of inter-process communication efficiency.
[0043] Figure 11 This is a comparison chart showing the performance of the output rectifier unit. Detailed Implementation
[0044] It should be noted that relational terms such as "first" and "second" are used merely to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.
[0045] The features and performance of the present invention will be further described in detail below with reference to embodiments.
[0046] Please see Figure 1-11 A video super-resolution method based on frame type differentiation processing, such as Figure 2 As shown, it includes the following steps: Differentiated frame processing strategy: According to the type of video frame ( frame, frame, Different super-resolution processing strategies are employed for each frame to maximize output quality while ensuring real-time performance. Specifically, these include: Frame type identification: Receive decoded video frames and identify them based on the frame type... The field determines its frame type. If it is... Frame, execution Frame CNN super-resolution steps; if it is Frame or Frame, then execute frame / Fast frame upsampling steps.
[0047] Frame-based CNN super-resolution: right Frame-based super-resolution processing using deep neural networks (DNNs) is performed. Convert the input frame from the YUV color space to the RGB color space; RGB images are fed into a pre-trained generative adversarial network (such as RealESRGAN) for super-resolution inference; In this embodiment, the deep neural network is based on the RealESRGAN pre-trained image super-resolution network, and the model structure and inference accuracy have been adapted to meet the requirements of real-time 1080p→4K transcoding scenarios. Specifically, it includes: (1) Network structure adaptation: Using RRDBNet as the generator backbone, the following parameters are configured for the 2x super-resolution task: , , , , , Load the official RealESRGAN pre-trained weights and... The key reads the exponential moving average weight; to adapt to the fixed input resolution requirements of real-time transcoding, a static shape is used for output: input 1×3×1080×1920, output 1×3×2160×3840, avoiding the additional memory allocation and shape inference overhead introduced by dynamic shape during inference.
[0048] (2) ONNX export and FP16 quantization: First, export the adapted PyTorch model to ONNX (realesrgan_x2plus_1080p.onnx) with FP32 precision. Enable constant folding (do_constant_folding=True) and do not add dynamic axes to ensure the inference graph is static and lightweight; then use onnxconverter_common.float16.convert_float_to_float16() to convert the entire FP32 ONNX model to FP16 precision (realesrgan_x2plus_1080p_fp16.onnx), converting all weights and input / output tensors to FLOAT16; The FP16 model reduces memory usage by about 50% and improves inference throughput while ensuring no significant loss of visual quality, making it more suitable for real-time 4K transcoding scenarios.
[0049] (3) Input preprocessing: Convert the input frame from the YUV color space to the RGB color space; Normalize the RGB pixel values from integers [0, 255] to floating-point values [0.0, 1.0]. Arranged in NCHW format as [1, 3, H, W], and converted to FP16 precision before being sent to ONNX Runtime CUDAProvider for inference.
[0050] (4) Why can image models be directly used for video? This model uses a single natural image as its training object, and its network structure is RRDBNet + U-Net + SpectralNormalization discriminator, without specific fine-tuning for video temporal consistency. Directly applying this image super-resolution model to single-frame video processing may introduce inter-frame detail and brightness fluctuations due to independent frame-by-frame inference. Therefore, this application only performs DNN super-resolution on I-frames, and uses less computationally expensive interpolation upsampling for P / B frames, thereby reducing computational overhead and minimizing frame-by-frame audio detail jitter and brightness fluctuations introduced by the image model's independent frame-by-frame processing. Furthermore, By introducing a "raw luminance statistical regression" step in the I-frame path, the mean and standard deviation of the Y channel of the super-resolution output frame are forcibly pulled back to the level of the original input frame. This regression operation is performed frame by frame based on the original frame and does not depend on the information of adjacent frames. Therefore, while compensating for the single-frame luminance / tone shift caused by the image model, it does not destroy the original temporal luminance changes of the video (such as fade-in and fade-out), avoids perceptible luminance jumps at the GOP start point, and improves the temporal stability of the video.
[0051] (5) Segmentation ( Processing strategy: When the target resolution is high (e.g., 1080p → 8K) or the GPU memory is insufficient to accommodate a full frame of DNN inference, the method in this application employs an overlapping block strategy. Specific parameters include: :single The side length is typically 512 pixels; Adjacent The width of the overlapping boundary between them, typically 10 pixels; Block triggering condition: Super-resolution multiple When to enable segmentation By default, the entire frame is sent directly. Blocking method: Divide the input frame into blocks in the width and height directions. indivual Adjacent Share between The width of the overlapping area; Reasoning and splicing: Each High resolution is obtained by feeding the data separately into the DNN. Output the data, then trim away the redundant boundaries caused by overlap around the edges, and stitch the effective area back into a complete high-resolution frame.
[0052] This strategy enables high-resolution I-frame super-resolution to be performed on memory-constrained GPUs without significantly increasing computational load, while mitigating the impact of overlapping regions. Potential seam artifacts at the boundary.
[0053] Optionally, a lightweight sharpening process (such as unsharp mask) can be performed on the Y channel to improve detail clarity; For the Y channel amplified by bicubic or Lanczos interpolation, unsharp mask sharpening is performed to compensate for edge blurring caused by interpolation. The specific implementation is as follows: 1. Use a Gaussian blur kernel (e.g., Blurring the Y channel yields the low-frequency components. ; 2. Through Calculate the sharpened Y channel, where This is the sharpening intensity factor, typically ranging from 0.2 to 0.5; 3. Recombined with the UV channels, the processed YUV frame is output.
[0054] This sharpening effect only applies to the Y channel of P / B frames. The intensity coefficient α can be adjusted online according to the target bitrate or user quality preferences, but by default it does not change with the scene to ensure constant processing latency.
[0055] Preserve the original luminance characteristics of P / B frames.
[0056] Frame sequence output: After processing Frames and / The frames are combined according to the original timing sequence and output to the encoding module.
[0057] A long-standing, overlooked technical bias exists in existing technologies: when adjusting the brightness consistency of video frames obtained from different sources or through different processing paths, those skilled in the art typically employ a globally uniform brightness mapping strategy, aligning the brightness statistics of all frames to a fixed reference frame or the average statistics of the entire video. While this strategy can eliminate brightness differences between frames, it inevitably disrupts the original temporal brightness variation intent of the video, such as director-intentioned fade-ins and fade-outs, flash effects, or brightness gradients during scene transitions. For videos using differentiated super-resolution processing, how to completely preserve the original temporal brightness characteristics of the video while eliminating brightness jumps between I-frames and P / B frames is a problem that existing technologies have not solved or even recognized.
[0058] Luminance Statistical Regression: Because DNN models alter the overall tonal statistics of an image (including mean and standard deviation of brightness) during super-resolution, if The brightness statistics of the frame are shifted after DNN processing. / If the frame retains its original statistics, a brightness jump will occur within the GOP. Therefore, this application addresses this issue. Frame super-resolution output performs luminance statistical regression, specifically including: Raw brightness statistics extraction: exist Extract the original input before or during the super-resolution step of the frame CNN. Frame Y-channel (luminance channel) statistics: Calculate the mean luminance of the Y channel of the original input frame; Calculate the standard deviation of the brightness of the Y channel of the original input frame.
[0059] Tone Match: Perform luminance statistical regression on the obtained preliminary super-resolution frames: Calculate the current average luminance of the Y channel in the initial super-resolution frame. and current brightness standard deviation ; Based on the original brightness statistics ( , ) and current brightness statistics, calculate the brightness scaling factor and brightness offset ; (3) Perform a linear transformation on the Y channel of the initial super-resolution frame: ; (4) After the above transformation, the mean and standard deviation of the brightness of the Y channel of the super-division frame are consistent with those of the original input frame.
[0060] It should be particularly noted that the luminance statistical regression strategy adopted in this application is fundamentally different from the global luminance alignment in the prior art: Selection of regression target: Existing technologies typically align the luminance statistics of all frames to a certain reference frame or global statistics, while this application only aligns the luminance statistics of all frames to a certain reference frame or global statistics. The luminance statistics of a frame are regressed to their own original input statistics, rather than being aligned to other frames or a global benchmark.
[0061] Selective processing: This application expressly waives the right to selective processing. Frames and The frame undergoes any luminance statistical adjustments to maintain the luminance characteristics of the original input frame. This "selective, local regression" strategy enables... The frame becomes the luminance reference within the GOP, while / The original temporal brightness variations of the frame are fully preserved.
[0062] The non-obviousness of the technical effect: When faced with the problem of brightness jumps within a GOP caused by differential super-resolution, those skilled in the art would conventionally adopt a "global alignment" scheme, that is, aligning all frames (including Frames and / The brightness statistics of each frame are unified to a single target value. However, this seemingly straightforward solution leads to the incorrect compression or elimination of temporal brightness variations such as fade-in / fade-out and flash. This application addresses this issue by deeply understanding the characteristics of human vision—specifically, the starting point of the Group of Pictures (GOP). It is highly sensitive to abrupt changes at a single frame, but insensitive to slow brightness changes between frames—creatively proposing the aforementioned "regression-only" approach. Frame, Hold The "frame" strategy simultaneously solves two contradictory technical problems: "eliminating jumps" and "preserving temporal changes," which has not been taught or implied by existing technologies.
[0063] Heterogeneous dual-queue and cross-process communication architecture: Unified decoding and distribution: A single decoding thread within the main process reads data packets from the demultiplexer, according to... Data packets are distributed to the video decoder and audio decoder. Decoded video frames are directly pushed into the video frame circular queue; decoded audio samples are optionally resampled and then pushed into the audio sample FIFO. The PTS of both queues undergoes uniform time base transformation and zero-based offset normalization to ensure audio and video timing alignment.
[0064] Master and child processes have a dual-queue buffer: The main process video queue uses a circular array structure of frame pointers (iqueue.c), with capacity configured according to the number of frames (approximately 600 frames by default). To achieve producer-consumer synchronization.
[0065] The main process audio queue uses a sample-level circular buffer structure (based on FFmpeg's AVAudioFifo), with capacity configured according to audio duration (default 60 seconds). It is accessed through a single... Space and data are available for coordination.
[0066] The child process's video and audio queues use the same implementation as the main process's (iqueue.c / iaudiQueue.c), with a video queue capacity of 300 frames and an audio queue capacity of 60 seconds of audio.
[0067] SR-sensory audio buffer: The audio FIFO is explicitly designed to have a capacity sufficient to absorb the startup latency caused by video SR processing. When the child process performs DNN super-resolution on the first I-frame, the video encoding path may be delayed by several seconds to tens of seconds before starting; during this period, the main process continuously decodes and writes audio samples to the audio FIFO. The 60-second audio FIFO ensures that no overflow occurs before the video path starts, fundamentally eliminating cross-process deadlock.
[0068] The capacity of the audio FIFO is designed according to the following principles: , in This refers to the audio sampling rate (e.g., 48000 Hz). This represents the maximum startup latency that the video SR path may cause. This takes into account the first frame. The time consumption of DNN super-resolution of frames is affected by model complexity, input resolution, and GPU computing power. A safety factor should be set. (Typically, 5 to 10 are used), then: , This can be obtained through actual measurement or estimation: , The time consumed by DNN inference per frame. YUV RGB color space conversion takes time. The time taken for brightness statistics calculation.
[0069] Taking this embodiment as an example: input 1080p, target 4K, RealESRGAN model, NVIDIA RTX 4060 GPU, Actual measurements show it to take approximately 1 to 3 seconds; ,but For 1 second, the sample size corresponding to 48000 Hz stereo 16-bit audio is 30 × 48000 = 1440000 samples.
[0070] If the model is upgraded (to a larger model) or the hardware is changed (to a lower-powered GPU / pure CPU), the field test should be repeated. Adjust Capacity_samples according to the formula above. The system reads the audio_fifo_seconds configuration item at startup; the default is 60 seconds, but this can be dynamically modified during deployment based on the hardware environment.
[0071] Independent consumer thread writes to SHM: Two independent consumer threads are created within the main process: (1) Video processing thread: pops frames from the video queue, serializes them into SHM data packets (containing metadata such as width, height, format, PTS, pict_type, flags, etc.), and writes them into the video SHM ring.
[0072] (2) Audio processing thread: Pop samples from the audio FIFO (aligned according to the encoder frame size, such as 1024 samples of AAC), serialize them into SHM data packets, and write them into the audio SHM ring.
[0073] Atomic Sequence SHM Protocol: To ensure the orderly transmission of audio and video frames in shared memory, the video SHM ring and the audio SHM ring adopt a hybrid synchronization mechanism based on a state machine with atomic sequence numbers and named semaphore event notifications. The "atomic sequence number" in this application is not used to completely replace the operating system synchronization primitives, but rather to push the state judgment down to each SHM slot.
[0074] Each SHM slot maintains a pair of named semaphores, representing "slot is free and writable" and "slot has new data to read," respectively. The write and read ends use `sem_wait` / `sem_post` to handle blocking waits and wake-up notifications, thus avoiding high CPU usage caused by pure polling. Simultaneously, the slot header maintains an `_Atomic(uint64_t) seq` field, which is set to odd after a write operation and even after a read operation. After being woken up by the semaphore, the read end does not immediately read the data but checks again to ensure the seq is indeed odd, verifying that the data has been truly written. If, due to memory order or extreme timing, the semaphore triggers before the seq has toggled, the read end enters a limited number of timed polling waits; if this times out, the slot is released and an error is returned. Therefore, the core advantage of this protocol is that the semaphore handles efficient process blocking / wake-up, while the atomic sequence number ensures observable data status verification. This separation of responsibilities avoids global lock contention and busy waiting while guaranteeing reliable cross-process data transfer.
[0075] Opaque marker EOF propagation: When decoding is complete, the decoding thread does not notify downstream processes via a global flag. Instead, it pushes a sentinel frame, with its opaque field set to 1, onto both the video and audio queues. This sentinel frame traverses the queues and the SHM ring along with the normal data stream, eventually reaching the child process. Upon detecting a sentinel frame with opaque=1, the child process knows the input stream has ended, performs a final flush, and gracefully exits. This mechanism eliminates race conditions for EOF notifications, ensuring that the child process never misses an end signal under any circumstances.
[0076] Figure 2 exhibit Frames and / The processing paths for each frame, and luminance statistical regression in Position in the frame processing path. Frame path: Save original luminance statistics → YUV → RGB → DNN super-resolution → RGB → YUV → luminance statistics regression → output. / Frame path: YUV bicubic interpolation scaling → unsharp mask sharpening → output (without brightness statistics adjustment).
[0077] This application also includes a video super-resolution system based on frame type differentiation processing, such as... Figure 1 As shown, it includes: Decoding module: Decodes the input video stream and outputs the original frame sequence; Frame type identification module: Identifies the frame type (I-frame, P-frame, B-frame) of the decoded video frames. Differentiated super-resolution processing module: Selects the corresponding super-resolution processing path based on the frame type; Luminance Statistical Regression Module: Performs luminance statistical regression processing on the DNN super-resolution output of I-frames; Encoding module: Encodes the super-resolution frame sequence; Encapsulation module: Encapsulates the encoded audio and video data into an output file; Dual queue module for main and child processes: Video frame queues and audio sample queues are set up on both the main process and the child process sides respectively, forming a dual backpressure design to achieve smooth flow control.
[0078] The system employs a dual-queue (dual backpressure) design for both main and child processes: The main process side sets up a circular queue for video frames (a custom circular queue, iqueue.c) and a circular queue for audio samples (encapsulating the underlying logic of AVAudioFifo). The child process side also sets up a circular queue for video frames and a circular queue for audio samples, with the SHM ring positioned between them. These four queues form a three-level buffer; when any queue is full, it automatically blocks flow upstream, achieving smooth flow control.
[0079] Figure 1 This demonstrates the dual-process structure of the main process and child processes, as well as the data flow between modules. The main process includes a demultiplexer, video decoder, audio decoder, video queue, audio queue, shared memory (SHM), and named semaphores. The child processes include threads for reading audio and video frames (reading frames from SHM, frame type identification, and differential super-resolution processing) and the main encoding / packaging threads (video encoder, audio encoder, and multiplexer).
[0080] Figure 3 The diagram shows that, without brightness regression, the starting point of each GOP in the existing technology (solid line in the legend) is... A brightness jump occurs at the frame level; this application (dashed line in the legend) shows... After luminance regression, the brightness of the entire GOP is consistent without any jumps. The horizontal axis represents the frame number, and the vertical axis represents the average brightness of the Y channel of the video frame.
[0081] Figure 4 The demonstration shows that there is a set of audio and video queues on both the main process side and the child process side (a total of four queue levels), with an SHM ring located between them to form a three-level buffer. This includes: the symmetrical structure of the main and child process dual queues; the location and function of the three-level backpressure points (main queue full / SHM full / child queue full); the capacity design principle of the SR-aware audio buffer; and the slot state switching of the atomic sequence SHM protocol (odd number = new data / even number = consumed).
[0082] Figure 5 This demonstrates the timing relationship between main process decoding, SHM transmission, child process reading, and processing. It includes the main process's unified decoding thread distributing video / audio frames, independent producer threads writing to the SHM, child process reading, and child process consumer threads reading SHM audio and video frames. Producers and consumers communicate via semaphores. , and synchronization serial number Precise collaboration enables cross-process transmission of audio and video data.
[0083] Figure 6 This embodiment demonstrates the overlapping block strategy used when the target resolution is high or the GPU memory is insufficient to accommodate the entire frame of DNN inference. The main approach involves setting a single inference block ( ) length (Typical value is 512 pixels), the original image is divided into blocks. The overlap area of adjacent blocks is set ( A typical value is 10 pixels. Blocking method: Divide the input frame into blocks in the width and height directions. indivual Adjacent Share between The width of the overlapping area; each High resolution is obtained by feeding the data separately into the DNN. Output the data, then trim away the redundant boundaries caused by overlap around the edges, and stitch the effective area back into a complete high-resolution frame.
[0084] In one specific embodiment, the system hardware environment of this application is as follows: The video transcoding system of this invention runs on a computing device equipped with the following hardware: CPU: x86_64 architecture multi-core processor; GPU (optional): NVIDIA GPU for accelerating DNN inference (via ONNX Runtime CUDA Provider); Memory: at least 16GB; Storage: SSD, used to cache video frame data.
[0085] The software environment includes: Linux operating system, FFmpeg audio and video processing library, OpenCV image processing library, and ONNXRuntime inference engine.
[0086] The detailed steps of the method in this application are as follows: Detailed implementation of the differentiated frame processing strategy: Detailed implementation of I-frame DNN super-resolution: via FFmpeg Create a YUV→RGB color space conversion context; Call Convert YUV frames to RGB format; Copy the RGB image data to OpenCV cv::Mat and send it into the ONNX Runtime inference session (RealESRGAN model). Obtain the high-resolution RGB image output by the DNN; pass Converting RGB back to YUV format yields preliminary super-resolution frames.
[0087] / Detailed implementation of fast frame upsampling: via FFmpeg Create a YUV→YUV color space conversion context and configure it for bicubic interpolation. ); Call Perform image scaling, increasing the resolution from the input size to the target size (e.g., 1920×1080 → 3840×2160). Unsharp masking is applied to the scaled Y channel: first, Gaussian blur is applied, and then the original image and the blurred image are blended by weight using addWeighted (e.g., original image weight 1.3, blurred image weight -0.3) to enhance edge details; The output frame does not undergo any brightness statistical adjustments.
[0088] Detailed implementation of luminance statistical regression: The detailed implementation of the luminance statistical regression module is as follows: First of all, Before the frames are fed into the DNN, the mean of the Y channel of the original input frame is calculated using OpenCV's cv::meanStdDev function. and standard deviation This computation is performed in an asynchronous thread, using std::async to start a background task, so the main thread is not blocked.
[0089] In this application, the luminance statistics of the Y channel of the original input frame are cached and synchronized through a combination of asynchronous tasks and global structures. Specifically, this includes: (1). Data Structure global structure ,Include , and Three members; std::future object This is used to asynchronously calculate the mean / standard deviation of the Y channel of the original input frame in the background; mutex Used to protect global Concurrent access.
[0090] (2). Processing flow for Before or simultaneously with feeding the original input frame into the DNN super-resolution, a background task is started using asynchronous mechanisms such as std::async to calculate the mean and standard deviation of the Y channel of the original input frame; after the calculation is completed, the result is written to the receiving... Overall protection and will Set to true; After the DNN super-resolution is completed and before the brightness regression, the main thread first checks stats_future.valid(). If it is valid, it calls stats_future.get() to block and wait for the background statistics task to complete and obtain the results. At the same time, it clears the future state to avoid repeated calls. Then the main thread Read global data under protection In , , After reading is complete, Setting it to false indicates that the statistic has been consumed, preventing misuse; Only when Only when the statistics are ready will the match_y_stats() brightness regression be performed on the DNN output frame; if the statistics are not ready, the brightness regression will be skipped and an alarm will be recorded.
[0091] (3) Timing correctness guarantee stats_future.get() ensures that the main thread must wait for the background statistics task to complete before brightness regression, and therefore will not use statistics that have not yet been calculated. The global i_stats are read and written under the protection of a mutex lock. Combined with the consumption zeroing mechanism of the valid flag, it is ensured that the read statistical values correspond to the I-frame currently being processed. Video frames are processed serially in the order of decoding to further ensure that global i_stats will not be mismatched between multiple frames.
[0092] (4). P / B frame processing For P / B frames, they do not participate in DNN super-resolution and luminance regression; the original luminance statistics of P / B frames remain unchanged.
[0093] Then, after the DNN performs super-resolution, a high-resolution output frame is obtained. The current mean is also calculated for the Y channel of the output frame. and standard deviation .
[0094] Next, calculate the linear transformation parameters: , , To avoid Values that are too close to zero can become unstable, so set a minimum tolerance value (e.g., 0.5).
[0095] Finally, a linear transformation is performed on the Y channel using OpenCV's cv::convertScaleAbs function: , After the above transformation, the first-order statistics (mean) and second-order statistics (standard deviation) of the output frame Y channel are accurately pulled back to the original input level.
[0096] Detailed implementation of master-child process dual queues (dual backpressure) and inter-process communication: Detailed implementation of the video queue: The video queue (iqueue.c) is implemented using a circular array, and its core data structures include: A fixed-size array of AVFrame pointers (default size approximately 600 frames). The head pointer and tail pointer are used to mark the read / write position; Protect queue state; two : (Notifying producers of available slots) and (Notify consumers that there is data); volatile The `stop` and `empty` flags are of type `stop` and `empty`, used for graceful termination.
[0097] Producer (decoding thread) operations: Lock the mutex; If the queue is full and no stop signal is received, wait on the not_full condition variable; Write the AVFrame pointer to the tail position, and update tail; If the last frame is written, set the opaque of that frame to 1 as a sentinel marker; Broadcast a not_empty message to consumers; Unlock the mutex.
[0098] Consumer (video SHM write thread) operations: Lock the mutex; If the queue is empty and no stop signal is received, wait on the not_empty condition variable; Pop the AVFrame pointer from the head position and update the head; Check the opaque field; if it is 1, mark EOF and pass it to the SHM write logic. Broadcast `not_full` to notify the producer; Unlock the mutex.
[0099] Detailed implementation of the audio queue: The audio queue (iaudiQueue.c) is implemented based on FFmpeg's AVAudioFifo, and its core data structures include: An AVAudioFifo instance, configured with a capacity of sample_rate × 60 seconds; pthread_mutex_t + pthread_cond_t (a single condition variable that simultaneously coordinates space availability and data availability); A PTS tracking model based on decoder base-PTS + sample-count accumulation.
[0100] Producer (decoding thread) operations: Optional resampling of the decoded audio samples (unified to the sampling format required by the encoder); Lock the mutex; If the FIFO has insufficient remaining space, wait on the condition variable; Write the sample to AVAudioFifo; Update the cumulative sample count for PTS calculation; Broadcast condition variables to notify consumers; Unlock the mutex.
[0101] Consumer (audio SHM write thread) operations: Lock the mutex; If the number of samples in the FIFO is less than the size of one encoder frame (such as 1024 samples in AAC), wait on the condition variable; Pop a sample of exactly one encoder frame size from AVAudioFifo; The PTS of the current frame is calculated based on the base-PTS and the number of consumed samples; In flush mode, if there are fewer than one frame of remaining samples in the FIFO, all remaining samples are retrieved at once. Broadcast condition variables to notify producers; Unlock the mutex.
[0102] Detailed implementation of the SHM protocol for atomic sequences: The video SHM ring and the audio SHM ring have the same structure, and each slot includes: Frame data area (video approximately 6MB / slot, audio approximately 128KB / slot); Metadata header (including width, height, format, pts, pict_type, flags, etc.); _Atomic(uint64_t) seq field.
[0103] Write-end (main process) operations: Get the seq value of the current slot; If seq is odd, it means that the child process has not yet consumed the slot and should either wait or poll. In this application, slot synchronization of the shared memory ring is accomplished by a combination of named semaphore master synchronization and atomic sequence number secondary verification.
[0104] (1). Semaphore Master Synchronization Each slot maintains a pair of named semaphores: a writable semaphore (initially 1) and a readable semaphore (initially 0); Before writing data, the writing end calls sem_wait(sem_write[idx]) to block and wait for the slot to become available; after writing is complete, it calls sem_post(sem_read[idx]) to notify the reading end. Before reading data, the reading end calls sem_wait(sem_read[idx]) to block and wait for the data to be ready; after reading is complete, it calls sem_post(sem_write[idx]) to notify the writing end.
[0105] (2). Secondary verification of atomic serial numbers The slot header maintains an _Atomic(uint64_t) seq field, initially set to an even number; After writing data, the writing end increments the seq value by 1 to make it odd, indicating that there is new data to be read in the slot; After reading data, the reader increments the seq value by 1 to make it even, indicating that the slot has been consumed and can be written to again. After sem_wait returns, the read end additionally checks whether seq is indeed an odd number. If the semaphore has been triggered but seq has not been toggled due to extreme timing, it will enter a timed polling wait of up to 3000 times, each time lasting 1ms. After the timeout, the write semaphore will be released and an error will be returned to avoid waiting indefinitely.
[0106] (3). Design Intent Semaphores perform the main process blocking / wake-up functions, avoiding busy waiting and reducing CPU usage; Atomic sequence numbers serve as a secondary check for data validity, preventing data inconsistencies caused by semaphore mis-triggering or memory order issues. The combination of the two ensures low overhead for cross-process synchronization while retaining explicit state machine verification.
[0107] Write frame data and metadata to the slot; Set seq to seq + 1 (to make it odd, indicating that new data is waiting to be read); Release the write semaphore (optional, used to notify child processes).
[0108] Read-end (child process) operations: Get the seq value of the current slot; If seq is even, it means that the main process has not yet written new data and is waiting or polling. Read frame data and metadata; Set seq to seq + 1 (to make it an even number, indicating that the data has been consumed); Release the read semaphore (optional, used to notify the main process).
[0109] Detailed implementation of Opaque sentinel frame EOF propagation: When the demultiplexer reaches the end of the file, the decoding thread pushes a special AVFrame onto the video queue, with its opaque field set to 1. When the consumer thread of the video queue detects that opaque=1, it retains the flag when serializing the frame to SHM; After the child process's Reader thread reads a frame marked with opaque=1 from SHM, it knows that the video stream has ended; Similarly, when the audio stream ends, sample frames marked with opaque=1 are pushed onto the audio queue; After detecting that both video and audio channels have received sentinel frames, the child process executes Final Flush and exits gracefully.
[0110] The child process maintains separate EOF flags, video_eof and audio_eof, for the video and audio streams respectively. Initially, if the input file contains only a single stream (such as pure video or pure audio), the EOF flag corresponding to that stream is directly set to true during system initialization, without participating in dual-channel waiting.
[0111] Normal procedure: (1). When the video SHM reading thread receives a video sentinel frame with opaque = 1, set video_eof = true; (2). When the audio SHM reading thread receives an audio sentinel frame with opaque = 1, set audio_eof = true; (3). The main loop of the child process continues to process the remaining frames until video_eof && audio_eof are both true; (4). At this point, execute Final Flush (flush encoder, multiplexer buffer) and exit.
[0112] Anomaly Protection: (1). If a sentinel frame does not arrive within the preset timeout (e.g., 120 seconds), an error log is recorded and the EOF flag of that sentinel is forcibly set to true. At the same time, the Final Flush is triggered to avoid deadlock of the child process. (2). If the video sentinel arrives first but the audio sentinel has not arrived, the child process continues to consume and encode the remaining samples in the audio SHM ring without exiting prematurely; (3). If the audio sentinel arrives before the video sentinel arrives, continue processing the video frames in the same way; (4) If an error occurs at the decoding end and the decoding is terminated prematurely, the demultiplexer will also push a sentinel frame into the corresponding queue to ensure the reliable propagation of the EOF signal.
[0113] Detailed implementation of the dual-process architecture: The system adopts a master-child dual-process architecture: Main process (main transcoder): Create a demultiplexer to read the input file; Create video decoders and audio decoders to decode the audio and video streams separately; Create a shared memory pool (SHM Pool) and semaphores for inter-process communication; A child process is created using fork(); The decoded frame sequence is passed to the child process through shared memory and semaphore synchronization mechanism.
[0114] Subprocess (GPL encoder): Read frame data from shared memory; Perform differential super-resolution processing on video frames (I-frame DNN + P / B fast upsampling); Create video encoders (such as h264_nvenc, libx264) and audio encoders (such as libfdk_aac) for encoding; Create a multiplexer (Muxer) to encapsulate the encoded data into an output file.
[0115] The main process and child processes use named semaphores to achieve read-write synchronization, ensuring the orderly transmission of frame data.
[0116] Experiment content: (1) Experimental setup Test sequences: football matches, news studio videos; Super-resolution target: 4K (3840×2160); Hardware: NVIDIA GPU / Intel x86_64 / 32 GB RAM; Software: FFmpeg 6.x, OpenCV 4.x, ONNX Runtime 1.27; DNN model: Based on RealESRGAN_x2plus pre-trained weights, adapted to a 2x super-resolution static shape model (1080p input, 4K output), and inferred in FP16 ONNX format; Comparison method: 1. Transcoding quality Baseline A: Full-frame DNN super-resolution (frame-by-frame processing of the adapted RealESRGAN_x2plus FP16 model); Baseline B: Full-frame bicubic interpolation up-transform; Baseline C: Full-frame LANSZOS interpolation up-transform; The method used in this application is: I-frame DNN + P / B frame interpolation + original luminance statistical regression.
[0117] 2. Transcoding efficiency Baseline A: Full-frame DNN super-resolution; The method used in this application is: I-frame DNN + P / B frame interpolation + original luminance statistical regression.
[0118] 3. Inter-process communication performance Baseline A: Inter-process Pipe data transfer Baseline B: Data transmission based on UDS (Unix Domain Socket) This application uses the method of inter-process shared memory data transfer. 4. Output bitrate stability Baseline A: The method in this application does not enable pacing. Comparison method: This application method enables pacing. (2) Objective indicators 1. To verify the differences in super-resolution quality between the proposed method and traditional interpolation-based transformation and full-frame AI super-resolution methods, comparative experiments were conducted on baselines A, B, C, and the proposed method using metrics such as PSNR, SSIM, VMAF-4K, and VMAF-1080P. Two types of 4K videos (football match and news studio) were selected as 4K reference videos. First, the 4K reference videos were down-converted from 4K to 1080P to serve as high-definition reference videos. Baselines A, B, C, and the proposed method were then used to super-resolution the high-definition reference videos back to 4K. Using the reference 4K videos as a benchmark, the super-resolution videos from each baseline and the proposed method were compared and scored. Then, the 4K videos up-converted from baselines A, B, C, and the proposed method were down-converted to 1080P. VMAF scoring was then performed using the high-definition reference videos as a benchmark to analyze the scores of baselines A, B, C, and the proposed method in enhancing the image quality of high-definition videos.
[0119] 2. To verify the difference in transcoding efficiency between the method of this application and the full-frame AI super-resolution method, a comparative experiment was conducted on baseline A and the method of this application based on indicators such as transcoding time, FPS, and real-time speed. The high-definition reference video (football match scene) in Experiment 1 was selected as the source video. The source video was super-recoded using both the full-frame AI super-resolution method and the method of this application. Data such as transcoding time, FPS, and real-time speed were recorded, and the transcoding rate of the two super-resolution methods was scored.
[0120] 3. To verify the difference in data transmission rate between the method of this application and the Pipe pipeline during inter-process data transmission, experiments were conducted to transmit the same video data using baseline A and the method of this application, based on metrics such as throughput, latency (ms / frame), and transmission time. Data throughput, transmission latency, and transmission time were recorded to compare the two transmission methods.
[0121] 4. The experiment sets up indicators such as average bitrate, standard deviation of bitrate (STD), coefficient of variation (CV), and maximum bitrate, and compares the experimental results of the proposed method for transcoding the same video under pacing_on and pacing_off modes.
[0122] (3) Experimental results 1. Transcoding quality In the context of a football match, the results of the comparative experiment are as follows: Figure 7 As shown; In fast-paced football scenes, the full DNN super-resolution method performs worse than other methods in terms of PSNR and SSIM. Our proposed method performs similarly to bicubic interpolation and LANCZOS interpolation in terms of PSNR and SSIM, with VMAF-4K (67.06) outperforming full DNN (62.48), bicubic interpolation (64.77), and LANCZOS interpolation (65.80). The fact that the full DNN super-resolution method performs worse than other methods in football scenes suggests that using an image model frame-by-frame for video processing can lead to quality degradation due to issues such as brightness / texture fluctuations and temporal consistency. Our proposed method, with its selective I-frame super-resolution combined with a brightness regression strategy, effectively mitigates this problem. In a news studio setting, the results of the comparative experiment are as follows: Figure 8 As shown; In the low-speed scene of a news studio, the full DNN VMAF-4K score (99.63) is higher than that of this application (97.56), and the method of this application is higher than that of bicubic interpolation (95.57) and LANCZOS interpolation (96.62). However, the full DNN super-resolution method is lower than that of this application in terms of PSNR (35.15) and SSIM (0.98), because the full DNN super-resolution method reconstructs the texture and brightness of the image. It can be seen that under the VMAF-1080P metric, the full DNN super-resolution method scores higher than other methods (99.38), and the method of this application ranks second (93.77). This shows that, from a visual perspective, the full DNN super-resolution method introduces more details and provides the best viewing experience. This comparative experiment proves that in low-speed scenes, the problems of brightness / texture fluctuations and temporal consistency caused by using the image model frame by frame for video are significantly alleviated.
[0123] 2. Transcoding efficiency The results of the transcoding efficiency comparison between the full DNN super-resolution method and the method in this application are as follows: Figure 9 As shown; Experimental results show that for transcoding the same test video (fps=50p), the full DNN super-resolution method takes 1351.8 seconds, while the method in this application takes 9.9 seconds. The output FPS of the full DNN super-resolution method is 0.4fps, while that of the method in this application is 51.2fps. The real-time speed multiplier of the full DNN super-resolution method (0.007) is much less than 1, while that of the method in this application (1.024) is greater than 1. This indicates that the method in this application, while ensuring output image quality superior to traditional super-resolution methods (even superior to full DNN super-resolution in football match scenarios), can still meet the requirements of 4K50P real-time live streaming.
[0124] 3. Inter-process communication performance The comparative experimental results of data transfer between processes using shared memory, UDS, and Pipe are as follows: Figure 10 As shown; In inter-process communication, SHM (Shared Memory) offers significantly higher throughput than UDS (Unix Domain Socket) and Pipe, with much lower latency per frame. Compared to UDS and Pipe, SHM boasts a substantial performance advantage, but requires manual handling of synchronization and mutual exclusion. In audio and video data transmission scenarios (especially 4K scenarios), SHM is the optimal choice for inter-process audio and video data transfer.
[0125] 4. Output bitrate stability The bitrate output results using the I-frame super-resolution strategy of this application under pacing_on and pacing_off conditions are shown in Table 1. Figure 11 As shown; Table 1
[0126] The core technical solution of this application is "DNN super-resolution of I-frames and fast upsampling of P / B frames." While this differentiated processing improves quality and real-time performance, it introduces a new engineering problem: uneven time intervals in the encoder's output bitstream. I-frames, due to DNN inference, color space conversion, and luminance statistical regression, are encoded significantly later than adjacent P / B frames. P / B frames are processed quickly, continuously outputting a large number of encoded packets. Therefore, the audio and video packets output by the encoder to the multiplexer exhibit a "burst" characteristic: no packets are output for a period, followed by a rapid output of multiple packets within a short time. For real-time transmission protocols based on TSBPD (Time-based Packet Delivery) mechanisms, such as SRT (Secure Reliable Transport), the sending end needs to send data packets at a uniform rate according to the target PTS time base. If the multiplexer directly writes the encoded burst into the SRT, the sending end will send packets at the wrong time, resulting in: TSBPD time base confusion at the receiving end; video stuttering, audio acceleration, or audio / video desynchronization; and drastic fluctuations in output bitrate, affecting network transmission stability. After enabling the pacing_on function, the transcoding output bitrate can be seen to be stable at around 20Mbps, which can meet the requirements of SRT live streaming scenarios.
[0127] The embodiments described above merely illustrate specific implementation methods of this application, and while the descriptions are detailed and specific, they should not be construed as limiting the scope of protection of this application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of the technical solution of this application, and these modifications and improvements all fall within the scope of protection of this application.
Claims
1. A video super-resolution method based on frame type differentiation processing, characterized in that, Includes the following steps: Identify the frame type of the decoded video frames. Frame types include... frame, Frames and frame; If the current frame is Frame, then the The frame performs super-resolution processing based on a deep neural network, and then performs super-resolution... The luminance statistics of the frame regressed to this Brightness statistics of the original input frame; If the current frame is Frame or Frame, then the Frame or The frame undergoes fast upsampling processing, but no luminance statistical alignment is performed.
2. The video super-resolution method based on frame type differentiation processing according to claim 1, characterized in that, The super-resolution The brightness statistics of the frame are regressed to the brightness statistics of the original input frame, specifically including: Extract raw input Average brightness of the Y channel of a frame and brightness standard deviation ; After extracting the super-resolution Current average brightness of the Y channel of the frame and current brightness standard deviation ; Calculate the brightness scaling factor and brightness offset ; After exceeding the score Y channel pixel value of the frame Perform a linear transformation: The Y channel values after brightness regression were obtained. .
3. The video super-resolution method based on frame type differentiation processing according to claim 2, characterized in that, The extraction of raw input Frame luminance mean and brightness standard deviation The process is executed via an asynchronous thread, which runs concurrently before or during the deep neural network super-resolution process. The luminance statistics are cached in a global structure protected by a mutex lock. The structure contains the luminance mean, luminance standard deviation, and validity flag. After the deep neural network super-resolution is completed, the main thread waits for the asynchronous statistics task to complete, reads the statistics from the global structure, and performs luminance regression when the validity flag is true.
4. A video super-resolution method based on frame type differentiation processing according to claim 2, characterized in that, When calculating the brightness scaling factor (scale), if If the value is less than the preset minimum tolerance value, then the minimum tolerance value will be used instead. Perform the calculation.
5. A video super-resolution method based on frame type differentiation processing according to claim 1, characterized in that, right Frame or The frame undergoes fast upsampling processing, specifically including: directly upscaling the frame resolution to the target resolution in the YUV color space using bicubic interpolation or Lanczos interpolation, and then performing unsharpened mask sharpening on the upscaled Y channel, specifically including: Use Gaussian blur kernel , Blurring the Y channel yields the low-frequency components. ; pass Calculate the sharpened The passage, in which This is the sharpening intensity factor, typically ranging from 0.2 to 0.
5.
6. The video super-resolution method based on frame type differentiation processing according to claim 1, characterized in that, right The frame performs super-resolution processing based on a deep neural network, and also includes: [the following is unclear and likely incomplete: "to perform super-resolution processing based on a deep neural network"] After frames are converted from the YUV color space to the RGB color space, they are input into a deep neural network. The high-resolution RGB image output by the deep neural network is then converted back to the YUV color space to obtain the initial super-resolution. frame.
7. A video super-resolution method based on frame type differentiation processing according to claim 1, characterized in that, It also includes the step of using a master-child dual-process architecture for cross-process communication: setting up a circular queue of video frames and a FIFO queue of audio samples on the master process side, and setting up corresponding circular queues of video frames and FIFO queues of audio samples on the child process side; wherein, the capacity of the audio sample FIFO queue is preset to be able to absorb the startup delay required for the child process to perform deep neural network super-resolution processing on the first I-frame; the capacity of the audio sample FIFO queue is set according to the product of the maximum super-resolution time of the first I-frame and the audio sampling rate, and a safety factor is set, specifically: , in For audio sampling rate, For a single The time required for DNN super-resolution of frames. This represents the maximum startup latency that the video SR path may cause; taking into account the first frame. The time consumption of DNN super-resolution of frames is affected by model complexity, input resolution, and GPU computing power. A safety factor should be set. (Typically, 5 to 10 are used), then: 。 8. A video super-resolution system based on frame type differentiation processing, characterized in that, include: The decoding module is used to decode the input video stream and output the original frame sequence; The frame type identification module is used to identify the frame type of the decoded video frames. Frame types include... frame, Frames and frame; The differentiated super-resolution processing module is used to select the corresponding processing path based on the frame type: if it is For each frame, super-resolution processing based on a deep neural network is performed; If Frame or For frames, fast upsampling is performed; The luminance statistical regression module is used to convert the light after super-resolution by the deep neural network. The luminance statistics of the frame regressed to this Brightness statistics of the original input frame, and for Frames and Frames are not luminance-statistical aligned.
9. A video super-resolution system based on frame type differentiation processing according to claim 8, characterized in that, It also includes an inter-process communication module, which adopts a master-child dual-process architecture and uses a hybrid strategy that combines a slot state machine based on atomic sequence numbers with named semaphore event notifications for synchronization. The main process and child processes exchange frame data through a shared memory ring. Each slot of this shared memory ring contains a named semaphore and an atomic sequence number field seq. The writing and reading ends use named semaphores to complete blocking waits and wake-up notifications, thereby avoiding high CPU usage caused by pure polling. The atomic sequence number serves as a secondary check for data validity, preventing data inconsistency caused by semaphore mis-triggering or memory order issues. The initial value of the sequence number field seq is an even number; the writing end executes: waits for seq to be even, writes data, and then increments seq by 1 to make it odd; the reading end executes: waits for seq to be odd, reads data, and then increments seq by 1 to make it even; if the shared slot data is not yet ready, it waits by a limited number of timed polling, and after the timeout, it releases the write semaphore and returns an error to avoid waiting indefinitely.
10. A video super-resolution system based on frame type differentiation processing according to claim 9, characterized in that, The cross-process communication module also includes an end-of-process propagation unit: when decoding is finished, a sentinel frame with a preset flag value in the opaque field is pushed into the video queue and the audio queue respectively, and the sentinel frame is transmitted to the child process along with the data stream; After the child process detects both video and audio sentinel frames, it performs a final refresh and exits; the video stream and audio stream maintain independent EOF flags respectively; if the input file contains only a single stream, the EOF flag corresponding to that stream is directly set to true during system initialization, and it does not participate in dual-channel waiting; It also includes: caching the video packets and audio packets after differential super-resolution processing and encoding to separate video packet queues and audio packet queues; selecting the next packet to be sent from the two queues according to the display timestamp of each packet; calculating the target transmission time based on the display timestamp, and outputting the packet to the multiplexer when the target transmission time is reached, so that the output rhythm is aligned with the original audio and video timestamps.
