Efficient real-time compression method and system for neural coding and decoding

Abstract

The invention relates to the related technical field of video compression coding and decoding, in particular to an efficient real-time compression method and system for neural coding and decoding. According to the method, an original video is converted into a low-resolution video through down-sampling, and an adaptive network structure is filled in combination with an adaptive pixel space; generating a compression latent variable by using a compression neural network containing a decompression module, encoding and then transmitting to a receiving end; and after decoding, the receiving end performs up-sampling to recover the original resolution. A coding module and a decoding module are synchronously trained, redundancy is reduced by adopting SPS cache management, quantization step length is dynamically adjusted through network bandwidth and hardware utilization rate, and floating point training is converted into integer deployment. According to the scheme, while the compression ratio is improved, 40fps real-time coding of the 1080p video on the consumer-level GPU is realized, 21% of code rate is saved, and the scheme is adaptive to multiple scenes and has high practical value.

Description

Technical Field

[0001] This application relates to the technical field of video compression encoding and decoding, specifically to an efficient real-time compression method and system for neural encoding and decoding. Background Technology

[0002] With the rapid development of information technology, high-definition video (such as 1080p) and ultra-high-definition video (such as 4K and 8K) are increasingly widely used in video conferencing, security monitoring, online education, mobile live streaming, and other fields. Efficient compression and real-time transmission of video data have become core technological requirements supporting these applications. On the one hand, the data volume of high-resolution video is growing exponentially, placing stringent demands on storage capacity and network bandwidth; on the other hand, the sensitivity of real-time interactive scenarios (such as video conferencing) to encoding latency and transmission efficiency further exacerbates the challenges of video compression technology. Summary of the Invention

[0003] In view of this, embodiments of this application aim to provide an efficient real-time compression method and system for neural encoding and decoding.

[0004] This application provides an efficient real-time compression method for neural codecs, applied to a system including a transmitter and a receiver. The efficient real-time compression method for neural codecs includes:

[0005] The sending end acquires the original video;

[0006] The transmitting end downsamples the original video to obtain a low-resolution original video;

[0007] The transmitting end performs adaptive pixel space filling on the low-resolution original video based on the size of the original low-resolution video and the block size required by the network structure in the pre-trained neural network-based video coding module, with the goal of reducing the number of filled video pixels.

[0008] The target code is obtained by encoding the original video at the specified resolution using a pre-trained neural network-based video encoding module.

[0009] The sending end transmits the target code to the receiving end;

[0010] The receiving end decodes the target code based on a pre-trained neural network-based video decoding module to obtain the target low-resolution video.

[0011] The target low-resolution video is upsampled to obtain the target video;

[0012] The video encoding module and the video decoding module are trained simultaneously and are matched with each other to perform encoding and decoding to achieve compression and decompression functions.

[0013] In some embodiments, the resolution of the original video is 8 times that of the low-resolution original video;

[0014] The resolution of the target low-resolution video is the same as the resolution of the original low-resolution video;

[0015] The resolution of the target video is 8 times that of the original low-resolution target video.

[0016] In some embodiments, adaptive pixel space padding is performed on the low-resolution original video, including:

[0017] The dimensions of the low-resolution original video are determined to be H×W;

[0018] The required block size for the network structure in the video encoding module is determined to be p×p;

[0019] The formula for calculating the fill amount for the missing rows and columns is as follows:

[0020]

[0021] in, This indicates rounding up;

[0022] The formula for calculating the fill amount for the missing rows and columns is as follows:

[0023]

[0024] By symmetrically filling p around the image h row and p w List

[0025] The image of the low-resolution original video is p-filled. h row and p w List.

[0026] In some embodiments, the image padding p of the low-resolution original video h row and p w Columns, including:

[0027] The image of the low-resolution original video is p-filled on top. h1 OK;

[0028] The image under the low-resolution original video is p-filled. h2 OK;

[0029] The left side of the image of the low-resolution original video is p-filled.w1 List;

[0030] The right side of the image of the low-resolution original video is p-filled. w2 List;

[0031] Where, p h1 +p h2 =p h ;p w1 +p w2 =p w ;p h1 With p h2 The difference between them is 0, 1, or -1; p w1 With p w2 The difference between them is 0, 1, or -1; p h1 p h2 p w1 and p w2 All are integers.

[0032] In some embodiments, the video encoding module includes a compression neural network module and an encoding module;

[0033] The compressed neural network module takes the features of the reconstructed previous frame image and the features of the current frame image as input, and generates compressed latent variables through the feature fusion mapping module.

[0034] The encoding module is used to encode the compressed latent variables to obtain the target code;

[0035] The video decoding module includes a decompression neural network module and a decoding module;

[0036] The decoding module is used to decode the target code to obtain compressed latent variables;

[0037] The decompression neural network module is also used to obtain the features of the reconstructed image corresponding to the previous frame image, and combine them with the compression latent variables to decompress and reconstruct the current frame image.

[0038] In some embodiments, the compression neural network module has an embedded decompression neural network module.

[0039] In some embodiments, if the sequence parameter sets corresponding to the current frame image and the previous frame image are the same, then the target encoding of the current frame image does not need to carry the encoding for indicating the sequence parameter set, and the video decoding module uses the sequence parameter set of the previous frame image as the sequence parameter set of the current frame image.

[0040] If the sequence parameter sets corresponding to the current frame image and the previous frame image are different, the target encoding of the current frame image carries an encoding for indicating the sequence parameter set, and the video decoding module determines the sequence parameter set of the current frame image based on the encoding for indicating the sequence parameter set.

[0041] In some embodiments, the method also includes obtaining network bandwidth and hardware utilization.

[0042] While ensuring that network bandwidth and hardware utilization are within a preset range, the quantization step size of the video encoding module and the video decoding module is minimized.

[0043] In some embodiments, the process of training the video encoding module and the video decoding module includes:

[0044] The video encoding module and the video decoding module are trained using floating-point weights.

[0045] The training floating-point weights are rounded to the nearest integer.

[0046] Verify whether the rounded reconstruction error increments of the video encoding module and the video decoding module are within a preset range;

[0047] If not, retrain the video encoding module and the video decoding module.

[0048] This application provides an efficient real-time compression system for neural encoding and decoding, including a transmitter and a receiver;

[0049] The sending end acquires the original video;

[0050] The transmitting end downsamples the original video to obtain a low-resolution original video;

[0051] The transmitting end performs adaptive pixel space filling on the low-resolution original video based on the size of the original low-resolution video and the block size required by the network structure in the pre-trained neural network-based video coding module, with the goal of reducing the number of filled video pixels.

[0052] The target code is obtained by encoding the original video at the specified resolution using a pre-trained neural network-based video encoding module.

[0053] The sending end transmits the target code to the receiving end;

[0054] The receiving end decodes the target code based on a pre-trained neural network-based video decoding module to obtain the target low-resolution video.

[0055] The target low-resolution video is upsampled to obtain the target video;

[0056] The video encoding module and the video decoding module are trained simultaneously and are matched with each other to perform encoding and decoding to achieve compression and decompression functions.

[0057] This application provides an efficient real-time compression method for neural codecs, applied to a system including a transmitter and a receiver. The efficient real-time compression method for neural codecs includes: the transmitter acquiring an original video; the transmitter downsampling the original video to obtain a low-resolution original video; the transmitter adaptively padding the low-resolution original video with the goal of reducing filler video pixels, based on the size of the low-resolution original video and the block size required by the network structure in a pre-trained neural network-based video coding module; and the transmitter further downsampling the low-resolution original video based on the pre-trained neural network-based video coding module. The video encoding module and the video decoding module are trained simultaneously and matched to perform encoding and decoding to achieve compression and decompression functions. The size of the space fill is adaptively adjusted based on the size of the original low-resolution video and the block size required by the network structure in the video encoding module to reduce the number of video pixels filled. The transmitting end transmits the target encoding to the receiving end. The receiving end decodes the target encoding based on a pre-trained neural network-based video decoding module to obtain a target low-resolution video. The receiving end up upsamples the target low-resolution video to obtain a target video. The transmitting end transmits the target encoding to the receiving end. The receiving ...

[0058] This setup, by downsampling the original video to obtain a low-resolution original video, concentrates the encoding and decoding operations in the low-resolution space, significantly reducing the amount of data processing (as mentioned in the disclosure, reducing the space size to 1 / 64 of the original), reducing the storage and transmission overhead of intermediate data, and improving compression efficiency from a fundamental level. The adaptive pixel space filling mechanism aims to reduce fill pixels, dynamically calculating the filling amount based on the low-resolution original video size and the block size required by the network structure. Compared with the traditional fixed-size filling method, it can save more than 20% of the extra pixel transmission, further reducing redundant data and optimizing the utilization of storage and transmission bandwidth. The design of low-resolution encoding and decoding combined with adaptive filling reduces the computing power requirements of hardware devices, making this method easier to deploy on low-to-medium computing power platforms such as consumer-grade GPUs and mobile devices, expanding its application scenarios (such as video conferencing, mobile live streaming, etc.). The entire process forms a complete closed loop from original video acquisition to target video reconstruction, with each step working closely together, balancing compression ratio and real-time performance. It can meet the needs of practical scenarios such as efficient video transmission, storage, and real-time communication, and has high practical value. Attached Figure Description

[0059] The above and other objects, features, and advantages of this application will become more apparent from the more detailed description of the embodiments of this application in conjunction with the accompanying drawings. The drawings are provided to further illustrate the embodiments of this application and form part of the specification. They are used together with the embodiments of this application to explain this application and do not constitute a limitation thereof. In the drawings, the same reference numerals generally represent the same components or steps.

[0060] Figure 1 This is a flowchart illustrating an efficient real-time compression method for neural encoding and decoding provided in one embodiment of this application.

[0061] Figure 2 This is a schematic diagram of the structure of an efficient real-time compression system for neural encoding and decoding provided in one embodiment of this application.

[0062] Figure 3 This is a schematic diagram of an electronic device structure provided in one embodiment of this application. Detailed Implementation

[0063] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0064] This application proposes a neural video compression system designed for high compression ratios, high real-time performance, and cross-platform deployment. The core idea is that at the sending end, the original video sequence is first downsampled by a fixed ratio to form a low-resolution representation. To reduce the size of data for subsequent processing; then, using an adaptive space-filling strategy that matches the network block size, only... The outermost layer is padded with necessary rows and columns to avoid introducing redundant pixels. The encoding end employs a lightweight neural network encoder with end-to-end joint training. Code networks perform implicit fusion of inter-frame information in a low-resolution latent space, eliminating the need for explicit motion estimation or optical flow computation, significantly improving performance. Reduce memory access and synchronization latency. Decouple quantization parameters from network weights, and use a runtime branching and plugging mechanism to adjust parameters based on real-time data. The bitrate is dynamically adjusted based on bandwidth and hardware utilization, without requiring a model restart. The encoded bitstream is then transmitted over the network to the receiving end. The symmetrically designed decoding network directly restores the low-resolution frames, and then restores them to the original resolution through single-stage upsampling. The entire... The link maintains consistent inference at int8 quantization precision, is compatible with consumer-grade GPUs, mobile devices, and edge devices, and achieves low power consumption. Real-time video communication with low latency and high compression efficiency.

[0065] After introducing the basic principles of this application, various non-limiting embodiments of this application will be described in detail below with reference to the accompanying drawings.

[0066] Figure 1 This is a flowchart illustrating an efficient real-time compression method for neural encoding and decoding provided in one embodiment of this application. Figure 1 As shown, the method includes the following:

[0067] Step S110: The sending end acquires the original video;

[0068] The sending end first collects or receives the raw video data that needs to be processed. This video can be high-definition / ultra-high-definition video (such as 4K, 8K, etc.) from cameras, storage devices or other data sources, providing input for subsequent compression processing.

[0069] Step S120: The transmitting end downsamples the original video to obtain a low-resolution original video;

[0070] To reduce data processing volume, the sending end downsamples the original video resolution by a fixed ratio (e.g., reducing it to 1 / 8 of the original resolution). By reducing the video size, the computational load and intermediate data volume during encoding and decoding are reduced, optimizing the system's operational efficiency from the ground up, while also laying the foundation for subsequent encoding and decoding in the lower resolution space.

[0071] In step S130, the transmitting end performs adaptive pixel space filling on the low-resolution original video based on the size of the original low-resolution video and the block size required by the network structure in the pre-trained neural network-based video coding module, with the goal of reducing the number of filled video pixels.

[0072] Because neural network operations such as convolution and block partitioning have integer divisibility requirements on the size of the input video (which must match the block size in the network structure), the sending end dynamically calculates the number of pixels to be filled based on the actual size of the low-resolution video and the block size required by the network. The filling process is performed symmetrically around the perimeter of the video to minimize the additional pixel redundancy. Compared with traditional fixed-size filling methods, this significantly reduces unnecessary pixel transmission and optimizes storage and bandwidth utilization.

[0073] In step S140, the transmitting end encodes the original video of the specified resolution based on a pre-trained neural network-based video encoding module to obtain the target encoding.

[0074] The padded low-resolution video is input into a pre-trained neural network video coding module. This module, based on the feature extraction and compression capabilities of the neural network, encodes the video to generate a compact target code (containing compressed video information). The coding module's design incorporates strategies such as implicit inter-frame fusion and low-resolution latent space modeling, reducing operational complexity while maintaining the compression ratio.

[0075] Step S150: The sending end transmits the target code to the receiving end;

[0076] The generated target code is transmitted to the receiving end via a network link (such as the Internet, local area network, etc.). Because the target code is efficiently compressed, the data volume is greatly reduced, which can reduce transmission latency, improve real-time performance, and is suitable for real-time scenarios such as video conferencing and live streaming.

[0077] In step S160, the receiving end decodes the target code based on a pre-trained neural network-based video decoding module to obtain the target low-resolution video.

[0078] The receiving end uses a neural network video decoding module that is trained synchronously and matched with the encoding module to decode the received target encoding. The decoding process corresponds to the encoding process, and through the reconstruction capability of the neural network, the low-resolution video data (i.e., the target low-resolution video) is recovered from the encoded information.

[0079] Step S170: Upsample the target low-resolution video to obtain the target video;

[0080] The receiving end upsamples the decoded low-resolution video to restore it to the same resolution as the original video, ultimately outputting a viewable or further processed target video. This upsampling process, combined with optimizations to the decoding module, ensures the quality of the reconstructed video and reduces distortion during resolution restoration.

[0081] The video encoding module and the video decoding module are trained simultaneously and are matched to each other to perform encoding and decoding to achieve compression and decompression functions. The synchronous training of the video encoding and decoding modules ensures precise matching, guaranteeing consistency between the encoding and decoding processes, which is the core foundation for achieving efficient compression and high-quality reconstruction.

[0082] Furthermore, the resolution of the original video is 8 times that of the low-resolution original video; the resolution of the target low-resolution video is the same as that of the low-resolution original video; and the resolution of the target video is 8 times that of the target low-resolution original video.

[0083] When downsampling the original video, the transmitting end strictly follows a 1 / 8 ratio, meaning the resolution of the low-resolution original video is 1 / 8 of the original video. For example, if the original video is 8K (7680×4320), the downsampled low-resolution original video will have a resolution of 960×540. By compressing the data volume to 1 / 64 of the original, the computational load and memory consumption of subsequent encoding and decoding are significantly reduced. This aligns with the "unified low-resolution latent space modeling" strategy in the patent, laying the foundation for real-time processing.

[0084] The target low-resolution video obtained by the receiving end through the decoding module has the exact same resolution as the original low-resolution video generated by the sending end. This strict matching ensures the consistency of the feature space during the encoding and decoding process, avoiding feature distortion caused by resolution mismatch. It is a direct manifestation of the accurate mapping achieved after the synchronous training of the neural network encoding and decoding modules.

[0085] When the receiving end upsamples the target low-resolution video, it magnifies it by a factor of 8 to make the resolution of the target video completely consistent with the original video sent from the receiving end. For example, if the low-resolution video is 960×540, it can be restored to 7680×4320 (8K) after upsampling. This satisfies the original clarity requirement of the video, and through the "downscaling and upscaling" resolution transformation, the core encoding and decoding operations are completed in the low-resolution latent space, reducing redundant calculations caused by high-resolution data, and balancing compression efficiency and reconstruction quality.

[0086] Upsampling and downsampling are core technologies for balancing data volume and image quality. While reducing data volume, they preserve as many key features of the original video as possible (such as edges, textures, and tonal transitions), providing high-quality low-resolution input for subsequent encoding and reconstruction. For example, if the original video is 8K resolution (7680×4320), this invention uses a 1 / 8 downsampling ratio (i.e., downsampling to 960×540). Specifically:

[0087] Anti-aliasing filtering downsampling is employed: Gaussian filtering or bilateral filtering is first applied to the original video frame to suppress high-frequency noise and moiré patterns (to avoid jagged edges or artifacts after downsampling), and then low-resolution pixels are generated through pixel aggregation (such as taking a weighted average of each 8×8 pixel area) to ensure the smoothness of edge contours and detail transitions.

[0088] Optimization for motion regions: For fast-moving objects in the video (such as people's movements in a video conference or moving objects in a surveillance video), adaptive window downsampling is used—the filter window is enlarged in areas of intense motion to reduce blur, and the window is reduced in static areas to preserve details, thus balancing the contradiction between motion blur and loss of details.

[0089] Below are two practical strategies for "both reducing and enlarging the image while maintaining clarity as much as possible," illustrated with examples of typical scenarios. Each strategy follows the same principle:

[0090] When downsampling, first perform bandpass filtering to avoid high-frequency aliasing; when upsampling, first perform anti-aliasing or detail reconstruction, then sharpen; try to complete the scaling down / expansion in one go to avoid multiple cascading damage.

[0091] Strategy A: Traditional "filtering + interpolation" link (hardware-friendly, high real-time performance)

[0092] Downsampling (reduction) scenario: First, use a low-pass filter for anti-aliasing, then perform region average downsampling; Result: No jagged edges on text, and readability is maintained in 540p preview.

[0093] Upsampling (amplification) is first performed using interpolation amplification; then a lightweight Unsharp Mask is superimposed for restoration.

[0094] Strategy B: Downsample (reduce) low-resolution frames using Content Adaptive Filtering Network (AAFNet); the network training objective simultaneously constrains high-frequency energy and perceptual loss to ensure that texture is preserved after reduction.

[0095] Upsampling (enlargement) in real time runs the lightweight ESRGAN-Tiny, directly upsampling and adding detail-enhancing convolutions, making the texture visibly sharper.

[0096] Adaptive pixel space filling is performed on the low-resolution original video, including:

[0097] The dimensions of the low-resolution original video are determined to be H×W;

[0098] The required block size for the network structure in the video encoding module is determined to be p×p;

[0099] The formula for calculating the fill amount in rows and columns is as follows:

[0100]

[0101] in, This indicates rounding up;

[0102] The formula for calculating the fill amount in rows and columns is as follows:

[0103]

[0104] By symmetrically filling p around the image h row and p w List

[0105] The image of the low-resolution original video is p-filled. h row and p w List.

[0106] The image padding p of the low-resolution original video h row and p w Columns, including:

[0107] The image of the low-resolution original video is p-filled on top. h1 OK;

[0108] The image under the low-resolution original video is p-filled. h2 OK;

[0109] The left side of the image of the low-resolution original video is p-filled. w1 List;

[0110] The right side of the image of the low-resolution original video is p-filled. w2 List;

[0111] Where, ph1 +p h2 =p h ;p w1 +p w2 =p w ;p h1 With p h2 The difference between them is 0, 1, or -1; p w1 With p w2 The difference between them is 0, 1, or -1; p h1 p h2 p w1 and p w2 All are integers.

[0112] Define the height and width of the low-resolution video frame to be processed, denoted as H×W (for example, the low-resolution video size obtained after 8x downsampling is 960×540, i.e., H=960, W=540).

[0113] Based on the neural network structure of the pre-trained video coding module, determine the fixed block size required for its convolution operation, attention window, or block partitioning, denoted as p×p (for example, the size of the convolution kernel or attention window in the network is 16×16, i.e., p=16).

[0114] The row fill amount is calculated as follows:

[0115] To ensure that the height H of the low-resolution video satisfies the divisibility requirement of the network block size p, the total number of rows to be added is calculated by rounding up:

[0116] Row fill amount (for example, if H=960, p=16, then) p h =60×16-960=0, no extra padding needed; if H=965, then... p h =61×16-965=11, 11 lines need to be added).

[0117] Similarly, the column fill amount calculation is as follows: To ensure that the width W satisfies the divisibility requirement of the block size p, calculate the total number of columns required: column fill amount. (For example, if W = 540, p = 16, p) w =34×16-540=4, 4 columns need to be added).

[0118] Symmetrically fill the edges of the low-resolution original video image, and evenly distribute the calculated ph rows on the top and bottom of the image (with a difference not exceeding 1). w The columns are evenly distributed to the left and right sides (with a difference of no more than 1).

[0119] For example, if p h=11, then fill the top with 5 rows and the bottom with 6 rows; if p w =4, then fill 2 columns on the left and 2 columns on the right.

[0120] The video size after filling is (H+p) h )×(W+p w It can perfectly match the divisibility requirements of size for operations such as convolution and attention window in the encoding module, while minimizing the additional pixel redundancy.

[0121] In some embodiments, the video encoding module includes a compression neural network module and an encoding module;

[0122] The compressed neural network module takes the features of the reconstructed previous frame image and the features of the current frame image as input, and generates compressed latent variables through the feature fusion mapping module.

[0123] The encoding module is used to encode the compressed latent variables to obtain the target code;

[0124] The video decoding module includes a decompression neural network module and a decoding module;

[0125] The decoding module is used to decode the target code to obtain compressed latent variables;

[0126] The decompression neural network module is also used to obtain the features of the reconstructed image corresponding to the previous frame image, and combine them with the compression latent variables to decompress and reconstruct the current frame image.

[0127] The video encoding module consists of a compression neural network module and an encoding module. The two work together to compress and encode the low-resolution original video. The specific process is as follows:

[0128] This module takes the features of the previous frame image and the features of the current frame image as input, and performs temporal correlation and redundancy compression on the features of the two frames through the built-in feature fusion mapping module, and finally generates compact compressed latent variables.

[0129] The “previous frame reconstruction features” here come from the decoding cache of historical frames by the encoding module, and are used to capture the temporal correlation between video frames (such as motion trends and background information).

[0130] "Current Frame Features" are the deep features of the low-resolution current frame after adaptive padding, containing spatial details of the current image;

[0131] The feature fusion process achieves implicit modeling of inter-frame redundancy through end-to-end training and optimization of neural networks (such as rate-distortion objective function), replacing the traditional explicit motion estimation module and reducing operational complexity.

[0132] The compressed latent variables received from the compressed neural network module are converted into a transmittable target code (i.e., bit stream) through entropy coding and other methods (such as context-based adaptive binary arithmetic coding), further reducing data redundancy and ensuring transmission efficiency.

[0133] The video decoding module consists of a decompression neural network module and a decoding module, forming a symmetrical design with the encoding module. The specific process is as follows:

[0134] The target encoding transmitted by the receiving end is restored to compressed latent variables through entropy decoding operation corresponding to the encoding module, providing original compressed information for subsequent image reconstruction.

[0135] The core function of the decompression neural network module is to acquire the features of the reconstructed image from the previous frame (consistent with historical features from the encoding end), and combine them with the compressed latent variables obtained from decoding. Through the feature mapping and reconstruction capabilities of the neural network, it restores the low-resolution image features of the current frame, ultimately outputting the target low-resolution video. The introduction of features from the previous frame ensures the temporal consistency of the decoding process, avoiding inter-frame jumps or distortion accumulation. The decompression process and the compressed neural network module at the encoding end achieve parameter matching through synchronous training, ensuring accurate mapping from latent variables to image features, laying a quality foundation for subsequent upsampling to restore the original resolution.

[0136] By fusing features from the previous frame and the current frame to generate latent variables, the traditional explicit motion estimation and compensation modules are abandoned, reducing the high complexity caused by pixel-level displacement calculations and lowering coding latency by more than 30%, making it suitable for real-time scenarios (such as video conferencing).

[0137] The compression / decompression neural network module and the encoding / decoding module are trained synchronously to ensure that the generation and restoration of latent variables are strictly corresponded, and that image details can still be maintained even at high compression ratios (e.g., when saving 21% bitrate compared to H.266 / VVC, there is no significant decrease in subjective image quality).

[0138] The separate design of the neural network module and the encoding / decoding module facilitates optimization for different hardware platforms (e.g., the neural network module is deployed in integer form to adapt to edge devices, while the encoding / decoding module is combined with TensorRT to accelerate throughput) and improves cross-scenario deployment capabilities.

[0139] This architecture achieves a balance between efficiency, image quality, and deployability in neural codecs through a "feature fusion-latent variable compression-cooperative decoding" chain, which is the core technical support for the "efficient real-time compression" of this invention.

[0140] Furthermore, the compression neural network module has an embedded decompression neural network module.

[0141] When processing the features of the current frame and the previous frame, the compressed neural network module simulates the decompression process at the receiving end through the embedded decompression neural network module. That is, it "pre-enacts" the decoding operation based on the generated compressed latent variables and outputs the reconstructed image of the current frame in order to determine the features of the reconstructed image of the current frame, which are then used as "features of the reconstructed image of the previous frame" when processing the next frame.

[0142] The embedded design allows the compression process to directly perceive the "reconstruction capability" of the decompression stage. By simulating decoding feedback, it optimizes the generation of latent variables, avoiding the problem in traditional separate designs where "the encoder cannot predict the decoder's error." This closed-loop optimization results in better rate-distortion performance (RD curve). At the same bitrate, the PSNR (Peak Signal-to-Noise Ratio) of the reconstructed image is improved by 0.5-1.2 dB compared to the separate design.

[0143] Because the compressed neural network module can preview the reconstruction result of the previous frame through the embedded decompression module, it can actively correct the errors that may be generated by the decoding of the previous frame when generating the latent variables of the current frame, avoid the accumulation of errors in time (such as the distortion of the picture of continuous movement of people in a video conference), and improve the overall reconstruction quality of long video sequences.

[0144] In summary, the "compressed neural network module embedded with a decompression neural network module" further enhances the advantages of this invention in terms of compression efficiency, real-time performance, and reconstruction quality through closed-loop optimization, parameter sharing, and error correction. It is one of the key designs for achieving "efficient real-time neural encoding and decoding".

[0145] Specifically, for the first frame, since there are no "reconstructed features from the previous frame" in the first frame, its compression process does not rely on the "inter-frame fusion" mechanism, but is mainly based on intra-frame feature modeling. Specifically, the specific method for handling the first frame is not limited in this embodiment.

[0146] Furthermore, if the sequence parameter set corresponding to the current frame image and the previous frame image is the same, then the target encoding of the current frame image does not need to carry the encoding for indicating the sequence parameter set, and the video decoding module uses the sequence parameter set of the previous frame image as the sequence parameter set of the current frame image.

[0147] If the sequence parameter sets corresponding to the current frame image and the previous frame image are different, the target encoding of the current frame image carries an encoding for indicating the sequence parameter set, and the video decoding module determines the sequence parameter set of the current frame image based on the encoding for indicating the sequence parameter set.

[0148] The Sequence Parameter Set (SPS) contains basic configuration information of the video sequence (such as resolution, frame rate, sampling format, etc.) and is a key parameter in the encoding and decoding process. This invention manages the SPS through an "on-demand carrying and buffered reuse" strategy, with the specific rules as follows:

[0149] SPS same scenario: If the SPS of the current frame is the same as that of the previous frame (e.g., the video sequence has not changed the resolution, frame rate and other basic configurations), the target encoding of the current frame does not contain SPS related encoding, and the video decoding module of the receiving end directly reuses the SPS buffered in the previous frame as the parameter set of the current frame.

[0150] Different SPS scenarios: If there is a difference between the SPS of the current frame and the previous frame (such as switching the video from 1080p to 4K, or adjusting the frame rate from 30fps to 60fps), the target encoding of the current frame needs to carry the updated SPS encoding. The receiving end decoding module parses the SPS of the current frame based on this encoding, overwrites the original buffer, and uses it for subsequent decoding.

[0151] This configuration reduces bitstream redundancy and improves compression efficiency. In video sequences, the SPS (Single Parameter Point) typically remains stable over a considerable period (e.g., the resolution remains constant throughout an entire video conference or a surveillance recording). By reusing the same SPS, it avoids carrying fixed parameters repeatedly in each frame, significantly reducing the amount of header data. According to test data from the "Modular Bitstream and SPS Buffer Management" section of the document, this mechanism can reduce SPS-related bit overhead by more than 60%, making it particularly suitable for the compression, storage, and transmission of long video sequences.

[0152] It ensures real-time performance and reduces transmission latency; by eliminating repetitive SPS encoding and transmission steps, it reduces the amount of encoding calculation and transmission bandwidth occupied at the sending end, thereby increasing the generation and transmission speed of target encoding by 10%-15%. Combined with the low-resolution latent space modeling of this invention, it further supports the real-time encoding performance of 1080p video at 40fps on consumer-grade GPUs.

[0153] Adapting to dynamic scenarios and enhancing flexibility: When video sequences need to adjust their basic configurations (such as switching resolution in live streaming or dynamically adjusting the frame rate in a monitoring system), the updated SPS encoding can be carried over to quickly synchronize parameters, ensuring that the decoding end accurately adapts to the new configuration and avoiding image distortion or decoding failure. This dynamic adjustment capability makes the system compatible with multiple scenarios (such as resolution switching in video conferencing and adaptive resolution in cloud live streaming).

[0154] SPS’s caching and reuse mechanism can be seamlessly integrated with the scheduling logic of underlying hardware acceleration libraries (such as TensorRT) to improve deployment efficiency. The receiver decoding module does not need additional memory interaction overhead for caching and calling SPS, which reduces hardware resource consumption and is especially suitable for the computing power and energy consumption constraints of edge devices.

[0155] This mechanism complements the "pluggable rate control" and "low-resolution latent space modeling" strategies of this invention: when the rate control module dynamically adjusts the quantization parameters, if no SPS change is involved, the bit rate fluctuation can be further reduced by reusing the SPS; in the low-resolution latent space, the stable reuse of the SPS reduces the interference of additional parameter parsing on real-time decoding, ensuring the parameter consistency of the upsampling recovery process.

[0156] In summary, SPS's dynamic management mechanism further strengthens the core advantages of this invention in efficient compression and practical deployment through multiple functions such as "redundancy reduction, real-time performance improvement, and scenario adaptation."

[0157] Furthermore, it also includes: obtaining network bandwidth and hardware utilization; and minimizing the quantization step size of the video encoding module and the video decoding module while ensuring that the network bandwidth and hardware utilization are within a preset range.

[0158] This mechanism dynamically optimizes the quantization step size (QP) by sensing network transmission capacity and hardware operating status in real time. It minimizes image quality fluctuations while ensuring that network bandwidth and hardware resources are not overloaded. The specific process is as follows:

[0159] The sending end collects the current available bandwidth in real time through the network interface (such as the link bandwidth fed back through the RTCP protocol in a video conferencing scenario) and determines the bandwidth upper limit threshold (the upper limit of the preset range);

[0160] At the same time, the current hardware utilization (u) is obtained through hardware monitoring interfaces (such as GPU CUDA utilization and CPU load rate), and the safe range of utilization is determined in combination with the device performance (such as the preset range of utilization for consumer-grade GPUs is 30%-70% to avoid latency caused by full load).

[0161] The bit rate R(QP) and the quantization step size QP satisfy an exponential decay relationship:

[0162] R(QP)≈a exp(-b QP)

[0163] Where a and b are constants calibrated offline. The system measures the current utilization rate u in real time and selects the optimal quantization branch:

[0164]

[0165] To achieve the preset target utilization rate u target Down-balanced bitrate and real-time performance.

[0166] The selected branch is immediately plugged into the coding process without restarting or reloading the entire model, ensuring that the system can smoothly switch compression strategies based on hardware load and bandwidth limitations in various scenarios such as video conferencing, remote monitoring, and mobile live streaming, achieving adaptive adjustment of "perceptual computing capabilities".

[0167] Specifically, the quantization step size corresponding to the rated value can be determined based on the utilization rate u of each hardware and the network bandwidth; the largest quantization step size is selected so that the utilization rate u of each hardware and the network bandwidth do not exceed the corresponding rated value.

[0168] In some embodiments, the process of training the video encoding module and the video decoding module includes:

[0169] The video encoding module and the video decoding module are trained using floating-point weights.

[0170] The training floating-point weights are rounded to the nearest integer.

[0171] Verify whether the rounded reconstruction error increments of the video encoding module and the video decoding module are within a preset range;

[0172] If not, retrain the video encoding module and the video decoding module.

[0173] This training process employs a closed-loop design of "floating-point training - integer conversion - error verification" to enable the model to adapt to integer deployment while ensuring reconstruction quality. The specific steps are as follows:

[0174] First, the video encoding and decoding modules are trained using floating-point precision (e.g., 32-bit floating-point numbers). The training objective is based on an end-to-end rate-distortion optimization function to ensure the model achieves an optimal balance between compression ratio and reconstruction quality. This stage focuses on optimizing the neural network's feature extraction, fusion, and latent variable mapping capabilities, laying a high-quality foundation for subsequent integer conversion.

[0175] After training, the floating-point weights W are converted into integer weights through a quantization mechanism.

[0176]

[0177] Where b is the quantization bit width, and the function round(·) represents rounding. After deployment, determine the reconstruction error increment:

[0178] By using iterative training, the reconstruction error increment is strictly controlled below the allowable threshold, reducing the impact of floating-point operations on cross-device consistency, optimizing inference consistency deployed on consumer-grade GPUs and embedded chips, and significantly reducing power consumption and memory overhead.

[0179] Specifically, when retraining the video encoding module and the video decoding module, several integer parameters can be randomly selected and kept constant. The video encoding module and the video decoding module are then trained using floating-point weights, while other parameters are adjusted. This process is iterated continuously to find suitable integer-based video encoding and decoding modules that meet the requirements.

[0180] The system embodiments of this application can be used to execute the method embodiments of this application. For details not disclosed in the system embodiments of this application, please refer to the method embodiments of this application.

[0181] Figure 2 The diagram shown is a block diagram of an efficient real-time compression system for neural encoding and decoding according to an embodiment of this application. Figure 2 As shown, the system includes: a transmitter 1 and a receiver 2;

[0182] The sending end 1 acquires the original video;

[0183] The transmitting end 1 downsamples the original video to obtain a low-resolution original video.

[0184] The transmitting end 1 performs adaptive pixel space filling on the low-resolution original video based on the size of the original low-resolution video and the block size required by the network structure in the pre-trained neural network-based video coding module, with the goal of reducing the number of filled video pixels.

[0185] The target code is obtained by encoding the original video at the specified resolution using a pre-trained neural network-based video encoding module.

[0186] The sending end 1 transmits the target code to the receiving end;

[0187] The receiving end 2 decodes the target code based on a pre-trained neural network-based video decoding module to obtain a target low-resolution video; and upsamples the target low-resolution video to obtain the target video.

[0188] The video encoding module and the video decoding module are trained simultaneously and are matched with each other to perform encoding and decoding to achieve compression and decompression functions.

[0189] Specifically, in practical applications, the sending end can be a cloud server, and there can be multiple receiving ends.

[0190] In summary, the solution provided in this application significantly reduces the overall complexity of the video compression system while maintaining or even surpassing the rate-distortion performance of traditional and existing neural codecs, enabling practical deployment of an end-to-end video compression system. It is suitable for scenarios such as online video transmission, remote conferencing, and edge device monitoring, possessing significant practicality and industrial value. This invention, by proposing several innovative optimization strategies from the perspective of operational complexity, achieves the following significant technical effects:

[0191] 1) Significantly improve real-time performance:

[0192] By removing high-overhead modules, reconstructing the encoding architecture, and deploying an integer model, the data transfer, memory access, and computation latency in the encoding process were significantly reduced, achieving a 1080p video encoding rate of 40fps on a general-purpose GPU platform with real-time execution capability.

[0193] (2) Ensure high compression ratio performance:

[0194] While maintaining superior compression ratio (saving 21% bitrate compared to VTM / H.266), it effectively improves rate-distortion curve performance through implicit frame modeling and unified latent space compression, outperforming existing state-of-the-art models on multiple public datasets.

[0195] (3) Enhance cross-device compatibility:

[0196] It supports consistent inference of models deployed on edge devices, consumer-grade GPUs, and the cloud, adapting to multiple hardware platforms and reducing operation and integration costs.

[0197] (4) Implement adaptive compression control:

[0198] The rate control mechanism based on the module library enables the system to dynamically adjust the compression configuration according to computing resources and network bandwidth, adapting to different application needs (such as video conferencing, monitoring, live streaming, etc.).

[0199] (5) Saves system resources and power consumption:

[0200] By using integer models and unified low-resolution modeling, the overhead of video memory and power consumption is significantly reduced, making it particularly suitable for video transmission and storage tasks in edge computing scenarios.

[0201] Specifically, this invention can be widely applied in the following fields:

[0202] Video conferencing and online education systems: Improve real-time coding efficiency and reduce latency;

[0203] Security monitoring system: Improve bandwidth utilization and storage compression capabilities;

[0204] Mobile video applications and social platforms: Adapting to edge devices and mobile inference;

[0205] Autonomous driving and intelligent transportation: Achieving low-power real-time compression of multiple video streams;

[0206] Cloud video processing and CDN transmission system: improving transmission efficiency and controlling operating costs.

[0207] This invention possesses outstanding engineering practicality and technological foresight, solving the key technical bottlenecks that make existing neural video compression schemes difficult to deploy practically, and has important reference value for promoting the formation of the next generation of video coding standards.

[0208] Below, for reference Figure 3 This describes an electronic device according to embodiments of the present application. Figure 3 A block diagram of an electronic device according to an embodiment of this application is illustrated.

[0209] like Figure 3 As shown, the electronic device 300 includes one or more processors 310 and memory 320.

[0210] The processor 310 may be a central processing unit (CPU) or other form of processing unit with data processing and / or instruction execution capabilities, and may control other components in the electronic device 300 to perform desired functions.

[0211] The memory 320 may include one or more computer program products, which may include various forms of computer-readable storage media, such as volatile memory and / or non-volatile memory. The volatile memory may include, for example, random access memory (RAM) and / or cache memory. The non-volatile memory may include, for example, read-only memory (ROM), hard disk, flash memory, etc. One or more computer program instructions may be stored on the computer-readable storage medium, and the processor 310 may execute the program instructions to implement the efficient real-time compression method for neural encoding and decoding described in the various embodiments of this application above, and / or other desired functions. Various contents, such as category correspondences, may also be stored in the computer-readable storage medium.

[0212] In one example, the electronic device 300 may also include an input device 330 and an output device 340, which are interconnected via a bus system and / or other forms of connection mechanism (not shown).

[0213] In addition, the input device 330 may also include, for example, a keyboard, mouse, interface, etc. The output device 340 can output various information to the outside, including analysis results, etc. The output device 340 may include, for example, a display, speaker, printer, and communication network and its connected remote output devices, etc.

[0214] Of course, for the sake of simplicity, Figure 3 Only some of the components of the electronic device relevant to this application are shown in this illustration; components such as buses, input / output interfaces, etc., are omitted. In addition, the electronic device may include any other suitable components depending on the specific application.

[0215] In addition to the methods and devices described above, embodiments of this application may also be computer program products, which include computer program instructions that, when executed by a processor, cause the processor to perform portions of the efficient real-time compression method for neural encoding and decoding described in the "Exemplary Methods" section above, either at the sending end or at the receiving end.

[0216] The computer program product can be written in any combination of one or more programming languages ​​to perform the operations of the embodiments of this application. The programming languages ​​include object-oriented programming languages ​​such as Java and C++, as well as conventional procedural programming languages ​​such as C or similar languages. The program code can be executed entirely on the user's computing device, partially on the user's computing device, as a standalone software package, partially on the user's computing device and partially on a remote computing device, or entirely on a remote computing device or server.

[0217] Furthermore, embodiments of this application may also be computer-readable storage media storing computer program instructions that, when executed by a processor, cause the processor to perform portions of the efficient real-time compression method for neural encoding and decoding according to various embodiments of this application described in the "Exemplary Methods" section of this specification, either the sending end or the receiving end.

[0218] The computer-readable storage medium may be any combination of one or more readable media. A readable medium may be a readable signal medium or a readable storage medium. A readable storage medium may, for example, include, but is not limited to, electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatuses, or devices, or any combination thereof. More specific examples of readable storage media (a non-exhaustive list) include: electrical connections having one or more wires, portable disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.

[0219] The above description has been given for purposes of illustration and description. Furthermore, this description is not intended to limit the embodiments of this application to the forms disclosed herein. Although numerous exemplary aspects and embodiments have been discussed above, those skilled in the art will recognize certain variations, modifications, alterations, additions, and sub-combinations thereof.

Claims

1. A highly efficient real-time compression method for neural encoding and decoding, characterized in that, The efficient real-time compression method for neural codecs, applicable to systems including a transmitter and a receiver, includes: The sending end acquires the original video; The transmitting end downsamples the original video to obtain a low-resolution original video; The transmitting end performs adaptive pixel space filling on the low-resolution original video based on the size of the original low-resolution video and the block size required by the network structure in the pre-trained neural network-based video coding module, with the goal of reducing the number of filled video pixels. The target code is obtained by encoding the original video at the specified resolution using a pre-trained neural network-based video encoding module. The sending end transmits the target code to the receiving end; The receiving end decodes the target code based on a pre-trained neural network-based video decoding module to obtain the target low-resolution video. The target low-resolution video is upsampled to obtain the target video; The video encoding module and the video decoding module are trained simultaneously and are matched with each other to perform encoding and decoding to achieve compression and decompression functions.

2. The efficient real-time compression method for neural encoding and decoding according to claim 1, characterized in that, The resolution of the original video is 8 times that of the low-resolution original video; The resolution of the target low-resolution video is the same as the resolution of the original low-resolution video; The resolution of the target video is 8 times that of the original low-resolution target video.

3. The efficient real-time compression method for neural encoding and decoding according to claim 1, characterized in that, Adaptive pixel space filling is performed on the low-resolution original video, including: The dimensions of the low-resolution original video are determined to be H×W; The required block size for the network structure in the video encoding module is determined to be p×p; The formula for calculating the fill amount in rows and columns is as follows: in, This indicates rounding up; The formula for calculating the fill amount in rows and columns is as follows: By symmetrically filling p around the image h row and p w List The image of the low-resolution original video is p-filled. h row and p w List.

4. The efficient real-time compression method for neural encoding and decoding according to claim 1, characterized in that, The image padding p of the low-resolution original video h row and p w Columns, including: The image of the low-resolution original video is p-filled on top. h1 OK; The image under the low-resolution original video is p-filled. h2 OK; The left side of the image of the low-resolution original video is p-filled. w1 List; The right side of the image of the low-resolution original video is p-filled. w2 List; Where, p h1 +p h2 =p h ;p w1 +p w2 =p w ;p h1 With p h2 The difference between them is 0, 1, or -1; p w1 With p w2 The difference between them is 0, 1, or -1; p h1 p h2 p w1 and p w2 All are integers.

5. The efficient real-time compression method for neural encoding and decoding according to claim 1, characterized in that, The video encoding module includes a compression neural network module and an encoding module; The compressed neural network module takes the features of the reconstructed previous frame image and the features of the current frame image as input, and generates compressed latent variables through the feature fusion mapping module. The encoding module is used to encode the compressed latent variables to obtain the target code; The video decoding module includes a decompression neural network module and a decoding module; The decoding module is used to decode the target code to obtain compressed latent variables; The decompression neural network module is also used to obtain the features of the reconstructed image corresponding to the previous frame image, and combine them with the compression latent variables to decompress and reconstruct the current frame image.

6. The efficient real-time compression method for neural encoding and decoding according to claim 5, characterized in that, The compression neural network module has a built-in decompression neural network module.

7. The efficient real-time compression method for neural encoding and decoding according to claim 1, characterized in that, If the sequence parameter set corresponding to the current frame image and the previous frame image is the same, then the target encoding of the current frame image does not need to carry the encoding for indicating the sequence parameter set, and the video decoding module uses the sequence parameter set of the previous frame image as the sequence parameter set of the current frame image. If the sequence parameter sets corresponding to the current frame image and the previous frame image are different, the target encoding of the current frame image carries an encoding for indicating the sequence parameter set, and the video decoding module determines the sequence parameter set of the current frame image based on the encoding for indicating the sequence parameter set.

8. The efficient real-time compression method for neural encoding and decoding according to claim 1, characterized in that, This also includes obtaining network bandwidth and hardware utilization; While ensuring that network bandwidth and hardware utilization are within a preset range, the quantization step size of the video encoding module and the video decoding module is minimized.

9. The efficient real-time compression method for neural encoding and decoding according to claim 1, characterized in that, The process of training the video encoding module and the video decoding module includes: The video encoding module and the video decoding module are trained using floating-point weights. The training floating-point weights are rounded to the nearest integer. Verify whether the rounded reconstruction error increments of the video encoding module and the video decoding module are within a preset range; If not, retrain the video encoding module and the video decoding module.

10. A high-efficiency real-time compression system for neural encoding and decoding, characterized in that, Includes the sending end and the receiving end; The sending end acquires the original video; The transmitting end downsamples the original video to obtain a low-resolution original video; The transmitting end performs adaptive pixel space filling on the low-resolution original video based on the size of the original low-resolution video and the block size required by the network structure in the pre-trained neural network-based video coding module, with the goal of reducing the number of filled video pixels. The target code is obtained by encoding the original video at the specified resolution using a pre-trained neural network-based video encoding module. The sending end transmits the target code to the receiving end; The receiving end decodes the target code based on a pre-trained neural network-based video decoding module to obtain the target low-resolution video. The target low-resolution video is upsampled to obtain the target video; The video encoding module and the video decoding module are trained simultaneously and are matched with each other to perform encoding and decoding to achieve compression and decompression functions.

Images