A method for spatial semantic perception and adaptive compression oriented to an encoding end
By generating a semantic importance map using a dual-stream deep neural network and combining it with a rate-distortion optimization objective function, the problem of semantic blindness in video coding technology is solved, adaptive video compression is achieved, and the fidelity of key information and compression efficiency are improved.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- NANTONG UNIV
- Filing Date
- 2026-03-12
- Publication Date
- 2026-07-31
AI Technical Summary
Existing video coding technologies cannot understand the semantic meaning of video content, resulting in wasted bit resources and loss of key information. Traditional ROI coding cannot adaptively adjust.
A two-stream deep neural network is used to generate pixel-level semantic importance maps. The objective function is optimized by combining semantic perception rate distortion, and the encoding decision is dynamically adjusted to achieve adaptive compression.
It effectively avoids inaccurate bit resource allocation, ensures the integrity of critical information, achieves pixel-level precision adaptive quantization parameter allocation, and improves adaptability in complex scenarios.
Smart Images
Figure CN122496631A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of video coding technology, and in particular to a method and system that combines computer vision and deep learning to perform semantic understanding of video spatial content at the encoding end, and thereby achieve adaptive bit allocation and compression. Background Technology
[0002] With the widespread adoption of ultra-high-definition video (UHD, such as 4K / 8K), the amount of video data is growing exponentially, putting enormous pressure on storage and transmission. Efficient video coding technology is key to addressing this challenge. New-generation video coding standards, such as AVS3 and H.266 / VVC, have significantly improved compression efficiency by introducing more complex block partitioning structures (such as QTMT), intra-frame prediction modes, and transform techniques. However, the core of these standards—rate-distortion optimization (RDO)—is essentially still a pixel- or block-based mathematical optimization process, aiming to balance pixel distortion with the number of encoded bits.
[0003] This pixel-based RDO has a fundamental flaw: "semantic blindness." The encoder cannot understand the semantic meaning of the video content. For example, it treats a human face and a solid-color background the same, attempting to minimize their mean squared errors. This leads to two problems. The first is a waste of bit resources: the encoder uses precious bits to smooth the subtle textures of the background to reduce negligible mean squared errors, but may suffer from insufficient bit budget, resulting in blurred details in critical areas such as faces. The second is the loss of critical information: in tasks such as remote surgery and industrial defect detection, critical information such as the sharp points of surgical instruments and tiny cracks in products may not be visually significant, but is essential to the task. Traditional encoders lack mechanisms to protect this information, which may be lost after compression.
[0004] Based on the above problems, the academic community has made some explorations in this area. For end-to-end learning-based coding, although it has excellent performance, its design goal is mostly pixel-level fidelity (such as MS-SSIM, PSNR), lacking explicit modeling of the importance of high-level semantics; for region of interest (ROI) coding, traditional ROI coding usually relies on handmade features or simple detectors (such as face detection), which is difficult to handle complex and dynamically changing "semantic" concepts (such as "surgical instrument tip" in surgical scenarios or "specific type of defect" in industrial scenarios), and its ROI weights are fixed and cannot be adaptively adjusted.
[0005] Therefore, the key technical problem to be solved by this invention is how to enable the encoder to overcome "semantic blindness", establish a universal and computable semantic value model, and seamlessly embed it into the standard encoding process to achieve true semantic adaptive compression. Summary of the Invention
[0006] This invention aims to address the aforementioned problems in existing technologies by providing a spatial semantic awareness and adaptive compression method for the encoding end. The core idea of this method is to equip traditional encoders with a pair of "semantic eyes," enabling them to "understand" video content and dynamically and precisely allocate bit resources based on the semantic value of the content.
[0007] The inventive concept of this invention is as follows: This invention provides a spatial semantic awareness and adaptive compression method for the encoding end. First, a video frame to be encoded is acquired, and the video frame is input into a pre-trained two-stream deep neural network to obtain a pixel-level semantic importance map. Then, based on the semantic importance map, a rate-distortion optimization objective function for semantic awareness is constructed. Finally, according to the rate-distortion optimization objective function for semantic awareness, adaptive encoding decisions are made for the encoding units in the video frame.
[0008] To achieve the aforementioned objectives, the technical solution adopted by this invention is specifically as follows: a spatial semantic awareness and adaptive compression method for the encoding end, characterized by comprising the following steps:
[0009] 1.1 A two-stream deep neural network is used to process the input ultra-high-definition video frames, and the global visual saliency and local instance semantic information are fused to generate a pixel-level, multi-channel semantic importance map. This map not only marks the location of key objects, but also quantifies their importance.
[0010] 1.2. The semantic importance graph generated in 1.1 is used as a weighting factor and injected into the rate-distortion cost function of the traditional encoder to reconstruct the Lagrange multiplier and distortion calculation method, forming a new semantically aware optimization objective, namely, establishing a semantically aware rate-distortion optimization model.
[0011] 1.3 By solving the optimization problem in 1.2, the optimal coding mode, partitioning depth and quantization parameter (QP) are dynamically selected for coding tree unit (CTU) or smaller coding blocks, and adaptive coding decisions are executed to achieve "high fidelity in high semantic value regions and efficient compression in low semantic value regions".
[0012] Further, the two-stream deep neural network in step 1.1 includes:
[0013] 2.1 An improved lightweight Swing Transformer is used to process the downsampled global image and capture eye-catching regions in the scene;
[0014] 2.2. Using a CNN-Transformer hybrid architecture, scene-specific semantics (such as faces, scalpels, and defects) in video frames are accurately segmented at the original resolution or after appropriate downsampling.
[0015] 2.3. The fusion module receives the feature maps, semantic category masks, and semantic value maps from both outputs. Through a learnable gating mechanism, it dynamically calculates the final semantic importance weight for each position (e.g., when instance segmentation confidence is high, the fusion result leans towards the instance segmentation stream; conversely, it relies more on global saliency). The final generated importance map is denoted as... Its range is [0, 1].
[0016] Furthermore, the core of the semantic perception rate-distortion optimization model in step 1.2 is the reconstructed Lagrangian cost function: ,in:
[0017] 3.1 Candidate encoding parameters (including QP, mode, partitioning method, etc.);
[0018] 3.2 To use parameters The number of bits produced by encoding;
[0019] 3.3 It is semantic weighted distortion, which is not a simple pixel-level mean square error, but rather calculates the error based on the original pixels first. With reconstructed pixels The difference map is then compared with the semantic importance map. The formula for performing a pixel-by-pixel weighted summation is: This means that in regions of high semantic importance ( In high-fidelity regions, even small pixel errors are amplified, forcing the optimizer to choose a higher-fidelity mode; while in low-semantic regions ( Small pixel errors can be tolerated;
[0020] 3.4 It is a semantically adaptive Lagrange multiplier, which dynamically adjusts according to the semantic importance of the current coding block, for example, ,in It is the average value of the importance spectrum within the block. The original Lagrange multipliers are calculated based on the encoding configuration (QP, frame type, temporal level), which ensures that for highly important regions, the optimization process tends to reduce distortion (because...). As the value decreases, the weight of the distortion term relatively increases.
[0021] Furthermore, the adaptive coding decision process in step 1.3 is as follows: for each coding block, traverse the candidate coding parameters Q, calculate using the new cost function J from 1.2, and select the parameter combination that minimizes J. This process will eventually generate a spatially varying QP matrix, with smaller QP allocated to high semantic regions (high quality) and larger QP allocated to low semantic regions (high compression ratio).
[0022] Meanwhile, this invention proposes a spatial semantic awareness and adaptive compression system for the encoding end, comprising:
[0023] A semantic graph generation module is used to perform the step of generating a semantic importance graph as described in this invention;
[0024] The encoding decision optimization module is used to execute the steps of constructing a semantic awareness rate distortion optimization objective function and making adaptive encoding decisions as described in this invention.
[0025] The video encoding module is used to encode video frames based on encoding decisions.
[0026] Meanwhile, the present invention proposes an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the computer program is executed, it implements the steps of the method described in the present invention.
[0027] Furthermore, the present invention proposes a computer-readable storage medium having a computer program stored thereon, the computer program being configured to implement the steps of the method described in the present invention when invoked by a processor.
[0028] Finally, the present invention provides a computer program product comprising a computer program / instructions that, when executed by a processor, implement the steps of the method described in the present invention.
[0029] Compared with the prior art, the beneficial effects of the present invention are as follows:
[0030] (1) Traditional video encoders are usually optimized for rate distortion based on pixel-level mathematical errors (such as MSE, SAD), which makes it difficult to distinguish between key content such as faces and surgical instruments and non-key content such as background. This invention constructs a multi-dimensional semantic importance graph, enabling the encoder to understand video content and realize compression from not understanding to understanding, effectively avoiding the problem of inaccurate bit resource allocation.
[0031] (2) The semantic importance map generated by this invention tends to be pixel-level accurate. It is transmitted to the coding decision through semantic perception rate distortion optimization, and finally forms a spatially variable quantization parameter matrix. That is, small quantization parameters are allocated to high semantic regions (high quality coding) to ensure that key information is not lost, while large quantization parameters are allocated to low semantic regions (high compression rate) to maximize bit saving. This achieves the accuracy improvement from traditional block-level fixed quantization parameters to pixel-level adaptive quantization parameters.
[0032] (3) This invention uses the semantic importance graph as weights, proposes semantic adaptive Lagrange multipliers, and reconstructs the traditional rate-distortion optimization objective function. Compared with the traditional ROI encoding weighted distortion term, it is more in line with the physical meaning of rate-distortion theory. Moreover, it does not require modification of the core distortion calculation module of the encoder and is compatible with the existing standard H.266 / VVC.
[0033] (4) This invention dynamically fuses global visual saliency features, pixel-level semantic category masks and semantic values through a gating attention mechanism. It can not only identify known objects, but also perform generalized prediction of unknown important content, which significantly improves the model's adaptability in complex and ever-changing scenarios. Attached Figure Description
[0034] The accompanying drawings are provided to further illustrate the invention and form part of the specification. They are used together with the embodiments of the invention to explain the invention and do not constitute a limitation thereof.
[0035] Figure 1 This is an overall flowchart of the method of the present invention;
[0036] Figure 2 This is a diagram of the improved lightweight Swing Transformer network structure of the present invention;
[0037] Figure 3 This is a diagram of the CNN-Transformer hybrid architecture in this invention;
[0038] Figure 4 This is a diagram of the fusion module in this invention;
[0039] Figure 5 This is a schematic diagram of the system structure of the present invention. Detailed Implementation
[0040] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. Of course, the specific embodiments described herein are merely illustrative and not intended to limit the invention.
[0041] Example 1: See Figures 1 to 5 This embodiment provides a technical solution: a spatial semantic awareness and adaptive compression method for the encoding end, comprising the following steps:
[0042] 1. Using a two-stream deep neural network, the input ultra-high-definition video frames are processed, and global visual saliency and local instance semantic information are fused to generate a pixel-level, multi-channel semantic importance map. This map not only marks the location of key objects, but also quantifies their importance.
[0043] 2. The semantic importance graph generated in 1.1 is used as a weighting factor and injected into the rate-distortion cost function of the traditional encoder. The Lagrange multipliers and distortion calculation method are reconstructed to form a new semantically aware optimization objective, namely, to establish a semantically aware rate-distortion optimization model.
[0044] 3. By solving the optimization problem in 1.2, the optimal coding mode, partitioning depth and quantization parameter (QP) are dynamically selected for coding tree unit (CTU) or smaller coding blocks, and adaptive coding decisions are executed to achieve "high fidelity in high semantic value regions and efficient compression in low semantic value regions".
[0045] Specifically, in step 1, refer to Figure 2-4 The input ultra-high-definition video frames are processed to fuse global visual saliency and local instance semantic information to generate a pixel-level, multi-channel semantic importance map, including the following steps:
[0046] 1) An improved lightweight Swing Transformer is used to process the downsampled global image and capture striking regions in the scene;
[0047] 2) Employing a CNN-Transformer hybrid architecture, scene-specific semantics (such as faces, scalpels, and defects) in video frames are accurately segmented at the original resolution or after appropriate downsampling.
[0048] 3) The fusion module receives the feature maps, semantic category masks, and semantic value maps from both outputs. Through a learnable gating mechanism, it dynamically calculates the final semantic importance weight for each position (e.g., when instance segmentation confidence is high, the fusion result leans towards the instance segmentation stream; conversely, it relies more on global saliency). The final generated importance map is denoted as... Its range is [0, 1].
[0049] Specifically, in step 2, the semantic importance graph generated in step 1 is used as a weighting factor and injected into the rate-distortion cost function of the traditional encoder. The Lagrange multipliers and distortion calculation method are reconstructed to establish a semantic-aware rate-distortion optimization model. ,in:
[0050] 1) Candidate encoding parameters (including QP, mode, partitioning method, etc.);
[0051] 2) To use parameters The number of bits produced by encoding;
[0052] 3) It is semantic weighted distortion, which is not a simple pixel-level mean square error, but rather calculates the error based on the original pixels first. With reconstructed pixels The difference map is then compared with the semantic importance map. The formula for performing a pixel-by-pixel weighted summation is: This means that in regions of high semantic importance ( In high-fidelity regions, even small pixel errors are amplified, forcing the optimizer to choose a higher-fidelity mode; while in low-semantic regions ( Small pixel errors can be tolerated;
[0053] 4) It is a semantically adaptive Lagrange multiplier, which dynamically adjusts according to the semantic importance of the current coding block, for example, ,in It is the average value of the importance spectrum within the block. The original Lagrange multipliers are calculated based on the encoding configuration (QP, frame type, temporal level), which ensures that for highly important regions, the optimization process tends to reduce distortion (because...). As the value decreases, the weight of the distortion term relatively increases.
[0054] Specifically, the adaptive coding decision process in step 3 is as follows: for each coding block, traverse the candidate coding parameters. We use the new cost function J from section 1.2 to perform the calculation and select the parameter combination that minimizes J. This process will eventually generate a spatially varying QP matrix, with smaller QP allocated to high semantic regions (high quality) and larger QP allocated to low semantic regions (high compression ratio).
[0055] This embodiment uses industrial defect datasets such as MVTec AD to train the instance segmentation model, enabling it to accurately identify dozens of common defects such as scratches, dents, and foreign objects. During the semantic graph generation stage, defect regions are assigned extremely high weights when generating the importance graph. Simultaneously, since defects can appear anywhere, global saliency flow is also used to assist in preventing the loss of crucial information due to missed detections by the segmentation model. In the adaptive encoding stage, the encoder performs high-quality compression on defect regions, while applying extremely high-intensity compression to the normal product surface or background such as the conveyor belt, even allowing some background blocks to be discarded directly.
[0056] Test data shows that, while ensuring that the defect detection rate does not decrease, the method of the present invention can compress the average size of video files to 1 / 5 of the original size, which greatly reduces storage costs.
[0057] Example 2: This example proposes a spatial semantic awareness and adaptive compression system for the encoding end, including:
[0058] A semantic graph generation module is used to perform the step of generating a semantic importance graph as described in this invention;
[0059] The encoding decision optimization module is used to execute the steps of constructing a semantic awareness rate distortion optimization objective function and making adaptive encoding decisions as described in this invention.
[0060] The video encoding module is used to encode video frames based on encoding decisions.
[0061] Example 3: This example proposes an electronic system, including: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to perform the method steps of the present invention.
[0062] Example 4: This example proposes a computer-readable storage medium storing a computer program thereon. When the computer program is executed by a processor, it implements the steps of the method described in this invention, which will not be repeated here.
[0063] Example 5: This example proposes a computer program product, including a computer program / instructions. When the computer program / instructions are executed by a processor, they implement the steps of the method described in this invention, which will not be repeated here.
[0064] It should be noted that the processing flow of embodiments 2-5 corresponds to the specific steps of the method provided in embodiment 1 of the present invention, and has the corresponding functional modules and beneficial effects of the method. Technical details not described in detail in this embodiment can be found in the method provided in embodiment 1 of the present invention.
[0065] The program code used to implement the methods of this application may be written in any combination of one or more programming languages. This program code may be provided to a processor or controller of a general-purpose computer, special-purpose computer, or other programmable data processing device, such that when executed by the processor or controller, the functions / operations specified in the flowcharts and / or block diagrams are implemented. The program code may be executed entirely on a machine, partially on a machine, as a standalone software package partially on a machine and partially on a remote machine, or entirely on a remote machine or server.
[0066] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.
Claims
1. A spatial semantic awareness and adaptive compression method for the encoding end, characterized in that, Includes the following steps: S1. Input the video frame to be encoded into a pre-trained dual-stream deep neural network to obtain a pixel-level semantic importance map, which is used to indicate the semantic value of different spatial locations in the video frame. S2. Based on the semantic importance graph, construct a rate-distortion optimization objective function for semantic perception; S3. Based on the semantically aware rate-distortion optimization objective function, make adaptive coding decisions for the coding units in the video frame.
2. The spatial semantic awareness and adaptive compression method for the encoding end according to claim 1, characterized in that, In S1, the two-stream deep neural network includes: S21. Improved Swin-Transformer, a global visual saliency extraction sub-network, for extracting global visual saliency features of video frames; S22, Local Instance Segmentation Subnetwork CNN-Transformer Hybrid Architecture, used to segment scene-specific semantics in video frames, generating pixel-level semantic category masks and semantic value maps; S23. The fusion module is connected to the global saliency extraction subnetwork and the local instance segmentation subnetwork to dynamically fuse global visual saliency features, pixel-level semantic category masks, and semantic value maps to generate the final semantic importance map.
3. The spatial semantic awareness and adaptive compression method for the encoding end according to claim 1, characterized in that, In S3, the rate-distortion optimization objective function for semantic awareness is: ； in, For semantic importance graph, Candidate encoding parameters, For the corresponding bitrate, For semantic weighting distortion, Semantically adaptive Lagrange multipliers; The semantic weighted distortion By calculating the error map between the original pixels and the reconstructed pixels, and then comparing this error map with the semantic importance map... The result is obtained by weighted summation pixel by pixel. The semantically adaptive Lagrange multiplier The average semantic importance of the current coding unit Inversely proportional.
4. The spatial semantic awareness and adaptive compression method for the encoding end according to claim 1, characterized in that, The adaptive encoding decision includes the following decisions: quantization parameters, encoding mode, and partition depth.
5. A spatial semantic awareness and adaptive compression method for the encoding end according to claim 1, characterized in that, The method described applies the H.266 / VVC video coding standard framework.
6. A spatial semantic awareness and adaptive compression system for the encoding end, characterized in that, include: S81, a semantic graph generation module, used to perform the step of generating a semantic importance graph as described in any one of claims 1-2; S82, Encoding decision optimization module, used to execute the steps of constructing the semantic awareness rate distortion optimization objective function and making adaptive encoding decisions as described in any one of claims 1 and 3; S83, Video Encoding Module, is used to encode video frames according to encoding decisions.
7. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the computer program is executed, it implements the steps of the method as described in any one of claims 1 to 5.
8. A computer-readable storage medium having a computer program stored thereon, characterized in that, The computer program is configured to implement the steps of the method according to any one of claims 1 to 5 when invoked by a processor.
9. A computer program product comprising a computer program / instructions, characterized in that, When the computer program / instructions are executed by the processor, they implement the steps of the method according to any one of claims 1 to 5.