Video compression method and device, computer equipment and storage medium

Abstract

The invention discloses a video compression method and device, computer equipment and a storage medium, and the method at least comprises the steps: collecting video data, and carrying out the preprocessing to obtain a standard video sequence; performing framing processing and classifying video frames; performing multi-modal feature extraction to obtain feature data including spatio-temporal features, depth features and motion features, and generating an entropy value for quantifying video content complexity; determining a coding strategy for the video frame based on the video frame type and the entropy value, the coding strategy comprising a quantization parameter; and encoding according to the encoding strategy and outputting a compressed code stream. The method has the advantages that video frame classification and context management can be effectively realized on the basis of multi-modal features, local texture details and long-distance space-time dependence are considered, space-time features, depth features and motion features are effectively integrated to generate entropy representing content complexity, different coding and dynamic bit allocation strategies are realized on the basis of frame types and the entropy, and the coding efficiency is improved. And the balance between the compression efficiency and the complexity of video compression is ensured.

Description

A video compression method, apparatus, computer equipment, and storage medium. Technical Field

[0001] This invention relates to the field of video image processing technology, and specifically to a video compression method, apparatus, computer equipment, and storage medium. Background Technology

[0002] In recent years, with the explosive growth of 4K / 8K ultra-high-definition, VR / AR and AIGC-generated video content, video data accounts for more than 80% of global network traffic, which puts forward unprecedented requirements for high efficiency, low latency and low energy consumption for video compression technology. Although existing video coding technology is constantly evolving, there are still significant technical bottlenecks and inherent defects in dealing with the above-mentioned complex requirements, mainly reflected in the following four aspects: (1) Traditional coding standards cannot achieve both efficiency and complexity. The core algorithms of mainstream coding standards such as H.264 / AVC and H.265 / HEVC are based on manually designed prediction, transformation and entropy coding modules, which limit the compression efficiency when facing complex scenarios such as ultra-high-definition and high dynamic range. Although the new generation standard H.266 / VVC has improved the compression rate by about 50% through more refined block division and prediction tools, its coding complexity has increased sharply, reaching more than 10 times that of H.264, which requires extremely high hardware computing power, resulting in a significant increase in coding latency, making it difficult to meet the needs of low-latency application scenarios such as live streaming and cloud gaming.

[0003] (2) Using convolutional neural networks (CNNs) as the basic architecture, the inherent local receptive field characteristics make it difficult to effectively model long-distance spatiotemporal dependencies between video frames (such as complex motion trajectories and global illumination changes), resulting in unstable phenomena such as screen flickering and increased artifacts when reconstructing videos. Network models designed to pursue high compression rates usually have a huge number of parameters and complex structures, resulting in slow inference speed at the encoding end, which cannot meet the requirements of real-time processing.

[0004] (3) The rigid encoding resource allocation strategy cannot adapt to dynamic changes in content. Traditional encoders and existing bitrate control and bit allocation strategies are mostly based on fixed rules or simple local features (such as inter-frame differences and gradient information), lacking a deep understanding of the semantics and spatiotemporal complexity of video content. Its rigid bit allocation mechanism cannot accurately and adaptively allocate bit resources according to the dynamic complexity of scene content (such as areas of intense movement, facial details, and static backgrounds), resulting in the contradiction of "insufficient bitrate in important areas and wasted bitrate in simple areas", and the overall compression efficiency cannot reach the optimal level.

[0005] The quantization parameter (QP) is a key control variable in video coding, used to adjust the quantization step size to balance compression efficiency and image quality. Essentially, QP is the index of the quantization step size (Qstep), and its value directly affects the coarseness of the quantization process. In luma coding, QP typically ranges from 0 to 51 (52 steps), while chroma coding ranges from 0 to 39. A smaller QP value results in finer quantization, preserving more spatial details and higher image quality, but at a higher bitrate. Conversely, a larger QP value results in coarser quantization, increasing compression efficiency but leading to distortion and decreased quality. For example, QP=0 represents the finest quantization, while QP=51 represents the coarsest. QP is linearly related to Qstep; for every increase of 6, Qstep roughly doubles. Summary of the Invention

[0006] The technical problem to be solved by this invention is to provide a video compression method, apparatus, device, and medium that can effectively classify and manage video frames based on multimodal features, take into account local texture details and long-distance spatiotemporal dependencies during feature extraction, effectively integrate spatiotemporal features, depth features, and motion features to generate entropy values ​​that characterize content complexity, and implement different encoding and dynamic bit allocation strategies based on frame type and entropy values, thereby ensuring a balance between compression efficiency and complexity in video compression.

[0007] To address the aforementioned technical problems, the present invention provides a video compression method, comprising at least the following steps: acquiring video data and preprocessing the video data to obtain a standard video sequence; performing frame segmentation on the standard video sequence and classifying the video frames into different frame types based on the features of each video frame; extracting multimodal features from the classified video frames to obtain feature data including spatiotemporal features, depth features, and motion features, and generating an entropy value that quantifies the complexity of the video content based on the feature data; determining different encoding strategies for different types of video frames based on the type of the video frame and the corresponding entropy value, wherein the encoding strategy includes at least a quantization parameter associated with the regional complexity of the video frame; and encoding according to the encoding strategy corresponding to each video frame to output a compressed bitstream.

[0008] In a preferred embodiment, the standard video sequence is processed by frame segmentation, specifically including the following steps: establishing a frame buffer queue with a capacity of N; adding the standard video sequence frame by frame to the frame buffer queue, and extracting and associating context information for the video frames added to the frame buffer queue, wherein the context information includes at least temporal context information; dynamically adjusting the priority of the video frames in the frame buffer queue based on the content of the video frames in the frame buffer queue; deleting video frames based on the adjusted priority; and outputting video frames frame by frame from the frame buffer queue.

[0009] In a preferred embodiment, classifying video frames into different frame types based on the features of each video frame specifically includes the following steps: inputting video frames from the frame buffer queue into a classification model for frame classification to obtain the corresponding frame types, wherein the frame types include static scene frames, dynamic scene frames, and key information frames; labeling the corresponding frame types on the video frames based on the classification results; wherein the classification model is trained based on sample video frames and the corresponding frame type labels of the sample video frames, and the classification model includes at least a first extraction module for extracting shallow features, a second extraction module for extracting mid-level features, a third extraction module for extracting deep features, a feature enhancement module for feature fusion based on a channel attention mechanism, and a classification head for classifying frame types based on fused features, all connected in sequence.

[0010] In a preferred embodiment, determining different encoding strategies for different types of video frames based on the type of the video frame and the corresponding entropy value includes: if the frame type of the video frame is a key information frame, then a first quantization parameter range is used; if the frame type of the video frame is a dynamic scene frame, then a second quantization parameter range is used, wherein the overall value of the second quantization parameter range is higher than that of the first quantization parameter range; if the frame type of the video frame is a static scene frame, then a third quantization parameter range is used, wherein the overall value of the third quantization parameter range is higher than that of the second quantization parameter range.

[0011] In a preferred embodiment, multimodal feature extraction of video frames specifically includes the following steps: extracting spatiotemporal features from a video segment of T consecutive frames using a hybrid 3D deformable convolutional and Transformer network; extracting depth features from a single frame image using a monocular depth estimation network; and extracting motion features from adjacent frames using an optical flow estimation network.

[0012] In a preferred embodiment, generating an entropy value that quantifies the complexity of video content based on the feature data includes: extracting texture complexity parameters and motion pattern complexity parameters based on the spatiotemporal features; obtaining spatial complexity parameters based on the depth features; obtaining temporal complexity parameters based on the motion features; and weightedly fusing the texture complexity parameters, motion pattern complexity parameters, semantic complexity parameters, spatial complexity parameters, and temporal complexity parameters to obtain the entropy value.

[0013] In a preferred embodiment, the step of determining the encoding strategy based on the frame type and the entropy value further includes bit allocation based on the entropy value: identifying a first region in the video frame with an entropy value higher than a first threshold, and allocating an increased number of encoding bits relative to the average bit budget to the first region; identifying a second region in the video frame with an entropy value lower than a second threshold, and allocating a decreased number of encoding bits relative to the average bit budget to the second region.

[0014] The present invention also provides a video compression apparatus, comprising at least: an acquisition and preprocessing unit for acquiring video data and preprocessing the video data to obtain a standard video sequence; a classification unit for performing frame segmentation processing on the standard video sequence and classifying the video frames into different frame types based on the features of each video frame; a feature extraction unit for performing multimodal feature extraction on the classified video frames to obtain feature data containing spatiotemporal features, depth features, and motion features, and generating an entropy value for quantifying the complexity of video content based on the feature data; a strategy decision unit for adaptively determining different encoding strategies for different types of video frames based on the frame type and the entropy value, wherein the encoding strategy includes at least quantization parameters associated with the frame type and regional complexity; and an encoding execution unit for encoding according to the encoding strategy corresponding to each video frame and outputting a compressed bitstream.

[0015] The present invention also provides a computer device, comprising: a processor, a memory, and a bus, wherein the memory stores machine-readable instructions executable by the processor, and when the computer device is running, the processor communicates with the memory via the bus, and when the machine-readable instructions are executed by the processor, the steps of the video compression method as described in any of the preceding claims are performed.

[0016] The present invention also provides a computer-readable storage medium storing a computer program, which, when executed by a processor, performs the steps of the video compression method described in any of the preceding claims.

[0017] Compared with the prior art, the video compression method, apparatus, computer equipment and storage medium of the present invention have the following beneficial effects: (1) The video compression method of the present invention includes at least the following steps: acquiring video data and preprocessing the video sequence to obtain a standard video sequence; performing frame processing on the standard video sequence and classifying the video frames into different frame types based on the characteristics of each video frame. By classifying the video frames into different types (such as static, dynamic and key information), the most suitable encoding strategy can be adopted for the characteristics of different types of frames, avoiding the efficiency loss caused by "one-size-fits-all" encoding, providing a decision basis for subsequent encoding, realizing the reasonable allocation of encoding resources, avoiding the problem that the existing encoding technology adopts the same or simple classification processing strategy for all video frames, and lacking targeted adaptation to the characteristics of video content.

[0018] Multimodal feature extraction is performed on the classified video frames to obtain feature data including spatiotemporal features, depth features, and motion features. Based on the feature data, an entropy value is generated to quantify the complexity of the video content. On the one hand, by extracting "spatiotemporal features," long-distance inter-frame correlations beyond the local receptive field of CNNs are captured, which helps to improve the flickering and artifact problems in reconstructed images caused by insufficient dependency modeling. By introducing "depth features" and "motion features," together with spatiotemporal features, a multi-dimensional and three-dimensional content analysis system is formed. Depth features provide spatial geometry and hierarchical information of the scene, while motion features quantify the dynamic changes of objects. Together, these achieve a deep and quantitative perception of the complexity of video content, far exceeding the simple analysis based on pixel differences or gradients in traditional methods, providing a rich and reliable data foundation for subsequent accurate resource allocation. On the other hand, the entropy value transforms the complexity of video content from a qualitative judgment to a quantitative assessment, providing an objective and quantifiable basis for coding decisions. Based on the entropy value, the coding difficulty of different regions in the video can be accurately identified, providing a direct basis for subsequent differentiated bit allocation and solving the contradiction of "insufficient bitrate in important regions and wasted bitrate in simple regions."

[0019] Based on video frame type and corresponding entropy value, different encoding strategies are determined for different types of video frames. Each encoding strategy includes at least quantization parameters related to the complexity of the video frame region. This avoids the rigidity of existing encoding resource allocation strategies, which are unable to adapt to dynamic content changes and struggle to balance efficiency and complexity. On one hand, by combining frame type (macroscopic characteristics) and entropy value (microscopic complexity) to determine the encoding strategy, the overall content category of the video is considered, while also taking into account the encoding difficulty of specific regions. On the other hand, appropriate quantization parameters are allocated to different types of frames and regions of varying complexity, avoiding the problems of insufficient quality in complex regions or wasted resources in simple regions in traditional encoding. This achieves better quality at the same bitrate, or saves more bitrate at the same quality. Through differentiated strategies, efficient encoding is used to reduce computation for simple content, while fine-grained encoding is used to ensure quality for complex content, achieving an overall optimized balance between encoding efficiency and computational complexity. Encoding is performed according to the encoding strategy corresponding to each video frame, and a compressed bitstream is output.

[0020] (2) The video compression method of the present invention performs frame-by-frame processing on a standard video sequence, specifically including the following steps: establishing a frame buffer queue with a capacity of N; adding the standard video sequence frame by frame to the frame buffer queue, and extracting and associating context information for the video frames added to the frame buffer queue, the context information including at least temporal context information; dynamically adjusting the priority of the video frames in the frame buffer queue based on the content of the video frames in the frame buffer queue; when the capacity of the frame buffer queue reaches the upper limit, deleting the video frames with the lowest priority; and outputting video frames frame by frame from the frame buffer queue. On the one hand, by managing the queue with a fixed capacity and using a priority-based elimination mechanism, the time complexity of the system processing is constrained within a controllable range, avoiding excessive consumption of computing resources caused by processing infinitely long historical frames; on the other hand, the retention priority of the video frame in the cache can be dynamically determined according to the content value of the video frame (such as whether it contains key objects or whether it is a scene switching point), which is essentially a forward-looking resource scheduling, ensuring that key reference frames that have a significant impact on the subsequent coding quality are retained, thereby indirectly guiding resources such as coding bits to tilt towards high-value content, and providing upstream decision support for solving the problem of "insufficient bitrate in important areas".

[0021] (3) The video compression method of the present invention determines different encoding strategies for different types of video frames based on the video frame type and the corresponding entropy value. Specifically, if the video frame type is a key information frame, a first quantization parameter range is used; if the video frame type is a dynamic scene frame, a second quantization parameter range is used, where the overall value of the second quantization parameter range is higher than that of the first quantization parameter range; if the video frame type is a static scene frame, a third quantization parameter range is used, where the overall value of the third quantization parameter range is higher than that of the second quantization parameter range. Bitrate resources (directly controlled by quantization parameters) are allocated as needed to content of different importance. Key information frames (such as faces and text) use the lowest first quantization parameter range, allocating the most bits to ensure high fidelity and high definition of core semantic content, solving the problem of "insufficient bitrate in important areas." Static scene frames (such as static backgrounds) use the highest third quantization parameter range, significantly reducing bit consumption and effectively avoiding "bitrate waste in simple areas." Dynamic scene frames use the second quantization parameter range, achieving efficient compression while ensuring motion continuity and optimizing overall bitrate utilization. Attached Figure Description

[0022] Figure 1 is a flowchart of the extraction method of a video compression method, apparatus, computer equipment, and storage medium according to Embodiment 1 of the present invention; Figure 2 is a schematic diagram of the classification model structure of a video compression method, apparatus, computer equipment, and storage medium according to Embodiment 1 of the present invention; Figure 3 is a schematic diagram of the residual block structure of a video compression method, apparatus, computer equipment, and storage medium according to Embodiment 1 of the present invention; Figure 4 is a schematic diagram of the SE module structure of a video compression method, apparatus, computer equipment, and storage medium according to Embodiment 1 of the present invention; Figure 5 is a schematic diagram of the classification header structure of a video compression method, apparatus, computer equipment, and storage medium according to Embodiment 1 of the present invention; Figure 6 is a schematic diagram of the hybrid network structure of a video compression method, apparatus, computer equipment, and storage medium according to Embodiment 1 of the present invention. Detailed Implementation

[0023] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention.

[0024] In the description of this invention, it should be understood that the terms "upper", "lower", "front", "rear", "inner", "outer", etc., indicate the orientation or positional relationship based on the orientation or positional relationship shown in the accompanying drawings. They are only for the convenience of describing this invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, they should not be construed as limitations on this invention.

[0025] In the description of this invention, it should be noted that, unless otherwise explicitly specified and limited, the terms "installation", "connection" and "linking" should be interpreted broadly. For example, they can refer to a fixed connection, an integral connection, or a detachable connection; they can refer to the internal connection of two components; they can refer to a direct connection or an indirect connection through an intermediate medium. Those skilled in the art can understand the specific meaning of the above terms in this invention based on the specific circumstances.

[0026] An embodiment of a video compression method, as shown in Figure 1, includes at least the following steps: Step S1. Acquire video data and preprocess the video data to obtain a standard video sequence.

[0027] Step S11. Acquire video data and decode the video into raw pixel data. In this embodiment, the acquired video data includes video files in MP4, AVI, MKV, and MOV formats, as well as real-time video files in RTSP, RTMP, and HLS formats. The FFmpeg library is used to detect the container format of the acquired video and decode it into raw pixel data in YUV format and corresponding video metadata. The video metadata includes the original resolution, original frame rate, color space, bitrate information, and total number of frames.

[0028] Step S12. Adjust the decoded raw pixel data to a preset standard resolution to achieve resolution unification. This eliminates the impact of rate differences on subsequent processing. In this embodiment, high-resolution data is unified by downsampling, and low-resolution data is unified by upsampling.

[0029] Step S13. Standardize the frame rate of the original pixel data after unifying the resolution to obtain a standard video sequence, ensuring temporal consistency. In this embodiment, the frame rate is set to 30fps. High frame rate data is processed by frame extraction, gradually extracting frames by retaining one frame every two frames until the frame rate reaches 30fps. Low frame rate data is processed by frame interpolation using light streaming until the frame rate reaches 30fps. It should be noted that generating intermediate frames based on light streaming is existing technology and will not be described in detail here.

[0030] Step S2. Perform frame segmentation on the standard video sequence and classify the video frames into different frame types based on the characteristics of each frame. By classifying video frames into different types (such as static, dynamic, and key information), the most suitable encoding strategy can be adopted for the characteristics of different frame types, avoiding the efficiency loss caused by "one-size-fits-all" encoding. This provides a decision-making basis for subsequent encoding, realizes the rational allocation of encoding resources, and avoids the problem of existing encoding technologies using the same or simple classification processing strategy for all video frames, lacking targeted adaptation to the characteristics of video content. Step S2 specifically includes the following steps: Step S21. Establish a frame buffer queue with a capacity of N. In this embodiment, a frame buffer queue with a capacity of 30 is established, which includes at least YUV format frame data and frame indexes established by timestamps.

[0031] Step S22. Add the standard video sequence frame by frame to the frame buffer queue, and extract and associate context information for the video frames added to the frame buffer queue. The context information includes at least temporal context information. In this embodiment, the frame buffer queue retains 30 frames, mainly to retain short-term context information for 5 adjacent frames, medium-term context information for 15 adjacent frames, and long-term context information for 30 frames.

[0032] Step S23. Input the video frames in the frame buffer queue into the classification model for frame classification to obtain the corresponding frame types. The frame types include static scene frames, dynamic scene frames, and key information frames. Based on the classification results, the corresponding frame types are labeled on the video frames. The classification model is trained based on sample video frames and the corresponding frame type labels. As shown in Figure 2, the classification model includes at least a first extraction module for extracting shallow features, a second extraction module for extracting mid-level features, a third extraction module for extracting deep features, a feature enhancement module for feature fusion based on channel attention, and a classification head for frame type classification based on the fused features, all connected in sequence. In this embodiment, the first extraction module includes a 5×5 convolutional layer, a batch normalization layer, a ReLU activation function layer, and a max pooling layer connected in sequence, used to extract shallow features from the input video frames. The stride of the 5×5 convolutional layer is set to 2, and the padding is set to 2. The pooling window of the max pooling layer is set to 3×3, the stride is set to 2, and the padding is set to 1.

[0033] The input of the second extraction module is connected to the output of the first extraction module, and includes a first residual block and a second residual block connected in sequence. The input of the third extraction module is connected to the output of the second extraction module, and includes a third residual block and a fourth residual block connected in sequence. In this embodiment, the first residual block, the second residual block, the third residual block and the fourth residual block adopt the same structural setting, as shown in Figure 3. It includes a 1×1 convolutional layer 1, a batch normalization layer, a ReLU activation function layer, a 3×3 convolutional layer, a batch normalization layer, a ReLU activation function layer, and a 1×1 convolutional layer 2 connected in sequence for channel dimensionality reduction, and a Shortcut connection connected to the input of the 1×1 convolutional layer 1 and the output of the 1×1 convolutional layer 2. The Shortcut connection is provided with a 1×1 convolutional layer 3 for adjusting the channel dimension of the input features. The output features of the 1×1 convolutional layer 3 and the output features of the 1×1 convolutional layer 2 are added element-wise, and the addition result is non-linearly activated by the ReLU activation function layer. For the first and second residual blocks, the stride of the 1×1 convolutional layer is set to 1, and the padding is set to 0; the stride of the 3×3 convolutional layer is set to 1, and the padding is set to 1; the stride of the second 1×1 convolutional layer is set to 1, and the padding is set to 0; the stride of the third 1×1 convolutional layer is set to 1, and the padding is set to 0. For the third and fourth residual blocks, the stride of the 1×1 convolutional layer is set to 2, and the padding is set to 0; the stride of the 3×3 convolutional layer is set to 1, and the padding is set to 1; the stride of the second 1×1 convolutional layer is set to 1, and the padding is set to 0; the stride of the third 1×1 convolutional layer is set to 2, and the padding is set to 0.

[0034] The feature enhancement module is used to fuse the features extracted by the first, second, and third extraction modules. As shown in Figure 2, the channel dimensions of the output features from the first, second, and third residual blocks of the third extraction module are adjusted, and sampling is used to ensure consistent output feature scale. Specifically, the output features of the first extraction module are upsampled by a factor of 4, the output features of the second extraction module are upsampled by a factor of 2, and the output features of the third residual block of the third extraction module are downsampled by a factor of 2 to maintain the same size as the output features of the fourth residual block.

[0035] The fusion layer concatenates the output features of the first extraction module, the second extraction module, and the third and fourth residual blocks of the third extraction module, after adjusting their dimensions and sizes, along the channel dimension. Furthermore, it applies a channel attention mechanism through the SE module to weight the importance of each channel. The SE module, as shown in Figure 4, includes a globally average pooling layer, a fully connected layer one for channel dimension reduction, a ReLU layer, a fully connected layer two for restoring the channel dimension, and a Simgoid activation function layer for generating channel attention weight parameters. The fused features are obtained by multiplying the input data of the globally average pooling layer with the channel attention weight parameters channel by channel.

[0036] The classification head, as shown in Figure 5, consists of a globally average pooling layer, a fully connected layer three, a batch normalization layer, a ReLU activation function layer, a first Dropout layer, a fully connected layer four, a batch normalization layer, a ReLU activation function layer, a second Dropout layer, and a fully connected layer five, all connected sequentially. The classification result is obtained by sequentially reducing the dimensionality of the fused features through fully connected layers three, four, and five.

[0037] Step S24. Dynamically adjust the priority of video frames in the frame buffer queue based on the content of the video frames in the frame buffer queue. Step S24 specifically includes the following steps: Step S241. Assign a basic priority weight to each video frame based on the classification result of the video frames in the frame buffer queue. In this embodiment, priority weight parameters are assigned to video frames whose classification result is a key information frame. The classification result assigns priority weight parameters to video frames that are dynamic scene frames. The video frames classified as static scene frames are assigned priority weight parameters. Among them, the priority weight parameter .

[0038] Step S242. Adjust the priority weight parameters of consecutive static scene frames based on content similarity and temporal distance. If consecutive video frames in the frame buffer queue are classified as static scene frames, calculate the similarity between consecutive static scene frames using the following calculation method: ,in, Total number of pixels They are different video frames, For each static scene frame, calculate its average content similarity to all other static scene frames in the cache queue. (The coordinates are in pixels.) The specific calculation method is as follows: ,in, This represents the number of consecutive static scene frames.

[0039] Calculate the time distance between consecutive static scene frames using timestamps. The specific calculation method is as follows: ,in, The latest timestamp of the current static scene frame. This is the timestamp corresponding to other static scene frames in a series of static scene frames.

[0040] Based on time distance and average content similarity The priority weight parameters for static scene frames are adjusted using the following calculation method: ,in, These are the average content similarity adjustment factor and the time distance adjustment factor, respectively.

[0041] Step S243. Adjust the priority weight parameters of consecutive dynamic scene frames based on motion intensity. If consecutive video frames in the frame buffer queue are classified as dynamic scene frames, then the motion intensity is calculated using optical flow. The specific calculation method is as follows: ,in, Total number of pixels The pixel coordinates of the nth dynamic scene frame and the (n-1)th dynamic scene frame are respectively The displacements in the x and y directions. After normalizing the calculated motion intensity, the priority weight parameters of the dynamic scene frames are adjusted, specifically using the following calculation method: ,in, Motion intensity in continuous dynamic scene frames The maximum value, This is the exercise intensity adjustment factor.

[0042] Step S25. Based on the adjusted priority, the video frames are reduced. In this embodiment, if the adjusted priority weight parameters of static scene frames and dynamic scene frames are less than a preset threshold, reduction processing is performed. For key information frames, they are retained. On the one hand, low-priority frames are eliminated in advance, avoiding the need to enter the subsequent time-consuming multimodal feature extraction, entropy calculation, and encoding processes, reducing the average processing time per frame, improving encoding throughput, and making it more suitable for real-time video processing scenarios. On the other hand, by adjusting the preset threshold, the complexity can be actively reduced when computing power is limited, ensuring real-time performance, achieving an adaptive balance between encoding efficiency and computational complexity, and enhancing the applicability of the system on different hardware platforms.

[0043] Step S26. Output video frames one by one from the frame buffer queue.

[0044] Step S3. Multimodal feature extraction is performed on the classified video frames to obtain feature data including spatiotemporal features, depth features, and motion features. Based on the feature data, an entropy value quantifies the complexity of the video content is generated. On the one hand, by extracting "spatiotemporal features," long-distance inter-frame correlations beyond the local receptive field of CNNs are captured, which helps to improve the flickering and artifact problems in reconstructed images caused by insufficient dependency modeling. By introducing "depth features" and "motion features," together with spatiotemporal features, a multi-dimensional and three-dimensional content analysis system is formed. Depth features provide spatial geometry and hierarchical information of the scene, while motion features quantify the dynamic changes of objects. Together, these achieve a deep and quantitative perception of the complexity of video content, far exceeding the simple analysis based on pixel differences or gradients in the past, providing a rich and reliable data foundation for subsequent accurate resource allocation. On the other hand, the entropy value transforms the complexity of video content from a qualitative judgment to a quantitative assessment, providing an objective and quantifiable basis for coding decisions. Based on the entropy value, the coding difficulty of different regions in the video can be accurately identified, providing a direct basis for subsequent differentiated bit allocation and solving the contradiction of "insufficient bitrate in important regions and wasted bitrate in simple regions."

[0045] Multimodal feature extraction of video frames specifically includes the following steps: Step S31. Extract spatiotemporal features from a continuous T-frame video segment using a hybrid network of 3D deformable convolution and Transformer. The hybrid network, as shown in Figure 6, includes a parallel 3D deformable convolution branch for extracting texture features F1 and a weight matrix A for generating temporal attention. tempThe Transformer branch is described. The 3D deformable convolution branch includes a first 3D deformable convolutional layer, a batch normalization layer, a ReLU layer, a max pooling layer, a second 3D deformable convolutional layer, a batch normalization layer, and a ReLU layer, connected sequentially. The Transformer branch includes an image block embedding layer connected sequentially to segment tiles for each video frame and obtain feature vectors through linear projection mapping; a spatial location encoding layer connected to the output of the image block embedding layer to add location information to each tile; a first Transformer layer connected to the output of the spatial location encoding layer to apply an attention mechanism in the spatial dimension; a temporal location encoding layer connected to the output of the first Transformer layer to add temporal order information; a sequence reshaping layer connected to the output of the temporal location encoding layer to convert spatiotemporally separated features into a sequence suitable for temporal modeling; and a second Transformer layer connected to the output of the sequence reshaping layer to apply an attention mechanism in the temporal dimension. In this embodiment, the continuous video segments have 3 channels, 16 frames, a height of 224, and a width of 224. The image patch embedding layer divides each frame into 196 patches of size 16×16 and maps them to a feature vector of dimension 256, outputting (16, 196, 256). The first Transformer layer has two Transformer encoders and eight attention heads, processing 16 frames in parallel without interaction between frames. The sequence reshaping layer concatenates all spatial features of each frame into a single vector. The second Transformer layer has two Transformer encoders and eight attention heads, using a sliding window approach with a window size of 4, calculating the temporal attention weight matrix A by comparing it with the preceding and following frames.

[0046] Step S32. Extract depth features from a single frame image using a monocular depth estimation network. In this embodiment, the monocular depth estimation network uses the existing lightweight encoder MobileNetV2 to extract depth features at different scales, and a decoder with a U-Net architecture adapted to the MobileNetV2 encoder to establish skip connections for features at different scales, fuse depth features at different scales, upsample to restore the original resolution, and output depth features. The resolutions of the different scale depth features output by the lightweight encoder MobileNetV2 are 1 / 2, 1 / 4, 1 / 8, 1 / 16, 1 / 16, 1 / 32, and 1 / 32 of the input, respectively, corresponding to 16, 24, 32, 64, 96, 160, and 1280 channels. The corresponding U-Net architecture decoder performs 1×1 convolutional dimensionality reduction on the encoder's highest-level features to obtain the first pyramid feature. Then, it performs deconvolutional upsampling to fuse the first pyramid feature with the encoder's corresponding scale features element-wise, resulting in the second pyramid feature. This upsampling fusion operation is performed three times, fusing with the encoder's 5th, 3rd, and 2nd stage output features respectively, to obtain fused features at different scales. It should be noted that before each feature fusion in the decoder, the number of encoder feature channels is first adjusted to match the number of upsampled feature channels using a 1×1 convolution, then element-wise fusion is performed, and finally, a 3×3 convolution is used to refine the features to obtain the deep feature D.

[0047] Step S33. Extract motion features from adjacent frames using an optical flow estimation network. In this embodiment, the optical flow estimation network adopts the existing FlowNetS architecture, which overlays two adjacent video frames along the channel dimension and inputs them to obtain the optical flow field F2 representing the motion features. This is existing technology and will not be described in detail here.

[0048] Step S34. Generate the entropy value of quantified video content complexity based on feature data, specifically including the following steps: Step S341. Extract texture complexity parameters and motion mode complexity parameters based on spatiotemporal features. In this embodiment, the following method is specifically adopted: (1) Obtain the texture complexity parameter by extracting texture features F1 through 3D deformable convolution branches. The specific calculation method is as follows: ,in, For the number of channels, Let be the variance of the c-th channel.

[0049] (2) Obtain the motion pattern complexity parameters by using the temporal attention weight matrix A generated through the Transformer branch. .

[0050] (1) Obtain the multi-head average temporal attention weight matrix based on the temporal attention weight matrix A. The specific calculation method is as follows: ,in, This is the multi-head average temporal attention weight matrix. To focus on the number of heads, Let be the i-th temporal attention weight matrix.

[0051] (2) Based on multi-head average temporal attention weight matrix Obtain the position attention parameters per unit time. The specific calculation method is as follows: ,in, Indexed by the current time. For all time positions traversed, For the preset threshold, This refers to the duration of the time.

[0052] (3) Based on positional attention parameters per unit time Obtain the motion pattern complexity parameters The specific calculation method is as follows: ,in, This refers to the duration of the time.

[0053] Step S342. Obtain the space complexity parameter based on depth features. In this embodiment, the space complexity parameter is obtained from the depth features D obtained by the monocular depth estimation network. The specific implementation method is as follows: (1) Divide the depth feature D into M×N blocks; (2) Calculate the depth variance based on the partitioned blocks to obtain the space complexity parameter. The specific calculation method is as follows: ,in, These represent the number of blocks in the vertical and horizontal directions, respectively. For the first Each partition block For the first The depth variance of each partitioned block.

[0054] Step S343. Obtain the time complexity parameter based on the motion features. In this embodiment, the time complexity parameter is obtained from the optical flow field F2 obtained through the optical flow estimation network. The specific calculation method is as follows: ,in, These are the height and width of the optical flow field, respectively. For the first The range of motion of a pixel between consecutive frames.

[0055] Step S344. The texture complexity parameter, motion pattern complexity parameter, space complexity parameter, and time complexity parameter are weighted and fused to obtain the entropy value. In this embodiment, the entropy value is based on the texture complexity parameter. Motion pattern complexity parameters Space complexity parameters and time complexity parameters The entropy value is obtained by weighted fusion, and is specifically calculated using the following method: ,in, These are texture complexity parameters. Motion pattern complexity parameters Space complexity parameters and time complexity parameters The corresponding weight parameters.

[0056] Step S4. Based on the video frame type and its corresponding entropy value, determine different encoding strategies for different types of video frames. The encoding strategy includes at least quantization parameters related to the complexity of the video frame region. This avoids the problems of rigid encoding resource allocation strategies in existing technologies, which cannot adapt to dynamic changes in content and are difficult to balance between efficiency and complexity. On the one hand, by combining the two dimensions of frame type (macroscopic characteristics) and entropy value (microscopic complexity) to determine the encoding strategy, the overall content category of the video is considered, while also taking into account the encoding difficulty of specific regions. On the other hand, by allocating appropriate quantization parameters to different types of frames and regions with different complexities, the problems of insufficient quality in complex regions or wasted resources in simple regions in traditional encoding are avoided. This achieves better quality at the same bitrate, or saves more bitrate at the same quality. Through differentiated strategies, efficient encoding is used to reduce computation for simple content, while fine encoding is used to ensure quality for complex content, thus achieving an optimized balance between encoding efficiency and computational complexity overall.

[0057] Static scene frames have minimal motion, with texture and spatial structure being the primary information sources; dynamic scene frames exhibit dramatic motion changes, making temporal continuity crucial; key information frames have high information density, requiring consideration of all dimensions, but with a focus on distinguishability. In this embodiment, the texture complexity parameter of static scene frames... Motion pattern complexity parameters Space complexity parameters and time complexity parameters The corresponding weight parameters are set to The weight parameters corresponding to the dynamic scene frames are set to... The weight parameters corresponding to the key information frames are set to... .

[0058] Step S4 specifically includes the following steps: Step S41. Determine the quantization parameter range based on the frame type of the video frame. Specifically, if the frame type of the video frame is a key information frame, then the first quantization parameter range is used; if the frame type of the video frame is a dynamic scene frame, then the second quantization parameter range is used, wherein the overall value of the second quantization parameter range is higher than that of the first quantization parameter range; if the frame type of the video frame is a static scene frame, then the third quantization parameter range is used, wherein the overall value of the third quantization parameter range is higher than that of the second quantization parameter range.

[0059] In this embodiment, key information frames, such as faces, subtitles, and important objects, need to ensure high quality and clarity in key areas, so the first quantization parameter range is set to 18-26. Dynamic scene frames, such as moving objects and scene transitions, are compressed appropriately while ensuring motion continuity, so the second quantization parameter range is set to 28-36. Static scene frames, such as still backgrounds and skies, use a high compression rate to save bits, so the third quantization parameter range is set to 38-46.

[0060] Step S42. Based on the entropy value in the video frames, judge consecutive video frames. If the entropy value is higher than a first threshold for a first region, allocate an increase in coding bits relative to the average bit budget to the first region; if the entropy value is lower than a second threshold for a second region, allocate a decrease in coding bits relative to the average bit budget to the second region. In this embodiment, the first threshold... The following method can be used to set it up: ,in, For average bit budget, The corresponding weight parameter for the first threshold. .

[0061] Second threshold The following method can be used to set it up: ,in, The weight parameters corresponding to the second threshold. .

[0062] Step S5. Encode each video frame according to the encoding strategy corresponding to it, and output the compressed bitstream.

[0063] Example 2 This example provides a video compression device, comprising at least: an acquisition and preprocessing unit for acquiring video data and preprocessing the video data to obtain a standard video sequence; a classification unit for performing frame segmentation on the standard video sequence and classifying the video frames into different frame types based on the features of each video frame; a feature extraction unit for performing multimodal feature extraction on the classified video frames to obtain feature data containing spatiotemporal features, depth features, and motion features, and generating an entropy value based on the feature data to quantify the complexity of the video content; a strategy decision unit for adaptively determining different encoding strategies for different types of video frames based on the frame type and the entropy value, wherein the encoding strategy includes at least quantization parameters related to the frame type and regional complexity; and an encoding execution unit for encoding according to the encoding strategy corresponding to each video frame and outputting a compressed bitstream.

[0064] This embodiment also provides a computer device, including: a processor, a memory, and a bus. The memory stores machine-readable instructions that can be executed by the processor. When the computer device is running, the processor communicates with the memory through the bus. When the machine-readable instructions are executed by the processor, the steps of the video compression method described in any of the preceding embodiments are performed.

[0065] This embodiment also provides a computer-readable storage medium storing a computer program, which, when executed by a processor, performs the steps of the video compression method described in any of the preceding embodiments.

[0066] In summary, the above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A video compression method, characterized in that, The method includes at least the following steps: acquiring video data and preprocessing the video data to obtain a standard video sequence; performing frame segmentation on the standard video sequence and classifying the video frames into different frame types based on the features of each video frame; extracting multimodal features from the classified video frames to obtain feature data including spatiotemporal features, depth features, and motion features, and generating an entropy value that quantifies the complexity of the video content based on the feature data; determining different encoding strategies for different types of video frames based on the type of the video frame and the corresponding entropy value, wherein the encoding strategy includes at least a quantization parameter associated with the regional complexity of the video frame; and encoding according to the encoding strategy corresponding to each video frame to output a compressed bitstream.

2. A video compression method according to claim 1, characterized in that, The standard video sequence is processed by frame segmentation, specifically including the following steps: establishing a frame buffer queue with a capacity of N; adding the standard video sequence frame by frame to the frame buffer queue, and extracting and associating context information for the video frames added to the frame buffer queue, wherein the context information includes at least temporal context information; and dynamically adjusting the priority of the video frames in the frame buffer queue based on the content of the video frames in the frame buffer queue. Based on the adjusted priority, the video frames are reduced; video frames are output frame by frame from the frame buffer queue.

3. A video compression method according to claim 2, characterized in that, The process of classifying video frames into different frame types based on the features of each video frame includes the following steps: inputting video frames in the frame buffer queue into a classification model for frame classification to obtain the corresponding frame types, wherein the frame types include static scene frames, dynamic scene frames, and key information frames; Based on the classification results, the corresponding frame type is labeled on the video frame; wherein, the classification model is trained based on sample video frames and the frame type labels corresponding to the sample video frames, and the classification model includes at least a first extraction module for extracting shallow features, a second extraction module for extracting mid-level features, a third extraction module for extracting deep features, a feature enhancement module for feature fusion based on channel attention mechanism, and a classification head for classifying frame types based on fused features, which are connected in sequence.

4. A video compression method according to claim 3, characterized in that, The step of determining different encoding strategies for different types of video frames based on the type of the video frame and the corresponding entropy value includes: if the frame type of the video frame is a key information frame, then a first quantization parameter range is used; if the frame type of the video frame is a dynamic scene frame, then a second quantization parameter range is used, wherein the overall value of the second quantization parameter range is higher than that of the first quantization parameter range; if the frame type of the video frame is a static scene frame, then a third quantization parameter range is used, wherein the overall value of the third quantization parameter range is higher than that of the second quantization parameter range.

5. A video compression method according to claim 4, characterized in that, Multimodal feature extraction of video frames includes the following steps: extracting spatiotemporal features from a continuous T-frame video segment using a hybrid 3D deformable convolutional and Transformer network; and extracting depth features from a single-frame image using a monocular depth estimation network. Motion features are extracted from adjacent frames using an optical flow estimation network.

6. A video compression method according to claim 5, characterized in that, The step of generating an entropy value for quantifying the complexity of video content based on the feature data includes: extracting texture complexity parameters and motion pattern complexity parameters based on the spatiotemporal features; obtaining spatial complexity parameters based on the depth features; obtaining time complexity parameters based on the motion features; and weightedly fusing the texture complexity parameters, motion pattern complexity parameters, spatial complexity parameters, and time complexity parameters to obtain the entropy value.

7. A video compression method according to claim 6, characterized in that: The method of determining the encoding strategy based on the frame type and the entropy value further includes bit allocation based on the entropy value: identifying a first region in the video frame with an entropy value higher than a first threshold, and allocating an increased number of encoding bits relative to the average bit budget to the first region; identifying a second region in the video frame with an entropy value lower than a second threshold, and allocating a decreased number of encoding bits relative to the average bit budget to the second region.

8. A video compression device, characterized in that, At least including: The acquisition and preprocessing unit is used to acquire video data and preprocess the video data to obtain a standard video sequence. A classification unit is used to perform frame segmentation processing on the standard video sequence and classify the video frames into different frame types based on the features of each video frame; a feature extraction unit is used to perform multimodal feature extraction on the classified video frames to obtain feature data containing spatiotemporal features, depth features, and motion features, and generate an entropy value for quantifying the complexity of video content based on the feature data; a strategy decision unit is used to adaptively determine different encoding strategies for different types of video frames based on the frame type and the entropy value, wherein the encoding strategy includes at least quantization parameters associated with the frame type and regional complexity; and an encoding execution unit is used to encode according to the encoding strategy corresponding to each video frame and output a compressed bitstream.

9. A computer device, characterized in that, include: The computer device includes a processor, a memory, and a bus. The memory stores machine-readable instructions executable by the processor. When the computer device is running, the processor communicates with the memory via the bus. When the machine-readable instructions are executed by the processor, they perform the steps of the video compression method as described in any one of claims 1-7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, performs the steps of the video compression method as described in any one of claims 1-7.

Images