A deep learning-based video compression method

Abstract

The application discloses a kind of video compression methods based on deep learning, specifically related to video compression field, for solving the problem that entropy modeling in deep learning video compression cannot track residual error distribution dynamic migration in real time, is through multi-scale self-attention mechanism accurate description residual error complex distribution in spatial and time dimension, adaptive entropy model configuration is realized on each block of data using content-based hybrid density generator, two-way evaluation is sent into pre-training attention model Dynamic threshold value drives high-precision encoding template to implement fine compression for high-frequency area, while entropy parameters are locally contracted based on real-time bit feedback to balance output, and through parameter freezing synchronization metadata, the consistency of decoding end model is guaranteed, The synchronous optimization of compression efficiency and reconstruction quality is realized, so that fluctuation converges and suppresses artifact deviation, so as to approach distortion limit.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of video compression, and more particularly, to a deep learning-based video compression method. BACKGROUND

[0002] The deep learning video compression framework extracts residual features with convolution and self-attention network on the encoding side, and then outputs latent symbols through quantization mapping. The residuals present multi-modal statistical patterns such as sparse spikes, Gaussian clustering and long-tail mixing in the space-time dimension, and the distribution changes continuously with the content and motion.

[0003] However, the mainstream entropy modeling still relies on fixed window context or shallow neighbor probability network, assuming that the latent symbols follow a single Gaussian or its linear variant. The static model cannot track the dynamic shift of residual distribution in real time, resulting in sharp expansion of motion boundary redundancy code rate, over-compression of smooth area estimation bias, and long-tail mismatch of high-frequency texture, which causes reconstruction artifacts, and becomes the core bottleneck of deep learning video compression approaching rate-distortion limit.

[0004] To solve the above problems, a technical solution is provided. SUMMARY

[0005] To overcome the above-mentioned defects of the prior art, embodiments of the present application provide a deep learning-based video compression method, which accurately depicts the complex distribution of residuals in spatial and temporal dimensions through a multi-scale self-attention mechanism, and uses a content-based hybrid density generator to realize adaptive entropy model configuration on each block of data. The bidirectional evaluation quantity is fed into a pre-trained attention model to dynamically generate a threshold-driven high-precision encoding template for fine compression of high-frequency regions. At the same time, based on real-time bit feedback, the entropy parameters are locally contracted to balance the output, and through parameter freezing and synchronization metadata, the consistency of the decoding end model is guaranteed, realizing the synchronous optimization of compression efficiency and reconstruction quality, making the fluctuation converge and suppressing artifacts bias, thereby approaching the distortion limit, to solve the problems raised in the background art.

[0006] To achieve the above-mentioned purposes, the present application provides the following technical solutions:

[0007] A deep learning-based video compression method, comprising the steps of:

[0008] S1: performing spatial-temporal joint scanning on the complete residual tensor with a multi-scale self-attention statistic at the encoding entrance to generate a dynamic distribution mapping and write it into a mapping buffer;

[0009] S2: the hybrid density generator reads the dynamic distribution mapping, instantiates a set of probability density parameters for each spatial block, and returns an entropy parameter matrix;

[0010] S3: Before quantization, evaluate the residual spectrum in the displacement difference direction to obtain the displacement evaluation quantity, and evaluate the texture in the coherent texture direction in the multi-scale wavelet domain to obtain the texture evaluation quantity, input the two direction evaluation quantities into the pre-trained hybrid attention model to output the results, if it exceeds the standard, call the corresponding precision arithmetic coding template to complete the corresponding residual segment coding;

[0011] S4: Record the bit consumption of the coded segment to generate a code rate feedback vector, and implement local flexible contraction on the entropy parameter matrix of the remaining block according to the corresponding vector to reduce redundant bits;

[0012] S5: When the code rate feedback vector is in the convergence interval, freeze the entropy parameter matrix, and send the bit stream together with the distribution metadata to the decoding end to complete video reconstruction.

[0013] In a preferred embodiment, step S1 includes the following content:

[0014] By calculating the pixel-by-pixel difference between the original video frame and the predicted frame, a residual tensor is generated; a multi-scale self-attention mechanism is applied to extract local details, regional structures and global pattern features of the residual tensor at small, medium and large scales respectively; the spatial features of each frame are spliced and the temporal correlation between consecutive frames is analyzed to generate a temporal feature tensor, and then the spatial feature tensor and the temporal feature tensor are fused into a spatio-temporal feature tensor; the spatio-temporal feature tensor is processed through convolution and full connection operation to generate the probability distribution center value and width value of the residual tensor at different positions, a dynamic distribution mapping is constructed, and it is stored in the mapping buffer.

[0015] In a preferred embodiment, step S2 includes the following content:

[0016] Divide the residual tensor into a plurality of non-overlapping spatial blocks in the spatial dimension; instantiate a hybrid density model for each spatial block; generate a corresponding set of probability density parameters based on the hybrid density model of each spatial block; construct the entropy parameters of each spatial block using the set of probability density parameters of each spatial block; and integrate the entropy parameters of all spatial blocks to form an entropy parameter matrix.

[0017] In a preferred embodiment, step S3 includes the following content:

[0018] Evaluating the residual spectrum in the displacement difference direction to obtain the displacement evaluation quantity, i.e. the displacement entropy difference spectrum amplitude:

[0019] Performing gradient spectrum analysis on the residual tensor in the spatial dimension, calculating the horizontal and vertical gradient values, merging to generate the gradient amplitude spectrum, and generating the displacement entropy difference spectrum amplitude through a rectifier function.

[0020] In a preferred embodiment, step S3 further includes the following content:

[0021] The texture evaluation quantity is obtained by evaluating the multi-scale wavelet domain according to the coherent texture direction, that is, the residual coherence wavelength index:

[0022] The multi-scale wavelet transform is performed on the residual tensor, which is decomposed into subbands of different scales, and the coherence index of each subband is calculated to generate the residual coherence wavelength index by weighted average.

[0023] In a preferred embodiment, step S3 further comprises the following:

[0024] The displacement entropy difference spectrum amplitude and the residual coherence wavelength index are input into a pre-trained hybrid attention model, which is processed by a spatial attention module and a channel attention module to generate a dynamic tension coefficient.

[0025] In a preferred embodiment, step S3 further comprises the following:

[0026] The dynamic tension coefficient is compared with a preset threshold, and according to the comparison result, it is determined whether to call the corresponding arithmetic coding template to encode the current residual segment.

[0027] In a preferred embodiment, step S4 comprises the following:

[0028] The bit consumption of the encoded residual segment is recorded in real time to generate a code rate feedback vector containing code rate deviation, volatility and trend information, and the entropy parameter matrix of the remaining block is locally flexibly contracted according to the code rate feedback vector to dynamically adjust the entropy parameter to reduce redundant bits. The adjusted entropy parameter matrix is applied to the entropy coding of the remaining block to optimize the code rate allocation and compression efficiency of the coding process.

[0029] In a preferred embodiment, step S5 comprises the following:

[0030] The bit rate feedback vector is evaluated to determine whether it is in a predefined convergence region; if it is determined to be in the predefined convergence region, the entropy parameter matrix is frozen and the parameter value of the entropy parameter matrix is fixed to be applied to the entropy coding of the subsequent coding block; then the compressed bit stream and the distribution metadata are integrated into a compressed data packet, and the distribution metadata contains the dynamic distribution mapping and the frozen entropy parameter matrix; finally, the compressed data packet is parsed at the decoding end, the entropy model is reconstructed using the distribution metadata, and the compressed bit stream is decoded to recover the residual tensor, combined with the predicted frame to generate the original video frame, completing the video reconstruction.

[0031] In a preferred embodiment, step S5 further comprises the following:

[0032] The evaluation conditions include that the bit rate deviation absolute value of the bit rate feedback vector is less than a preset tolerance threshold, the average absolute dispersion of bit consumption is lower than a preset limit value, and the absolute value of the sequence correlation coefficient of the bit consumption is greater than a preset minimum value; when the bit rate feedback vector satisfies all the above conditions, it is judged that it is in the pre-defined convergence area.

[0033] The technical effects and advantages of the video compression method based on deep learning are as follows:

[0034] The present application accurately depicts the complex distribution of residual in spatial and temporal dimensions through multi-scale self-attention mechanism, uses a content-based hybrid density generator to realize adaptive entropy model configuration on each block of data, sends the bidirectional evaluation quantity into a pre-trained attention model to dynamically generate a threshold-driven high-precision encoding template for fine compression of high-frequency areas, simultaneously performs local contraction on the entropy parameters based on real-time bit feedback to balance the output, and guarantees the consistency of the decoding end model through parameter freezing and synchronization metadata, realizes the synchronous optimization of compression efficiency and reconstruction quality, makes the fluctuation converge and suppresses the artifact deviation, and thus approaches the distortion limit. BRIEF DESCRIPTION OF DRAWINGS

[0035] Figure 1 The flowchart of the video compression method based on deep learning is shown. DETAILED DESCRIPTION

[0036] The technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor fall within the scope of protection of the present application.

[0037] Embodiment 1: Figure 1 The video compression method based on deep learning is given, which comprises:

[0038] S1: At the encoding entrance, a multi-scale self-attention statistic is performed on the complete residual tensor for spatial-temporal joint scanning to generate a dynamic distribution mapping and write it into a mapping buffer.

[0039] S2: The hybrid density generator reads the dynamic distribution mapping, instantiates a set of probability density parameters for each spatial block, and returns an entropy parameter matrix.

[0040] S3: Before quantization, the displacement evaluation quantity is obtained by evaluating the residual spectrum according to the displacement difference direction, and the texture evaluation quantity is obtained by evaluating the multi-scale wavelet domain according to the coherent texture direction, the two direction evaluation quantities are input into a pre-trained hybrid attention model to output results, and if the results exceed the standard, the corresponding arithmetic encoding template is called to complete the corresponding residual segment coding.

[0041] S4: generating a code rate feedback vector for the coded segment instantaneous recording bit consumption, and performing local flexible shrinkage on the entropy parameter matrix of the remaining blocks according to the vector to reduce redundant bits.

[0042] S5: freezing the entropy parameter matrix when the code rate feedback vector is in a convergence interval, and sending the bit stream together with the distribution metadata to the decoding end to complete video reconstruction.

[0043] With the explosive growth of video content demand, especially the widespread application of high dynamic range (HDR), high frame rate (HFR) and ultra-high definition (UHD) video, the amount of video data has shown explosive growth, and higher requirements have been put forward for storage and transmission efficiency. Traditional video compression standards such as H.264 and H.265 rely on fixed rules and shallow statistical models, and gradually show limitations in dealing with complex scenes, such as increased motion boundary redundant code rate, over-compression of smooth areas, and high-frequency texture reconstruction distortion. The video compression method based on deep learning can more accurately capture the spatio-temporal distribution characteristics of the residual by the powerful feature extraction capability of the neural network, thereby breaking through the bottleneck of traditional methods. The present application proposes a video compression method based on deep learning, which uses multi-scale self-attention mechanism and dynamic entropy modeling technology to adaptively process the multi-modal statistical features of the residual tensor to approach the rate-distortion limit. Step S1 as the entrance of the encoding process aims to lay the foundation for subsequent entropy modeling and encoding optimization.

[0044] Step S1 includes the following contents:

[0045] S1.1, obtaining of the residual tensor:

[0046] The residual tensor is obtained by comparing and calculating the pixel values of the original video frame and the predicted frame one by one.

[0047] The original video frame is the original image data in the video sequence without processing, which contains complete visual information; the predicted frame is the approximate image data generated by the encoder based on the existing information. The processing logic is:

[0048] For each frame of image, subtract the pixel value of the corresponding position of the predicted frame from each pixel value of the original video frame, and arrange the obtained difference value according to the spatial position of the original frame to form a tensor with the same dimension as the original video frame, which is called residual tensor. Each element in the residual tensor represents the difference between the original image and the predicted image at the corresponding position, which represents the detail information that has not been captured by the prediction model.

[0049] The original video frame contains a large amount of redundant information, while the predicted frame has approximately expressed the predictable part. By calculating the difference between the two, the key information that needs further compression can be extracted. This way avoids repeated processing of redundant data and focuses on the unique content needed for encoding. The introduction of residual tensors significantly reduces the amount of redundancy in data processing, allowing the subsequent encoding process to focus on the difference information, thereby improving compression efficiency and optimizing the use of computing resources.

[0050] S1.2, Construction of multi-scale self-attention mechanism:

[0051] The multi-scale self-attention mechanism is used to extract features from the residual tensor. The processing logic is divided into three levels of calculation:

[0052] First, in the small-scale layer, such as a 3x3x3 local window, the residual tensor is divided into multiple small local regions, and the correlation analysis is performed on the pixel points in each small region to calculate the mutual influence strength between each pixel point and other pixel points, generating features reflecting local details;

[0053] Second, in the medium-scale layer, such as a 16x16x5 local window, the region range is expanded, and similar correlation analysis is performed on medium-sized regions to obtain features of local structures and short-term changes;

[0054] Finally, in the large-scale layer, the entire residual tensor is taken as the range, and the global correlation between all pixel points is calculated to generate information describing the overall pattern. The features of the three levels are saved separately to form a multi-scale feature set.

[0055] The residual tensor contains complex characteristics in space and time, and single-scale analysis cannot fully capture its information distribution. The small-scale layer is suitable for extracting details such as edges, the medium-scale layer can reflect regional structures, and the large-scale layer reveals global trends. Multi-scale design adapts to the multi-level characteristics of the residual tensor through hierarchical processing. The multi-scale self-attention mechanism can fully depict the feature distribution of the residual tensor, covering from details to the whole, thereby improving the accuracy and integrity of feature extraction and providing more abundant information support for subsequent processing.

[0056] S1.3, Implementation of space-time joint scanning:

[0057] Spatial dimension processing: for each residual tensor in a frame (where is the time index), extract local features in the small-scale layer , extract regional features in the medium-scale layer , and integrate global spatial information through the large-scale layer to obtain the spatial feature tensor :

[0058]

[0059] Temporal dimension processing: residual tensor between consecutive frames Apply self-attention within a temporal window (e.g. frame) to calculate inter-frame dependencies and obtain a temporal feature tensor :

[0060]

[0061] Joint integration: integrate the spatial feature tensor and the temporal feature tensor into a spatio-temporal feature tensor through linear transformation :

[0062]

[0063] where is a trainable fusion parameter matrix.

[0064] The core of video data lies in the combination of spatial texture and temporal motion, and analyzing either dimension alone will miss key information. Spatial-temporal joint scanning ensures a complete description of the spatio-temporal characteristics of the residual tensor by processing and finally integrating the two types of features.

[0065] S1.4, generation of dynamic distribution map:

[0066] The dynamic distribution map is generated by statistical analysis of the spatio-temporal feature tensor. The specific processing logic is as follows:

[0067] First, use multi-layer convolution operations to process the spatio-temporal feature tensor, extracting its deep statistical properties layer by layer, and then convert these properties into parameters of the probability distribution through fully connected operations, including the distribution center value and the distribution width value of each position;

[0068] Next, according to these parameters, a mapping structure with the same dimension as the residual tensor is constructed, where each position stores the distribution center value and the distribution width value of the corresponding pixel point, forming a dynamic distribution map;

[0069] Finally, store the dynamic distribution map in the mapping buffer for subsequent steps to call.

[0070] The statistical properties of the residual tensor vary with video content and motion, and a fixed distribution model cannot adapt to its changes. By analyzing the spatio-temporal feature tensor in real time and generating a dynamic distribution map, the dynamic characteristics of the residual tensor can be accurately reflected.

[0071] Step S1 serves as the starting link of the entire encoding process. By obtaining the residual tensor, constructing the multi-scale self-attention mechanism, implementing the spatial-temporal joint scanning, and generating the dynamic distribution map, key information is gradually extracted and refined from the original data, and finally the dynamic distribution map is formed. This process fully utilizes the multi-modal characteristics of the residual tensor, and the generated dynamic distribution map provides accurate statistical basis for subsequent entropy modeling and encoding optimization.

[0072] Step S1 performs spatial-temporal joint scanning on the residual tensor by the multi-scale self-attention statistic, generates the dynamic distribution map and stores it to the mapping buffer. The dynamic distribution map accurately describes the statistical characteristics of the residual tensor in the space-time dimension, providing a data basis for subsequent processing. However, entropy modeling, as a core link of video compression, directly affects the compression efficiency and reconstruction quality. Traditional methods rely on fixed context windows or shallow probability networks, assuming that the latent symbols follow a single Gaussian distribution or its linear variant, which cannot adapt to the dynamic changes of the residual tensor distribution, leading to problems such as code rate redundancy, over-compression, and reconstruction artifacts. Step S2 aims to use a hybrid density generator to generate a set of adaptive probability density parameters for each spatial block based on the dynamic distribution map, thereby constructing an entropy parameter matrix to achieve accurate and flexible entropy modeling.

[0073] Step S2 includes the following content:

[0074] S2.1, reading of the dynamic distribution map:

[0075] The hybrid density generator reads the dynamic distribution map from the mapping buffer. The dynamic distribution map is a data structure that records the statistical characteristics of the residual tensor in the spatial and temporal dimensions, specifically recording the distribution center value and distribution width value of the residual tensor at each position. The hybrid density generator accesses the storage location in the mapping buffer to extract the data content of the dynamic distribution map for subsequent processing.

[0076] S2.2, division of spatial blocks:

[0077] The residual tensor is divided into multiple non-overlapping spatial blocks in the spatial dimension.

[0078] According to the pre-set block height and block width, each frame of the residual tensor is divided into several rectangular regions, and each rectangular region is regarded as a spatial block. In the time dimension, each spatial block covers all frames of the video sequence, i.e., each spatial block contains the residual values of this position in all frames.

[0079] Dividing the residual tensor into spatial blocks can decompose the global entropy modeling problem into multiple local problems, facilitating independent processing of the statistical characteristics of different regions.

[0080] S2.3, instantiation of the mixture density model:

[0081] The mixture density generator instantiates a mixture density model for each spatial block.

[0082] The mixture density generator analyzes the statistical properties of the parts within the corresponding spatial block range according to the dynamic distribution mapping, determines the number and type of distribution components in the mixture density model. The mixture density model is composed of multiple distribution components, each of which adopts a Gaussian distribution or other suitable probability distribution form to capture the multi-modal characteristics of the residual tensor within the spatial block.

[0083] Instantiating a mixture density model for each spatial block enhances the expressiveness of the entropy model, making the probability estimation more accurate, thereby optimizing the coding efficiency.

[0084] S2.4, generation of the probability density parameter set:

[0085] The mixture density generator generates a probability density parameter set for each spatial block.

[0086] First, the mixture density generator performs clustering analysis on the distribution center values of the corresponding spatial block in the dynamic distribution mapping, determines the weights of each distribution component in the mixture density model, and the weights represent the contribution proportion of each distribution component in the overall distribution; second, calculate the average value of the distribution center value in each cluster as the mean value of the corresponding distribution component; finally, based on the weighted average of the distribution width value within the cluster, calculate the standard deviation of each distribution component, the weight is set according to the spatial and temporal correlation to reflect the importance of different positions.

[0087] Through clustering analysis and weighted calculation, parameters that match the actual distribution characteristics of the residual tensor can be extracted from the dynamic distribution mapping, ensuring the accuracy of the mixture density model. The generated probability density parameter set can accurately describe the probability distribution of the residual tensor within the corresponding spatial block, providing reliable model support for entropy coding.

[0088] S2.5, construction of the entropy parameter matrix:

[0089] The mixture density generator constructs the entropy parameter for each spatial block.

[0090] Based on the generated probability density parameter set, the probability density function of the mixture density model is calculated and discretized into a probability distribution table, which represents the probability distribution of the residual tensor within the spatial block. All the entropy parameters of the spatial blocks are integrated into a matrix structure, called the entropy parameter matrix.

[0091] The entropy parameter matrix collects the probability distribution information of all spatial blocks, and is the core input data of entropy coding. The construction of the entropy parameter matrix enables entropy coding to adaptively adjust according to the characteristics of different spatial blocks. Reduces the redundant bits in the coding process, and improves the compression performance.

[0092] S2.6, return of the entropy parameter matrix:

[0093] The hybrid density generator returns the constructed entropy parameter matrix for subsequent steps. The entropy parameter matrix contains the entropy parameters of all spatial blocks, providing adaptive model support for subsequent quantization, coding and code rate control.

[0094] The entropy parameter matrix is the final output of step S2, and returning the entropy parameter matrix ensures that the subsequent processing can directly use the adaptive entropy model.

[0095] Through the processing technology logic of step S2, the present application realizes adaptive entropy modeling of the dynamic distribution of the residual tensor. The hybrid density generator uses dynamic distribution mapping to generate a customized set of probability density parameters for each spatial block, and further constructs an entropy parameter matrix. The entropy parameter matrix provides accurate model support for subsequent entropy coding, ensuring that the coding process can be optimized according to the actual distribution characteristics of the residual tensor.

[0096] Step S2 reads the dynamic distribution mapping generated by step S1 through the hybrid density generator, instantiates a set of probability density parameters for each spatial block, and returns an entropy parameter matrix. However, the efficiency of entropy coding not only depends on the accuracy of the entropy model, but also is limited by the accurate processing of high-frequency details and motion boundaries in the residual tensor. Traditional static models are difficult to adapt to the dynamic migration of residual distribution, resulting in code rate redundancy or reconstruction distortion in high-frequency regions. Step S3 aims to dynamically adjust the coding strategy by evaluating the displacement difference and texture characteristics of the residual tensor, and implement high-precision coding for high-frequency regions to optimize compression performance.

[0097] Step S3 includes the following:

[0098] S3.1, evaluate the residual spectrum according to the displacement difference direction to obtain the displacement evaluation quantity, i.e. the displacement entropy difference spectrum amplitude:

[0099] The calculation of the displacement entropy difference spectrum amplitude is obtained by analyzing the change intensity of the residual tensor in the spatial dimension. The specific processing logic is as follows:

[0100] First, for each frame of the residual tensor, the gradient values in the horizontal and vertical directions are calculated respectively, and the gradient values reflect the change rate of pixel values in space;

[0101] Then, the gradient values in horizontal direction and vertical direction are combined into a comprehensive gradient magnitude spectrum, which represents the overall change intensity of the residual tensor in space.

[0102] Finally, the gradient magnitude spectrum is processed by a rectification function, which retains the gradient values exceeding a preset threshold and filters out the low-intensity changes below the preset threshold, thereby generating the displacement entropy difference spectrum amplitude.

[0103] For example, the following method can be used to obtain:

[0104] The residual tensor is subjected to gradient spectrum analysis in spatial dimensions. For each frame of residual tensor , the horizontal gradient and the vertical gradient are calculated respectively, and then the gradient magnitude spectrum is calculated by the formula . Here represents the change intensity of the residual tensor in space.

[0105] Based on the gradient magnitude spectrum , the gradient spectrum is rectified by a rectification function (where is a preset threshold for filtering out low-intensity changes) to obtain the displacement entropy difference spectrum amplitude .

[0106] The displacement entropy difference spectrum amplitude quantifies the entropy change intensity of the residual tensor caused by displacement difference.

[0107] The displacement difference in the residual tensor is mainly reflected in the motion boundary and rapidly changing areas, where the pixel value change rate is large. The calculation of the displacement entropy difference spectrum amplitude accurately quantifies the entropy change intensity of the residual tensor caused by displacement difference, which helps to identify the motion boundary area that needs to be finely coded, thereby improving the pertinence and efficiency of the coding process.

[0108] S3.2, evaluate the texture evaluation quantity, i.e. residual coherence wavelength index, in the multi-scale wavelet domain according to the coherent texture direction:

[0109] The calculation of the residual coherence wavelength index is obtained by analyzing the frequency characteristics of the residual tensor in the multi-scale wavelet domain. The specific processing logic is as follows:

[0110] First, the residual tensor is subjected to multi-scale wavelet transform, which decomposes the residual tensor into subbands of different scales, each subband containing the frequency information of the residual tensor at the corresponding scale;

[0111] Then, for each subband, a coherency index is calculated, which is derived by analyzing the local consistency of pixel values within the subband, specifically the cosine similarity of wavelength consistency within the neighborhood.

[0112] Finally, the coherency indices of all scales are merged by weighted average to generate the residual coherency wavelength index.

[0113] For example, the following approach can be used to obtain:

[0114] A multi-scale wavelet transform is performed on the residual tensor to decompose it into subbands at different scales (scaled by a scale index ). Each subband contains the frequency information of the residual tensor at the corresponding scale, reflecting the texture characteristics.

[0115] For each subband , a coherency index is calculated, where is a local coherency function defined as the cosine similarity of wavelength consistency within the neighborhood. Finally, the residual coherency wavelength index is obtained by merging the residual coherency wavelength indices of all scales by weighted average: , where is a predefined weight for scale , satisfying .

[0116] : denotes the row number in the subband , used to identify the position of the pixel in the vertical direction.

[0117] : denotes the column number in the subband , used to identify the position of the pixel in the horizontal direction.

[0118] The residual coherency wavelength index difference reflects the frequency coherency of the residual tensor in the texture direction, used to identify high-frequency texture regions.

[0119] The texture characteristics in the residual tensor are manifested as frequency coherency at different scales, especially in high-frequency texture regions, where the coherency index can reflect the complexity of these regions. The calculation of the residual coherency wavelength index provides a quantitative evaluation of the texture characteristics of the residual tensor, which can accurately identify high-frequency texture regions and provide important basis for the optimization of subsequent encoding strategies.

[0120] S3.3, generation of dynamic tension coefficient:

[0121] The generation of the dynamic tension coefficient is achieved by a pre-trained hybrid attention model. The specific processing logic is as follows:

[0122] The pre-trained hybrid attention model includes a spatial attention module and a channel attention module. The spatial attention module processes the displacement entropy difference spectrum amplitude through convolution and normalization operations to generate a spatial feature map, which emphasizes important regions in the spatial dimension. The channel attention module processes the residual coherent wavelength index through global pooling and fully connected operations to generate a channel weight vector, which adjusts the importance of different scales. Finally, the spatial feature map and the channel weight vector are fused, and a scalar value, i.e., the dynamic tension coefficient, is mapped through a fully connected layer.

[0123] The displacement entropy difference spectrum amplitude reflects the displacement difference of the residual tensor, and the residual coherent wavelength index reflects the texture characteristics of the residual tensor. The hybrid attention model dynamically integrates these two types of information through spatial and channel dual attention mechanisms to achieve comprehensive evaluation of the complexity of the residual tensor.

[0124] The generation of the dynamic tension coefficient accurately quantifies the complexity of the residual tensor, providing a dynamic adjustment basis for the encoding process to ensure that complex regions are given appropriate processing precision.

[0125] S3.4, calling of the arithmetic coding template:

[0126] The calling of the arithmetic coding template is based on the decision of the dynamic tension coefficient.

[0127] The dynamic tension coefficient is compared with a preset threshold. If the dynamic tension coefficient exceeds the preset threshold, it is determined that the residual segment has high displacement or texture complexity, and a high-precision arithmetic coding template is called to finely encode the current residual segment. If the dynamic tension coefficient does not exceed the preset threshold, a standard encoding template is used to encode the current residual segment.

[0128] The following is an example of a high-precision arithmetic coding template:

[0129] For each symbol in the current residual segment, a dynamic context is constructed, which integrates the spatial and temporal information of adjacent symbols and introduces features extracted based on the displacement entropy difference spectrum amplitude and the residual coherent wavelength index, which are mapped to the context space by a pre-trained model to enhance information expression. Subsequently, a multi-layer perceptron (MLP) network is used to generate the conditional probability distribution of the current symbol in real time using the dynamic context as input. The MLP network is continuously updated during the encoding process to adapt to the local characteristics of the residual segment. Based on this probability distribution, arithmetic coding is performed to map the symbol to the real number interval according to the conditional probability calculated by the dynamic context. At the same time, a feedback mechanism is introduced to adjust the MLP network parameters by monitoring the encoding efficiency and reconstruction quality in real time, optimizing the probability estimation precision.

[0130] The calling of the high-precision arithmetic coding template realizes fine compression of high-complexity regions, improves coding efficiency, and ensures optimization of reconstruction quality.

[0131] The present application realizes dynamic evaluation and fine coding of high-frequency details and motion boundaries of residual tensors through the specific processing technology logic of step S3. First, the displacement entropy difference spectrum amplitude and residual coherence wavelength index are calculated to quantize the displacement difference and texture characteristics of the residual tensor, respectively. Then, a pre-trained hybrid attention model is used to generate a dynamic tension coefficient to comprehensively evaluate the complexity of the residual tensor. Finally, the dynamic tension coefficient is used to determine whether to call the high-precision arithmetic coding template, ensuring that the coding process can be optimized according to the actual characteristics of the residual tensor.

[0132] Step S3 evaluates the displacement difference and texture characteristics of the residual tensor before quantization to generate a dynamic tension coefficient, and implements high-precision coding for high-frequency regions to optimize compression performance. However, step S3 optimizes the coding strategy through the dynamic tension coefficient, but does not involve real-time rate control issues, which is the key to balancing compression efficiency and reconstruction quality. Step S4 aims to record the bit consumption of the coded segments in real time, generate a rate feedback vector, and implement local flexible contraction of the entropy parameter matrix of the remaining blocks based on the vector to reduce redundant bits and achieve dynamic optimization of the code rate.

[0133] Step S4 includes the following contents:

[0134] S4.1, real-time recording of coded segment bit consumption:

[0135] For each residual segment that has completed coding, record the actual number of bits consumed, and integrate the bit consumption of all coded residual segments into a bit consumption vector. The bit consumption vector is an ordered set, where each element corresponds to the bit consumption value of a coded residual segment, reflecting the distribution of the code rate in the current coding process.

[0136] The purpose of recording the bit consumption of the coded residual segment is to monitor the change of the code rate in the coding process in real time. By recording the bit consumption of the coded residual segment in real time, the code rate distribution of the current coding process can be accurately mastered, providing a reliable basis for subsequent generation of a rate feedback vector and adjustment of the entropy parameter matrix, thereby improving the accuracy of rate control.

[0137] S4.2, generation of a rate feedback vector:

[0138] First, the average of the bit consumption of all coded residual segments is calculated, that is, the sum of the values of the elements in the bit consumption vector is divided by the total number of elements to obtain the average bit consumption, which represents the average code rate level of the current coding process. Then, according to the preset target code rate, the difference between the current average bit consumption and the target code rate is calculated to obtain the code rate deviation, which reflects the degree of deviation of the current code rate from the target code rate. Then, the average of the absolute difference between each element in the bit consumption vector and the average bit consumption is calculated to obtain the average absolute dispersion of the bit consumption, which is used to measure the amplitude of the code rate fluctuation. Finally, the correlation between adjacent elements in the bit consumption vector is calculated to obtain the sequence correlation coefficient of the bit consumption, which is used to represent the trend characteristics of the code rate change. Finally, the code rate deviation, the average absolute dispersion of the bit consumption and the sequence correlation coefficient of the bit consumption are combined into a code rate feedback vector to comprehensively reflect the deviation degree, fluctuation characteristics and trend information of the current code rate.

[0139] The purpose of generating the code rate feedback vector is to comprehensively evaluate the code rate state of the current coding process through multi-dimensional indicators. The code rate feedback vector contains code rate deviation, fluctuation and trend information, which can comprehensively describe the code rate state, so that the adjustment of the entropy parameter matrix is more accurate, thereby optimizing the code rate distribution of the coding process and improving the compression efficiency.

[0140] S4.3, local flexible shrinkage of the entropy parameter matrix:

[0141] First, a shrinkage factor is calculated according to the code rate deviation, the average absolute dispersion of the bit consumption and the sequence correlation coefficient in the code rate feedback vector. The calculation method of the shrinkage factor is to add the absolute value of the code rate deviation to the average absolute dispersion of the bit consumption, and then multiply it by a weighted value of the sequence correlation coefficient of the bit consumption, which is used to control the amplitude of the entropy parameter adjustment. Then, a local flexible shrinkage transformation is applied to each element in the entropy parameter matrix. The specific transformation process is as follows: according to the size of the shrinkage factor, the value of each element in the entropy parameter matrix is reduced by a certain proportion while its basic characteristics are preserved, ensuring that the adjustment amplitude is consistent with the information of the code rate feedback vector. Through this transformation, the redundant components in the entropy parameter matrix are dynamically reduced, and the adjusted entropy parameter matrix can adapt to the current code rate demand.

[0142] For example, the following method is used:

[0143] For the remaining uncoded spatial blocks, the entropy parameter matrix is denoted as , where each element in the matrix represents the entropy coding parameter of the corresponding block, where is the row and column index of the matrix. For each element in the entropy parameter matrix , a local flexible shrinkage transformation is applied:

[0144]

[0145] wherein:

[0146] is a preset upper limit of the dispersion, used for normalizing the fluctuation of bit consumption .

[0147] controls the amplitude of shrinkage, introduces the impact of fluctuation, adjusts the inhibitory effect of trend continuity on shrinkage.

[0148] The transformation reduces the redundant components in the entropy parameter matrix by integrating the rate deviation, fluctuation, and trend information.

[0149] The locally flexible shrinkage transformation can adjust the entropy parameter matrix according to the rate deviation, fluctuation, and trend information, making the encoding process more adaptive to the current rate state, thereby improving compression efficiency and optimizing bit allocation.

[0150] S4.4, updating and applying of the entropy parameter matrix:

[0151] The entropy parameter matrix adjusted by the locally flexible shrinkage transformation is applied to the entropy encoding process of the remaining spatial blocks.

[0152] During the entropy encoding of the remaining spatial blocks, the parameter values in the adjusted entropy parameter matrix are directly used to guide the encoding operations. At the same time, the updated entropy parameter matrix is passed to the subsequent encoding process to generate the final compressed bitstream, ensuring that the entire encoding process uses consistent parameter configurations.

[0153] The purpose of updating and applying the entropy parameter matrix is to ensure that the entropy encoding process of the remaining spatial blocks can utilize the optimized parameters, thereby achieving dynamic adjustment of the rate and improvement of the compression performance.

[0154] The bit consumption of the encoded residual segments is recorded in real time to generate a rate feedback vector containing rate deviation, fluctuation, and trend information. Then, the locally flexible shrinkage transformation is performed on the entropy parameter matrix according to the rate feedback vector. Finally, the adjusted entropy parameter matrix is applied to the encoding of the remaining spatial blocks and generates a compressed bitstream. This complete process realizes dynamic optimization of the rate, reduces redundant bits in the encoding process, and improves the compression efficiency and reconstruction quality of the deep learning-based video compression method.

[0155] Step S4 records the bit consumption of the encoded segments in real time, generates a rate feedback vector, and performs local flexible shrinkage on the entropy parameter matrix of the remaining blocks according to the vector, so as to realize dynamic optimization of the code rate. However, in order to ensure the stability of the code rate and complete the video reconstruction, step S5 needs to freeze the entropy parameter matrix when the rate feedback vector meets a specific condition, and send the compressed bit stream and distribution metadata to the decoding end, so as to realize the closed-loop process of video compression and reconstruction.

[0156] Step S5 includes the following contents:

[0157] S5.1, judge the convergence of the bit rate feedback vector:

[0158] First, the bit rate feedback vector is evaluated to determine whether it is within a predefined convergence region.

[0159] The bit rate feedback vector includes three parts: the bit rate deviation, the average absolute dispersion of the bit consumption, and the serial correlation coefficient of the bit consumption.

[0160] The convergence region is defined by three specific conditions: the absolute value of the bit rate deviation must be less than a preset tolerance threshold, the average absolute dispersion of the bit consumption must be lower than a specific limit value, and the absolute value of the serial correlation coefficient of the bit consumption must be greater than a minimum value.

[0161] The evaluation process is completed by checking whether the three parts of the current bit rate feedback vector meet the above conditions at the same time. The bit rate deviation is obtained by calculating the absolute difference between the current cumulative bit consumption and the target total bit amount, and then dividing the difference by the target total bit amount. The average absolute dispersion of the bit consumption is calculated by taking the absolute difference between the bit consumption of each encoding segment within a time window and the average bit consumption within the window, and then taking the average of these differences. The serial correlation coefficient of the bit consumption is determined by analyzing the correlation between the bit consumptions of consecutive encoding segments within the same time window using standard correlation calculation methods. If the three parts of the bit rate feedback vector meet the conditions of the convergence region, it is considered that the bit rate has reached a stable state, and the subsequent processing is entered.

[0162] By defining the convergence region and evaluating the bit rate feedback vector, the purpose is to ensure that the bit rate approaches the target value during encoding, with small fluctuations and consistent trends. In the case of stable bit rate, subsequent processing can avoid the decline of encoding quality or efficiency caused by bit rate fluctuations, thereby ensuring the stability of the compression process and the high quality of video reconstruction.

[0163] S5.2, freeze the entropy parameter matrix:

[0164] When the bit rate feedback vector satisfies all the conditions of the convergence region, the entropy parameter matrix is immediately frozen. The entropy parameter matrix is a key tool for entropy coding of the remaining coding blocks. During this process, the entropy parameter matrix is kept in the current state and no further adjustments are made. The frozen entropy parameter matrix will be directly applied to the entropy coding of all subsequent coding blocks, ensuring that the encoding process uses a fixed entropy model.

[0165] The purpose of freezing the entropy parameter matrix is to lock the model of entropy coding when the bit rate is stable, in order to maintain the consistency of the encoding process. The stability of the bit rate indicates that the current entropy parameter matrix has adapted to the target compression requirements and does not need further adjustment. By freezing the entropy parameter matrix, it is possible to prevent the decline in coding efficiency caused by excessive adjustment, while ensuring that the entropy coding of the remaining coding blocks remains consistent with the previous optimization results, thereby improving the overall performance of compression and simplifying the decoding process.

[0166] S5.3, Integration of compressed bitstream and distribution metadata:

[0167] After freezing the entropy parameter matrix, the compressed bitstream and the distribution metadata are integrated. The compressed bitstream contains the encoded data of all residual segments, while the distribution metadata includes the dynamic distribution mapping and the frozen entropy parameter matrix. The integration process encapsulates the compressed bitstream and the distribution metadata into a compressed data packet for transmission to the decoding end.

[0168] The purpose of integrating the compressed bitstream and the distribution metadata is to ensure that the decoding end receives complete information for correct video reconstruction. The distribution metadata provides key parameters of the entropy model, while the compressed bitstream carries the actual encoded content, both of which are indispensable. By transmitting the compressed bitstream and the distribution metadata together, the decoding end can accurately restore the entropy model and data content during encoding, thereby ensuring the accuracy of video reconstruction and maintaining the high-quality output of the compression method.

[0169] S5.4, Completion of video reconstruction:

[0170] At the decoding end, after receiving the compressed data packet, it is first parsed to extract the compressed bitstream and the distribution metadata. Using the dynamic distribution mapping and the frozen entropy parameter matrix in the distribution metadata, the decoding end reconstructs an entropy model consistent with the encoding end. Then, using this entropy model, the compressed bitstream is decoded to recover the residual tensor. Subsequently, the predicted frame is combined with the recovered residual tensor to generate the original video frame, thereby completing the video reconstruction process.

[0171] The purpose of completing the video reconstruction is to realize the complete cycle of compression and decompression, and to ensure that the decoding end can accurately restore the original video content using the information transmitted by the encoding end. The frozen entropy parameter matrix and distribution metadata are used to ensure the matching of the decoding process and the encoding process. This process ensures that the compression method can still provide high-quality video reconstruction results while reducing the amount of data. The accurate reconstruction process maintains the integrity of the video content, reflecting the practical value and efficiency of the compression method.

[0172] The above formulas are dimensionless values calculated, and the formulas are obtained by collecting a large amount of data to simulate a formula of the most recent real situation, and the preset parameters in the formula are set by a person skilled in the art according to the actual situation.

[0173] It should be noted that the system of the present application can be deployed in the device itself to realize embedded application, or can be run on PC or other terminal with user interface, so as to meet various hardware environments and use requirements.

[0174] The above only describes some exemplary embodiments of the present application by way of illustration, and it is needless to say that those skilled in the art can modify the described embodiments in various ways without departing from the spirit and scope of the present application. Therefore, the above figures and descriptions are illustrative in nature and should not be understood as limiting the scope of protection of the claims of the present application.

[0175] It should be noted that in this paper, if there are relationship terms such as first and second, they are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between the entities or operations. Moreover, the term "includes" "contains" or any other variant thereof is intended to cover non-exclusive inclusion, so that the process, method, article or device including a series of elements not only includes those elements, but also includes other elements not explicitly listed or inherent to such process, method, article or device. Without more limitations, the element defined by the statement "includes a" does not exclude the presence of another identical element in the process, method, article or device including the element.

[0176] The above is only a specific embodiment of the present application, but the protection scope of the present application is not limited thereto, and any person skilled in the art can easily think of changes or replacements within the technical range disclosed in the present application, which should be covered within the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the protection scope of the claims.

Claims

1. A video compression method based on deep learning, characterized in that, Including the following steps: S1: At the encoding entry point, a multi-scale self-attention mechanism is used to perform a joint spatial-temporal scan on the complete residual tensor to generate a spatiotemporal feature tensor. The spatiotemporal feature tensor is statistically analyzed to generate a dynamic distribution mapping and written into the mapping buffer. S2: The hybrid density generator reads the dynamic distribution map, instantiates a set of probability density parameters for each spatial block, and returns an entropy parameter matrix; S3: Before quantization, gradient spectrum analysis is performed on the residual tensor in the spatial dimension. The gradient values ​​in the horizontal and vertical directions are calculated and merged to generate the gradient magnitude spectrum. The displacement entropy difference spectrum amplitude is generated through the rectification function. Multi-scale wavelet transform is performed on the residual tensor to decompose it into sub-bands of different scales. The coherence index of each sub-band is calculated and the residual coherence wavelength index is generated by weighted averaging. The displacement entropy difference spectrum amplitude and the residual coherence wavelength index are input into the pre-trained hybrid attention model to output the dynamic tension coefficient. If the dynamic tension coefficient exceeds the threshold, a high-precision arithmetic coding template is called to complete the corresponding residual segment encoding; otherwise, the standard coding template is used to encode the residual segments. S4: Real-time recording of bit consumption for encoded segments to generate a rate feedback vector, calculation of the shrinkage factor based on the rate feedback vector, and adjustment of each element in the parameter matrix of the remaining blocks according to the shrinkage factor; S5: When the bitrate feedback vector is in the convergence interval, freeze the entropy parameter matrix and send the bitstream along with the distribution metadata to the decoding end to complete video reconstruction.

2. The video compression method based on deep learning according to claim 1, characterized in that, Step S1 includes the following: A residual tensor is generated by calculating the pixel-by-pixel difference between the original video frame and the predicted frame; a multi-scale self-attention mechanism is applied to extract local details, regional structure and global pattern features of the residual tensor at small, medium and large scales respectively. The spatial features of each frame are stitched together, and the temporal correlation between consecutive frames is analyzed to generate a temporal feature tensor. The spatial feature tensor and the temporal feature tensor are then fused into a spatiotemporal feature tensor. The spatiotemporal feature tensor is processed through convolution and fully connected operations to generate the probability distribution center value and width value of the residual tensor at different locations. A dynamic distribution mapping is constructed and stored in the mapping buffer.

3. The video compression method based on deep learning according to claim 2, characterized in that, Step S2 includes the following: The residual tensor is divided into multiple non-overlapping spatial blocks in the spatial dimension; a hybrid density model is instantiated for each spatial block; a set of probability density parameters is generated based on the hybrid density model of each spatial block; the entropy parameters of each spatial block are constructed using the set of probability density parameters of each spatial block; and the entropy parameters of all spatial blocks are integrated into an entropy parameter matrix.

4. The video compression method based on deep learning according to claim 1, characterized in that, Step S3 also includes the following: The displacement entropy difference spectral amplitude and residual coherence wavelength exponent are input into a pre-trained hybrid attention model, which is then processed by the spatial attention module and the channel attention module to generate dynamic tension coefficients.

5. The video compression method based on deep learning according to claim 1, characterized in that, Step S4 includes the following: The bit consumption of the encoded residual segments is recorded in real time, and a rate feedback vector containing rate deviation, volatility and trend information is generated. The shrinkage factor is calculated based on the rate feedback vector. Each element in the parameter matrix of the remaining blocks is adjusted according to the shrinkage factor, and the entropy parameter is dynamically adjusted. The adjusted entropy parameter matrix is ​​then applied to the entropy coding of the remaining blocks.

6. The video compression method based on deep learning according to claim 1, characterized in that, Step S5 includes the following: The bitrate feedback vector is evaluated to determine if it falls within a predefined convergence region. If it does, the entropy parameter matrix is ​​frozen, and its parameter values ​​are fixed for application to entropy coding of subsequent coding blocks. Then, the compressed bitstream and distribution metadata are integrated into a compressed data packet, which contains a dynamic distribution map and the frozen entropy parameter matrix. Finally, the compressed data packet is parsed at the decoding end, the entropy model is reconstructed using the distribution metadata, the compressed bitstream is decoded to recover the residual tensor, and the original video frame is generated by combining the predicted frame to complete video reconstruction.

7. The video compression method based on deep learning according to claim 6, characterized in that, Step S5 also includes the following: The evaluation conditions include that the absolute value of the bit rate deviation of the bit rate feedback vector is less than the preset tolerance threshold, the average absolute dispersion of bit consumption is lower than the preset limit value, and the absolute value of the sequence correlation coefficient of bit consumption is greater than the preset minimum value. When the bit rate feedback vector meets all the above conditions, it is determined that it is within the predefined convergence region.

Images