A video compression method and system based on attention mechanism

Abstract

This invention provides a video compression method and system based on an attention mechanism. The method includes: receiving an original video to be compressed; dividing the original video into multiple image groups; compressing a first type of frames to be compressed using a first compression model to obtain reference frames; inputting the reference frames and second type of frames to be compressed into a second compression model, wherein the second compression model includes a motion information encoder based on an attention mechanism to extract motion information, and a motion information decoder included in the second compression model to reconstruct the extracted motion information; inputting the reconstructed motion information and the reference frames into a motion compensation module and outputting reconstructed frames obtained by compressing each second type of frames to be compressed; and sequentially combining the reference frames and reconstructed frames obtained after processing each image group using the first and second compression models to obtain the compressed video. This invention can enhance the robustness of optical flow estimation video compression by introducing an attention mechanism.

Description

Technical Field

[0001] This invention relates to the field of video image compression technology, and in particular to a video compression method and system based on an attention mechanism. Background Technology

[0002] Compared to other forms of content such as text, images, and audio, video content can carry more information. In today's world, video content is increasingly widely disseminated and plays an increasingly important role in people's daily lives. Video compression technology is a crucial prerequisite for video representation, transmission, and storage, especially in the field of real-time video compression. Video compression has always been a cutting-edge technology in computer vision, and research on video compression methods has been flourishing for many years.

[0003] Before the rapid development of artificial intelligence, traditional block-based video compression technology dominated research and industrial development in the field of video compression. Block-based methods were commonly used to predict and transform motion information, leading to the development of various effective and widely used protocols such as H.264, H.265, H.266, VP8, VP9, ​​and AV1. However, with the continuous advancements in deep learning technology, data-driven neural networks have achieved remarkable results in visual tasks such as image classification, object detection, and image segmentation. Deep learning methods have gradually been applied to video compression, such as convolutional-based DVC and C2F, generative adversarial network-based SGANC, and Transformer-based VCT.

[0004] In the existing technology, an end-to-end video compression model framework based on deep learning is provided. Following the traditional video encoding and decoding framework, the traditional components—motion estimation and motion compensation—are mapped to a convolutional neural network module. Motion estimation is achieved by using optical flow estimation based on convolutional neural networks. In the motion estimation stage, the optical flow value between the current frame and the reference frame is obtained through the SpyNet optical flow model. Then, motion information is extracted from the optical flow using 8 layers of convolution and the extracted motion information is quantized.

[0005] However, existing motion estimation modules based on convolutional neural networks heavily rely on optical flow estimation. When the motion information extracted by optical flow estimation is inaccurate, the quality of the compressed video is unsatisfactory. Furthermore, motion estimation modules using fully convolutional neural networks are prone to losing global information during feature extraction because they need to utilize local information to obtain a tight representation of the video or image. Summary of the Invention

[0006] In view of this, embodiments of the present invention provide a video compression method and system based on an attention mechanism to eliminate or improve one or more defects existing in the prior art.

[0007] One aspect of the present invention provides a video compression method and system based on an attention mechanism, the method comprising the following steps:

[0008] Receive the original video to be compressed and divide the original video into multiple image groups; wherein, each image group contains a first type of frame to be compressed and multiple second type of frame to be compressed, and the first type of frame to be compressed is the first image frame in time order from the original video contained in each image group;

[0009] The first type of frames to be compressed is compressed using a preset first compression model, and the compression result is used as a reference frame.

[0010] The reference frame and the second type of frames to be compressed are input into the second compression model. The second compression model contains a motion information encoder based on an attention mechanism to extract the motion information of each second type of frame to be compressed relative to the reference frame. The motion information decoder in the second compression model reconstructs the extracted motion information. The reconstructed motion information and the reference frame are input into the motion compensation module and the reconstructed frame obtained by compressing each second type of frame to be compressed is output. The motion information encoder based on the attention mechanism contains multiple motion information extraction units. Each motion information extraction unit extracts sub-motion information based on the input. The final motion information is obtained by weighted calculation based on the preset weights of the extraction results of each motion information extraction unit.

[0011] The compressed video is obtained by sequentially combining the reference frames and reconstructed frames obtained after processing each image group through the first compression model and the second compression model.

[0012] In some embodiments of the present invention, in the step of inputting the reconstructed motion information and reference frame into the motion compensation module and outputting the reconstructed frame obtained by compressing each second type of frame to be compressed, the method further includes: performing a subtraction calculation on each second type of frame to be compressed and the corresponding reconstructed frame output by the motion compensation module to obtain the residual corresponding to the second type of frame to be compressed; inputting the residual corresponding to the second type of frame to be compressed into the residual encoder and performing quantization processing on the output of the residual encoder; inputting the quantized output of the residual encoder into the residual decoder for reconstruction; and performing an addition calculation on the output of the residual decoder and the reconstructed frame output by the motion compensation module, with the result of the addition calculation serving as the reconstructed frame after removing residual redundancy.

[0013] In some embodiments of the present invention, before the motion information decoder included in the second compression model reconstructs the extracted motion information, the method further includes: quantizing the motion information extracted by the motion information encoder; after the step of quantizing the output of the residual encoder, the method further includes: inputting the quantized output of the residual encoder and the quantized motion information into the bit rate estimation model to estimate the size of the compressed and encoded bit stream.

[0014] In some embodiments of the present invention, the preset first compression model is an end-to-end image compression model based on the Ballé algorithm.

[0015] In some embodiments of the present invention, the motion information encoder includes a first motion information extraction unit, which includes a downsampling module, an optical flow model, a motion information compression network, and a convolutional layer. The step of the motion information extraction unit extracting sub-motion information based on the input includes: inputting a second type of frame to be compressed and a reference frame at the original resolution into the first motion information extraction unit; obtaining a second type of frame to be compressed and a reference frame at a reduced resolution based on the downsampling module; extracting motion information from the second type of frame to be compressed and the reference frame at a reduced resolution based on the optical flow model; inputting the motion information extracted by the optical flow model into a motion information compression network consisting of a joint attention mechanism network and a convolutional neural network to compress the motion information; and processing the output of the motion information compression network through a convolutional layer to obtain the sub-motion information extracted by the first motion information extraction unit.

[0016] In some embodiments of the present invention, the step of inputting motion information extracted by an optical flow model into a motion information compression network combining an attention mechanism and a convolutional neural network to compress the motion information includes: processing the motion information extracted by the optical flow model using a predetermined number of 1×1 convolutional kernels in the motion information compression network; outputting the processing results of the predetermined number of convolutional kernels to a fully connected layer and an attention mechanism network respectively; concatenating the processing results of the predetermined number of convolutional kernels using a fully connected layer; inputting the concatenated processing results into a convolutional neural network; uniformly dividing the input into a predetermined number of tensor blocks using the convolutional neural network; processing each tensor block using a different convolutional kernel; concatenating the outputs of the convolutional neural network; using the processing results of the predetermined number of convolutional kernels as the query, key, and value of the attention mechanism network respectively; calculating the attention matrix using the query and key; performing matrix multiplication using the attention matrix and value to obtain the attention feature matrix; concatenating the outputs of the attention mechanism network; and weighting and summing the outputs of the convolutional neural network and the attention mechanism network based on predetermined weights to obtain the final output of the motion information compression network.

[0017] In some embodiments of the present invention, the motion information encoder further includes a second motion information extraction unit, which includes an optical flow model, a convolutional layer, and a generalized division normalization function. The step of the motion information extraction unit extracting sub-motion information based on the input further includes: inputting the second type of frame to be compressed and the reference frame at the original resolution into the second motion information extraction unit to extract motion information at the original resolution; and processing the motion information at the original resolution extracted by the second motion information extraction unit through the convolutional layer and the generalized division normalization function to obtain the sub-motion information extracted by the second motion information extraction unit.

[0018] In some embodiments of the present invention, the motion information encoder further includes a third motion information extraction unit, which includes a 1×1 convolutional kernel, a self-attention mechanism network, and a convolutional layer. The step of the motion information extraction unit extracting sub-motion information based on the input further includes: inputting a reference frame into a 1×1 convolutional kernel and using the output of the 1×1 convolutional kernel as the key and value of the self-attention mechanism network; inputting the second type of frame to be compressed at the original resolution into another 1×1 convolutional kernel and using the output of the other 1×1 convolutional kernel as the query of the self-attention mechanism network; and inputting the output of the self-attention mechanism network based on the above key, value, and query into the convolutional layer for processing to obtain the sub-motion information extracted by the third motion information extraction unit.

[0019] Another aspect of the present invention provides a video compression system based on an attention mechanism, including a processor and a memory, wherein the memory stores computer instructions, and the processor is configured to execute the computer instructions stored in the memory, wherein when the computer instructions are executed by the processor, the system implements the steps of the method as described in any of the above embodiments.

[0020] Another aspect of the present invention provides a computer-readable storage medium having a computer program stored thereon that, when executed by a processor, implements the steps of the method as described in any of the above embodiments.

[0021] The video compression method and system based on the attention mechanism proposed in this invention introduces a motion information encoder based on the attention mechanism to extract motion information. Compared with the motion information encoder based on the convolutional neural network, the motion information encoder based on the attention mechanism can better preserve global information. At the same time, the multiple motion information extraction units contained in the motion information encoder based on the attention mechanism extract motion information based on different strategies and calculate a final motion information by weighting, avoiding a single motion information calculation method, thereby avoiding excessive dependence on the results of optical flow estimation and ensuring the stability of video compression quality.

[0022] Additional advantages, objects, and features of the invention will be set forth in part in the description which follows, and will also become apparent in part to those skilled in the art upon studying the description, or may be learned by practice of the invention. The objects and other advantages of the invention can be realized and obtained by means of the structures specifically pointed out in the description and drawings.

[0023] Those skilled in the art will understand that the objectives and advantages achievable with the present invention are not limited to those specifically described above, and that the above and other objectives achievable with the present invention will become clearer from the following detailed description. Attached Figure Description

[0024] The accompanying drawings, which are included to provide a further understanding of the invention and form part of this application, are not intended to limit the scope of the invention. In the drawings:

[0025] Figure 1 This is a flowchart of a video compression method based on an attention mechanism in one embodiment of the present invention.

[0026] Figure 2 This is an overall framework diagram of a video compression method based on an attention mechanism in one embodiment of the present invention.

[0027] Figure 3 This is an overall framework diagram of a video compression method based on an attention mechanism in another embodiment of the present invention.

[0028] Figure 4 This is a schematic diagram of a motion information encoder structure based on an attention mechanism in one embodiment of the present invention.

[0029] Figure 5 This is a flowchart of motion information compression using a motion information compression network in one embodiment of the present invention.

[0030] Figure 6 This is a schematic diagram of the structure of a motion information decoder in one embodiment of the present invention. Detailed Implementation

[0031] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the embodiments and accompanying drawings. Here, the illustrative embodiments and descriptions of this invention are used to explain the invention, but are not intended to limit the invention.

[0032] It should also be noted that, in order to avoid obscuring the invention with unnecessary details, only the structures and / or processing steps closely related to the solution according to the invention are shown in the accompanying drawings, while other details that are not closely related to the invention are omitted.

[0033] It should be emphasized that the term "including / comprises" as used herein refers to the presence of a feature, element, step, or component, but does not exclude the presence or addition of one or more other features, elements, steps, or components.

[0034] It should also be noted that, unless otherwise specified, the term "connection" in this article can refer not only to a direct connection, but also to an indirect connection involving an intermediary.

[0035] In the following description, embodiments of the invention will be illustrated with reference to the accompanying drawings. In the drawings, the same reference numerals represent the same or similar parts, or the same or similar steps.

[0036] Existing optical flow-based video compression models suffer from high computational complexity, a large number of model parameters, and lengthy training and testing times. Furthermore, their compression performance largely depends on optical flow estimation results, making it difficult to obtain appropriate motion information when the video to be compressed is not suitable for an optical flow network. To address the over-reliance on a single optical flow estimation result in existing technologies, and the potential loss of global information that may result from using convolutional neural networks exclusively, this invention proposes a video compression method and system based on an attention mechanism. This method and system incorporate an attention mechanism into a convolutional neural network and design a motion encoder containing multiple motion information extraction units. Motion information is extracted based on different strategies and weighted to calculate the final motion information. On the one hand, combining an attention mechanism with a convolutional neural network better preserves global information; on the other hand, it reduces reliance on a single motion information calculation method, avoiding image quality fluctuations caused by quality variations in single-path acquisition. Furthermore, this invention obtains approximate optical flow information at the original resolution using an Optical Flow Net, then uses downsampled frames as additional input to obtain optical flow information at different scales. The original frame and reference frame are then used as inputs, serving as the Query and Key / Value for a Self-Attention Mechanism, respectively, simultaneously yielding spatial and temporal feature information. Optical flow refers to the perceived visual motion in the world of a moving person, and it can be used to determine the relative distance and angle between objects. It is the instantaneous velocity of pixel movement of spatial objects on the observation plane. It is a method that uses the temporal changes of pixels in an image sequence and the correlation between adjacent frames to find the correspondence between the previous and current frames, thereby calculating the motion information of objects between adjacent frames. For example, the distance to us can be determined by the speed of different moving targets; the farther the target, the closer it appears to be stationary, and the closer the target, the faster it moves. For example, an object moving at a 90-degree angle to the eye moves faster than objects moving at other angles, while an object moving perpendicular to the direction the eye is looking at appears stationary even as it grows larger. Optical flow models are pre-trained models used to analyze two-dimensional motion fields on time-varying images contained in video data to extract important visual cues. They have wide applications in action recognition, video understanding, video segmentation, object tracking, and panoramic stitching. Currently known optical flow models include FlowNet, MMFlow (developed by OpenMMLab), and PWC-Net.

[0037] Figure 1 This is a flowchart of a video compression method based on an attention mechanism according to an embodiment of the present invention. The method includes the following steps:

[0038] Step S110: Receive the original video to be compressed and divide the original video into multiple image groups; wherein, each image group contains a first type of frame to be compressed and multiple second type of frame to be compressed, and the first type of frame to be compressed is the image frame from the original video that is ranked first in time in each image group.

[0039] Step S120: Compress the first type of frame to be compressed using a preset first compression model, and use the compression result as a reference frame.

[0040] In one embodiment of the present invention, the preset first compression model can be an end-to-end image compression model based on the Ballé algorithm, published in the ICLR conference as End-to-end optimized image compression. However, the present invention is not limited to this; the first compression model is merely an example used to compress a first type of frame to be compressed, thereby obtaining a reference frame. Based on the reference frame and the extracted motion information, the second type of frame to be compressed within the image group is then compressed, and a reconstructed frame is generated.

[0041] Step S130: Input the reference frame and the second type of frames to be compressed into the second compression model. The second compression model includes a motion information encoder based on an attention mechanism to extract the motion information of each second type of frame to be compressed relative to the reference frame. The motion information decoder included in the second compression model reconstructs the extracted motion information. The reconstructed motion information and the reference frame are input into the motion compensation module, and the reconstructed frame obtained by compressing each second type of frame to be compressed is output. The motion information encoder based on the attention mechanism includes multiple motion information extraction units. Each motion information extraction unit extracts sub-motion information based on the input. Combined with the preset weights of the extraction results of each motion information extraction unit, the final motion information is calculated by weighting. The motion compensation module is a mapping of traditional video compression technology to the field of deep learning. First, the motion information is extracted by the motion information encoder, and then the motion information is reconstructed through quantization processing and motion information decoder processing. A reconstructed frame is generated based on the reconstructed motion information and the reference frame. This reconstructed frame is actually already compressed. In the attention mechanism, Query is the query, Key is the key, and Value is the value. The query is mapped to the relevant Key in the database to obtain the most matching Value.

[0042] Optionally, after quantizing the output of the residual encoder in step S130, the method further includes: inputting the quantized output of the residual encoder and the quantized motion information into a bitrate estimation model to estimate the size of the compressed bitstream. Quantization is performed to compress data usage; quantization further sparsifies and discretizes continuous weights. After discretization, the original continuous and dense values ​​can be represented by discrete values. Bitrate estimation is used to estimate the size of the compressed bitstream. In one embodiment of the invention, the probabilistic estimation model proposed by Ballé can be used to estimate the required bit size.

[0043] Further, in the step of inputting the reconstructed motion information and reference frame into the motion compensation module and outputting the reconstructed frame obtained by compressing each second type of frame to be compressed, the method may further include: (1) performing a subtraction calculation on each second type of frame to be compressed and the reconstructed frame output by the corresponding motion compensation module to obtain the residual corresponding to the second type of frame to be compressed; (2) inputting the residual corresponding to the second type of frame to be compressed into the residual encoder and performing quantization processing on the output of the residual encoder; (3) inputting the output of the quantized residual encoder into the residual decoder for reconstruction; (4) performing an addition calculation on the output of the residual decoder and the reconstructed frame output by the motion compensation module, and using the result of the addition calculation as the reconstructed frame after removing residual redundancy. Calculating the residual and encoding the residual is to further reduce residual redundancy, and the decoding operation corresponds to dequantization. Optionally, in model training, noise is added during residual encoding and quantization in this step to improve the generalization ability of the model. In another embodiment of the present invention, the reconstructed frame without residual processing is called the prediction frame.

[0044] Following the step of quantizing the output of the residual encoder mentioned above, the method further includes: inputting the quantized output of the residual encoder and the quantized motion information into a bitrate estimation model to estimate the size of the compressed bitstream. By estimating the bitstream size, the required degree of video compression can be determined, thereby achieving an optimal balance between compressed video quality and transmission bandwidth, ensuring the highest possible compressed video quality.

[0045] Step S140: Combine the reference frames and reconstructed frames obtained after processing each image group through the first compression model and the second compression model in sequence to obtain the compressed video.

[0046] In step S140, each image group contains all image frames that have been compressed. Based on the order of the image frames in the original video, the compressed image frames are combined to obtain the compressed video.

[0047] The video compression method and system based on the attention mechanism proposed in this invention introduces a motion information encoder based on the attention mechanism to extract motion information. Compared with the motion information encoder based on the convolutional neural network, the motion information encoder based on the attention mechanism can better preserve global information. At the same time, the multiple motion information extraction units contained in the motion information encoder based on the attention mechanism extract motion information based on different strategies and calculate a final motion information by weighting. This avoids a single motion information calculation method, thereby avoiding excessive dependence on the results of optical flow estimation and avoiding the image quality fluctuations in video compression caused by the quality fluctuations of a single motion information acquisition path, thus ensuring the stability of video compression quality.

[0048] More specifically, in one embodiment of the present invention, the motion information encoder includes a first motion information extraction unit, which includes a downsampling module, an optical flow model, a motion information compression network, and a convolutional layer.

[0049] Accordingly, the step S130, which involves the motion information extraction unit extracting sub-motion information based on the input, includes: inputting the second type of frame to be compressed and the reference frame at the original resolution into the first motion information extraction unit; obtaining the second type of frame to be compressed and the reference frame at a reduced resolution based on the downsampling module; extracting motion information from the second type of frame to be compressed and the reference frame at a reduced resolution based on the optical flow model; and inputting the motion information extracted by the optical flow model into a motion information compression network that combines an attention mechanism network and a convolutional neural network to compress the motion information. The output of the motion information compression network is processed by a convolutional layer to obtain the sub-motion information extracted by the first motion information extraction unit. The motion information encoder includes the first motion information extraction unit, which includes a motion information compression network. The motion information compression network mainly consists of two parts: an attention mechanism network and a convolutional neural network.

[0050] Furthermore, the step of inputting the motion information extracted by the optical flow model into the motion information compression network of the joint attention mechanism and convolutional neural network to compress the motion information includes: (1) The motion information compression network contains a preset number of 1×1 convolutional kernels to process the motion information extracted by the optical flow model, and outputs the processing results of the preset number of convolutional kernels to the fully connected layer and the attention mechanism network respectively. The preset number can be 3, which can be consistent with the number of Q, K and V, but other numbers are also feasible; (2) The processing results of the preset number of convolutional kernels are concatenated using the fully connected layer, and the concatenated processing results are output to the fully connected layer and the attention mechanism network respectively. The input is fed into a convolutional neural network, which divides the input into a preset number of tensor blocks. Each tensor block is processed using a different convolutional kernel, and the output of the convolutional neural network is concatenated. (3) The processing results of the preset number of convolutional kernels are used as the Query, Key, and Value of the attention mechanism network, respectively. The attention matrix is ​​calculated using the Query and Key, and the attention feature matrix is ​​obtained by matrix multiplication using the attention matrix and Value. The output of the attention mechanism network is concatenated. (4) Based on the preset weights of the output of the convolutional neural network and the output of the attention mechanism network, the weighted sum is used to obtain the final output of the motion information compression network. It should be noted that the Query is a query statement, the Value is a data item, and the Key is the key corresponding to each data item. The Query, Key, and Value can be understood as functions contained in the attention mechanism network. The processing results of the preset number of convolutional kernels are used as the input of the function. This is actually a tensor calculation operation. The output of the preset number of convolutional kernels is a tensor of the same size. This technical content is understandable to those skilled in the art.

[0051] In the above embodiments of the invention, in the first motion information extraction unit of the motion information encoder based on the attention mechanism, the resolution of the input image is reduced by the downsampling module, thereby reducing the complexity of the second compression model in image compression, thus reducing the number of model parameters. In the process of introducing the attention mechanism and combining it with the convolutional neural network, the huge computational complexity is avoided, the training and inference time is shortened, and the long model inference time caused by the use of optical flow models with a large number of overall parameters and images with original resolution, which is difficult to apply in practice, is avoided in the prior art.

[0052] In addition, the motion information encoder also includes a second motion information extraction unit, which comprises an optical flow model, convolutional layers, and a generalized division normalization function. The motion information encoder also includes a third motion information extraction unit, which comprises a 1×1 convolutional kernel, a self-attention mechanism network, and convolutional layers.

[0053] Accordingly, the step of inputting the motion information extracted by the optical flow model into the motion information compression network of the joint attention mechanism and the convolutional neural network to compress the motion information further includes: inputting the second type of frame to be compressed and the reference frame at the original resolution into the second motion information extraction unit to extract the motion information at the original resolution; and processing the motion information at the original resolution extracted by the second motion information extraction unit through a convolutional layer and a generalized division normalization function to obtain the sub-motion information extracted by the second motion information extraction unit.

[0054] Accordingly, the step of inputting the motion information extracted by the optical flow model into the motion information compression network of the joint attention mechanism and the convolutional neural network to compress the motion information further includes: inputting the reference frame into a 1×1 convolutional kernel, and using the output of the 1×1 convolutional kernel as the key and value of the self-attention mechanism network; inputting the second type of frame to be compressed at the original resolution into another 1×1 convolutional kernel, and using the output of the other 1×1 convolutional kernel as the query of the self-attention mechanism network; and inputting the output of the self-attention mechanism network based on the above key, value and query into the convolutional layer for processing to obtain the sub-motion information extracted by the third motion information extraction unit.

[0055] In summary, by adding two auxiliary information channels, namely multiple motion information extraction units, this invention compensates for the impact of inaccurate optical flow estimation on motion estimation, avoids excessive dependence of motion estimation on optical flow estimation results, and can still achieve good video compression results even when the motion information extracted by optical flow estimation is not accurate enough.

[0056] Figure 2 This is a schematic diagram of the overall framework of a video compression method based on an attention mechanism according to an embodiment of the present invention. The overall framework of the present invention includes modules for encoding and decoding motion information, and obtaining compressed frames in an image group excluding the reference frame based on the motion information. Here, the motion information is understood as the motion information of the frame to be compressed relative to the reference frame. Figure 3 This is a diagram illustrating the overall framework of a video compression method based on an attention mechanism according to another embodiment of the present invention. The overall framework of the present invention includes modules for encoding and decoding motion information and residual information, a model for compensating for the residual portion using motion information, a network for compressing the residual information, and a network for estimating the bit rate. Furthermore, the present invention proposes a novel motion estimation module, called an attention-based motion information encoder, to enhance the robustness of extracting motion information from between frames.

[0057] In a specific embodiment of the present invention, the entire video compression process includes the following steps:

[0058] (1) Taking the Ultra Video Group (UVG) dataset as an example, the 1920x1080 YUV420 video in the UVG video dataset is used as the original dataset. The YUV420 video is converted into a PNG image with RGB24 color mode using the ffmpeg command, which is used as the original input of this invention.

[0059] (2) The default group of pictures (GOP) size is 12. The Ballé algorithm is used as the I-frame compression model, and the compressed I-frame is used as the reference frame for the subsequent 11 original P-frames. and P in This means that the model's input and output, i.e., the compressed frame, are the inputs to the model. It should be explained that I-frame is an industry abbreviation for the first type of frame to be compressed, which is compressed using the first image compression model (i.e., the Ballé algorithm). P-frame is the second type of compressed frame, which is compressed using the second image compression model based on a reference frame. The second image compression model includes the motion estimation module based on the attention mechanism designed and proposed in this invention. This motion estimation module includes multiple motion information extraction units, wherein the first motion information extraction unit includes a motion information compression network that combines an attention mechanism network and a convolutional neural network.

[0060] (3) Encoding end: Specifically, the model (the second image compression model, used to compress the second type of images to be compressed) has two frames of input, namely the reference frame obtained by compressing the first type of images to be compressed contained in the image group. Second type of frame to be compressed x t The input is fed into an attention-based motion encoder to obtain motion information m. t The predicted motion information to be transmitted is obtained through quantization. The reconstructed motion information is then obtained through a motion information decoder. The input reference frame and the decoded motion information are used as inputs for motion compensation to obtain the model's predicted frame. Input frame x t With the predicted frame Subtracting them, we get the residual r. t The residual is encoded to obtain y t To further improve the compression ratio, y is quantized. t get The residual is reconstructed by decoding the residual. Finally, the predicted frames will be... Adding the reconstructed residual to the original data yields the reconstructed frame. This refers to the compression result of the second type of frame to be compressed, output by the second compression model. It's important to note that the predicted frame can also be directly used as the reconstructed frame; subsequent residual calculation, residual encoding, and decoding are optional steps. Encoding the residual can further reduce residual redundancy. During model training, the residual encoding operation here adds noise during quantization to improve the model's generalization ability, while the decoding operation corresponds to the dequantization operation.

[0061] Figure 6 This is a schematic diagram of the motion information decoder in one embodiment of the present invention. In the motion information decoder, the input is motion information. The resolution is continuously increased through Deconv (transposed convolution). Each transposed convolution multiplies the resolution by 2. Multiple IGDN (Inverse GDN) processes are also performed in between, ultimately resulting in the same initial resolution as the input frame. IGDN is the inverse of generalized divisive normalization (GDN).

[0062] (4) Network transmission part: based on predicted motion information and The estimated bitstream size, i.e., the file size transmitted over the network, can be obtained from the model. This invention maximizes the quality of the compressed image while considering the network channel environment, avoiding the poor image quality that may result from a single motion information extraction channel in existing technologies.

[0063] (5) Decoding end: The model directly transmits I-frames and reconstructs P-frames in units of GOPs. Each GOP's I-frame serves as the reference frame for the current GOP. Transmitted over the network, the decoding end obtains the quantized motion information of the P-frames. and the residual after encoding and quantization With reference frame and As input to the motion information decoder, the predicted frame is obtained. by As input to the residual decoder, the residual is obtained. Predicted frames With residual Add them together to get the decoded frame.

[0064] Figure 4 This is a schematic diagram of a motion information encoder structure based on an attention mechanism according to an embodiment of the present invention. The motion information encoder (Attention-guided Motion Encoder) includes a first motion information extraction unit, a second motion information extraction unit, and a third roadside information extraction unit. Figure 2The system is divided into three paths: the middle one is the first motion information extraction unit, the bottom one is the second motion information extraction unit, and the top one is the third motion information extraction unit. The three motion information extraction units are described below:

[0065] (1) For the first motion information extraction unit, the current frame and the reference frame at the original resolution are used as input. The input frame is first downsampled, and then the optical flow model is used to extract the lower-resolution optical flow information. The obtained optical flow information is then input into the joint attention mechanism and convolutional neural network motion compression network (Attention-CNN MotionCompression network, ACMC) to obtain motion information at different scales. Figure 4 v in t,↓2 This indicates that the output resolution is half of the original. Through convolution processing, multiple motion information can be restored to the same size for subsequent weighted averaging calculations.

[0066] (2) For the second motion information extraction unit, the current frame (the second type of frame to be compressed) at the original resolution and the reference frame (the first type of frame to be compressed that has already been compressed) are used as inputs. The motion information at the original resolution is extracted using an optical flow model. Then, the output motion information (existing in the form of optical flow values) is enhanced using a convolutional neural network (CNN). The extraction of motion information can be understood as a type of feature extraction. Among them, generalized divisive normalization (GDN) is a biologically inspired activation function that can capture the statistical characteristics of images well and convert them into a Gaussian distribution. Empirically, GDN can effectively reduce the number of parameters / layers of the model while achieving similar quality performance.

[0067] (3) For the third motion information extraction unit, the current frame and the reference frame at the original resolution are used as inputs. After 1x1 convolution processing, Key(K), Query(Q), and Value(V) in the attention mechanism are obtained. The difference is that K and V come from the reference frame, and Q comes from the current frame. Then, the self-attention mechanism is used to calculate the motion information in both the spatial and temporal domains.

[0068] Finally, weights are assigned to the outputs of (1)-(3), and the three outputs are merged as the final motion information, m. t =α*f path1 +β*f path2 +γ*f path3 , where α, β, and γ are weight values, and f represents motion information. In one embodiment of the present invention, the weights for α, β, and γ are preset to 0.25, 0.25, and 0.5, respectively, which is only an example.

[0069] Figure 5 This is a flowchart illustrating motion information compression using a motion information compression network in one embodiment of the present invention. The first motion information extraction network includes a motion information compression network (Attention-CNN Motion Compression Network, ACMC), which further includes an attention mechanism network and a convolutional neural network. First, in stage I, the optical flow extracted in the previous step is processed through three identical 1x1 convolutional kernels. These three identical feature sets are then fed into the fully connected layer and the attention mechanism network in stage II, respectively, where the feature information is processed using convolutional paradigms and attention mechanism paradigms. Specifically, in stage II:

[0070] (1) The processing results of the fully connected layer (FC) are concatenated and fed into a convolution operation. This convolutional neural network divides the output of the fully connected layer into multiple chunks, and processes each chunk using a different convolutional kernel. The fully connected layer is used to concatenate the output of stage I along the Channel dimension. It should be noted that the number of chunks in step (1) is a preset number (N). The concatenation operation (also abbreviated as Concat) refers to the concatenation of a new vector after the original vector, resulting in an increase in the corresponding dimension.

[0071] (2) The outputs of the three identical 1x1 convolutions are used as Query(Q), Key(K), and Value(V) respectively. The attention matrix M is calculated using Q and K, and then the VM is calculated using matrix multiplication to obtain the feature matrix extracted based on the attention mechanism. It should be noted that the number of steps in (2) is a preset number (N).

[0072] Finally, the N outputs from (1) and (2) are concatenated respectively, and the sum of the two is obtained using the weighting parameters μ and σ, which is used as the final feature output, f. ACMC =μ*f cnn +σ*f attn .

[0073] The video compression method and system based on the attention mechanism proposed in this invention introduces a motion information encoder based on the attention mechanism to extract motion information. Compared with the motion information encoder based on the convolutional neural network, the motion information encoder based on the attention mechanism can better preserve global information. At the same time, the multiple motion information extraction units contained in the motion information encoder based on the attention mechanism extract motion information based on different strategies and calculate a final motion information by weighting. This avoids a single motion information calculation method, thereby avoiding excessive dependence on the results of optical flow estimation and avoiding the image quality fluctuation of video compression caused by the quality fluctuation of a single motion information acquisition path. This ensures the stability of video compression quality and enhances the robustness of video compression based on optical flow estimation.

[0074] In some embodiments of the present invention, in the first motion information extraction unit of the motion information encoder based on the attention mechanism, the resolution of the input image is reduced by a downsampling module. Then, the downsampled frame is used as another input to obtain optical flow information at different scales. The original frame and the reference frame are then used as inputs, serving as the Query and Key / Value of the self-attention mechanism network, respectively. Simultaneously, spatial and temporal feature information is obtained, thereby reducing the complexity of image compression by the second compression model and reducing the number of model parameters. In the process of introducing the attention mechanism and combining it with the convolutional neural network, the huge computational complexity is avoided, and the training and inference time is shortened. This avoids the problems in the prior art, such as the long model inference time caused by using optical flow models with a large overall number of parameters and images with original resolution, which makes it difficult to apply in practice.

[0075] In some embodiments of the present invention, by adding two auxiliary information channels, namely multiple motion information extraction units, the impact of inaccurate optical flow estimation on motion estimation is compensated, and the excessive dependence of motion estimation on optical flow estimation results is avoided. Even when the motion information extracted by optical flow estimation is not accurate enough, a better video compression effect can still be achieved.

[0076] This invention introduces motion information obtained from motion estimation based on an attention mechanism into a video compression model based on optical flow estimation. By utilizing temporal information at different scales and spatiotemporal information based on an attention mechanism, the accuracy of motion estimation is improved. This invention can flexibly adapt to different types of videos and further improve the video compression rate while ensuring video quality as much as possible, so as to adapt to the bandwidth of data transmission.

[0077] Corresponding to the above method, the present invention also provides a video compression system based on an attention mechanism. The system includes a computer device, which includes a processor and a memory. The memory stores computer instructions, and the processor is used to execute the computer instructions stored in the memory. When the computer instructions are executed by the processor, the system implements the steps of the method described above.

[0078] This invention also provides a computer-readable storage medium storing a computer program thereon, which, when executed by a processor, performs the steps of the method as described above. The computer-readable storage medium may be a tangible storage medium, such as random access memory (RAM), main memory, read-only memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, register, floppy disk, hard disk, removable storage disk, CD-ROM, or any other form of storage medium known in the art.

[0079] Those skilled in the art will understand that the exemplary components, systems, and methods described in conjunction with the embodiments disclosed herein can be implemented in hardware, software, or a combination of both. Whether implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this invention. When implemented in hardware, it can be, for example, electronic circuits, application-specific integrated circuits (ASICs), appropriate firmware, plug-ins, function cards, etc. When implemented in software, the elements of this invention are programs or code segments used to perform the desired tasks. The programs or code segments can be stored in a machine-readable medium or transmitted over a transmission medium or communication link via data signals carried in a carrier wave.

[0080] It should be clarified that the present invention is not limited to the specific configurations and processes described above and shown in the figures. For the sake of brevity, detailed descriptions of known methods are omitted here. In the above embodiments, several specific steps are described and shown as examples. However, the method process of the present invention is not limited to the specific steps described and shown. Those skilled in the art can make various changes, modifications, and additions, or change the order of steps, after understanding the spirit of the present invention.

[0081] In this invention, features described and / or illustrated for one embodiment may be used in the same or similar manner in one or more other embodiments, and / or combined with or in place of features of other embodiments.

[0082] The above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, various modifications and variations of the embodiments of the present invention are possible. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A video compression method based on an attention mechanism, characterized in that, The method includes the following steps: Receive the original video to be compressed and divide the original video into multiple image groups; wherein, each image group contains a first type of frame to be compressed and multiple second type of frame to be compressed, and the first type of frame to be compressed is the first image frame in time order from the original video contained in each image group; The first type of frames to be compressed is compressed using a preset first compression model, and the compression result is used as a reference frame. The reference frame and the second type of frames to be compressed are input into the second compression model. The second compression model contains a motion information encoder based on an attention mechanism to extract the motion information of each second type of frame to be compressed relative to the reference frame. The motion information decoder in the second compression model reconstructs the extracted motion information. The reconstructed motion information and the reference frame are input into the motion compensation module and the reconstructed frame obtained by compressing each second type of frame to be compressed is output. The motion information encoder based on the attention mechanism contains three motion information extraction units. Each motion information extraction unit extracts sub-motion information based on the input. The final motion information is obtained by weighted calculation based on the preset weights of the extraction results of each motion information extraction unit. The compressed video is obtained by sequentially combining the reference frames and reconstructed frames obtained after processing each image group through the first compression model and the second compression model. The motion information encoder includes a first motion information extraction unit, a second motion information extraction unit, and a third motion information extraction unit; The first motion information extraction unit includes a downsampling module, an optical flow model, a motion information compression network, and a convolutional layer; the second motion information extraction unit includes an optical flow model, a convolutional layer, and a generalized division normalization function; the third motion information extraction unit includes a module of size [missing information]. Convolutional kernels, self-attention mechanism networks, and convolutional layers; The motion information extraction unit extracts sub-motion information based on the input, including the following steps: The original resolution of the second type of frame to be compressed and the reference frame are input into the first motion information extraction unit. The reduced resolution of the second type of frame to be compressed and the reference frame are obtained based on the downsampling module. Motion information is extracted from the reduced resolution of the second type of frame to be compressed and the reference frame based on the optical flow model. The motion information extracted by the optical flow model is input into the motion information compression network of the joint attention mechanism network and the convolutional neural network to compress the motion information. The output of the motion information compression network is processed by the convolutional layer to obtain the sub-motion information extracted by the first motion information extraction unit. The second type of frame to be compressed and the reference frame at the original resolution are input into the second motion information extraction unit to extract the motion information at the original resolution. The motion information at the original resolution extracted by the second motion information extraction unit is processed by a convolutional layer and a generalized division normalization function to obtain the sub-motion information extracted by the second motion information extraction unit. Input the reference frame to In the convolution kernel, and The output of the convolutional kernel serves as the key and value of the self-attention mechanism network, and the second type of frame to be compressed at the original resolution is input into another... In the convolution kernel, and the other one The output of the convolutional kernel is used as the request query of the self-attention mechanism network. The output of the self-attention mechanism network based on the above Key, Value and Query is input into the convolutional layer for processing to obtain the sub-motion information extracted by the third motion information extraction unit.

2. The method according to claim 1, characterized in that, In the step of inputting the reconstructed motion information and reference frame into the motion compensation module and outputting the reconstructed frame obtained by compressing each second type of frame to be compressed, the method further includes: Subtraction is performed on each second type of frame to be compressed and its corresponding reconstructed frame output by the motion compensation module to obtain the residual corresponding to the second type of frame to be compressed. The residual corresponding to the second type of frame to be compressed is input into the residual encoder, and the output of the residual encoder is quantized. The output of the quantized residual encoder is input into the residual decoder for reconstruction; The output of the residual decoder and the reconstructed frame output by the motion compensation module are added together, and the result of the addition is used as the reconstructed frame after removing residual redundancy.

3. The method according to claim 2, characterized in that, Before the motion information decoder in the second compression model reconstructs the extracted motion information, the method further includes: quantizing the motion information extracted by the motion information encoder; After quantizing the output of the residual encoder, the method further includes: inputting the quantized output of the residual encoder and the quantized motion information into the bit rate estimation model to estimate the size of the compressed bit stream.

4. The method according to claim 1, characterized in that, The preset first compression model is an end-to-end image compression model based on the Ballé algorithm.

5. The method according to claim 1, characterized in that, The step of inputting motion information extracted by the optical flow model into a motion information compression network combining a joint attention mechanism and a convolutional neural network to compress the motion information includes: The motion information compression network contains a preset number of elements of size [number to be filled in]. The convolutional kernels process the motion information extracted by the optical flow model, and the processing results of the preset number of convolutional kernels are output to the fully connected layer and the attention mechanism network respectively. The processing results of a preset number of convolutional kernels are concatenated using fully connected layers, and the concatenated processing results are input into a convolutional neural network. The convolutional neural network evenly divides the input into a preset number of tensor blocks, processes each tensor block using a different convolutional kernel, and concatenates the output of the convolutional neural network. The processing results of a preset number of convolutional kernels are used as the Query, Key, and Value of the attention mechanism network, respectively. The attention matrix is ​​calculated using the Query and Key, and the attention feature matrix is ​​obtained by matrix multiplication using the attention matrix and Value. The outputs of the attention mechanism network are then concatenated. Based on the preset weights of the outputs of the convolutional neural network and the attention mechanism network, the output of the final motion information compression network is obtained by weighted summation.

6. A video compression system based on an attention mechanism, comprising a processor and a memory, characterized in that, The memory stores computer instructions, and the processor executes the computer instructions stored in the memory. When the computer instructions are executed by the processor, the system implements the steps of the method as described in any one of claims 1 to 5.

7. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it implements the steps of the method as described in any one of claims 1 to 5.

Images