Video compression method, apparatus, device, storage medium and computer program product

Abstract

This application discloses a video compression method, apparatus, device, storage medium, and computer program product, belonging to the field of video processing technology. The method, applied at the encoding end, includes: determining a motion vector based on the original frame of the current frame and the compressed frame of the previous frame adjacent to the current frame; obtaining a motion information representation by inputting the motion vector to a first recurrent autoencoder; the first recurrent autoencoder includes a coding layer, a quantization layer, and a decoding layer connected in sequence, the coding layer including a first convolutional layer, a first DatRNN module, and a second convolutional layer, and the decoding layer including a first deconvolutional layer, a second DatRNN module, and a second deconvolutional layer; determining a prediction frame based on the compressed frame of the previous frame and the motion information representation; and sending the prediction frame and the residual between the original frame and the prediction frame to the decoding end. This method can better adapt to changes in different time and spatial scales, improving video compression efficiency.

Description

Technical Field

[0001] This application relates to the field of video processing technology, and in particular to a video compression method, apparatus, device, storage medium, and computer program product. Background Technology

[0002] Significant progress has been made in the application of deep learning technology in the field of video processing. Related methods propose a video compression framework based on recurrent neural networks. This framework, through two key components—a recurrent autoencoder (RAE) and a recurrent probability model (RPM)—fully utilizes the temporal correlation between video frames to achieve more efficient video compression.

[0003] However, when dealing with complex changes in time series, such as fast-moving scenes, the video compression framework RAE will be unable to accurately capture the changes between frames, resulting in a decrease in compression efficiency and reconstruction quality. Summary of the Invention

[0004] This application provides a video compression method, apparatus, device, storage medium, and computer program product to at least solve the problem that related video compression methods cannot accurately capture inter-frame differences when dealing with complex changes in fast-moving scenes.

[0005] To solve the above-mentioned technical problems, this application is implemented as follows: In a first aspect, embodiments of this application provide a video compression method applied at an encoding end, comprising: determining a motion vector based on the original frame of the current frame and the compressed frame of the previous frame adjacent to the current frame; obtaining a motion information representation by inputting the motion vector to a first recurrent autoencoder; wherein the first recurrent autoencoder includes an encoding layer, a quantization layer, and a decoding layer connected in sequence, the encoding layer including a first convolutional layer, a first DatRNN module, and a second convolutional layer, and the decoding layer including a first deconvolutional layer, a second DatRNN module, and a second deconvolutional layer; the first DatRNN module and the second DatRNN module are both recurrent neural networks based on a deformable attention mechanism, used to capture inter-frame dependencies in a time series; determining a prediction frame based on the compressed frame of the previous frame and the motion information representation; and sending the prediction frame and the residual between the original frame and the prediction frame to the decoding end. Secondly, embodiments of this application provide a video compression method applied at a decoding end, comprising: acquiring a predicted frame of the current frame sent by an encoding end, and a residual between the original frame of the current frame and the predicted frame; acquiring a residual estimate by inputting the residual to a second recursive autoencoder; wherein the second recursive autoencoder has the same structure as the first recursive autoencoder described in the first aspect above; and fusing the predicted frame and the residual estimate to obtain a compressed frame of the current frame.

[0006] Thirdly, embodiments of this application provide a video compression apparatus applied at an encoding end, comprising: a first determining module, configured to determine a motion vector based on the original frame of the current frame and the compressed frame of the previous frame adjacent to the current frame; an encoding module, configured to obtain a motion information representation by inputting the motion vector to a first recurrent autoencoder; wherein the first recurrent autoencoder includes an encoding layer, a quantization layer, and a decoding layer connected in sequence, the encoding layer including a first convolutional layer, a first DatRNN module, and a second convolutional layer, and the decoding layer including a first deconvolutional layer, a second DatRNN module, and a second deconvolutional layer; the first DatRNN module and the second DatRNN module are both recurrent neural networks based on a deformable attention mechanism, used to capture inter-frame dependencies in a time series; a second determining module, configured to determine a predicted frame based on the compressed frame of the previous frame and the motion information representation; and a sending module, configured to send the predicted frame and the residual between the original frame and the predicted frame to the decoding end.

[0007] Fourthly, embodiments of this application provide a video compression apparatus applied at a decoding end, comprising: a first acquisition module, configured to acquire a predicted frame of the current frame sent by an encoding end, and a residual between the original frame of the current frame and the predicted frame; a second acquisition module, configured to acquire a residual estimate by inputting the residual to a second recursive autoencoder; wherein the second recursive autoencoder has the same structure as the first recursive autoencoder described in the third aspect above; and a fusion processing module, configured to perform fusion processing on the predicted frame and the residual estimate to obtain a compressed frame of the current frame.

[0008] Fifthly, embodiments of this application provide an electronic device, the electronic device including a processor and a memory, the memory storing programs or instructions executable on the processor, the programs or instructions being executed by the processor to implement the steps of the method described in the first or second aspect above.

[0009] In a sixth aspect, embodiments of this application provide a computer-readable storage medium on which a program or instructions are stored, which, when executed by a processor, implement the steps of the method described in the first or second aspect above.

[0010] In a seventh aspect, embodiments of this application provide a computer program product, the computer program product including a computer program stored on a non-transitory computer-readable storage medium, the computer program including program instructions, which, when executed by a computer, cause the computer to perform the steps of the method described in the first or second aspect above.

[0011] In this embodiment, motion vectors are determined based on the original frame of the current frame and the compressed frame of the previous frame adjacent to the current frame. Motion information representation is obtained by inputting the motion vectors to a first recurrent autoencoder. The first recurrent autoencoder includes a coding layer, a quantization layer, and a decoding layer connected in sequence. The coding layer includes a first convolutional layer, a first DatRNN module, and a second convolutional layer. The decoding layer includes a first deconvolutional layer, a second DatRNN module, and a second deconvolutional layer. Both the first and second DatRNN modules are recurrent neural networks based on deformable attention mechanisms, used to capture inter-frame dependencies in the time series. A prediction frame is determined based on the compressed frame of the previous frame and the motion information representation. The prediction frame and the residual between the original frame and the prediction frame are sent to the decoding end. Thus, by performing spatiotemporal correlation analysis based on motion vectors between adjacent frames and combining this with a recurrent neural network based on a deformable attention mechanism to capture inter-frame dependencies in the time series, it is possible to better adapt to changes in different time and spatial scales and improve video compression efficiency.

[0012] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and do not limit this application. Attached Figure Description

[0013] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0014] Figure 1 A flowchart illustrating some embodiments of the video compression method provided in this application is shown; Figure 2 The diagram shows structural schematics of the encoding and decoding ends provided in some embodiments of this application; Figure 3 The diagram shows a schematic representation of the structure of a DatRNN module provided in some embodiments of this application; Figure 4 The diagram shows a structural schematic of a deformable attention unit provided in some embodiments of this application; Figure 5 A flowchart illustrating a video compression method provided in other embodiments of this application is shown; Figure 6This application shows one of the structural schematic diagrams of a video compression apparatus provided in some embodiments; Figure 7 This is shown as a second schematic diagram of the structure of a video compression apparatus provided in some embodiments of this application; Figure 8 The diagram shows a schematic representation of the structure of an electronic device provided in some embodiments of this application. Detailed Implementation

[0015] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numbers in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this application. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this application as detailed in the appended claims.

[0016] Most deep learning-based video compression methods employ acyclic structures, utilizing a limited number of reference frames to compress new frames. This significantly restricts the exploitation of temporal correlations between video sequences. To address this issue, Recurrent Learned Video Compression (RLVC) proposes a video compression framework based on recurrent neural networks. This framework, through two core components—a recurrent autoencoder (RAE) and a recurrent probability model (RPM)—fully exploits the temporal correlations between video frames, thereby achieving more efficient compression.

[0017] However, the above-mentioned RLVC has at least the following problems: 1) Limited ability to capture spatial features In RLVC, Reconstructive Aspect-Oriented Earth (RAE) primarily relies on Convolutional Neural Networks (CNNs) to extract spatial features. While CNNs can capture local spatial information, their ability to capture complex global spatial relationships and significant spatial changes (such as object edges and textures) is limited. This can negatively impact compression efficiency and reconstruction quality when processing video frames with complex spatial structures.

[0018] 2) Time dependency modeling is not efficient enough. Temporal dependency modeling in RLVC is primarily achieved through the cyclic structures of RPM and RAE. While these mechanisms can capture the temporal correlations between frames, they may not be efficient enough when dealing with complex changes in time series. For example, in fast-moving scenes, RAE may not accurately capture the changes between frames, leading to decreased compression efficiency and reconstruction quality.

[0019] 3) The model lacks flexibility and adaptability. RLVC's model structure is relatively fixed, primarily relying on RAE and RPM. This fixed structure may not be flexible enough to handle different types of video content (such as natural landscapes, motion scenes, etc.). Different video content has different spatial and temporal characteristics, and a fixed-structure model may perform poorly in certain scenarios.

[0020] 4) Higher bit rate In RLVC, reducing the bitrate is key to improving compression efficiency. While RAE and RPM can effectively capture spatiotemporal features, in some cases, it may not be possible to further reduce the bitrate, especially when processing complex video content. This can result in compressed video files that are still large, impacting transmission efficiency.

[0021] 5) The quality of reconstruction needs to be improved. In video compression, reconstruction quality is a crucial metric for evaluating compression effectiveness. While Reconstructive Imagery (RAE) can generate latent representations and reconstruct video frames, the reconstruction quality may be less than ideal in certain situations. For example, for video frames with complex spatial structures, RAE may fail to accurately generate latent representations, resulting in distortion in the reconstructed video frames.

[0022] To address the aforementioned problems in video compression, this application provides a video compression method. This method performs spatiotemporal correlation analysis based on motion vectors between adjacent frames and combines a recurrent neural network based on a deformable attention mechanism to capture inter-frame dependencies in the time series, thereby better adapting to changes in different time and spatial scales and improving video compression efficiency.

[0023] Please see Figure 1 , Figure 1The illustration shows a flowchart of a video compression method provided in some embodiments of this application. The execution entity of this method can be a terminal device or a server. The terminal device can be a personal computer, a mobile terminal device such as a mobile phone or tablet, or a user-used terminal device. The server can be a standalone server or a server cluster composed of multiple servers. Furthermore, the server can be a backend server for a specific service, or a backend server for a platform or application (e.g., streaming media platform, video conferencing application, social media, monitoring system, virtual reality and augmented reality applications, etc.). This application uses a server as the execution entity for illustration. For the case of a terminal device, the following related content can be used, and it will not be repeated here. Figure 1 As shown, method 100, applied to the encoding end, may include the following steps: Step 101: Determine the motion vector based on the original frame of the current frame and the compressed frame of the previous frame adjacent to the current frame.

[0024] In practical implementation, objects or scenes in the original frame of the current frame and the compressed frame of the adjacent previous frame can be identified. Motion vectors are generated based on the displacement of the objects or scenes. These motion vectors characterize the motion of various parts of the scene or object from the previous frame to the current frame. In a deep learning-based video compression framework, this motion estimation process can be implemented using convolutional neural networks or optical flow networks.

[0025] In one exemplary embodiment, such as Figure 2 As shown, the compressed frame of the previous frame (Also known as the reconstructed frame) and the original frame of the current frame. The data is input into the motion estimation module, which analyzes adjacent frames and predicts the motion vector for the current frame. This process aims to capture motion information of objects or scenes in a video for motion compensation in subsequent steps, thereby optimizing the compression and reconstruction process. Motion vectors This will be used to guide predictive coding during the coding process, thereby reducing the amount of data that needs to be encoded. Motion estimation is represented as follows: .

[0026] Step 102: Obtain motion information representation by inputting motion vectors into the first recurrent autoencoder; wherein the first recurrent autoencoder includes an encoding layer, a quantization layer and a decoding layer connected in sequence, the encoding layer includes a first convolutional layer, a first DatRNN module and a second convolutional layer, and the decoding layer includes a first deconvolutional layer, a second DatRNN module and a second deconvolutional layer.

[0027] The first and second DatRNN modules mentioned above are both recurrent neural networks based on deformable attention mechanisms, used to capture inter-frame dependencies in time series.

[0028] Continuing with the above embodiments, as follows: Figure 2 As shown, by using motion vectors The input is fed into the first recursive autoencoder to obtain motion information representation. The first recurrent autoencoder comprises an encoder layer, a quantize layer, and a decoder layer connected in sequence. The encoder includes a first convolutional layer (conv), a first DatRNN module (N DatRNNs), and a second convolutional layer (conv). The decoder includes a first deconvolutional layer (deconv), a second DatRNN module (N DatRNNs), and a second deconvolutional layer (deconv). Both the first and second DatRNN modules are recurrent neural networks based on a deformable attention mechanism, used to capture inter-frame dependencies in the time series.

[0029] In some possible implementations, step 102 above, which involves obtaining a motion information representation by inputting a motion vector to a first recursive autoencoder, includes: Motion vectors are input into the encoding layer, and spatial features are extracted through the first convolutional layer. After the spatial features are captured by the first DatRNN module to capture inter-frame dependencies, deep spatial features are extracted through the second convolutional layer. The deep spatial features are then passed through the quantization layer to obtain a quantized latent representation. The quantized latent representation is then passed through the first deconvolutional layer to recover the spatial features. After the spatial features are captured by the second DatRNN module to capture inter-frame dependencies, the motion information representation is reconstructed through the second deconvolutional layer.

[0030] Continuing with the above embodiments, the encoder (i.e. Figure 2 The encoder on the left receives the motion vector output by Motion Estimation. The motion vector First, the spatial features of the input data are extracted using the first convolutional layer (Conv). These spatial features are then fed into the first DatRNN module for processing. By integrating a deformable Transformer, the first DatRNN module can more effectively focus on important spatial features. After N iterations of processing by the DatRNN module, capturing inter-frame dependencies, the features are further processed by the second convolutional layer (Conv) to extract deeper spatial features. Finally, the processed deep spatial features are converted into discrete quantized latent representations through a quantization layer. This prepares for the entropy coding that follows.

[0031] decoder (i.e.) Figure 2 The Decoder (on the left) receives the quantized latent representation from the encoder. First, a first deconvolutional layer (Deconv) is used to recover spatial features. Then, these features are processed by N second DatRNN modules to reconstruct spatiotemporal features and capture dependencies between previous frames. Finally, after processing by the second deconvolutional layer (Deconv), a motion information representation is reconstructed. .

[0032] Step 103: Determine the prediction frame based on the compressed frame and motion information representation of the previous frame.

[0033] Continuing with the above embodiment, the compressed frame of the previous frame... The motion information obtained in step 102 above represents The input is processed by the motion compensation module, which compensates for changes caused by motion to obtain the predicted frame. Its expression is as follows: ; Motion compensation uses motion vectors obtained during the motion estimation stage to adjust a reference frame (i.e., the reconstructed frame adjacent to the previous frame) to better match the content of the current frame. This method corrects image changes caused by object movement or camera movement, thereby improving compression efficiency and reconstruction quality.

[0034] Step 104: Send the predicted frame and the residual between the original frame and the predicted frame to the decoding end.

[0035] Continuing with the above embodiment, the predicted frame determined in step 103 is sent. and the original frame With the predicted frame The residual between them is sent to the decoding end.

[0036] In this way, spatiotemporal correlation analysis based on motion vectors between adjacent frames, combined with a recurrent neural network based on deformable attention mechanism, can capture inter-frame dependencies in time series, better adapt to changes in different time and spatial scales, and improve video compression efficiency.

[0037] In some possible implementations, the first DatRNN module described above includes a deformable attention unit, an input gate, a forget gate, and an input modulation gate. The second DatRNN module described above has the same structure as the first DatRNN module, specifically including: Based on the spatiotemporal features and the first spatiotemporal memory information from the previous moment, a first feature map is determined. The first feature map is processed by a deformable attention unit to capture global spatial dependencies, resulting in a first attention feature. Based on the weights of the first attention feature and the first spatiotemporal memory information, the first parameters of the input gate, forget gate, and input modulation gate are determined. The input gate controls the writing of the first attention feature according to the corresponding first parameters, the forget gate controls the forgetting of the first attention feature according to the corresponding first parameters, and the input modulation gate modulates the intensity of the first attention feature according to the corresponding first parameters, thus obtaining the second spatiotemporal memory information for the current moment. Based on the second spatiotemporal memory information, the spatiotemporal features, and the first hidden state from the previous moment, the output feature is determined.

[0038] In one exemplary embodiment, such as Figure 3 As shown, based on spatiotemporal characteristics First spatiotemporal memory information from the previous moment The first feature map is determined; the first feature map is processed by a deformable attention unit to capture global spatial dependencies, resulting in the first attention feature. Based on the characteristics of first attention Weight of first-time memory information Determine the input gate Forgotten Gate and input modulation gate The first parameter. Specifically: Input gate Process the current input using convolution operations. And the first spatiotemporal memory information from the spatiotemporal memory unit transmitted from the previous layer. It determines how much spatiotemporal memory information from the previous layer is written into the current layer's spatiotemporal memory unit; the input gate... .

[0039] Forgotten Gate Process the current input using convolution operations. And the first spatiotemporal memory information from the spatiotemporal memory unit transmitted from the previous layer. It determines which information in the spatiotemporal memory unit needs to be forgotten; the forgetting gate. .

[0040] Input modulation gate Process the current input using convolution operations. And the first spatiotemporal memory information from the spatiotemporal memory unit transmitted from the previous layer. It modulates the intensity of the spatiotemporal information transmitted from the previous layer, input to the modulation gate. .

[0041] The second spatiotemporal memory information at the current moment is obtained by controlling the writing of the first attention feature according to the corresponding first parameter through the input gate, controlling the forgetting of the first attention feature according to the corresponding first parameter through the forget gate, and modulating the intensity of the first attention feature according to the corresponding first parameter through the input modulation gate. Specifically, the spatiotemporal memory unit updates the second spatiotemporal memory information at the current moment. It is the sum of the results of the forget gate and the input gate operations. First, the information retention portion controlled by the forget gate... New information writing section with input gate control Adding them together yields the updated state of the spacetime memory unit, which is then updated to... .

[0042] The output gate controls the contribution of the memory cell content to the hidden state. It does this by combining the current input (i.e., the spatiotemporal features) and the hidden state from the previous time step. and the state of the spatiotemporal memory unit The calculation is performed, and the output gate is... .

[0043] After determining the output features based on the second spatiotemporal memory information, spatiotemporal features, and the first hidden state of the previous moment, the process also includes: Based on the spatiotemporal features and the first hidden state, a second feature map is determined; the second feature map is processed by the deformable attention unit to capture global spatial dependencies, resulting in a second attention feature; based on the weights of the second attention feature and the first hidden state, second parameters of the input gate, forget gate, and input modulation gate are determined; the writing of the attention feature is controlled by the input gate according to the corresponding second parameters, the forgetting of the attention feature is controlled by the forget gate according to the corresponding second parameters, and the intensity of the attention feature is modulated by the input modulation gate according to the corresponding second parameters to obtain the memory information at the current moment; based on the memory information, spatiotemporal features, and spatiotemporal memory information, the second hidden state at the current moment is generated.

[0044] Continuing with the above embodiments, as follows: Figure 3 As shown, based on spatiotemporal characteristics and the first hidden state of the previous moment The second feature map is determined; the second feature map is then processed by a deformable attention unit to capture global spatial dependencies, resulting in the second attention feature. According to the characteristics of second attention Weights of the first hidden state Determine the input gate Forgotten Gate and input modulation gate The second parameter. Specifically: Input gate Process the current input using convolution operations. and the first hidden state of the previous moment It determines how much of the current input information is written into the memory unit; the input gate... .

[0045] Forgotten Gate Process the current input using convolution operations. and the first hidden state of the previous moment Which information in a memory cell needs to be forgotten? Forgetting gates. .

[0046] Input modulation gate Process the current input using convolution operations. and the first hidden state of the previous moment It modulates the intensity of the input information to make it suitable for storage in memory. (Input modulation gate) .

[0047] The memory information at the current moment is obtained by controlling the writing of the attention feature through the input gate according to the corresponding second parameter, controlling the forgetting of the attention feature through the forgetting gate according to the corresponding second parameter, and modulating the intensity of the attention feature through the input modulation gate according to the corresponding second parameter. Specifically, the memory information at the current moment in the memory unit... Updates are performed using a forget gate and an input gate. The forget gate controls which information is forgotten. What information is written to the input gate control? This mechanism enables memory cells to capture long-term dynamics. Memory cells are updated to... .

[0048] The second hidden state at the current time step is the output of the DatRNN module. It carries information from the sequence up to the current time step and is either passed to the next time step or used to generate the final output. Based on the memory information... Spatiotemporal memory information and output features Generate the second hidden state at the current moment. Its expression is .

[0049] In this way, the adaptive nature of the DatRNN module allows the model to dynamically adjust the feature extraction process based on the input data, thus better adapting to changes in different temporal and spatial scales. Introducing the DatRNN module allows for more flexible adaptation to these changes, resulting in excellent performance across different types of video content. For example, the DatRNN module helps the model capture motion information more accurately when processing fast-moving scenes, thereby improving compression efficiency and reconstruction quality.

[0050] In some possible implementations, the deformable attention unit (Transformer) computes attention weights based on the current input and historical information. These weights reflect the importance of features from different spatiotemporal locations to the current prediction. Input After passing through the linear projection layer module, and comparing with the first hidden state of the previous time step... Spacetime memory information transmitted from the upper level This combination provides fundamental information for the attention mechanism. By introducing the deformable Transformer, PredRNN's information flow becomes more flexible and dynamic, adaptively adjusting attention based on the characteristics of the input data to focus on the spatiotemporal features most helpful for prediction. This mechanism significantly improves the model's ability to capture complex spatiotemporal dynamics, thereby enhancing prediction accuracy and the model's generalization ability.

[0051] like Figure 4 As shown, one or more reference points are selected on the feature maps (i.e., the first and / or second feature maps mentioned above). A small neural network (OffsetNet) is used to predict the offset relative to the reference points. This small neural network calculates the offset based on the input features and the reference points. The offset is added to the reference points to obtain def points, which are the feature locations that the model will focus on. Bilinear interpolation is used on the def points to extract features. This process considers the feature values ​​around the def points and performs a weighted average to obtain the feature representation at the def points. A multi-head attention mechanism is applied to the def points. This step involves calculating attention weights, which indicate the importance of different def point features to the current task. The features at the def points are weighted using the calculated attention weights to highlight important features and suppress unimportant features. The attention-weighted features are fused with the original features or features from other sources to form a richer feature representation, resulting in attention features.

[0052] In this way, by introducing the deformable Transformer module, important information in the video can be identified and processed more intelligently, thereby improving the compression ratio while maintaining video quality.

[0053] The DatRNN module described above can more effectively capture complex spatial relationships, especially when processing video frames with significant spatial variations. Therefore, introducing the DatRNN module can significantly improve the model's ability to capture spatial features, thereby improving compression efficiency and reconstruction quality. This DatRNN module not only captures spatial features but also dynamically adjusts feature weights through an attention mechanism, thus better handling changes in time series. Introducing the DatRNN module further enhances the modeling ability for temporal dependencies, dynamically focusing on key regions in video frames, thereby more accurately capturing changes between frames and further reducing the bitrate.

[0054] Furthermore, by more effectively capturing spatial and temporal features, the DatRNN module can reduce redundant information, thereby lowering the bit rate required for encoding. Introducing the DatRNN module allows for further reduction in bit rate through more accurate feature extraction and modeling, while maintaining or improving reconstruction quality. For example, the DatRNN module can help the model more accurately estimate the conditional probability quality function of the latent representation, thereby reducing conditional entropy and further decreasing the bit rate.

[0055] Please see Figure 5 , Figure 5 The following are schematic flowcharts illustrating video compression methods provided in other embodiments of this application, such as... Figure 5 As shown, this video compression method is applied at the decoding end and specifically includes the following steps: Step 501: Obtain the predicted frame of the current frame sent by the encoding end, and the residual between the original frame and the predicted frame of the current frame.

[0056] In one exemplary embodiment, such as Figure 2 As shown, the predicted frame of the current frame sent by the encoding end is obtained. and the original frame of the current frame. With the predicted frame The residuals between them.

[0057] Step 502: Obtain residual estimates by inputting the residuals into the second recursive autoencoder.

[0058] The second recursive autoencoder described above has the same structure as the first recursive autoencoder described above.

[0059] Continuing with the above embodiment, the residual is input to a second recursive autoencoder, which has the same structure as the first recursive autoencoder, to obtain the residual estimate. Specifically, the encoder (i.e. Figure 2The input to the Encoder (on the right) comes from the residual between the original video frame and the predicted frame obtained after motion compensation processing. Its internal processing flow is similar to... Figure 2 The Encoder on the left is the same.

[0060] decoder (i.e.) Figure 2 The Decoder on the right also receives the quantized latent representation from the encoder, and its processing is similar to... Figure 2 The decoder on the left is similar, but its output is different. The decoder on the right outputs the reconstructed residual estimate. The signal is similar to the original output.

[0061] Step 503: Perform fusion processing on the predicted frame and the residual estimate to obtain the compressed frame of the current frame.

[0062] Continuing with the above embodiments, the predicted frame after motion compensation... Residual estimates from reconstruction The frames are then fused to obtain the final reconstructed frames. (That is, the compressed frame of the current frame). This final reconstructed frame is based on the original frame. The high-quality reconstruction is expressed as follows: .

[0063] This application provides a video compression method applied at the decoding end. It obtains the predicted frame of the current frame sent by the encoding end, and the residual between the original frame and the predicted frame. The residual is input into a second recursive autoencoder to obtain a residual estimate. The second recursive autoencoder has the same structure as the first recursive autoencoder described above. The predicted frame and the residual estimate are fused to obtain the compressed frame of the current frame. By introducing a DatRNN module into the second recursive autoencoder, which adaptively focuses on key regions, video frames can be reconstructed more accurately, thereby improving reconstruction quality. For example, the second DatRNN module can help the model generate latent representations more accurately, thus reconstructing video frames more accurately during the decoding process.

[0064] Please refer to Figure 6. Figure 6 This illustration shows one of the structural schematic diagrams of a video compression apparatus provided in some embodiments of this application. This video compression apparatus can achieve, for example... Figure 1 The video compression device 600, which is used at the encoding end, includes all or part of the content shown in the embodiment. The first determining module 610 is used to determine the motion vector based on the original frame of the current frame and the compressed frame of the previous frame adjacent to the current frame. The encoding module 620 is used to obtain motion information representation by inputting the motion vector into a first recurrent autoencoder; wherein, the first recurrent autoencoder includes an encoding layer, a quantization layer and a decoding layer connected in sequence, the encoding layer includes a first convolutional layer, a first DatRNN module and a second convolutional layer, the decoding layer includes a first deconvolutional layer, a second DatRNN module and a second deconvolutional layer; the first DatRNN module and the second DatRNN module are both recurrent neural networks based on a deformable attention mechanism, used to capture inter-frame dependencies in the time series; The second determining module 630 is used to determine the predicted frame based on the compressed frame of the previous frame and the motion information representation; The sending module 640 is used to send the predicted frame and the residual between the original frame and the predicted frame to the decoding end.

[0065] In some possible implementations, the encoding module 620, when used to obtain a motion information representation by inputting the motion vector to a first recursive autoencoder, is specifically used for: The motion vector is input into the coding layer, and spatial features are extracted through the first convolutional layer. After the spatial features are captured by the first DatRNN module to capture inter-frame dependencies, deep spatial features are extracted through the second convolutional layer. The deep spatial features are processed by the quantization layer to obtain a quantized latent representation; The quantized latent representation recovers spatial features through the first deconvolution layer. After the spatial features are captured by the second DatRNN module to capture inter-frame dependencies, they are reconstructed through the second deconvolution layer to obtain motion information representation.

[0066] In some possible implementations, the first DatRNN module includes a deformable attention unit, an input gate, a forget gate, and an input modulation gate; the second DatRNN module has the same structure as the first DatRNN module; and the encoding module 620 is used for: Based on the aforementioned spatiotemporal characteristics and the first spatiotemporal memory information from the previous moment, the first feature map is determined; The first feature map is processed by the deformable attention unit to capture global spatial dependencies, resulting in a first attention feature. Based on the weights of the first attention feature and the first spatiotemporal memory information, the first parameters of the input gate, forget gate, and input modulation gate are determined; The input gate controls the writing of the first attention feature according to the corresponding first parameter, the forget gate controls the forgetting of the first attention feature according to the corresponding first parameter, and the input modulation gate modulates the intensity of the first attention feature according to the corresponding first parameter to obtain the second spatiotemporal memory information at the current moment. The output features are determined based on the second spatiotemporal memory information, the spatiotemporal features, and the first hidden state of the previous moment.

[0067] In some possible implementations, the encoding module 620 is also used for: Based on the spatiotemporal features and the first hidden state, a second feature map is determined; The second feature map is processed by the deformable attention unit to capture global spatial dependencies, resulting in a second attention feature. Based on the second attention feature and the weights of the first hidden state, determine the second parameters of the input gate, forget gate, and input modulation gate; The input gate controls the writing of the attention feature according to the corresponding second parameter, the forget gate controls the forgetting of the attention feature according to the corresponding second parameter, and the input modulation gate modulates the intensity of the attention feature according to the corresponding second parameter to obtain the memory information at the current moment. Based on the memory information, the spatiotemporal memory information, and the output features, a second hidden state is generated for the current moment.

[0068] This application provides a video compression device applied at the encoding end, including a first determining module, an encoding module, a second determining module, and a sending module. The first determining module determines motion vectors based on the original frame of the current frame and the compressed frame of the previous frame adjacent to the current frame. The encoding module obtains motion information representation by inputting the motion vectors to a first recurrent autoencoder. The first recurrent autoencoder includes an encoding layer, a quantization layer, and a decoding layer connected in sequence. The encoding layer includes a first convolutional layer, a first DatRNN module, and a second convolutional layer. The decoding layer includes a first deconvolutional layer, a second DatRNN module, and a second deconvolutional layer. The second determining module determines a predicted frame based on the compressed frame of the previous frame and the motion information representation. The sending module sends the predicted frame and the residual between the original frame and the predicted frame to the decoding end. In this way, spatiotemporal correlation analysis based on motion vectors between adjacent frames, combined with a recurrent neural network based on a deformable attention mechanism, captures inter-frame dependencies in the time series, better adapting to changes in different time and spatial scales and improving video compression efficiency.

[0069] Please refer to Figure 7. Figure 7This is shown as a second schematic diagram of the structure of a video compression apparatus provided in some embodiments of this application. This video compression apparatus can achieve, for example... Figure 5 The video compression device 700, which is used at both the encoding and decoding ends, includes all or part of the content shown in the embodiment. The first acquisition module 710 is used to acquire the predicted frame of the current frame sent by the encoding end, and the residual between the original frame of the current frame and the predicted frame. The second acquisition module 720 is used to acquire a residual estimate by inputting the residual to a second recursive autoencoder; wherein the second recursive autoencoder has the same structure as the first recursive autoencoder described above. The fusion processing module 730 is used to fuse the predicted frame and the residual estimate to obtain the compressed frame of the current frame.

[0070] This application provides a video compression device applied at a decoding end, including a first acquisition module, a second acquisition module, and a fusion processing module. The first acquisition module acquires the predicted frame of the current frame sent by the encoding end, and the residual between the original frame and the predicted frame of the current frame. The second acquisition module obtains a residual estimate by inputting the residual into a second recursive autoencoder. The second recursive autoencoder has the same structure as the first recursive autoencoder. The fusion processing module fuses the predicted frame and the residual estimate to obtain the compressed frame of the current frame. Thus, by introducing a DatRNN module into the second recursive autoencoder to adaptively focus on key regions, video frames can be reconstructed more accurately, thereby improving reconstruction quality.

[0071] Figure 8 The diagram illustrates the structure of an electronic device according to some embodiments of this application. Referring to the diagram, at the hardware level, the electronic device 800 includes a processor 810, and optionally includes an internal bus 820, a network interface 830, and a memory. The memory may include main memory 841, such as high-speed random-access memory (RAM), and may also include non-volatile memory 842, such as at least one disk storage device. Of course, the electronic device may also include other hardware required for other services.

[0072] The processor 810, network interface 830, and memory can be interconnected via an internal bus 820. This internal bus 820 can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus, etc. The bus can be categorized as an address bus, data bus, control bus, etc. For ease of illustration, only a single bidirectional arrow is used in this diagram, but this does not imply that there is only one bus or one type of bus.

[0073] The memory stores programs. Specifically, the program may include program code, which includes computer operation instructions. The memory may include main memory 841 and non-volatile memory 842, and provides instructions and data to the processor 810.

[0074] The processor 810 reads the corresponding computer program from the non-volatile memory 842 into memory and then runs it, forming a device for locating the target user at the logical level. The processor 810 executes the program stored in memory and specifically performs the following: Figure 1 or Figure 5 The methods disclosed in the embodiments shown achieve the functions and beneficial effects of the methods described in the preceding method embodiments, and will not be repeated here.

[0075] The above is as stated in this application. Figure 1 or Figure 5The methods disclosed in the illustrated embodiments can be applied to or implemented by processor 810. Processor 810 may be an integrated circuit chip with signal processing capabilities. During implementation, each step of the above methods can be completed by integrated logic circuits in the hardware of processor 810 or by instructions in software form. The processor 810 can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. It can implement or execute the methods, steps, and logic block diagrams disclosed in the embodiments of this application. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the methods disclosed in the embodiments of this application can be directly embodied in the execution of a hardware decoding processor, or executed by a combination of hardware and software modules in the decoding processor. The software module can reside in a mature storage medium in the field, such as random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, or registers. This storage medium is located in memory, and the processor reads information from the memory and, in conjunction with its hardware, completes the steps of the above method.

[0076] The computer device can also execute the methods described in the preceding method embodiments and achieve the functions and beneficial effects of the methods described in the preceding method embodiments, which will not be repeated here.

[0077] Of course, in addition to software implementation, the electronic device of this application does not exclude other implementation methods, such as logic devices or a combination of hardware and software, etc. In other words, the execution subject of the following processing flow is not limited to each logic unit, but can also be hardware or logic devices.

[0078] This application also proposes a computer-readable storage medium that stores one or more programs, which, when executed by an electronic device including multiple applications, cause the electronic device to perform... Figure 1 or Figure 5 The methods disclosed in the embodiments shown achieve the functions and beneficial effects of the methods described in the preceding method embodiments, and will not be repeated here.

[0079] The computer-readable storage medium mentioned above includes read-only memory (ROM), random access memory (RAM), magnetic disk or optical disk, etc.

[0080] Furthermore, embodiments of this application also provide a computer program product, the computer program product including a computer program stored on a non-transitory computer-readable storage medium, the computer program including program instructions, which, when executed by a computer, implement the following process: Figure 1 or Figure 5 The methods disclosed in the embodiments shown achieve the functions and beneficial effects of the methods described in the preceding method embodiments, and will not be repeated here.

[0081] The embodiments of this application can be applied to various scenarios of electronic device collaboration or interconnection, including: collaboration and interconnection between mobile phones and laptops / tablets; collaboration and interconnection between mobile terminals and smart TVs / monitors; collaboration and interconnection between mobile phones or tablets and in-vehicle entertainment systems; collaboration and interconnection between mobile terminals and smart conferencing systems, etc. This satisfies users' diverse needs in smart home, smart office, and smart travel scenarios.

[0082] In summary, the above description is merely a preferred embodiment of this application and does not limit the scope of protection of this application. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of protection of this application.

[0083] The systems, devices, modules, or units described in the above embodiments can be implemented by computer chips or entities, or by products with certain functions. A typical implementation device is a computer. Specifically, a computer can be, for example, a personal computer, laptop computer, cellular phone, camera phone, smartphone, personal digital assistant, media player, navigation device, email device, game console, tablet computer, wearable device, or any combination of these devices.

[0084] Computer-readable media includes both permanent and non-permanent, removable and non-removable media that can store information by any method or technology. Information can be computer-readable instructions, data structures, modules of programs, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other non-transferable medium that can store information accessible by a computing device. As defined herein, computer-readable media does not include transient computer-readable media, such as modulated data signals and carrier waves.

[0085] It should also be noted that the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0086] The various embodiments in this specification are described in a progressive manner. Similar or identical parts between embodiments can be referred to interchangeably. Each embodiment focuses on describing the differences from other embodiments. In particular, the system embodiments are basically similar to the method embodiments, so the description is relatively simple; relevant parts can be referred to the descriptions in the method embodiments.

Claims

1. A video compression method, characterized in that, Applied to the encoding end, including: The motion vector is determined based on the original frame of the current frame and the compressed frame of the previous frame adjacent to the current frame; Motion information representation is obtained by inputting the motion vector into a first recurrent autoencoder; wherein, the first recurrent autoencoder includes an encoding layer, a quantization layer and a decoding layer connected in sequence, the encoding layer includes a first convolutional layer, a first DatRNN module and a second convolutional layer, the decoding layer includes a first deconvolutional layer, a second DatRNN module and a second deconvolutional layer; the first DatRNN module and the second DatRNN module are both recurrent neural networks based on deformable attention mechanisms, used to capture inter-frame dependencies in time series; The predicted frame is determined based on the compressed frame of the previous frame and the motion information representation; The predicted frame and the residual between the original frame and the predicted frame are sent to the decoding end.

2. The method according to claim 1, characterized in that, The step of obtaining motion information representation by inputting the motion vector into a first recursive autoencoder includes: The motion vector is input into the coding layer, and spatial features are extracted through the first convolutional layer. After the spatial features are captured by the first DatRNN module to capture inter-frame dependencies, deep spatial features are extracted through the second convolutional layer. The deep spatial features are processed by the quantization layer to obtain a quantized latent representation; The quantized latent representation recovers spatial features through the first deconvolution layer. After the spatial features are captured by the second DatRNN module to capture inter-frame dependencies, they are reconstructed through the second deconvolution layer to obtain motion information representation.

3. The method according to claim 2, characterized in that, The first DatRNN module includes a deformable attention unit, an input gate, a forget gate, and an input modulation gate. The second DatRNN module has the same structure as the first DatRNN module, specifically including: Based on the aforementioned spatiotemporal characteristics and the first spatiotemporal memory information from the previous moment, the first feature map is determined; The first feature map is processed by the deformable attention unit to capture global spatial dependencies, resulting in a first attention feature. Based on the weights of the first attention feature and the first spatiotemporal memory information, the first parameters of the input gate, forget gate, and input modulation gate are determined; The input gate controls the writing of the first attention feature according to the corresponding first parameter, the forget gate controls the forgetting of the first attention feature according to the corresponding first parameter, and the input modulation gate modulates the intensity of the first attention feature according to the corresponding first parameter to obtain the second spatiotemporal memory information at the current moment. The output features are determined based on the second spatiotemporal memory information, the spatiotemporal features, and the first hidden state of the previous moment.

4. The method according to claim 3, characterized in that, After determining the output features based on the second spatiotemporal memory information, the spatiotemporal features, and the first hidden state at the previous moment, the method further includes: Based on the spatiotemporal features and the first hidden state, a second feature map is determined; The second feature map is processed by the deformable attention unit to capture global spatial dependencies, resulting in a second attention feature. Based on the second attention feature and the weights of the first hidden state, determine the second parameters of the input gate, forget gate, and input modulation gate; The input gate controls the writing of the attention feature according to the corresponding second parameter, the forget gate controls the forgetting of the attention feature according to the corresponding second parameter, and the input modulation gate modulates the intensity of the attention feature according to the corresponding second parameter to obtain the memory information at the current moment. Based on the memory information, the spatiotemporal memory information, and the output features, a second hidden state is generated for the current moment.

5. A video compression method, characterized in that, Applied to the decoding end, including: Obtain the predicted frame of the current frame sent by the encoding end, and the residual between the original frame of the current frame and the predicted frame; The residual is input to a second recursive autoencoder to obtain a residual estimate; wherein the second recursive autoencoder has the same structure as the first recursive autoencoder described in claim 1. The predicted frame and the residual estimate are fused to obtain the compressed frame of the current frame.

6. A video compression device, characterized in that, Applied to the encoding end, including: The first determining module is used to determine the motion vector based on the original frame of the current frame and the compressed frame of the previous frame adjacent to the current frame; An encoding module is used to obtain motion information representation by inputting the motion vector into a first recurrent autoencoder; wherein, the first recurrent autoencoder includes an encoding layer, a quantization layer and a decoding layer connected in sequence, the encoding layer includes a first convolutional layer, a first DatRNN module and a second convolutional layer, the decoding layer includes a first deconvolutional layer, a second DatRNN module and a second deconvolutional layer; the first DatRNN module and the second DatRNN module are both recurrent neural networks based on a deformable attention mechanism, used to capture inter-frame dependencies in the time series; The second determining module is used to determine the predicted frame based on the compressed frame of the previous frame and the motion information representation; The sending module is used to send the predicted frame and the residual between the original frame and the predicted frame to the decoding end.

7. A video compression device, characterized in that, Applied to the decoding end, including: The first acquisition module is used to acquire the predicted frame of the current frame sent by the encoding end, and the residual between the original frame of the current frame and the predicted frame; The second acquisition module is used to acquire a residual estimate by inputting the residual into a second recursive autoencoder; wherein the second recursive autoencoder has the same structure as the first recursive autoencoder described in claim 6; The fusion processing module is used to fuse the predicted frame and the residual estimate to obtain the compressed frame of the current frame.

8. An electronic device, characterized in that, The electronic device includes a processor and a memory, the memory storing programs or instructions that can run on the processor, the programs or instructions being executed by the processor to implement the steps of the method as described in any one of claims 1 to 5.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a program or instructions that, when executed by a processor, implement the steps of the method as described in any one of claims 1 to 5.

10. A computer program product, characterized in that, The computer program product includes a computer program stored on a non-transitory computer-readable storage medium, the computer program including program instructions that, when executed by a computer, cause the computer to perform the steps of the method as described in any one of claims 1 to 5.

Images