A transformer-based video decoding filtering method, system, terminal and storage medium

Abstract

This invention discloses a transformer-based video decoding and filtering method, system, terminal, and storage medium. The method includes: acquiring a reference frame and a current frame; passing the reference frame and the current frame through a motion estimation network to obtain motion vectors; performing motion encoding and decoding on the motion vectors to obtain decoded motion vectors; performing motion compensation on the decoded motion vectors to obtain multi-scale context information; inputting the current frame and the multi-scale context information into a residual coding network to obtain a feature representation of the current frame; inputting the feature representation and reference features of the reference frame into a preprocessing module of a temporal context filter for preprocessing to obtain key vectors, value vectors, and query vectors constituting an attention mechanism; inputting the key vectors, value vectors, and query vectors into the filtering main module of the temporal context filter for filtering optimization; outputting a final reconstructed frame; and participating the final reconstructed frame in subsequent video decoding and filtering processes. This invention significantly improves the quality of reconstructed video.

Description

Technical Field

[0001] This invention relates to the field of image processing technology, and in particular to a video decoding and filtering method, system and terminal based on transformer. Background Technology

[0002] With the widespread adoption of ultra-high-definition video (4K / 8K), virtual reality, video conferencing, and mobile streaming media, the amount of video data has exploded. To efficiently store and transmit this data, video codec standards (such as H.264 / AVC, H.265 / HEVC, and H.266 / VVC) use complex prediction, transformation, and quantization techniques to significantly compress video data while maintaining a certain level of visual quality.

[0003] However, high compression rates inevitably lead to information loss, resulting in various visual distortions in the decoded and reconstructed video frames, primarily including: blockiness, overall blurring and texture loss, and ringing and noise. These distortions severely impact the visual experience of end users. Therefore, how to effectively post-process and filter the reconstructed video at the decoding end to suppress distortion and restore details has become a key technology for improving video service quality and user experience.

[0004] To address the aforementioned issues, various attention-based inter-frame filtering techniques have been proposed, such as traditional in-loop filtering techniques and context modeling schemes based on global attention mechanisms. However, these schemes suffer from a sharp increase in computational complexity when processing high-resolution video frames, and they do not consider noise and other issues present in the reconstruction results, thus failing to effectively suppress the distortion of the reconstructed video.

[0005] Therefore, existing technologies still need to be improved and developed. Summary of the Invention

[0006] The main objective of this invention is to provide a video decoding and filtering method, system, terminal, and computer-readable storage medium based on transformer, aiming to solve the problems of limited recovery capability, excessive reliance on unreliable motion estimation, or high computational complexity in existing post-processing techniques at the decoding end.

[0007] To achieve the above objectives, the present invention provides a transformer-based video decoding and filtering method, which includes the following steps: A reference frame and a current frame are acquired. The reference frame and the current frame are passed through a motion estimation network to obtain motion vectors. The motion vectors are then subjected to motion encoding and motion decoding to obtain decoded motion vectors. Motion compensation is performed on the decoded motion vector to obtain multi-scale context information. The current frame and the multi-scale context information are then input into a conditional residual coding network for transformation, quantization, and fusion processing to obtain the feature representation of the current frame. The feature representation and the reference features of the reference frame are input into the preprocessing module of the temporal context filter for preprocessing to obtain the key vector, value vector and query vector that make up the attention mechanism; The key vector, the value vector, and the query vector are input into the filtering main module of the temporal context filter for filtering optimization, and the final reconstructed frame is output. The final reconstructed frame is then used as the reference frame for the next frame in the subsequent video decoding filtering process.

[0008] Optionally, the transformer-based video decoding and filtering method, wherein obtaining a reference frame and a current frame, passing the reference frame and the current frame through a motion estimation network to obtain motion vectors describing pixel block displacements, and performing motion encoding and decoding on the motion vectors to obtain decoded motion vectors, specifically includes: Receive a reference frame that has been decoded and reconstructed and a current frame to be encoded. Input the reference frame and the current frame to be encoded into a motion estimation network for processing and output a motion vector to describe the displacement of pixel blocks. The motion vector is input to the motion encoding module for entropy encoding compression, outputting a motion information bitstream. The motion information bitstream is then input to the motion decoding module for decoding, outputting a decoded motion vector.

[0009] Optionally, the transformer-based video decoding and filtering method, wherein performing motion compensation on the decoded motion vector to obtain multi-scale context information specifically includes: Based on the reference features of the reference frame, a multi-scale reference feature set is constructed, which includes: a first-scale reference feature, a second-scale reference feature, and a third-scale reference feature. Based on the decoded motion vector, motion compensation is performed on the first scale reference feature, the second scale reference feature, and the third scale reference feature respectively to obtain the first scale context information, the second scale context information, and the third scale context information.

[0010] Optionally, in the transformer-based video decoding and filtering method, the conditional residual coding network includes: an entropy model, a residual coding module, and a residual decoding module; The process of performing motion compensation on the decoded motion vector to obtain multi-scale context information, and then inputting the current frame and the multi-scale context information into a conditional residual coding network for transformation, quantization, and fusion processing to obtain the feature representation of the current frame, specifically includes: The third-scale context information is input into the entropy model for processing, and statistical parameters are output. Based on the feature space, the current frame is combined with the first scale context information and the second scale context information to perform temporal redundancy elimination to obtain the residual signal; The residual signal is input into the residual coding module for transformation and quantization processing, and auxiliary adaptive entropy coding is performed according to the statistical parameters to generate a residual information code stream. The residual information bitstream is input into the residual decoding module, and the residual information bitstream is decoded according to the statistical parameters and the first scale context information to obtain the residual information; The residual information is fused with the second scale context information to obtain the feature representation of the current frame.

[0011] Optionally, in the transformer-based video decoding and filtering method, the preprocessing module includes a block partitioning unit and a reference block selection unit, and the filtering main module includes a location embedding unit, a spatial attention modeling unit, and a quality-aware fusion unit.

[0012] Optionally, the transformer-based video decoding and filtering method, wherein the step of inputting the feature representation and the reference features of the reference frame into the preprocessing module of the temporal context filter for preprocessing to obtain the key vector, value vector, and query vector constituting the attention mechanism, specifically includes: The feature representation and the reference feature of the reference frame are input into the block partitioning unit for partitioning, and multiple feature representation blocks and reference feature blocks of uniform size are output. The feature representation block and the reference feature block are input into the reference block selection unit. Based on the spatial correspondence of the feature representation block, a preset number of reference feature blocks are selected to form the key vector and value vector of the attention mechanism, and the feature representation block is used as the query vector of the attention mechanism.

[0013] Optionally, the transformer-based video decoding and filtering method, wherein the step of inputting the key vector, the value vector, and the query vector into the filtering main module of the temporal context filter for filtering optimization and outputting the final reconstructed frame specifically includes: The key vector and the value vector are input into the embedding unit for learnable position encoding while keeping the feature dimension unchanged, to obtain the target key vector and the target value vector. The query vector is input into the embedding unit for learnable position encoding and then aligned in dimensions according to the preset number to obtain the target query vector. The target key vector and the target query vector are input into the spatial attention modeling unit for calculation to obtain the similarity between the target key vector and the target query vector. An attention weight matrix is ​​obtained based on the similarity. The attention weight matrix is ​​then weighted and summed with the target value vector to output the intermediate features that fuse the reference block context information. The intermediate features and the target query vector are input into the quality-aware fusion unit for feature evaluation calculation to obtain a quality confidence score that characterizes the reconstruction quality of the reference block. The fusion weights are dynamically adjusted based on the quality confidence score to obtain the target fusion weights. Based on the target fusion weights, the intermediate features and the query vector are residually connected to output the final reconstructed frame.

[0014] Furthermore, to achieve the above objectives, the present invention also provides a transformer-based video decoding and filtering system, wherein the transformer-based video decoding and filtering system comprises: The motion estimation module is used to acquire a reference frame and a current frame, pass the reference frame and the current frame through a motion estimation network to obtain motion vectors, and perform motion encoding and motion decoding on the motion vectors to obtain decoded motion vectors. The motion compensation and residual coding module is used to perform motion compensation on the decoded motion vector to obtain multi-scale context information, and input the current frame and the multi-scale context information into a condition-based residual coding network for transformation, quantization and fusion processing to obtain the feature representation of the current frame. The preprocessing module is used to input the feature representation and the reference features of the reference frame into the preprocessing module of the temporal context filter for preprocessing to obtain the key vector, value vector and query vector that make up the attention mechanism; The filtering optimization module is used to input the key vector, the value vector and the query vector into the filtering main module of the temporal context filter for filtering optimization, output the final reconstructed frame, and use the final reconstructed frame as the reference frame for the next frame in the subsequent video decoding filtering process.

[0015] Furthermore, to achieve the above objectives, the present invention also provides a terminal, wherein the terminal includes: a memory, a processor, and a transformer-based video decoding and filtering program stored in the memory and executable on the processor, wherein when the transformer-based video decoding and filtering program is executed by the processor, it implements the steps of the transformer-based video decoding and filtering method as described above.

[0016] Furthermore, to achieve the above objectives, the present invention also provides a computer-readable storage medium, wherein the computer-readable storage medium stores a transformer-based video decoding and filtering program, which, when executed by a processor, implements the steps of the transformer-based video decoding and filtering method as described above.

[0017] In this invention, a reference frame and a current frame are acquired. These are then passed through a motion estimation network to obtain motion vectors. The motion vectors are then subjected to motion encoding and decoding to obtain decoded motion vectors. Motion compensation is performed on the decoded motion vectors to obtain multi-scale context information. The current frame and the multi-scale context information are input into a conditional residual coding network for transformation, quantization, and fusion processing to obtain a feature representation of the current frame. The feature representation and the reference features of the reference frame are input into a preprocessing module of a temporal context filter for preprocessing to obtain key vectors, value vectors, and query vectors that constitute the attention mechanism. The key vectors, value vectors, and query vectors are then input into the filtering main module of the temporal context filter for filtering optimization, outputting a final reconstructed frame. This final reconstructed frame is used as a reference frame for the next frame in subsequent video decoding and filtering processes. This invention significantly improves the subjective and objective quality of reconstructed video, effectively suppressing compression distortions such as blockiness and blurring while restoring details, and exhibits higher robustness and efficiency. Attached Figure Description

[0018] Figure 1 This is a flowchart of a preferred embodiment of the video decoding and filtering method based on transformer of the present invention; Figure 2 This is a schematic diagram of the principle architecture of the video decoding and filtering method based on transformer of the present invention; Figure 3 This is a temporal context filter architecture diagram of the video decoding and filtering method based on transformer of the present invention; Figure 4 This is a structural diagram of a preferred embodiment of the video decoding and filtering system based on transformer of the present invention; Figure 5This is a structural diagram of a preferred embodiment of the terminal of the present invention. Detailed Implementation

[0019] This application provides a video decoding and filtering method, system, and terminal based on transformers. To make the objectives, technical solutions, and effects of this application clearer and more explicit, the following detailed description is provided with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only for explaining this application and are not intended to limit this application.

[0020] It will be understood by those skilled in the art that, unless otherwise defined, all terms used herein (including technical and scientific terms) have the same meaning as commonly understood by one of ordinary skill in the art to which this application pertains. It should also be understood that terms such as those defined in general dictionaries should be understood to have the same meaning as in the context of the prior art, and should not be interpreted in an idealized or overly formal sense unless specifically defined as herein.

[0021] Furthermore, if the embodiments of this invention involve descriptions such as "first" or "second," these descriptions are for descriptive purposes only and should not be construed as indicating or implying their relative importance or implicitly specifying the number of technical features indicated. Therefore, a feature defined with "first" or "second" may explicitly or implicitly include at least one of those features. Additionally, the technical solutions of the various embodiments can be combined with each other, but this must be based on the ability of those skilled in the art to implement them. If the combination of technical solutions is contradictory or impossible to implement, it should be considered that such a combination of technical solutions does not exist and is not within the scope of protection claimed by this invention.

[0022] The preferred embodiment of the video decoding and filtering method based on transformer described in this invention, such as... Figure 1 and Figure 2 As shown, the transformer-based video decoding and filtering method includes the following steps: Step S10: Obtain a reference frame and a current frame, pass the reference frame and the current frame through a motion estimation network to obtain motion vectors, and perform motion encoding and motion decoding on the motion vectors to obtain decoded motion vectors.

[0023] It is understood that the technical principle of the transformer-based video decoding and filtering method provided by this invention is as follows: Figure 2As shown, this invention covers two core stages: the compression encoding link at the encoding end and the reconstruction optimization link at the decoding end. The temporal context filter is a core innovative module of this invention, used to perform temporal optimization of reconstructed features at the decoding end to improve the reconstruction quality and temporal consistency of video frames. The core objective of the encoding end is to efficiently compress the original video frames to generate an encoded bitstream containing motion and residual information.

[0024] Specifically, the system receives a reference frame that has been decoded and reconstructed and a current frame to be encoded. The reference frame and the current frame to be encoded are then input into a motion estimation network for processing, and a motion vector describing the displacement of the pixel block is output.

[0025] In this embodiment, the encoding end receives the reference frame that has been decoded and reconstructed. (The final output of the previous frame) and the current frame to be encoded This serves as the input basis for subsequent motion estimation and residual calculation. Next, the motion estimation module receives a reference frame. With the current frame The motion vector describing the displacement of pixel blocks is obtained through a motion estimation network. .

[0026] Furthermore, the motion vector is input to the motion encoding module for entropy encoding compression, outputting a motion information bitstream, and the motion information bitstream is input to the motion decoding module for decoding, outputting a decoded motion vector.

[0027] In this embodiment, the motion encoding module performs entropy encoding compression on the motion vector to generate a motion information bitstream, which is then transmitted to the decoding end. At the decoding end, the motion information bitstream is decoded, and the decoded motion vector is output. .

[0028] Step S20: Perform motion compensation on the decoded motion vector to obtain multi-scale context information. Input the current frame and the multi-scale context information into a conditional residual coding network for transformation, quantization and fusion processing to obtain the feature representation of the current frame.

[0029] The step of performing motion compensation on the decoded motion vector to obtain multi-scale context information specifically includes: Based on the reference features of the reference frame, a multi-scale reference feature set is constructed, which includes: a first-scale reference feature, a second-scale reference feature, and a third-scale reference feature. Based on the decoded motion vector, motion compensation is performed on the first scale reference feature, the second scale reference feature, and the third scale reference feature respectively to obtain the first scale context information, the second scale context information, and the third scale context information.

[0030] It is understandable that a multi-scale reference feature set is constructed based on the reference features of the reference frame. Among these, the first-scale reference features... The reference features at the second and third scales are obtained by downsampling the reference frame. These features are obtained by downsampling or feature transformation of the reference frame with different parameters. The reference features at these three scales together constitute a multi-scale reference feature set, which is used to provide texture and structural information at different granularities.

[0031] Furthermore, motion compensation is performed on reference features at three scales based on the decoded motion vectors. Specifically, block matching with integer pixel or fractional pixel precision is performed on the first-scale reference features to obtain first-scale context information; in the second and third-scale reference features, motion compensation is performed after scaling the motion vectors accordingly to obtain context information for the corresponding scales. The three sets of context information obtained ultimately describe the predicted data at different spatial granularities.

[0032] Furthermore, the condition-based residual coding network includes: an entropy model, a residual coding module, and a residual decoding module; The process of performing motion compensation on the decoded motion vector to obtain multi-scale context information, and then inputting the current frame and the multi-scale context information into a conditional residual coding network for transformation, quantization, and fusion processing to obtain the feature representation of the current frame, specifically includes: The third-scale context information is input into the entropy model for processing, and statistical parameters are output. Based on the feature space, the current frame is combined with the first scale context information and the second scale context information to perform temporal redundancy elimination to obtain the residual signal; The residual signal is input into the residual coding module for transformation and quantization processing, and auxiliary adaptive entropy coding is performed according to the statistical parameters to generate a residual information code stream. The residual information bitstream is input into the residual decoding module, and the residual information bitstream is decoded according to the statistical parameters and the first scale context information to obtain the residual information; The residual information is fused with the second scale context information to obtain the feature representation of the current frame.

[0033] In this embodiment, based on the decoded motion vector The encoding end performs motion compensation internally to obtain the prediction temporal context, and then converts the current frame... In a conditional residual coding network, prediction context information is input to eliminate temporal redundancy in the feature space to obtain a residual signal. The residual coding module receives this residual signal and combines it with multi-scale context information obtained from motion compensation. The process involves transformation, quantization, and fusion. Simultaneously, the statistical parameters μ (mean) and σ (variance) output by the entropy model are used to assist adaptive entropy coding, ultimately generating a residual information bitstream that is transmitted to the decoding end.

[0034] like Figure 2 As shown, the motion decoding module receives the motion information bitstream and decodes it to obtain the reconstructed motion vector. The motion compensation module reconstructs the motion vectors and reference features based on this. Generate the temporal context of the current frame and simultaneously extract multi-scale context information of the predicted frame. The residual decoding module receives the residual information bitstream and combines it with the statistical parameters μ and σ of the entropy model and multi-scale context information. Decoding yields the reconstructed residual The feature reconstruction module will reconstruct the residuals. Multi-scale contextual information of motion compensation output Perform fusion to generate the feature representation of the current frame. .

[0035] Step S30: Input the feature representation and the reference features of the reference frame into the preprocessing module of the temporal context filter for preprocessing to obtain the key vector, value vector and query vector that make up the attention mechanism.

[0036] like Figure 3 As shown, the temporal context filter of this invention takes the reconstructed features of the current frame as its input. Reference features from the previous frame This module utilizes the temporal attention mechanism of the transformer model to model cross-frame contextual dependencies, effectively correcting encoding distortions (such as blocking and ringing artifacts) and improving temporal consistency and visual quality. After filtering optimization, the final reconstructed frame is output. At the same time, this frame will serve as a reference frame for the next frame in subsequent encoding and decoding processes.

[0037] The temporal context filter described in this invention is a core innovative module for decoding-side reconstruction optimization, and its input is the reconstruction features of the current frame. Reference features from the previous frame The output is a fine-grained feature block optimized across frame contexts, which is used to correct coding distortion and improve temporal consistency.

[0038] It should be noted that in other embodiments, other computationally more efficient models can be used to replace the transformer while retaining the core objective of cross-frame context modeling. For example: replacing the transformer with a 3D CNN / spatiotemporal CNN: modeling cross-frame dependencies through the spatiotemporal receptive field of convolutions, resulting in lower computational cost; using a CNN+transformer hybrid architecture (such as MobileViT): extracting features locally using CNNs and modeling cross-frame dependencies globally using lightweight attention, ensuring efficiency; using recurrent neural networks (RNN / LSTM): modeling inter-frame dependencies through temporal recursion, suitable for processing continuous video streams. All of the above-mentioned alternative solutions fall within the scope of protection of this invention.

[0039] Furthermore, in another embodiment, the filtering function can also be integrated into other modules of the decoding end without setting a separate temporal context filter. For example, the filtering logic can be embedded in the feature reconstruction module: cross-frame correction can be directly added in the residual decoding and feature fusion stage, instead of being a separate filtering module; or it can be jointly optimized with the motion compensation module: context correction can be directly introduced during motion compensation. The above-mentioned alternative solutions also fall within the protection scope of this invention.

[0040] In this embodiment, the preprocessing module includes a block partitioning unit and a reference block selection unit, and the filtering main module includes a position embedding unit, a spatial attention modeling unit, and a quality perception fusion unit.

[0041] The preprocessing module that inputs the feature representation and the reference features of the reference frame into the temporal context filter for preprocessing to obtain the key vector, value vector, and query vector that constitute the attention mechanism specifically includes: The feature representation and the reference feature of the reference frame are input into the block partitioning unit for partitioning, and multiple feature representation blocks and reference feature blocks of uniform size are output. The feature representation block and the reference feature block are input into the reference block selection unit. Based on the spatial correspondence of the feature representation block, a preset number of reference feature blocks are selected to form the key vector and value vector of the attention mechanism, and the feature representation block is used as the query vector of the attention mechanism.

[0042] It is understandable that, in the block partitioning unit, the input current frame features Reference features from the previous frame The feature blocks are uniformly divided into blocks of size (B, K, K, C) (where B is the batch size, K is the block space size, and C is the number of channels) to ensure dimensional consistency in subsequent calculations.

[0043] In the reference block selection unit, for each feature block of the current frame (denoted as...) Size (B, K, K, C)), the module is derived from the feature map of the previous frame (denoted as...). Within a size (B, 3K, 3K, C), nine 3×3 reference blocks are selected based on spatial location correspondence to form the key and value vectors input to the attention mechanism (size (B, 9, K, K, C)). The feature block of the current frame serves as the query vector input to the attention mechanism (size (B, K, K, C)). This application designs a reference block selection unit that effectively reduces computational complexity by limiting the spatial range of the reference blocks, ensuring sufficient contextual information.

[0044] Step S40: Input the key vector, the value vector and the query vector into the filtering main module of the temporal context filter for filtering optimization, output the final reconstructed frame, and use the final reconstructed frame as the reference frame for the next frame in the subsequent video decoding filtering process.

[0045] The filtering module that inputs the key vector, the value vector, and the query vector into the temporal context filter for filtering optimization and outputs the final reconstructed frame specifically includes: The key vector and the value vector are input into the embedding unit for learnable position encoding while keeping the feature dimension unchanged, to obtain the target key vector and the target value vector. The query vector is input into the embedding unit for learnable position encoding and then aligned in dimensions according to the preset number to obtain the target query vector. The target key vector and the target query vector are input into the spatial attention modeling unit for calculation to obtain the similarity between the target key vector and the target query vector. An attention weight matrix is ​​obtained based on the similarity. The attention weight matrix is ​​then weighted and summed with the target value vector to output the intermediate features that fuse the reference block context information. The intermediate features and the target query vector are input into the quality-aware fusion unit for feature evaluation calculation to obtain a quality confidence score that characterizes the reconstruction quality of the reference block. The fusion weights are dynamically adjusted based on the quality confidence score to obtain the target fusion weights. Based on the target fusion weights, the intermediate features and the query vector are residually connected to output the final reconstructed frame.

[0046] It is understandable that, in the Position Embedding unit, in order to enable the transformer attention mechanism to perceive the spatial location information of the feature blocks, the module adds learnable positional encodings to the input K&V (9 reference blocks) and Q (current block).

[0047] The process of adding learnable positional codes to the input K, V (9 reference blocks), and Q (current block) includes: By keeping the location encoding consistent with the original feature dimension, attention computation can distinguish feature blocks in different spatial locations, thus avoiding the loss of spatial information.

[0048] The Q-features are expanded by 9 (expand is a common dimensional expansion method in tensor operations; its core function is to "expand" a specified dimension of a tensor to a larger size without copying the data), expanding its size to (B, 9, K, K, C), aligning it with the dimensions of K and V, providing a matching input format for subsequent attention calculations. Through the above process, the target key vector, target value vector, and target query vector are finally obtained.

[0049] Furthermore, the Spatial Attention modeling unit is based on the transformer self-attention mechanism to model the spatial dependency between the current block and the reference block.

[0050] In the spatial attention modeling unit, the inputs are the target key vector, target value vector, and target query vector, which are positionally embedded and dimensionally aligned. Next, the similarity between the target query vector and the target key vector is calculated to obtain the attention weight matrix, which reflects the contribution of each reference block to the current block. After Softmax (normalization function) normalization, the weight matrix and the target value vector tensor are weighted and summed to output an intermediate feature that incorporates the contextual information of the reference blocks.

[0051] Furthermore, the Quality Aware Fusion unit is the final step in filtering optimization, and its core lies in dynamic weighted fusion and residual connection.

[0052] In the quality-aware fusion unit, the input consists of intermediate features output from spatial attention modeling and the original features of the current block (the original input of Q). A quality-aware mechanism is introduced: the fusion weights are dynamically adjusted based on the reconstruction quality of the reference block (e.g., the degree of coding distortion), prioritizing the preservation of effective information from high-quality reference blocks and suppressing noise interference from low-quality blocks. Finally, the fused intermediate features are residually concatenated with the original features of the current block (element-wise addition), resulting in a refined feature block of size (B, K, K, C). This design preserves the original details of the current block while incorporating temporal context information from the previous frame, effectively correcting coding distortions such as block artifacts and ringing artifacts, and significantly improving the visual quality and temporal consistency of the reconstructed frame.

[0053] Furthermore, the final reconstructed frame is used as a reference frame for the next frame in subsequent video decoding and filtering processes.

[0054] In the video decoding process, storing the final reconstructed frame after decoding into a reference frame buffer as a motion reference for subsequent frames is a crucial step in ensuring decoding continuity and compression efficiency. During the decoding of subsequent frames, the decoder uses motion vectors parsed from the bitstream to perform motion compensation prediction on specific regions within the reference frame.

[0055] The core protected object of this invention is a transformer-based temporal context filtering method for video decoding. Its innovation and protection revolve around "accurate cross-frame context modeling" and "distortion adaptive correction", mainly including a structured reference block selection mechanism, cross-frame spatial attention modeling assisted by position embedding, a quality-aware feature fusion strategy, and fine feature output of residual connections.

[0056] Among them, the structured reference block selection mechanism: for the feature block of the current frame, a 3×3 local neighborhood reference block is selected from the feature map of the previous frame to construct the structured input of the attention mechanism.

[0057] Location embedding-assisted cross-frame spatial attention modeling: Learnable location encoding is added to feature blocks, and the spatial dependency between the current block and the reference block is modeled by combining the transformer attention mechanism.

[0058] Quality-aware feature fusion strategy: Dynamically evaluate the encoding distortion of the reference block, adaptively adjust the fusion weights, and prioritize the preservation of effective information from high-quality reference blocks. Residual connection for refined feature output: Perform residual connections between the fused features and the original current block features to balance temporal context fusion with the preservation of original details.

[0059] The beneficial effects of this invention are as follows: 1. Significantly improves the subjective and objective quality of reconstructed videos, effectively suppressing compression distortions such as blockiness and blurring while restoring details; 2. It has higher robustness and efficiency, avoiding performance degradation caused by motion estimation failure, and the one-way reference mechanism reduces computation and storage overhead. 3. The module design is flexible and can be seamlessly integrated into existing video decoding pipelines as a general post-processing unit, making it especially suitable for latency-sensitive real-time video application scenarios.

[0060] Furthermore, such as Figure 4 As shown, based on the above-described transformer-based video decoding and filtering method, this invention also provides a transformer-based video decoding and filtering system, wherein the transformer-based video decoding and filtering system includes: The motion estimation module 51 is used to acquire a reference frame and a current frame, pass the reference frame and the current frame through a motion estimation network to obtain motion vectors, and perform motion encoding and motion decoding on the motion vectors to obtain decoded motion vectors. The residual coding module 52 is used to perform motion compensation on the decoded motion vector to obtain multi-scale context information, and input the current frame and the multi-scale context information into a condition-based residual coding network for transformation, quantization and fusion processing to obtain the feature representation of the current frame; Preprocessing module 53 is used to input the feature representation and the reference features of the reference frame into the preprocessing module of the temporal context filter for preprocessing to obtain the key vector, value vector and query vector that make up the attention mechanism; The filtering optimization module 54 is used to input the key vector, the value vector and the query vector into the filtering main module of the temporal context filter for filtering optimization, output the final reconstructed frame, and use the final reconstructed frame as the reference frame for the next frame in the subsequent video decoding filtering process.

[0061] Furthermore, such as Figure 5 As shown, based on the above-mentioned transformer-based video decoding and filtering method and system, the present invention also provides a terminal, which includes a processor 10, a memory 20 and a display 30. Figure 5 Only some of the terminal components are shown; however, it should be understood that it is not required to implement all of the components shown, and more or fewer components may be implemented instead.

[0062] In some embodiments, the memory 20 may be an internal storage unit of the terminal, such as a hard disk or memory. In other embodiments, the memory 20 may be an external storage device of the terminal, such as a plug-in hard disk, smart media card (SMC), secure digital card (SD), flash card, etc. Further, the memory 20 may include both internal and external storage devices. The memory 20 is used to store application software and various types of data installed on the terminal, such as the program code installed on the terminal. The memory 20 can also be used to temporarily store data that has been output or will be output. In one embodiment, the memory 20 stores a transformer-based video decoding and filtering program 40, which can be executed by the processor 10 to implement the transformer-based video decoding and filtering method of this application.

[0063] In some embodiments, the processor 10 may be a central processing unit (CPU), a microprocessor, or other data processing chip, used to run program code stored in the memory 20 or process data, such as executing the transformer-based video decoding and filtering method.

[0064] In some embodiments, the display 30 may be an LED display, a liquid crystal display, a touch-sensitive liquid crystal display, or an OLED (Organic Light-Emitting Diode) touchscreen. The display 30 is used to display information on the terminal and to display a visual user interface. The components of the terminal communicate with each other via a system bus.

[0065] In one embodiment, when processor 10 executes transformer-based video decoding and filtering program 40 in memory 20, the following steps are performed: A reference frame and a current frame are acquired. The reference frame and the current frame are passed through a motion estimation network to obtain motion vectors. The motion vectors are then subjected to motion encoding and motion decoding to obtain decoded motion vectors. Motion compensation is performed on the decoded motion vector to obtain multi-scale context information. The current frame and the multi-scale context information are then input into a conditional residual coding network for transformation, quantization, and fusion processing to obtain the feature representation of the current frame. The feature representation and the reference features of the reference frame are input into the preprocessing module of the temporal context filter for preprocessing to obtain the key vector, value vector and query vector that make up the attention mechanism; The key vector, the value vector, and the query vector are input into the filtering main module of the temporal context filter for filtering optimization, and the final reconstructed frame is output. The final reconstructed frame is then used as the reference frame for the next frame in the subsequent video decoding filtering process.

[0066] Specifically, the process of obtaining a reference frame and a current frame, passing the reference frame and the current frame through a motion estimation network to obtain motion vectors describing pixel block displacements, and performing motion encoding and decoding on the motion vectors to obtain decoded motion vectors includes: Receive a reference frame that has been decoded and reconstructed and a current frame to be encoded. Input the reference frame and the current frame to be encoded into a motion estimation network for processing and output a motion vector to describe the displacement of pixel blocks. The motion vector is input to the motion encoding module for entropy encoding compression, outputting a motion information bitstream. The motion information bitstream is then input to the motion decoding module for decoding, outputting a decoded motion vector.

[0067] Specifically, the step of performing motion compensation on the decoded motion vector to obtain multi-scale context information includes: Based on the reference features of the reference frame, a multi-scale reference feature set is constructed, which includes: a first-scale reference feature, a second-scale reference feature, and a third-scale reference feature. Based on the decoded motion vector, motion compensation is performed on the first scale reference feature, the second scale reference feature, and the third scale reference feature respectively to obtain the first scale context information, the second scale context information, and the third scale context information.

[0068] The condition-based residual coding network includes: an entropy model, a residual coding module, and a residual decoding module; The process of performing motion compensation on the decoded motion vector to obtain multi-scale context information, and then inputting the current frame and the multi-scale context information into a conditional residual coding network for transformation, quantization, and fusion processing to obtain the feature representation of the current frame, specifically includes: The third-scale context information is input into the entropy model for processing, and statistical parameters are output. Based on the feature space, the current frame is combined with the first scale context information and the second scale context information to perform temporal redundancy elimination to obtain the residual signal; The residual signal is input into the residual coding module for transformation and quantization processing, and auxiliary adaptive entropy coding is performed according to the statistical parameters to generate a residual information code stream. The residual information bitstream is input into the residual decoding module, and the residual information bitstream is decoded according to the statistical parameters and the first scale context information to obtain the residual information; The residual information is fused with the second scale context information to obtain the feature representation of the current frame.

[0069] The preprocessing module includes a block partitioning unit and a reference block selection unit, and the filtering main module includes a position embedding unit, a spatial attention modeling unit, and a quality perception fusion unit.

[0070] Specifically, the step of inputting the feature representation and the reference features of the reference frame into the preprocessing module of the temporal context filter for preprocessing to obtain the key vector, value vector, and query vector that constitute the attention mechanism includes: The feature representation and the reference feature of the reference frame are input into the block partitioning unit for partitioning, and multiple feature representation blocks and reference feature blocks of uniform size are output. The feature representation block and the reference feature block are input into the reference block selection unit. Based on the spatial correspondence of the feature representation block, a preset number of reference feature blocks are selected to form the key vector and value vector of the attention mechanism, and the feature representation block is used as the query vector of the attention mechanism.

[0071] Specifically, the step of inputting the key vector, the value vector, and the query vector into the filtering main module of the temporal context filter for filtering optimization and outputting the final reconstructed frame includes: The key vector and the value vector are input into the embedding unit for learnable position encoding while keeping the feature dimension unchanged, to obtain the target key vector and the target value vector. The query vector is input into the embedding unit for learnable position encoding and then aligned in dimensions according to the preset number to obtain the target query vector. The target key vector and the target query vector are input into the spatial attention modeling unit for calculation to obtain the similarity between the target key vector and the target query vector. An attention weight matrix is ​​obtained based on the similarity. The attention weight matrix is ​​then weighted and summed with the target value vector to output the intermediate features that fuse the reference block context information. The intermediate features and the target query vector are input into the quality-aware fusion unit for feature evaluation calculation to obtain a quality confidence score that characterizes the reconstruction quality of the reference block. The fusion weights are dynamically adjusted based on the quality confidence score to obtain the target fusion weights. Based on the target fusion weights, the intermediate features and the query vector are residually connected to output the final reconstructed frame.

[0072] The present invention also provides a computer-readable storage medium, wherein the computer-readable storage medium stores a transformer-based video decoding and filtering program, which, when executed by a processor, implements the steps of the transformer-based video decoding and filtering method as described above.

[0073] In summary, this invention provides a transformer-based video decoding and filtering method, system, terminal, and storage medium. The method includes: acquiring a reference frame and a current frame; passing the reference frame and the current frame through a motion estimation network to obtain motion vectors; performing motion encoding and decoding on the motion vectors to obtain decoded motion vectors; performing motion compensation on the decoded motion vectors to obtain multi-scale context information; inputting the current frame and the multi-scale context information into a conditional residual coding network for transformation, quantization, and fusion processing to obtain a feature representation of the current frame; inputting the feature representation and the reference features of the reference frame into a preprocessing module of a temporal context filter for preprocessing to obtain a key vector, value vector, and query vector constituting an attention mechanism; inputting the key vector, value vector, and query vector into the filtering main module of the temporal context filter for filtering optimization, outputting a final reconstructed frame, and using the final reconstructed frame as a reference frame for the next frame in subsequent video decoding and filtering processes. This invention significantly improves the subjective and objective quality of reconstructed video, effectively suppresses compression distortions such as blockiness and blurring, and restores details, while also exhibiting higher robustness and efficiency.

[0074] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or terminal that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or terminal. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or terminal that includes that element.

[0075] Of course, those skilled in the art will understand that all or part of the processes in the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. Any references to memory, storage, databases, or other media used in the embodiments provided by this invention can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), Rambus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0076] It should be understood that the application of the present invention is not limited to the examples above. Those skilled in the art can make improvements or modifications based on the above description, and all such improvements and modifications should fall within the protection scope of the appended claims.

Claims

1. A method for transformer-based video decoding filtering, characterized in that, The transformer-based video decoding and filtering method includes: A reference frame and a current frame are acquired. The reference frame and the current frame are passed through a motion estimation network to obtain motion vectors. The motion vectors are then subjected to motion encoding and motion decoding to obtain decoded motion vectors. Motion compensation is performed on the decoded motion vector to obtain multi-scale context information. The current frame and the multi-scale context information are then input into a conditional residual coding network for transformation, quantization, and fusion processing to obtain the feature representation of the current frame. The feature representation and the reference features of the reference frame are input into the preprocessing module of the temporal context filter for preprocessing to obtain the key vector, value vector and query vector that make up the attention mechanism; The key vector, the value vector, and the query vector are input into the filtering main module of the temporal context filter for filtering optimization, and the final reconstructed frame is output. The final reconstructed frame is then used as the reference frame for the next frame in the subsequent video decoding filtering process.

2. The video decoding and filtering method based on transformer according to claim 1, characterized in that, The process of acquiring a reference frame and a current frame, passing the reference frame and the current frame through a motion estimation network to obtain motion vectors describing pixel block displacements, and performing motion encoding and decoding on the motion vectors to obtain decoded motion vectors specifically includes: Receive a reference frame that has been decoded and reconstructed and a current frame to be encoded. Input the reference frame and the current frame to be encoded into a motion estimation network for processing and output a motion vector to describe the displacement of pixel blocks. The motion vector is input to the motion encoding module for entropy encoding compression, outputting a motion information bitstream. The motion information bitstream is then input to the motion decoding module for decoding, outputting a decoded motion vector.

3. The video decoding and filtering method based on transformer according to claim 1, characterized in that, The step of performing motion compensation on the decoded motion vector to obtain multi-scale context information specifically includes: Based on the reference features of the reference frame, a multi-scale reference feature set is constructed, which includes: a first-scale reference feature, a second-scale reference feature, and a third-scale reference feature. Based on the decoded motion vector, motion compensation is performed on the first scale reference feature, the second scale reference feature, and the third scale reference feature respectively to obtain the first scale context information, the second scale context information, and the third scale context information.

4. The video decoding and filtering method based on transformer according to claim 3, characterized in that, The condition-based residual coding network includes: an entropy model, a residual coding module, and a residual decoding module; The process of performing motion compensation on the decoded motion vector to obtain multi-scale context information, and then inputting the current frame and the multi-scale context information into a conditional residual coding network for transformation, quantization, and fusion processing to obtain the feature representation of the current frame, specifically includes: The third-scale context information is input into the entropy model for processing, and statistical parameters are output. Based on the feature space, the current frame is combined with the first scale context information and the second scale context information to perform temporal redundancy elimination to obtain the residual signal; The residual signal is input into the residual coding module for transformation and quantization processing, and auxiliary adaptive entropy coding is performed according to the statistical parameters to generate a residual information code stream. The residual information bitstream is input into the residual decoding module, and the residual information bitstream is decoded according to the statistical parameters and the first scale context information to obtain the residual information; The residual information is fused with the second scale context information to obtain the feature representation of the current frame.

5. The video decoding and filtering method based on transformer according to claim 1, characterized in that, The preprocessing module includes a block partitioning unit and a reference block selection unit, and the filtering main module includes a position embedding unit, a spatial attention modeling unit, and a quality perception fusion unit.

6. The video decoding and filtering method based on transformer according to claim 5, characterized in that, The preprocessing module that inputs the feature representation and the reference features of the reference frame into the temporal context filter for preprocessing to obtain the key vector, value vector, and query vector that constitute the attention mechanism specifically includes: The feature representation and the reference feature of the reference frame are input into the block partitioning unit for partitioning, and multiple feature representation blocks and reference feature blocks of uniform size are output. The feature representation block and the reference feature block are input into the reference block selection unit. Based on the spatial correspondence of the feature representation block, a preset number of reference feature blocks are selected to form the key vector and value vector of the attention mechanism, and the feature representation block is used as the query vector of the attention mechanism.

7. The video decoding and filtering method based on transformer according to claim 6, characterized in that, The filtering module that inputs the key vector, the value vector, and the query vector into the temporal context filter for filtering optimization and outputs the final reconstructed frame specifically includes: The key vector and the value vector are input into the embedding unit for learnable position encoding while keeping the feature dimension unchanged, to obtain the target key vector and the target value vector. The query vector is input into the embedding unit for learnable position encoding and then aligned in dimensions according to the preset number to obtain the target query vector. The target key vector and the target query vector are input into the spatial attention modeling unit for calculation to obtain the similarity between the target key vector and the target query vector. An attention weight matrix is ​​obtained based on the similarity. The attention weight matrix is ​​then weighted and summed with the target value vector to output the intermediate features that fuse the reference block context information. The intermediate features and the target query vector are input into the quality-aware fusion unit for feature evaluation calculation to obtain a quality confidence score that characterizes the reconstruction quality of the reference block. The fusion weights are dynamically adjusted based on the quality confidence score to obtain the target fusion weights. Based on the target fusion weights, the intermediate features and the query vector are residually connected to output the final reconstructed frame.

8. A video decoding and filtering system based on transformer, characterized in that, The transformer-based video decoding and filtering system includes: The motion estimation module is used to acquire a reference frame and a current frame, pass the reference frame and the current frame through a motion estimation network to obtain motion vectors, and perform motion encoding and motion decoding on the motion vectors to obtain decoded motion vectors. The motion compensation and residual coding module is used to perform motion compensation on the decoded motion vector to obtain multi-scale context information, and input the current frame and the multi-scale context information into a condition-based residual coding network for transformation, quantization and fusion processing to obtain the feature representation of the current frame. The preprocessing module is used to input the feature representation and the reference features of the reference frame into the preprocessing module of the temporal context filter for preprocessing to obtain the key vector, value vector and query vector that make up the attention mechanism; The filtering optimization module is used to input the key vector, the value vector and the query vector into the filtering main module of the temporal context filter for filtering optimization, output the final reconstructed frame, and use the final reconstructed frame as the reference frame for the next frame in the subsequent video decoding filtering process.

9. A terminal, characterized in that, The terminal includes: a memory, a processor, and a transformer-based video decoding and filtering program stored in the memory and executable on the processor. When the transformer-based video decoding and filtering program is executed by the processor, it implements the steps of the transformer-based video decoding and filtering method as described in any one of claims 1-7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a transformer-based video decoding and filtering program, which, when executed by a processor, implements the steps of the transformer-based video decoding and filtering method as described in any one of claims 1-7.

Images