Video coding method and device

Abstract

The embodiment of the invention provides a video decoding method and device, and a video coding method and device, and relates to the technical field of video coding and decoding. Comprising the following steps: performing feature extraction on a current frame and a reconstructed reference frame to obtain a current feature and a reference feature; performing splicing processing on the current feature and the reference feature to obtain a combined feature; carrying out convolution processing on the combined features to obtain pre-processing features of which the channel number is a preset value; processing the preprocessing features through a self-attention model based on a U-shaped structure to obtain a motion offset between the current frame and the reconstructed reference frame; and obtaining coded data of the current frame according to the motion offset. Some embodiments of the invention are used for improving the robustness of solving the motion offset between the reference frame and the current frame.

Description

Technical Field

[0001] Some embodiments of the present application relate to the technical field of video coding and decoding, and more specifically, to a video coding method and device. Background Art

[0002] With the widespread popularity of smart electronic products, video transmission has long become an indispensable part of people's lives. Video uploading and downloading in social media, live streaming, virtual reality (VR) games, etc. all rely on video transmission and processing technologies, and video encoding technology is the basis of many of the above-mentioned video transmission and processing technologies.

[0003] Video can be regarded as a combination of multiple consecutive frames, and the amount of motion of objects between adjacent frames is very small, especially the content between adjacent frames may be very similar, which means that there is a lot of redundant information in the time domain of the video. In order to compress the redundant information of the video in the time domain, a video coding method based on inter-frame prediction is proposed in the related technology. The video coding method based on inter-frame prediction first uses optical flow estimation and other methods to obtain the motion offset between the reference frame and the current frame, and encodes the offset, then uses the decoded offset to align the reference frame with the current frame to generate a predicted frame, and then uses the current frame to subtract the obtained predicted frame to obtain the prediction residual, and finally adds the encoded offset and the encoded prediction residual to the video bit stream, thereby reducing the bit stream of the video signal transmission. At present, the related technology generally uses the optical flow method to obtain the motion offset between the reference frame and the current frame. However, since the optical flow method is based on the assumption that the brightness is constant and the motion is small, the optical flow method is easily affected by illumination, object occlusion, etc., is sensitive to noise, has poor robustness, is prone to artifacts, and is difficult to perform stable motion alignment. Summary of the invention

[0004] An exemplary embodiment of the present application provides a video encoding method and apparatus for improving the robustness of obtaining a motion offset between a reference frame and a current frame.

[0005] Some embodiments of the present application provide the following technical solutions:

[0006] In a first aspect, some embodiments of the present application provide a video encoding method, including:

[0007] Perform feature extraction on the current frame and the reconstructed reference frame to obtain current features and reference features;

[0008] Performing splicing processing on the current feature and the reference feature to obtain a combined feature;

[0009] Performing convolution processing on the merged features to obtain preprocessed features with a preset number of channels;

[0010] Processing the preprocessed features through a self-attention model based on a U-shaped structure to obtain a motion offset between the current frame and the reconstructed reference frame;

[0011] The encoded data of the current frame is obtained according to the motion offset.

[0012] In a second aspect, some embodiments of the present application provide a video encoding device, including:

[0013] a memory configured to store a computer program;

[0014] The processor is configured to, when calling a computer program, enable the video encoding device to implement the video encoding method described in the first aspect.

[0015] In a third aspect, some embodiments of the present application provide a computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a computing device, the computing device implements the video encoding method described in the first aspect.

[0016] In a fourth aspect, some embodiments of the present application provide a computer program product, which, when executed on a computer, enables the computer to implement the video encoding method described in the first aspect.

[0017] It can be seen from the above technical scheme that when encoding the current frame, the video encoding method provided in the embodiment of the present application first performs feature extraction on the current frame and the reconstructed reference frame to obtain the current feature and the reference feature, then splices the current feature and the reference feature to obtain the merged feature, convolution is performed on the merged feature to obtain a preprocessing feature with a preset number of channels, and the preprocessing feature is processed by a self-attention model based on a U-shaped structure to obtain the motion offset between the current frame and the reconstructed reference frame, and finally the encoded data of the current frame is obtained according to the motion offset. Since motion alignment is to compensate for the changes in the reference frame caused by the motion of the object so that the reference frame is similar to the current frame, and in the process of processing the preprocessing features based on the self-attention model of the U-shaped structure, the relationship between the pixels of the current frame, the relationship between the pixels of the reconstructed reference frame, and the relationship between the pixels of the current frame and the reconstructed reference frame can all be reflected, so the operation of the self-attention model based on the U-shaped structure is consistent with the purpose of motion alignment and is suitable for obtaining the offset required for motion alignment. Therefore, the embodiment of the present application can process the preprocessing features through the self-attention model based on the U-shaped structure to obtain the motion offset between the current frame and the reconstructed reference frame. Compared with obtaining the motion offset between the current frame and the reconstructed reference frame based on the optical flow method, the embodiment of the present application obtains the motion offset between the current frame and the reconstructed reference frame based on the self-attention model of the U-shaped structure. Therefore, the process of obtaining the motion offset in the embodiment of the present application is not easily affected by illumination, object occlusion, etc. Therefore, the embodiment of the present application can improve the robustness of obtaining the motion offset between the reference frame and the current frame. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] In order to more clearly illustrate the implementation methods of some embodiments of the present application or related technologies, the following is a brief introduction to the drawings required for use in the description of the embodiments or related technologies. Obviously, the drawings described below are some embodiments of the present application, and a person skilled in the art can also obtain other drawings based on these drawings.

[0019] Figure 1 A schematic diagram showing the structure of a video encoding and decoding framework in some embodiments of the present application is shown;

[0020] Figure 2 One of the flow charts of the steps of the video encoding method in some embodiments of the present application is shown;

[0021] Figure 3 The second flowchart of the video encoding method in some embodiments of the present application is shown;

[0022] Figure 4 A schematic diagram showing the structure of a feature extraction model in some embodiments of the present application is shown;

[0023] Figure 5 A schematic diagram of a self-attention model based on a U-shaped structure in some embodiments of the present application is shown;

[0024] Figure 6 A schematic diagram of the structure of a local window enhanced self-attention layer in some embodiments of the present application is shown;

[0025] Figure 7 A schematic diagram showing the structure of an offset compression model in some embodiments of the present application is shown;

[0026] Figure 8 A schematic diagram of the structure of an enhanced fusion model in some embodiments of the present application is shown;

[0027] Fig. 9 A schematic diagram showing the structure of a fusion model in some embodiments of the present application is shown;

[0028] Fig.10 A schematic diagram of the structure of a residual encoder based on spatiotemporal fusion in some embodiments of the present application is shown. DETAILED DESCRIPTION

[0029] In order to make the purpose and implementation method of the present application clearer, the exemplary implementation method of the present application will be clearly and completely described below in conjunction with the drawings in the exemplary embodiments of the present application. Obviously, the described exemplary embodiments are only part of the embodiments of the present application, rather than all the embodiments.

[0030] It should be noted that the brief description of terms in this application is only for the convenience of understanding the embodiments described below, and is not intended to limit the embodiments of this application. Unless otherwise specified, these terms should be understood according to their common and usual meanings.

[0031] The terms "comprises," "comprising," and "having," and any variations thereof, are intended to cover but not exclude inclusion, for example, a product or device comprising a list of components is not necessarily limited to all the components expressly listed but may include other components not expressly listed or inherent to such product or device.

[0032] References to "some implementations", "some embodiments", etc. in the specification indicate that the described implementations or embodiments may include specific features, structures, or characteristics, but not every embodiment may include the specific features, structures, or characteristics. In addition, such phrases do not necessarily refer to the same implementation. In addition, when a specific feature, structure, or characteristic is described in connection with an embodiment, it is considered to be within the knowledge of those skilled in the art to implement such feature, structure, or characteristic in connection with other implementations (whether or not explicitly described herein).

[0033] The embodiments of the present application relate to the field of video coding and decoding technology. The following first describes a video coding and decoding framework for executing the video encoding method and video decoding method provided by the embodiments of the present application.

[0034] Reference Figure 1 As shown, a video coding and decoding framework for executing the video encoding method and video decoding method provided by the embodiments of the present application includes: a reconstructed frame buffer pool 101, a feature extraction model 102, a feature splicing module 103, a feature buffer pool 104, a self-attention model based on a U-shaped structure 105, a motion vector decoder 106, a deformable convolution layer 107, an enhanced fusion module 108, a residual calculation unit 109, a linear layer 110, a residual decoder based on spatiotemporal fusion 111, an entropy encoding and decoding module 112, a residual reconstruction unit 113 and an image reconstruction model 114.

[0035] In the video encoding process, the video encoding and decoding framework first uses the feature extraction model 102 to extract the current frame X t , reconstruct the reference frame reconstructed in the frame buffer pool 101 and at least one reconstructed auxiliary reference frame ( Figure 1 At least one reconstructed auxiliary reference frame includes As an example), feature extraction is performed to obtain the current feature F t , Reference Features and at least one auxiliary feature And the reference feature and at least one auxiliary feature Write the feature buffer pool 104. Then, the feature splicing module 103 splices the current feature F t and reference features To obtain the merged features The combined features are then processed by a self-attention model 105 based on a U-shaped structure. Processing is performed to obtain the current frame X t With the reconstructed reference frame The motion offset between off The motion offset mv is then decoded by the motion vector decoder 106. off Encode and decode to obtain reconstructed motion offset Then the motion offset is reconstructed by Dynamically adjust the sampling position of the convolution kernel of the deformable convolution layer 107 to the reference feature Sampling is performed to obtain the initial prediction feature F pre Then the enhancement fusion module 108 is based on the at least one auxiliary feature For the initial prediction feature F prePerform fusion enhancement to obtain prediction features F pre_enh , and the current feature F is calculated by the residual calculation unit 109 t With the prediction feature F pre_enh The residual of the prediction residual F resi , through the linear layer 110 to predict the feature F pre_enh and the prediction residual F resi Perform block partitioning and linear mapping to obtain the first query vector Q R , the first key vector K R , value vector V R , the second query vector Q P And the first key vector K P And through the residual decoder 111 based on time-space fusion, the first query vector Q R , the first key vector K R , value vector V R , the second query vector Q P , the second key vector K P And the adjustment coefficient modu to obtain the prediction residual F resi The compression result is finally obtained by entropy decoding module 112 for the motion offset mv off The compression result and prediction residual F resi The compression result is entropy encoded to obtain the current frame X t The corresponding coded data (b mv 、b resi ).

[0036] In the video decoding process, the video codec first decodes the current frame X through the entropy decoding module 112. t The corresponding coded data is entropy decoded to obtain the prediction residual compression result and the motion offset compression result, and then the prediction residual compression result is decoded by the residual decoder 111 based on time-space fusion to obtain the reconstructed prediction residual. The motion vector decoder 106 decodes the compressed result of the motion offset to obtain a reconstructed motion offset. On the other hand, the reconstructed reference frame is extracted by the feature extraction model 102. and at least one reconstructed auxiliary reference frame Perform feature extraction and obtain reference features and at least one auxiliary feature According to the reference characteristics and the reconstructed motion offset Get the initial prediction feature F pre ; Then, the enhanced fusion module 108 is based on the at least one auxiliary feature For the initial prediction feature F prePerform fusion enhancement to obtain prediction feature F pre_enh Then, through the residual reconstruction unit 113, the prediction feature F pre_enh And the reconstructed prediction residual Get the current features of the reconstruction Finally, the image reconstruction model 114 and the current features of the reconstruction are Get the current frame of reconstruction

[0037] The present application embodiment provides a video encoding method, referring to Figure 2 As shown, the video encoding method includes the following steps S21 to S25:

[0038] S21. Perform feature extraction on the current frame and the reconstructed reference frame to obtain current features and reference features.

[0039] That is, the current frame and the reconstructed reference frame are respectively converted from the pixel domain to the feature domain, so as to convert the information of the current frame and the reconstructed reference frame from the pixel information system to high-dimensional features, thereby making the model pay more attention to high-level semantic features while maintaining important information in the image.

[0040] The reconstructed reference frame in the embodiment of the present application refers to: first encoding the reference frame to obtain the encoding result of the reference frame, and then decoding the encoding result of the reference frame to obtain the video frame. Since the encoding process includes lossy compression, some information will be lost, so the reconstructed reference frame is not completely equivalent to the reference frame.

[0041] In some embodiments, the reference frame may be a video frame that is located before the current frame and adjacent to the current frame. t-1 The current frame is X t The previous video frame.

[0042] Since the above step S21 performs feature extraction on the current frame and the reconstructed reference frame to obtain the current features and the reference features, thereby converting the pixel information of the video frame into high-dimensional features, the above step S21 can make the model pay more attention to high-level semantic features while maintaining important information in the video frame.

[0043] S22: performing splicing processing on the current feature and the reference feature to obtain a merged feature.

[0044] In some embodiments, the above step S22 (stitching the current feature and the reference feature to obtain a merged feature) includes: stitching the current feature and the reference feature in a feature channel dimension to obtain the merged feature.

[0045] Exemplarily, the length, width and number of channels of the current feature are H, w and C1 respectively, and the length, width and number of channels of the reference feature are H, W and C2 respectively. Then, the length, width and number of channels of the merged feature obtained by splicing the current feature and the reference feature in the feature channel dimension are H, W and C1+C2 respectively.

[0046] In some embodiments, the current feature and the reference feature have the same length, width, and number of channels, the length and width of the merged feature are the same as the length and width of the current feature and the reference feature, and the number of channels of the merged feature is twice the number of channels of the current feature and the reference feature.

[0047] S23, performing convolution processing on the merged features to obtain preprocessed features with a preset number of channels.

[0048] In some embodiments, the merged features can be convolved using a convolution layer whose convolution kernel size is 1*1 and whose output channel number is the preset value to obtain preprocessed features whose channel number is the preset value.

[0049] In some embodiments, the preset value is the number of feature channels of the current feature. That is, the splicing process in step S22 changes the number of channels of the feature to twice the channel data of the current feature, so the number of channels of the merged feature obtained by the splicing process is compressed to the preprocessed feature with the same channel as the current feature through step S23.

[0050] The concatenation process is represented as <>, and the convolution process is represented as Conv 1×1 , the preprocessed features are represented as F c , the above steps S22 and S23 can be expressed as the following formula (1):

[0051]

[0052] S24. Process the preprocessed features through a self-attention model based on a U-shaped structure to obtain a motion offset between the current frame and the reconstructed reference frame.

[0053] Since motion alignment compensates for the changes in the reference frame caused by the motion of the object, so that the reference frame is similar to the current frame, and in the process of processing the pre-processed features by the self-attention model based on the U-shaped structure, the relationship between the pixels of the current frame, the relationship between the pixels of the reconstructed reference frame, and the relationship between the pixels of the current frame and the reconstructed reference frame can all be reflected, the operation of the self-attention model based on the U-shaped structure is consistent with the purpose of motion alignment and is suitable for obtaining the offset required for motion alignment. Therefore, the embodiment of the present application can process the pre-processed features through the self-attention model based on the U-shaped structure to obtain the motion offset between the current frame and the reconstructed reference frame.

[0054] S25. Obtain encoding data of the current frame according to the motion offset.

[0055] In some embodiments, the above step S25 (obtaining the encoded data of the current frame according to the motion offset) includes the following steps 251 to 254:

[0056] Step 251: Obtain a reconstructed motion offset according to the motion offset.

[0057] The reconstructed motion offset is a motion offset obtained by encoding the motion offset and decoding the encoding result of the motion offset.

[0058] Since the encoding process includes lossy compression, part of the information in the motion offset will be lost, so the reconstructed motion offset is not completely equivalent to the motion offset.

[0059] Step 252: Obtain prediction features according to the reference features and the reconstructed motion offset.

[0060] In some embodiments, the above step 252 (obtaining predicted features based on the reference features and the reconstructed motion offset) includes: dynamically adjusting the sampling position of the convolution kernel of the feasible variable convolution layer by the reconstructed motion offset to sample the reference features to obtain the predicted features.

[0061] In some other embodiments, the above step 252 (obtaining the predicted feature according to the reference feature and the reconstructed motion offset) includes:

[0062] Dynamically adjusting the sampling position of the convolution kernel of the deformable convolution layer by the reconstruction motion offset to sample the reference feature to obtain the initial prediction feature;

[0063] The initial prediction feature is enhanced and fused based on at least one auxiliary feature to obtain the prediction feature.

[0064] The at least one auxiliary feature is a feature obtained by performing feature extraction on at least one auxiliary reference frame.

[0065] In some embodiments, the at least one auxiliary reference frame may include a video frame that is located before the current frame and has intervals of 1 and 2 from the current frame, respectively.

[0066] At present, inter-frame coding based on deep learning mainly uses a single prediction frame for direct compensation to obtain residuals. However, in a more complex motion background, the information provided by a single reference frame is relatively limited. Due to the lack of multi-frame information supplement and the inability to use long-range time domain information, motion detail information may be lost, resulting in reduced coding efficiency. On the other hand, due to the use of motion alignment, objects in video frames are prone to deformation and artifacts to varying degrees, resulting in distortion of local detail information. In response to the above problems, an embodiment of the present application introduces at least one auxiliary reference frame to perform local detail enhancement on the initial prediction features after motion alignment, and improves the quality of the prediction features by using multiple video frames in the time domain.

[0067] Step 253: Obtain prediction residuals according to the current features and the predicted features.

[0068] In some embodiments, the above step S253 (obtaining a prediction residual according to the current feature and the predicted feature) includes: calculating a residual between the current feature and the predicted feature to obtain the prediction residual.

[0069] Step 254: Generate encoding data of the current frame according to the motion offset and the prediction residual.

[0070] In some embodiments, the above-mentioned step 254 (generating the encoded data of the current frame according to the motion offset and the prediction residual) includes: encoding the motion offset and the prediction residual respectively to obtain the encoding result of the motion offset and the encoding result of the prediction residual, and using the encoding result of the motion offset and the encoding result of the prediction residual as the encoded data of the current frame.

[0071] The video encoding method provided in the embodiment of the present application first extracts features from the current frame and the reconstructed reference frame to obtain current features and reference features when encoding the current frame, then performs splicing processing on the current features and the reference features to obtain merged features, performs convolution processing on the merged features to obtain preprocessing features with a preset number of channels, and processes the preprocessing features through a self-attention model based on a U-shaped structure to obtain the motion offset between the current frame and the reconstructed reference frame, and finally obtains the encoding data of the current frame according to the motion offset. Since motion alignment compensates for the changes in the reference frame caused by the motion of the object so that the reference frame is similar to the current frame, and in the process of processing the preprocessing features based on the self-attention model of the U-shaped structure, the relationship between the pixels of the current frame, the relationship between the pixels of the reconstructed reference frame, and the relationship between the pixels of the current frame and the reconstructed reference frame can all be reflected, so the operation of the self-attention model based on the U-shaped structure is consistent with the purpose of motion alignment, and is suitable for obtaining the offset required for motion alignment. Therefore, the embodiment of the present application can process the preprocessing features through a self-attention model based on a U-shaped structure to obtain the motion offset between the current frame and the reconstructed reference frame. Compared to obtaining the motion offset between the current frame and the reconstructed reference frame based on the optical flow method, the embodiment of the present application obtains the motion offset between the current frame and the reconstructed reference frame based on the self-attention model of the U-shaped structure. Therefore, the process of obtaining the motion offset in the embodiment of the present application is not easily affected by lighting, object occlusion, etc. Therefore, the embodiment of the present application can improve the robustness of obtaining the motion offset between the reference frame and the current frame.

[0072] The present application also provides another video encoding method. Figure 3 As shown, the video encoding method includes the following steps S301 to S318:

[0073] S301. Perform feature extraction on a current frame, a reconstructed reference frame, and at least one reconstructed auxiliary reference frame based on a feature extraction model to obtain current features, reference features, and at least one auxiliary feature.

[0074] Among them, refer to Figure 4 As shown, the feature extraction model includes: a first convolutional layer 41, a first activation layer 42, a first residual network 43 and a first addition fusion unit 44, wherein the first residual network is composed of a plurality of serially connected residual blocks ( Figure 4 In the figure, the first residual network 43 is taken as an example including three levels of cascaded residual blocks 430).

[0075] The feature extraction model is used to process the input video frame of the feature extraction model through the first convolution layer 41 and the first activation layer 42 in sequence to obtain the first convolution feature Fconv , the first convolution feature F is processed by the first residual network 43 conv Process and obtain the first residual feature The first convolution feature F is processed by the first addition fusion unit 44. conv and the first residual characteristic The features are added and fused to obtain the output features of the feature extraction model ( Figure 4 In the current frame X t Perform feature extraction to obtain the current feature F t as an example).

[0076] In some embodiments, the first convolution layer 41 is a layer convolution with a convolution kernel size of 5*5, a stride of 2, and an output channel of 64.

[0077] In some embodiments, the first activation layer 42 is a linear rectification function (Linear Rectification Function, ReLU) layer.

[0078] In some embodiments, reference Figure 4 As shown, any residual block 430 of the first residual network 43 includes: a convolution layer 431, an activation layer 432, a convolution layer 433, an activation layer 434 and an addition fusion module 435. The residual block 430 processes the input features by: sequentially processing the input features through the convolution layer 431, the activation layer 432, the convolution layer 433, the activation layer 434 Process and obtain residual features Then the input features are fused by the fusion module 435 and residual features Addition fusion is performed to obtain the input features of the residual block 430

[0079] In some embodiments, the convolution layer 431 and the convolution layer 433 are both convolution layers with a convolution kernel size of 3*3, and the activation layer 432 and the activation layer 434 are both ReLU activation layers.

[0080] The convolution process of the first convolutional layer 41 is represented as: Conv 5×5 (), the activation processing of the first activation layer 42 is represented by: ReLU(), the residual processing of the first residual network 43 is represented by: ResBlocks(), and the current frame is represented by: X t , the current feature is represented by F t , then the feature extraction model for the current frame X t Perform feature extraction to obtain the current feature F t The process can be expressed as the following formula (2):

[0081] F conv =ReLU(Conv 5×5 (X t ))

[0082] F t =ResBlocks(F conv )+F conv (2)

[0083] The representation of the process of the feature extraction model performing feature extraction on the reference frame and at least one auxiliary reference frame to obtain the reference feature and at least one auxiliary feature is similar to the above formula (2), and will not be repeated for the sake of redundancy.

[0084] S302: Perform concatenation processing on the current feature and the reference feature to obtain a combined feature.

[0085] S303: Perform convolution processing on the combined features to obtain preprocessed features with a preset number of channels.

[0086] The implementation of steps S302 and S303 may refer to the implementation of steps S22 and S23, and will not be repeated here to avoid redundancy.

[0087] S304: Process the preprocessed features through a self-attention model based on a U-shaped structure to obtain a motion offset between the current frame and the reconstructed reference frame.

[0088] Reference Figure 5 As shown, the self-attention model based on the U-shaped structure includes: a contraction network 51 and an expansion network 52; the contraction network 51 and the expansion network 52 are connected through a local window enhanced self-attention layer 53; the contraction network 51 is composed of L cascaded contraction modules 510, and the expansion network 52 is composed of L cascaded expansion modules 520 ( Figure 5 In the figure, the contraction network 51 is composed of four cascaded contraction modules 510, and the expansion network 52 is composed of four cascaded expansion modules 520 as an example.

[0089] The contraction module 510 includes: a local window enhanced self-attention layer 501 and a downsampling layer 502 connected in series, the i-th level contraction module 510 of the contraction network is used to process the output features of the i-1-th level contraction module 510 through the local window enhanced self-attention layer, obtain the jump connection features of the i-th level contraction module, input the jump connection features into the L-i+1-th level expansion module 520 of the expansion network, downsample the jump connection features through the downsampling layer to obtain the output features of the i-th level contraction module 520, and input the output features of the i-th level contraction module into the i+1-th level contraction module 520;

[0090] The expansion module 520 includes: a serially connected upsampling layer 503, a splicing layer 504 and a local window enhanced self-attention layer 501. The j-th level expansion module of the expansion network is used to upsample the output features of the j-1-th level expansion module 520 through the upsampling layer to obtain the expansion features of the j-th level expansion module 520, splice the expansion features and the jump connection features of the L-j+1-th level contraction module 510 of the contraction network 51 through the splicing layer 504 to obtain the splicing features of the j-th level expansion module, process the splicing features through the local window enhanced self-attention layer 501 to obtain the output features of the j-th level expansion module, and input the output features of the j-th level contraction module into the j+1-th level expansion module 520.

[0091] It should be noted that Figure 5 In the figure, the contraction network 51 is composed of four cascaded contraction modules 510, and the expansion network 52 is composed of four cascaded expansion modules 520 (L=4) as an example, but the embodiment of the present application is not limited to this, and L in the embodiment of the present application can also be other values.

[0092] The U-shaped attention model obtains the essential features with a larger receptive field in the deep layer by downsampling layer by layer, and retains the relatively high-frequency texture features in the shallow layer, taking into account the feature information at both coarse and fine granularity; at the same time, since deconvolution cannot make up for the information lost in the downsampling process, the U-shaped structure adopts jump connections to splice the feature maps of the same scale, supplementing the information of the upsampled features, so that the output features can retain the corresponding edge information.

[0093] In some embodiments, reference Figure 6 As shown, Figure 5 The local window enhanced self-attention layer 501 in the U-shaped structure-based self-attention model shown includes: an attention module 61 and an enhancement module 62.

[0094] The attention module 61 includes: a window division layer 611, a mosaic layer 612, a first normalization layer 613, a multi-head window self-attention layer 614 and a first fusion layer 615, which is used to enhance the input features of the self-attention layer of the local window through the window division layer 611. Perform window division of a preset spatial scale, embed the local two-dimensional relative position code pos in the window into the features in the window through the mosaic layer 612 to obtain mosaic features, process the mosaic features through the first normalization layer 613 and the multi-head window self-attention layer 614 in turn to obtain self-attention features, and fuse the input features of the attention module through the first fusion layer 615 and the self-attention feature to obtain the output feature F of the attention module 61 a .

[0095] Exemplarily, the feature block of the preset spatial scale is a feature block of a spatial scale of 8*8.

[0096] The embedding operation of the mosaic layer 612 is represented as f(), the normalization operation of the first normalization layer 613 is represented as LN(), and the multi-head self-attention calculation of the multi-head window self-attention layer 614 is represented as W_MSA(), then the attention module 61 of the local window enhanced self-attention layer performs the following operations on its input features: The operation can be expressed as the following formula (3):

[0097]

[0098] The enhancement module 62 includes: a second normalization layer 621, a second convolution layer 622, a depth-separable convolution layer 623, a third convolution layer 624, and a second fusion layer 625, which is used to sequentially transform the output feature F of the attention module through the second normalization layer 621, the second convolution layer 622, the depth-separable convolution layer 623, and the third convolution layer 624. a Processing is performed to obtain enhanced features, and the output features F of the attention module 61 are fused through the second fusion layer 625 a and the enhanced features to obtain the output features of the enhancement module 62

[0099] In some embodiments, the second convolution layer 622 and the third convolution layer 624 are both convolution layers with a convolution kernel size of 1*1.

[0100] In some embodiments, the depthwise separable convolution layer 623 is a depthwise separable convolution layer with a convolution kernel of 3*3 and a stride of 1.

[0101] The normalization operation of the second normalization layer 621 is represented as LN(), and the convolution operation of the second convolution layer 622 and the third convolution layer 624 is represented as Conv 1×1 (), the depthwise separable convolution operation of the depthwise separable convolution layer 623 is represented as DWConv 3×3 (), then the local window enhancement module 62 of the self-attention layer has its input features (the output features F of the attention module 61) a ) can be expressed as the following formula (4):

[0102]

[0103] S305: Perform lossy compression on the motion offset to obtain a compression result of the motion offset.

[0104] In some embodiments, the above step S305 (performing lossy compression on the motion offset to obtain a compression result of the motion offset) includes: performing lossy compression on the motion offset through an offset compression model to obtain a compression result of the motion offset.

[0105] Reference Figure 7 As shown, the offset compression model includes: a multi-stage serially connected offset compression module 70 ( Figure 7 In the example, the offset compression model includes three levels of serially connected offset compression modules 70, and any offset compression module 70 includes: an offset convolution layer 71, a second residual network 72, and a second addition fusion unit 73, wherein the second residual network 72 is composed of a plurality of serially connected residual blocks 720. Figure 7 In the figure, the second residual network 72 is taken as an example consisting of three levels of serially connected residual blocks 720).

[0106] The offset convolution layer 71 is used to compress the input features of the offset compression module 70 Process to obtain the second convolution feature F cv , the second residual network 72 is used to transform the second convolution feature F cv Processing is performed to obtain the second residual feature The second addition fusion unit 73 is used to add the second convolution feature F cv and the second residual feature Perform addition fusion to obtain the output features of the offset compression module The second residual network 72 is composed of a plurality of residual blocks 720 connected in series.

[0107] In some embodiments, the offset convolution layer 71 is a convolution layer with a convolution kernel size of 3*3 and a stride of 2.

[0108] In some embodiments, the second residual network 72 is composed of three residual blocks 720 connected in series.

[0109] In some embodiments, Figure 7 As shown, any residual block 720 of the second residual network 72 includes: a convolution layer 721, an activation layer 722, a convolution layer 723, an activation layer 724, and an addition fusion module 725. The residual block 720 processes the input features by: sequentially processing the input features through the convolution layer 721, the activation layer 722, the convolution layer 723, the activation layer 724 Process and obtain residual features Then the input features are fused by the fusion module 725 and residual features Add and fuse to obtain the input features of the residual block 720

[0110] S306: quantize the compression result of the motion offset based on a preset quantization parameter (Quantization Parameter) to obtain a quantization result of the compression result.

[0111] In some embodiments, the preset quantization parameter may be 512.

[0112] S307 . Dequantize the quantization result of the compression result based on the preset quantization parameter to obtain a dequantization result of the quantization result.

[0113] S308: Decompress the inverse quantization result of the quantization result to obtain the reconstructed motion offset.

[0114] The compression operation of the offset compression model is represented as Enc(), and the quantization operation based on the preset quantization parameter is represented as f Q (), the dequantization operation based on the preset quantization parameter is expressed as The decompression operation of the decompression model corresponding to the offset compression model is expressed as Dec(), and the above steps S305 to S308 can be expressed as the following formula (5):

[0115]

[0116] S309: Dynamically adjust the sampling position of the convolution kernel of the deformable convolution layer by using the reconstruction motion offset to sample the reference feature to obtain the initial prediction feature.

[0117] The convolution operation of the deformable convolutional layer is denoted as DConv() and the initial prediction feature is denoted as F pre , the reconstructed motion offset is expressed as Then the above step S309 can be expressed as the following formula (6):

[0118]

[0119] S310: Concatenate the at least one auxiliary feature and the initial prediction feature respectively to obtain at least one first feature.

[0120] In some embodiments, in the above step S310, the at least one auxiliary feature and the initial prediction feature may be concatenated in the feature channel dimension.

[0121] S311. Process the at least one first feature respectively through a self-attention model based on a U-shaped structure to obtain at least one second feature.

[0122] Please refer to the self-attention model based on the U-shaped structure Figure 5 and Figure 6 The U-shaped self-attention model shown will not be repeated here to avoid redundancy.

[0123] S312. Process the at least one second feature respectively through a deformable convolutional network to obtain at least one third feature.

[0124] The concatenation operation is represented as <>, the operation of the self-attention model based on the U-shaped structure on the feature is represented as U_Attn(), and the operation of the deformable convolutional network on the feature is represented as DCN(), then the above steps S310 to S312 can be expressed as the following formula (7):

[0125]

[0126] S313: Concatenate the initial prediction feature and the at least one third feature to obtain a fourth feature.

[0127] S314: Fusing the fourth feature and the initial prediction feature to obtain the prediction feature.

[0128] Reference Figure 8 As shown, Figure 8 At least one auxiliary feature includes as well as As an example, the enhanced fusion model used to execute the above steps S313 and S314 includes: a first splicing layer 81, a second splicing layer 82, a third splicing layer 83, a self-attention model 84 based on a U-shaped structure, a deformable convolutional network 85, a fourth splicing layer 86 and a fusion model 87.

[0129] Among them, the first splicing layer 81 is used to splice the initial prediction features F pre and auxiliary features Generate Features The second concatenation layer 82 is used to concatenate the initial prediction features F pre and auxiliary features Generate Features The third concatenation layer 83 is used to concatenate the initial prediction features F pre and auxiliary features Generate Features The U-shaped self-attention model 84 is used to separately Processing to obtain features The deformable convolutional network 85 is used to classify features Process to obtain feature F enh1 、F enh2 、F enh3 The fourth splicing layer 86 is used for the feature F pre 、F enh1 、F enh2 、F enh3 Perform splicing to generate the fourth feature F enh_cat , the fusion model 87 is used to analyze the fourth feature F enh_cat and initial prediction features F pre Fusion is performed to obtain the prediction feature F pre_enh .

[0130] In some embodiments, the above step S314 (merging the fourth feature and the initial prediction feature to obtain the prediction feature) includes the following steps 3141 to 3146:

[0131] Step 3141: Use the channel attention layer to analyze the fourth feature F enh_cat Processing is performed to obtain weight values ​​of each channel of the fourth feature.

[0132] Step 3142: The fourth feature F is calculated based on the weight values ​​of each channel of the fourth feature. enh_cat The characteristic values ​​of each channel are recalibrated to obtain the fifth characteristic F attn_ch .

[0133] The attention calculation operation of the channel attention layer is represented as Channel_attn(), then the above steps 3141 and 3142 can be expressed as the following formula (8):

[0134] F attn_ch = Channel_attn(F enh_cat )·F enh_cat (8)

[0135] Step 3143: sequentially perform the fourth convolution layer and the second activation layer on the fifth feature F attn_chProcessing is performed to obtain the sixth feature F enh_conv .

[0136] In some embodiments, the fourth convolution layer is a convolution kernel with a convolution kernel size of 1*1.

[0137] In some embodiments, the second activation layer is a ReLU activation layer.

[0138] The convolution process of the fourth convolutional layer is represented as Conv 1×1 (), the activation processing of the second activation layer is represented by ReLU(), then the above step 3143 can be expressed as the following formula (9):

[0139] F enh_conv =ReLU(Conv 1×1 (F attn_ch )) (9)

[0140] Step 3144: Use the spatial attention layer to analyze the sixth feature F enh_conv Processing is performed to obtain the sixth feature F enh_conv The weight value of each spatial position.

[0141] Step 3145: Based on the sixth feature F enh_conv The weight value of each spatial position on the sixth feature F enh_conv The eigenvalues ​​of each spatial position are recalibrated to obtain the seventh feature F attn_sp .

[0142] The attention calculation operation of the spatial attention layer is represented as Spatial_attn(), then the above steps 3144 and 3145 can be expressed as shown in the following formula (10):

[0143] F attn_sp =Spatial_attn(F enh_cat )·F enh_cat (10)

[0144] Step 3146: Initial prediction feature F pre And the seventh feature F attn_sp Perform addition fusion to obtain the prediction feature F are_enh .

[0145] That is, the above step 3146 can be expressed as shown in the following formula (11):

[0146] F pre_enh =F pre +F attn_sp (11)

[0147] Since there are differences in the similarities between different auxiliary features and prediction features, the importance of the enhanced feature information that can be provided is also different. Therefore, the channel attention layer is first used to assign weights to the spliced ​​channels, and then the number of channels is reduced to the number of prediction frame channels through a single layer of convolution. After activation through the activation layer, the spatial attention layer is used to assign different weights to the spatial positions. Finally, the enhanced part of the information after information fusion is added to the prediction frame features to complete the detail supplement of the coarse-grained prediction frame and form a prediction frame with enhanced quality under fine-grained conditions.

[0148] In some embodiments, reference Fig. 9 As shown, Figure 8 The fusion model 87 shown includes: a channel attention layer 91, a channel recalibration module 92, a fourth convolutional layer 93, a second activation layer 94, a spatial attention layer 95, a spatial recalibration module 96 and an additive fusion module 97.

[0149] Among them, the channel attention layer 91 is used to enh_cat Processing is performed to obtain the weight values ​​of each channel of the fourth feature, and the channel recalibration module 92 is used to recalibrate the fourth feature F based on the weight values ​​of each channel of the fourth feature. enh_cat The characteristic values ​​of each channel are recalibrated to obtain the fifth characteristic F attn_ch The fourth convolutional layer 93 and the second activation layer 94 are used to sequentially attn_ch Processing is performed to obtain the sixth feature F enh_conv The spatial attention layer 95 is used to process the sixth feature to obtain the weight value of each spatial position of the sixth feature, and the spatial recalibration module 96 is used to calculate the weight value of each spatial position of the sixth feature based on the sixth feature F enh_conv The weight value of each spatial position on the sixth feature F enh_conv The eigenvalues ​​of each spatial position are recalibrated to obtain the seventh feature F attn_sp The addition fusion module 97 is used to add the initial prediction feature F pre And the seventh feature F attn_sp Perform addition fusion to obtain the prediction feature F pre_enh .

[0150] S315: Calculate the residual between the current feature and the predicted feature to obtain the predicted residual.

[0151] Denote the prediction residual as F resi , then the above step S314 can be expressed as the following formula (12):

[0152] F resi =F t -F pre_enh (12)

[0153] S316: Perform lossy compression, quantization, and entropy coding on the motion offset in sequence to obtain first coded data.

[0154] Since the motion offset mv has been calculated and reconstructed in step S308, off Lossy compression and quantization are performed, so step S316 only needs to perform entropy coding on the quantized result of the motion offset based on step S308.

[0155] The entropy coding is represented as Entropy(), and the first coded data is represented as b mv , then the above step S316 can be expressed as shown in the following formula (13):

[0156] b mv =Entropy(f Q (Enc(mv off ))) (13)

[0157] S317: Encode the prediction residual based on the spatial domain priori information in the prediction feature to obtain second encoded data.

[0158] In some embodiments, the above step S317 (encoding the prediction residual based on the spatial domain prior information in the prediction feature to obtain the second encoded data) includes the following steps 3171 to 3174:

[0159] Step 3171: respectively calculate the prediction residual F resi and the prediction feature F pre_enh Block division is performed to obtain a first feature block set corresponding to the prediction residual and a second feature block set corresponding to the prediction feature.

[0160] Step 3172: Based on the first linear mapping function f embed-q (), the second linear mapping function f embed-k () and the third linear mapping function f embed-v () Linearly map the feature blocks in the first feature block set to obtain a first query vector Q of the first feature block set R , the first key vector K R and the value vector V R .

[0161] The block division operation is represented as: PatchEmbed(), and the linear mapping operation of the first linear mapping function is represented as f embed-q (), the linear mapping operation of the second linear mapping function is expressed as f embed-k (), the linear mapping operation of the third linear mapping function is expressed as fembed-v (), then the above steps 3171 and 3172 can be expressed as the following formulas (14) to (16):

[0162] Q R =f embed-q (PatchEmbed(F resi )) (14)

[0163] K R =f embed-k (PatchEmbed(F resi )) (15)

[0164] V R =f embed-v (PatchEmbed(F resi )) (16)

[0165] Step 3173: Based on the first linear mapping function f embed-q () and the second linear mapping function f embed-k () Linearly map the feature blocks in the second feature block set to obtain a second query vector Q of the second feature block set p and the second key vector K p .

[0166] The block division operation is represented as: PatchEmbed(), and the linear mapping operation of the first linear mapping function is represented as f embed-q (), the linear mapping operation of the second linear mapping function is expressed as f embed-k (), then the above steps 3171 and 3173 can be expressed as the following formulas (17) and (18):

[0167] Q P =f embed-q (PatchEmbed(F pre_enh )) (17)

[0168] K P =f embed-k (PatchEmbed(F pre_enh )) (18)

[0169] Step 3174: According to the first query vector Q R 、The first key vector K R , the value vector V R , the second query vector Q P , the second key vector K P , the spatial relative position encoding pos of the prediction feature band adaptive adjustment coefficient modu, to obtain the second coded data b resi .

[0170] In some embodiments, step 3174 (according to the first query vector Q R 、The first key vector K R , the value vector V R , the second query vector Q P , the second key vector K P , the spatial relative position encoding pos of the prediction feature b and an adaptive adjustment coefficient modulator, to obtain the second coded data b resi ),include:

[0171] Through the residual encoder based on spatiotemporal fusion, the first query vector Q R 、The first key vector K R , the value vector V R , the second query vector QP, the second key vector K P , the spatial relative position encoding pos of the prediction feature b And the adaptive adjustment coefficient modu, obtain the prediction residual F resi The compression result F resi-o , the compression result F of the prediction residual resi-o Quantization and entropy coding are performed to obtain the second coded data b resi .

[0172] In some embodiments, reference Fig.10 As shown, the residual encoder based on spatiotemporal fusion includes: N coding units 12 ( Fig.10 The residual encoder based on spatiotemporal fusion includes 4 encoding units 12 cascaded through block fusion units 11, N=4). Any of the encoding units 12 is composed of a plurality of serially connected attention blocks 120; the n-th level encoding unit is used to process the input features of the n-th level encoding unit through the plurality of serially connected attention layers to obtain the output features of the n-th level encoding unit, and input the output features of the n-th level encoding unit into the block fusion unit cascading the n-th level encoding unit and the n+1-th level encoding unit, and the attention block 120 is used to process the input features of the n-th level encoding unit according to the first query vector Q R and the first key vector K R Calculate the attention value Q R K R , according to the second query vector Q p and the second key vector K p Calculate the spatial prior information Qp K p , integrating the spatial domain prior information Q p K p , the spatial relative position encoding pos of the prediction feature b , adaptive adjustment coefficient adjustment modl and the attention value Q R K R , obtain fusion features, process the fusion features through the normalized exponential function softmax(), obtain normalized features, and calculate the normalized features and the value vector V R The block fusion unit 11 is used to downsample the output features of the previous encoding unit by inter-block reorganization to obtain the output features of the block fusion unit, input the output features of the block fusion unit into the next encoding unit, and perform the second query vector Q p and the second key vector K p Downsampling and normalization operations are performed to make the second query vector Q p and the second key vector K p The size of matches the size of the output feature of the block fusion unit 11.

[0173] In some embodiments, the encoding unit 12 may be a transformer block.

[0174] In some embodiments, the block fusion unit 11 is used to downsample the output features of the previous encoding unit, the second query vector, and the second key vector to 1 / 2 of the original size.

[0175] In some embodiments, the residual encoder based on spatiotemporal fusion includes: 4 encoding units 12 cascaded through block fusion 11 units;

[0176] The first-level coding unit, the second-level coding unit, and the fourth-level coding unit are all composed of 2 serially connected attention blocks, and the third-level coding unit is composed of 6 serially connected attention blocks. That is, the coding units in the residual encoder based on spatiotemporal fusion are 2, 2, 6, and 2 respectively.

[0177] The normalization operation based on the normalized exponential function is represented as softmax(), and the output feature of the attention block is represented as Attn st-r , then the processing of features by attention block 120 can be expressed as the following formula (19):

[0178] Attn st-r =softmax(Q R K R +pos b +Qp K p +modu)V R (19)

[0179] The operation of downsampling the feature map by inter-block reorganization is represented by PatchMerge(), and the output of the previous encoding unit 12 of the block fusion unit 11 is represented by Attn st-r(i-1) (F resi(i-1) , F pre_enh(i-1) ), the output of the block fusion unit 11 is expressed as F resi(i) , then the operation of the block fusion unit 11 on the input features can be expressed as the following formula (20):

[0180]

[0181] The second query vector downsampled by the previous block fusion unit 11 of the current block fusion unit 11 is represented as Q p(i-1) , the second query vector after downsampling by the current block fusion unit 11 is represented as Q p(i) , the normalization operation is represented by LN(), and the linear mapping downsampling is represented by Reshape(), then the block fusion unit 11 performs the downsampling of the second query vector Q by the previous block fusion unit 11 p(i-1) The processing can be expressed as the following formula (21):

[0182]

[0183] Similarly, the second key vector K after the current block fusion unit 11 downsamples the previous block fusion unit 11 p(i-1) The processing can be expressed as shown in the following formula (22):

[0184]

[0185] Denote the quantization operation as f Q (), the entropy coding operation is represented by Entropy(), then the compression result F of the prediction residual resi-o Quantization and entropy coding are performed to obtain the second coded data b resi , which can be expressed as shown in the following formula (23):

[0186] b resi =Entropy(f Q (F resi-o )) (twenty three)

[0187] The embodiment of the present application introduces the spatial correlation of the prediction features into the self-attention calculation of residual compression. On the one hand, by presetting the residual block Q R , K RThe traditional inter-block attention score is obtained by multiplying the vectors. On the other hand, the predicted feature inter-block Q p , K p Calculating the attention score provides prior information about spatial correlation. Since the predicted frame information can be obtained through motion vectors and reference frame features at the decoding end, the predicted frame position information can be used to assist the residual features to reduce spatial redundancy and improve its coding efficiency.

[0188] S318. Add the first coded data and the second coded data to the coded data of the current frame.

[0189] That is, the encoded data of the current frame includes first encoded data (a bit stream obtained by lossy compression, quantization and entropy encoding of the motion offset in sequence) and second encoded data (a bit stream obtained by encoding the prediction residual based on the spatial domain prior information in the prediction feature).

[0190] In some embodiments, some embodiments of the present application provide a video encoding device, the video encoding device comprising:

[0191] a memory configured to store a computer program;

[0192] The processor is configured to enable the video encoding device to implement the video encoding method described in any of the above embodiments when calling the computer program.

[0193] In some embodiments, some embodiments of the present application provide a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a computing device, the computing device implements the video encoding method described in any of the above embodiments or the video decoding method described in any of the above embodiments.

[0194] In some embodiments, some embodiments of the present application provide a computer program product, which, when executed on a computer, enables the computer to implement the video encoding method described in any of the above embodiments or the video decoding method described in any of the above embodiments.

[0195] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit it. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some or all of the technical features therein with equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present application.

[0196] For the convenience of explanation, the above description has been made in conjunction with specific embodiments. However, the above exemplary discussion is not intended to be exhaustive or limit the embodiments to the specific forms disclosed above. Based on the above teachings, various modifications and variations can be obtained. The selection and description of the above embodiments are to better explain the principles and practical applications, so that those skilled in the art can better use the embodiments and various different variations of the embodiments suitable for specific use considerations.

Claims

1. A video encoding method, characterized in that: include: Perform feature extraction on the current frame and the reconstructed reference frame to obtain current features and reference features; Performing splicing processing on the current feature and the reference feature to obtain a combined feature; Performing convolution processing on the merged features to obtain preprocessed features with a preset number of channels; Processing the preprocessed features through a self-attention model based on a U-shaped structure to obtain a motion offset between the current frame and the reconstructed reference frame; The encoded data of the current frame is obtained according to the motion offset.

2. The method according to claim 1, characterized in that The self-attention model based on the U-shaped structure includes: a contraction network, an expansion network, and a local window enhanced self-attention layer connecting the contraction network and the expansion network; the contraction network is composed of L cascaded contraction modules, and the expansion network is composed of L cascaded expansion modules; The contraction module includes: a local window enhanced self-attention layer and a downsampling layer connected in series, the i-th level contraction module of the contraction network is used to process the output features of the i-1-th level contraction module through the local window enhanced self-attention layer, obtain the jump connection features of the i-th level contraction module, input the jump connection features into the L-i+1-th level expansion module of the expansion network, downsample the jump connection features through the downsampling layer to obtain the output features of the i-th level contraction module, and input the output features of the i-th level contraction module into the i+1-th level contraction module; The expansion module includes: a serially connected upsampling layer, a splicing layer and a local window enhanced self-attention layer. The j-th expansion module of the expansion network is used to upsample the output features of the j-1-th expansion module through the upsampling layer to obtain the expansion features of the j-th expansion module, splice the expansion features and the jump connection features of the L-j+1-th contraction module of the contraction network through the splicing layer to obtain the splicing features of the j-th expansion module, process the splicing features through the local window enhanced self-attention layer to obtain the output features of the j-th expansion module, and input the output features of the j-th contraction module into the j+1-th expansion module.

3. The method according to claim 2, characterized in that The local window enhanced self-attention layer includes: an attention module and an enhancement module; The attention module includes: a window division layer, a mosaic layer, a first normalization layer, a multi-head window self-attention layer and a first fusion layer, which is used to perform window division of a preset spatial scale on the input features of the local window enhanced self-attention layer through the window division layer, embed the local two-dimensional relative position code in the window into the features in the window through the mosaic layer to obtain mosaic features, process the mosaic features through the first normalization layer and the multi-head window self-attention layer in turn to obtain self-attention features, and fuse the input features of the attention module and the self-attention features through the first fusion layer to obtain output features of the attention module; The enhancement module includes: a second normalization layer, a second convolution layer, a depth-separable convolution layer, a third convolution layer and a second fusion layer, which is used to process the output features of the attention module through the second normalization layer, the second convolution layer, the depth-separable convolution and the third convolution layer in sequence to obtain enhanced features, and fuse the output features of the attention module and the enhanced features through the second fusion layer to obtain the output features of the enhancement module.

4. The method according to claim 1, characterized in that: The extracting features of the current frame and the reconstructed reference frame to obtain current features and reference features includes: Performing feature extraction on the current frame and the reconstructed reference frame based on a feature extraction model to obtain the current feature and the reference feature; Among them, the feature extraction model includes: a first convolutional layer, a first activation layer, a first residual network and a first additive fusion unit, and the first residual network is composed of a plurality of residual blocks connected in series; the feature extraction model is used to process the input video frame of the feature extraction model through the first convolutional layer and the first activation layer in sequence to obtain a first convolutional feature, process the first convolutional feature through the first residual network to obtain a first residual feature, and additively fuse the first convolutional feature and the first residual feature through the first additive fusion unit to obtain the output feature of the feature extraction model.

5. The method according to claim 1, characterized in that The step of obtaining the encoded data of the current frame according to the motion offset comprises: Obtaining a reconstructed motion offset according to the motion offset, wherein the reconstructed motion offset is a motion offset obtained by encoding the motion offset and decoding the encoding result of the motion offset; Acquire a prediction feature according to the reference feature and the reconstructed motion offset; Obtaining a prediction residual according to the current feature and the predicted feature; Generate encoded data of the current frame according to the motion offset and the prediction residual.

6. The method according to claim 5, characterized in that The obtaining the reconstructed motion offset according to the motion offset comprises: Performing lossy compression on the motion offset to obtain a compression result of the motion offset; quantizing the compression result of the motion offset based on a preset quantization parameter to obtain a quantization result of the compression result; Dequantizing the quantization result of the compression result based on the preset quantization parameter to obtain a dequantization result of the quantization result; The inverse quantization result of the quantization result is decompressed to obtain the reconstructed motion offset.

7. The method according to claim 6, characterized in that The performing lossy compression on the motion offset to obtain a compression result of the motion offset includes: Performing lossy compression on the motion offset by using an offset compression model to obtain a compression result of the motion offset; Among them, the offset compression model includes: a multi-stage serially connected offset compression module, any offset compression module includes: an offset convolution layer, a second residual network and a second additive fusion unit, the second residual network is composed of a plurality of serially connected residual blocks; the offset convolution layer is used to process the input features of the offset compression module to obtain a second convolution feature, the second residual network is used to process the second convolution feature to obtain a second residual feature; the second additive fusion unit is used to perform additive fusion on the second convolution feature and the second residual feature to obtain the output feature of the offset compression module.

8. The method according to claim 5, characterized in that The obtaining of the initial prediction feature according to the reference feature and the reconstructed motion offset comprises: Dynamically adjust the sampling position of the convolution kernel of the feasible variable convolution layer by the reconstruction motion offset to sample the reference feature to obtain the initial prediction feature; The initial prediction feature is fused and enhanced based on at least one auxiliary feature to obtain a prediction feature, and the at least one auxiliary feature is a feature obtained by extracting features from at least one reconstructed auxiliary reference frame.

9. The method according to claim 8, characterized in that The fusing and enhancing the initial prediction feature based on at least one auxiliary feature to obtain the prediction feature includes: respectively concatenating the at least one auxiliary feature and the initial prediction feature to obtain at least one first feature; Processing the at least one first feature respectively through a self-attention model based on a U-shaped structure to obtain at least one second feature; Processing the at least one second feature respectively through a deformable convolutional network to obtain at least one third feature; Concatenating the initial prediction feature and the at least one third feature to obtain a fourth feature; The fourth feature and the initial prediction feature are fused to obtain the prediction feature.

10. The method according to claim 9, characterized in that The fusing the fourth feature and the initial prediction feature to obtain the prediction feature includes: Processing the fourth feature through a channel attention layer to obtain weight values ​​of each channel of the fourth feature; Recalibrating the feature values ​​of each channel of the fourth feature based on the weight values ​​of each channel of the fourth feature to obtain a fifth feature; The fifth feature is processed by a fourth convolutional layer and a second activation layer in sequence to obtain a sixth feature; Processing the sixth feature through a spatial attention layer to obtain weight values ​​of each spatial position of the sixth feature; Recalibrating the feature values ​​of each spatial position of the sixth feature based on the weight values ​​of each spatial position of the sixth feature to obtain a seventh feature; The initial prediction feature and the seventh feature are added and fused to obtain the prediction feature.

11. The method according to claim 5, characterized in that The obtaining a prediction residual according to the current feature and the prediction feature includes: A residual between the current feature and the predicted feature is calculated to obtain the prediction residual.

12. The method according to claim 5, characterized in that The step of generating the coded data of the current frame according to the motion offset and the prediction residual comprises: performing lossy compression, quantization, and entropy coding on the motion offset in sequence to obtain first coded data; Encoding the prediction residual based on the spatial domain priori information in the prediction feature to obtain second encoded data; The first encoded data and the second encoded data are added to the encoded data of the current frame.

13. The method according to claim 12, characterized in that The step of encoding the prediction residual based on the spatial domain priori information in the prediction feature to obtain second encoded data includes: Performing block division on the prediction residual and the prediction feature respectively to obtain a first feature block set corresponding to the prediction residual and a second feature block set corresponding to the prediction feature; Performing linear mapping on the feature blocks in the first feature block set based on a first linear mapping function, a second linear mapping function, and a third linear mapping function, respectively, to obtain a first query vector, a first key vector, and a value vector of the first feature block set; Performing linear mapping on the feature blocks in the second feature block set based on the first linear mapping function and the second linear mapping function respectively, so as to obtain a second query vector and a second key vector of the second feature block set; The second encoded data is obtained according to the first query vector, the first key vector, the value vector, the second query vector, the second key vector, the spatial relative position encoding of the prediction feature, and the adaptive adjustment coefficient.

14. The method according to claim 13, characterized in that The acquiring the second encoded data according to the first query vector, the first key vector, the value vector, the second query vector, the second key vector, the spatial relative position encoding of the prediction feature, and the adaptive adjustment coefficient includes: Obtaining a compression result of the prediction residual through a residual encoder based on spatiotemporal fusion, the first query vector, the first key vector, the value vector, the second query vector, the second key vector, spatial relative position encoding of the prediction feature, and an adaptive adjustment coefficient, and quantizing and entropy encoding the compression result of the prediction residual to obtain the second encoded data; The residual encoder based on spatiotemporal fusion comprises: N coding units cascaded through block fusion units, any of the coding units is composed of a plurality of serially connected attention blocks; the n-th level coding unit is used to process the input features of the n-th level coding unit through the plurality of serially connected attention layers to obtain the output features of the n-th level coding unit, and input the output features of the n-th level coding unit into the block fusion unit cascading the n-th level coding unit and the n+1-th level coding unit, the attention block is used to calculate the attention value according to the first query vector and the first key vector, calculate the spatial domain prior information according to the second query vector and the second key vector, fuse the spatial domain prior information, the spatial relative position of the predicted feature, and calculate the spatial domain prior information. The method comprises the steps of: adjusting the setting encoding, adaptively adjusting the coefficient and the attention value, obtaining a fusion feature, processing the fusion feature through a normalized exponential function, obtaining a normalized feature, and calculating the product of the normalized feature and the value vector to obtain the output feature of the attention block; the block fusion unit is used to downsample the output feature of the previous level coding unit through inter-block reorganization to obtain the output feature of the block fusion unit, input the output feature of the block fusion unit to the next level coding unit, and downsample and normalize the second query vector and the second key vector through linear projection to match the size of the second query vector and the second key vector with the size of the output feature of the block fusion unit.

15. The method according to claim 14, characterized in that The residual encoder based on spatiotemporal fusion includes: 4 encoding units cascaded through block fusion units; The first-level coding unit, the second-level coding unit, and the fourth-level coding unit are each composed of two serially connected attention blocks, and the third-level coding unit is composed of six serially connected attention blocks.

16. A video encoding device, characterized in that: include: a memory configured to store a computer program; The processor is configured to enable the video encoding device to implement the video encoding method according to any one of claims 1 to 15 when calling a computer program.

Images