Context enhancement method, apparatus, device, and medium

Abstract

The application discloses a context enhancement method, device, equipment and medium, and belongs to the technical field of video compression. Based on a to-be-compressed frame and a previously decoded frame, a reconstructed optical flow is obtained; multi-scale feature extraction is performed on the features of the previously decoded frame to obtain multi-scale features; the reconstructed optical flow and the multi-scale features are input into a context generation module to obtain multi-scale contexts, and local enhancement and global enhancement are respectively performed on the multi-scale contexts according to the deformable convolution offset and the cross attention of the multi-scale contexts and the multi-scale features; and the multi-scale contexts after local enhancement and the multi-scale contexts after global enhancement are fused to obtain multi-scale enhanced contexts. The multi-scale contexts are respectively subjected to local enhancement and global enhancement through the multi-scale features of the previously decoded frame, the to-be-compressed frame is enhanced without code rate overhead, and the multi-scale enhanced contexts are obtained, so that the quality of video compression is improved, and the efficiency of video compression is improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of video compression, in particular to a context enhancement method, device, equipment and medium. BACKGROUND

[0002] In recent years, deep learning-based video compression technology has also developed rapidly.

[0003] However, the context enhancement method in the current deep learning video compression technology mainly focuses on local inter-frame information, lacks attention to global information, and thus the existing context enhancement method is difficult to handle complex video content.

[0004] The above content is only used to assist in understanding the technical solutions of the present application, and does not represent the acknowledgement of the above content as prior art. SUMMARY

[0005] The main purpose of the present application is to provide a context enhancement method, device, equipment and medium, aiming to improve the video compression performance when processing complex video content.

[0006] To achieve the above purpose, the present application provides a context enhancement method, which comprises:

[0007] Based on the to-be-compressed frame and the previously decoded frame, a reconstructed optical flow is obtained;

[0008] Multi-scale feature extraction is performed on the features of the previously decoded frame to obtain multi-scale features, wherein the multi-scale features include preset minimum scale features;

[0009] The reconstructed optical flow and the multi-scale features are input into a context generation module to obtain multi-scale contexts, wherein the multi-scale contexts include preset minimum scale contexts;

[0010] Through the deformable convolution offset of the multi-scale context and the multi-scale features, and the reconstructed optical flow, the multi-scale features are subjected to optical flow guided deformable convolution compensation to obtain locally enhanced multi-scale contexts;

[0011] Through the cross-attention of the preset minimum scale context and the preset minimum scale features, the multi-scale contexts are globally enhanced to obtain globally enhanced multi-scale contexts;

[0012] The locally enhanced multi-scale contexts and the globally enhanced multi-scale contexts are fused to obtain multi-scale enhanced contexts.

[0013] Optionally, the multi-scale features further include preset medium-scale features and preset minimum-scale features; the step of inputting the reconstructed optical flow and the multi-scale features into a context generation module to obtain multi-scale contexts comprises:

[0014] performing bilinear down-sampling on the reconstructed optical flow to obtain a preset medium-scale reconstructed optical flow and a preset minimum-scale reconstructed optical flow;

[0015] performing compensation on the reconstructed optical flow, the preset medium-scale reconstructed optical flow and the preset minimum-scale reconstructed optical flow respectively through the original-scale features, the preset medium-scale features and the preset minimum-scale features to obtain multi-scale contexts.

[0016] Optionally, the multi-scale contexts further include preset medium-scale contexts and original-scale contexts; the step of performing optical flow guided deformable convolution compensation on the multi-scale features through the multi-scale contexts and deformable convolution offsets of the multi-scale features, and the reconstructed optical flow to obtain locally enhanced multi-scale contexts comprises:

[0017] performing optical flow guided deformable convolution compensation on the original-scale features through the original-scale contexts and deformable convolution offsets of the original-scale features, and the reconstructed optical flow to obtain first-scale contexts;

[0018] performing optical flow guided deformable convolution compensation on the preset medium-scale features through the preset medium-scale contexts and deformable convolution offsets of the preset medium-scale features, and the preset medium-scale reconstructed optical flow to obtain second-scale contexts;

[0019] performing optical flow guided deformable convolution compensation on the preset minimum-scale features through the preset minimum-scale contexts and deformable convolution offsets of the preset minimum-scale features, and the preset minimum-scale reconstructed optical flow to obtain third-scale contexts.

[0020] Optionally, the step of fusing the locally enhanced multi-scale contexts and the globally enhanced multi-scale contexts to obtain multi-scale enhanced contexts comprises:

[0021] fusing the third-scale contexts and globally enhanced preset minimum-scale contexts through channel space attention to obtain fused contexts;

[0022] obtaining a first residual of the first-scale contexts, a second residual of the second-scale contexts and a third residual of the fused contexts through a convolutional neural network;

[0023] The first residual, the second residual and the third residual are added to the first scale context, the second scale context and the fusion context respectively to obtain a multi-scale enhanced context.

[0024] Optionally, the step of obtaining the global enhanced multi-scale context according to the cross attention includes:

[0025] The preset minimum scale context is globally enhanced by the cross attention to obtain a global enhanced preset minimum scale context.

[0026] The global enhanced preset minimum scale context is taken as a new preset minimum scale context, and a new cross attention is obtained according to the new preset minimum scale context, and the step of globally enhancing the preset minimum scale context by the cross attention is continuously performed for a preset number of times to obtain the preset minimum scale context globally enhanced for a preset number of times.

[0027] Optionally, the step of performing multi-scale feature extraction on the features of the previously decoded frame to obtain a multi-scale feature includes:

[0028] The features of the previously decoded frame are input into a pyramid convolutional neural network to obtain a multi-scale feature.

[0029] Optionally, the step of obtaining a reconstructed optical flow based on the to-be-compressed frame and the previously decoded frame includes:

[0030] The to-be-compressed frame and the previously decoded frame are input into an optical flow estimation network to obtain an optical flow.

[0031] The optical flow is input into an optical flow encoder and an optical flow decoder to obtain a reconstructed optical flow.

[0032] The application also provides a context enhancement device, which includes:

[0033] A reconstruction module is configured to obtain a reconstructed optical flow based on a to-be-compressed frame and a previously decoded frame.

[0034] An extraction module is configured to perform multi-scale feature extraction on features of the previously decoded frame to obtain a multi-scale feature, and the multi-scale feature includes a preset minimum scale feature.

[0035] A context generation module is configured to input the reconstructed optical flow and the multi-scale feature into the context generation module to obtain a multi-scale context, and the multi-scale context includes a preset minimum scale context.

[0036] a local enhancement module configured to perform optical flow guided deformable convolution compensation on the multi-scale features based on the multi-scale context, the deformable convolution offset of the multi-scale features, and the reconstructed optical flow, to obtain locally enhanced multi-scale context;

[0037] a global enhancement module configured to perform global enhancement on the multi-scale context based on cross attention of the preset minimum scale context and the preset minimum scale feature, to obtain globally enhanced multi-scale context;

[0038] a fusion module configured to fuse the locally enhanced multi-scale context and the globally enhanced multi-scale context, to obtain multi-scale enhanced context.

[0039] The application also provides a context enhancement device, which comprises a memory, a processor, and a computer program stored on the memory and executable on the processor, and the computer program is configured to implement the steps of the context enhancement method according to any one of the preceding embodiments.

[0040] The application also provides a medium, which stores a computer program, and the computer program is executable on a processor to implement the steps of the context enhancement method according to any one of the preceding embodiments.

[0041] The application provides a context enhancement method, device, equipment and medium, based on a to-be-compressed frame and a previously decoded frame, a reconstructed optical flow is obtained; multi-scale feature extraction is performed on the features of the previously decoded frame to obtain multi-scale features, the multi-scale features include preset minimum scale features; the reconstructed optical flow and the multi-scale features are input into a context generation module to obtain multi-scale context, the multi-scale context includes preset minimum scale context; optical flow guided deformable convolution compensation is performed on the multi-scale features based on the multi-scale context, the deformable convolution offset of the multi-scale features, and the reconstructed optical flow, to obtain locally enhanced multi-scale context; global enhancement is performed on the multi-scale context based on cross attention of the preset minimum scale context and the preset minimum scale feature, to obtain globally enhanced multi-scale context; and the locally enhanced multi-scale context and the globally enhanced multi-scale context are fused to obtain multi-scale enhanced context.

[0042] The context enhancement method in the conventional deep learning video compression technology mainly focuses on the local inter-frame information in the context, and ignores the global information in the context. After compression of complex video content, important details and motion features in the global information are lost, resulting in low performance of the existing video compression method in processing complex video content.

[0043] Unlike conventional techniques, the application provides a context enhancement method, device, equipment and medium. First, a reconstructed optical flow is obtained based on a to-be-compressed frame and a previously decoded frame, so as to reduce redundant data. Then, a multi-scale feature of the previously decoded frame is obtained, and a multi-scale context is obtained based on the reconstructed optical flow and the multi-scale feature. The multi-scale context is obtained by extracting the multi-scale feature of the previously decoded frame, so that the local and global information of the previously decoded frame is fully utilized, and redundant code rate overhead is avoided. Next, a deformable convolution offset and cross attention of the multi-scale context and the multi-scale feature are calculated. The local information extraction capability of the deformable convolution offset and the global information extraction capability of the cross attention are used to respectively perform local enhancement and global enhancement on the multi-scale context, so as to obtain a minimum scale context after global enhancement and a multi-scale context after local enhancement. The minimum scale context after global enhancement and the multi-scale context after local enhancement are fused to obtain a final multi-scale enhanced context. Thus, the inter-frame information of the context is focused on, and the global information in the context is retained. In this way, the problem that the global information is lost when the existing video compression method processes complex video content, resulting in low video compression quality, is solved, and the video compression performance is improved. BRIEF DESCRIPTION OF DRAWINGS

[0044] Figure 1 is a structural schematic diagram of a context enhancement device of a hardware running environment related to an embodiment scheme of the application;

[0045] Figure 2 is a flow schematic diagram of a first embodiment of the application;

[0046] Figure 3 is a schematic diagram of a local and global context enhancement method related to an embodiment scheme of the application;

[0047] Figure 4 is a structural schematic diagram of a local and global context fusion module related to an embodiment scheme of the application;

[0048] Figure 5 is a structural schematic diagram of a context generation module related to an embodiment scheme of the application;

[0049] Figure 6 is a structural schematic diagram of a local context enhancement module related to an embodiment scheme of the application;

[0050] Figure 7 is a structural schematic diagram of a global context enhancement module related to an embodiment scheme of the application;

[0051] Figure 8 is a schematic diagram of an ablation experiment result related to an embodiment scheme of the application;

[0052] Figure 9A functional module diagram related to the embodiment of the present application is shown.

[0053] The implementation, functional features and advantages of the present application will be further described with reference to the embodiments and the accompanying drawings. DETAILED DESCRIPTION

[0054] It should be understood that the specific embodiments described herein merely exemplify the present application and do not limit the present application.

[0055] Reference Figure 1 , Figure 1 A context-enhanced device structure diagram is shown for the hardware running environment related to the embodiment of the present application.

[0056] As shown in Figure 1 , the context-enhanced device can include a processor 1001, such as a central processing unit (CPU), a communication bus 1002, a user interface 1003, a network interface 1004, and a memory 1005. The communication bus 1002 is used to realize the connection and communication between these components. The user interface 1003 can include a display screen, an input unit such as a keyboard, and can also include a standard wired interface, a wireless interface. The network interface 1004 can optionally include a standard wired interface, a wireless interface (such as a wireless fidelity (WIreless-FIdelity, WI-FI) interface). The memory 1005 can be a high-speed random access memory (RAM) memory, or a stable non-volatile memory (Non-Volatile Memory, NVM), such as a magnetic disk memory. The memory 1005 can also be a storage device independent of the aforementioned processor 1001.

[0057] Those skilled in the art can understand that Figure 1 the structure shown in the figure does not constitute a limitation on the context-enhanced device, and can include more or fewer components than the figure, or combine certain components, or different component arrangements.

[0058] As shown in Figure 1 , the memory 1005 as a storage medium can include an operating system, a network communication module, a user interface module, and a computer program.

[0059] In Figure 1In the illustrated context enhancement device, the network interface 1004 is mainly used for data communication with other devices; the user interface 1003 is mainly used for data interaction with the user; the processor 1001 and the memory 1005 in the context enhancement device of the present application can be arranged in the context enhancement device, the processor 1001 calls the computer program stored in the memory 1005, and executes the context enhancement method provided by the embodiments of the present application:

[0060] Based on the to-be-compressed frame and the previously decoded frame, a reconstructed optical flow is obtained;

[0061] Multi-scale feature extraction is performed on the features of the previously decoded frame to obtain multi-scale features, the multi-scale features including preset minimum scale features;

[0062] The reconstructed optical flow and the multi-scale features are input into a context generation module to obtain multi-scale contexts, the multi-scale contexts including preset minimum scale contexts;

[0063] The multi-scale features are subjected to optical flow guided deformable convolution compensation through a deformable convolution offset of the multi-scale contexts and the multi-scale features, and the reconstructed optical flow, to obtain locally enhanced multi-scale contexts;

[0064] The multi-scale contexts are subjected to global enhancement through cross-attention of the preset minimum scale contexts and the preset minimum scale features, to obtain globally enhanced multi-scale contexts;

[0065] The locally enhanced multi-scale contexts and the globally enhanced multi-scale contexts are fused to obtain multi-scale enhanced contexts.

[0066] In an embodiment, the processor 1001 can call the computer program stored in the memory 1005, and further perform the following operations:

[0067] The multi-scale features further include preset medium scale features and preset minimum scale features; the step of inputting the reconstructed optical flow and the multi-scale features into the context generation module to obtain multi-scale contexts includes:

[0068] The reconstructed optical flow is subjected to bilinear down-sampling to obtain a preset medium scale reconstructed optical flow and a preset minimum scale reconstructed optical flow;

[0069] The reconstructed optical flow, the preset medium scale reconstructed optical flow, and the preset minimum scale reconstructed optical flow are compensated through the original scale features, the preset medium scale features, and the preset minimum scale features, respectively, to obtain multi-scale contexts.

[0070] In an embodiment, the processor 1001 can invoke a computer program stored in the memory 1005, and further perform the following operations:

[0071] The multiscale context further includes a preset medium scale context and an original scale context; the step of performing flow-guided deformable convolution compensation on the multiscale feature by means of the deformable convolution offset of the multiscale context and the multiscale feature, and the reconstructed optical flow, to obtain a locally enhanced multiscale context, includes:

[0072] Performing flow-guided deformable convolution compensation on the original scale feature by means of the deformable convolution offset of the original scale context and the original scale feature, and the reconstructed optical flow, to obtain a first scale context;

[0073] Performing flow-guided deformable convolution compensation on the preset medium scale feature by means of the deformable convolution offset of the preset medium scale context and the preset medium scale feature, and the reconstructed optical flow of the preset medium scale, to obtain a second scale context;

[0074] Performing flow-guided deformable convolution compensation on the preset minimum scale feature by means of the deformable convolution offset of the preset minimum scale context and the preset minimum scale feature, and the reconstructed optical flow of the preset minimum scale, to obtain a third scale context.

[0075] In an embodiment, the processor 1001 can invoke a computer program stored in the memory 1005, and further perform the following operations:

[0076] The step of fusing the locally enhanced multiscale context and the globally enhanced multiscale context to obtain a multiscale enhanced context, includes:

[0077] Fusing the third scale context and the globally enhanced preset minimum scale context by means of channel space attention to obtain a fused context;

[0078] Obtaining, by means of a convolutional neural network, a first residual of the first scale context, a second residual of the second scale context, and a third residual of the fused context;

[0079] Adding the first residual, the second residual, and the third residual to the first scale context, the second scale context, and the fused context, respectively, to obtain a multiscale enhanced context.

[0080] In an embodiment, the processor 1001 can invoke a computer program stored in the memory 1005, and further perform the following operations:

[0081] The step of obtaining the multi-scale context enhanced globally according to the cross attention comprises:

[0082] The preset minimum scale context is globally enhanced by the cross attention to obtain the preset minimum scale context enhanced globally.

[0083] The preset minimum scale context enhanced globally is taken as a new preset minimum scale context, and a new cross attention is obtained according to the new preset minimum scale context, and the step of globally enhancing the preset minimum scale context by the cross attention is continuously performed for a preset number of times to obtain the preset minimum scale context enhanced globally for a preset number of times.

[0084] In an embodiment, the processor 1001 can invoke the computer program stored in the memory 1005, and further perform the following operations:

[0085] The step of performing multi-scale feature extraction on the features of the previously decoded frame to obtain multi-scale features comprises:

[0086] The features of the previously decoded frame are input into a pyramid convolutional neural network to obtain multi-scale features.

[0087] In an embodiment, the processor 1001 can invoke the computer program stored in the memory 1005, and further perform the following operations:

[0088] The step of obtaining the reconstructed optical flow based on the to-be-compressed frame and the previously decoded frame comprises:

[0089] The to-be-compressed frame and the previously decoded frame are input into an optical flow estimation network to obtain an optical flow.

[0090] The optical flow is input into an optical flow encoder and an optical flow decoder to obtain a reconstructed optical flow.

[0091] Embodiments of the present application provide a context enhancement method, referring to Figure 2 , Figure 2 The flowchart of a first embodiment of the context enhancement method of the present application.

[0092] In the embodiment, the context enhancement method comprises:

[0093] Step S10: obtaining a reconstructed optical flow based on a to-be-compressed frame and a previously decoded frame;

[0094] In the embodiment, the reconstructed optical flow is obtained based on a previously decoded frame and a frame to be compressed, wherein the previously decoded frame refers to a video frame that has been compressed and decoded, the frame to be compressed refers to a video frame that needs to be compressed at present, and the optical flow refers to displacement or motion information of each pixel over time. The optical flow estimation and optical flow recovery operations are performed on the optical flow to obtain the reconstructed optical flow.

[0095] Step S20: performing multi-scale feature extraction on the features of the previously decoded frame to obtain multi-scale features, wherein the multi-scale features include preset minimum scale features.

[0096] In the embodiment, the features of the previously decoded frame are obtained, and multi-scale feature extraction is performed on the features of the previously decoded frame to obtain multi-scale features of the features of the previously decoded frame. The multi-scale features at least include preset minimum scale features. For example, the multi-scale features of the previously decoded frame include original scale features, preset medium scale features and preset minimum scale features, wherein the original scale features are the features of the previously decoded frame; the original scale > the preset medium scale > the preset minimum scale. Optionally, the preset medium scale is 1 / 2 scale, and the preset minimum scale is 1 / 4 scale, which can be adjusted according to actual conditions. For example, the preset medium scale is adjusted to 1 / 4 scale, and the preset minimum scale is adjusted to 1 / 8 scale. The feature extraction methods include but are not limited to image preprocessing-based feature extraction methods, deep learning-based feature extraction methods and local feature descriptor-based feature extraction methods. The multi-scale feature extraction methods include but are not limited to scale space feature extraction methods, pyramid feature extraction methods and convolutional neural network feature extraction methods.

[0097] Step S30: inputting the reconstructed optical flow and the multi-scale features into a context generation module to obtain multi-scale contexts, wherein the multi-scale contexts include preset minimum scale contexts.

[0098] In the embodiment, the reconstructed optical flow and the multi-scale features of the previously decoded frame are input into the context generation module to obtain multi-scale contexts. The multi-scale contexts at least include preset minimum scale contexts. For example, the multi-scale contexts include original scale contexts, preset medium scale contexts and preset minimum scale contexts. The preset minimum scale is 1 / 4 scale, and the preset medium scale is 1 / 2 scale, which can be adjusted according to actual conditions. For details, refer to the description of the context generation module in the first aspect. Figure 3 Figure 3 The input image x i The input image x is the frame to be compressed involved in the embodiment. The reconstructed optical flow and the multi-scale features are input into the context generation module to obtain multi-scale contexts, and the deformable convolution offset and the cross attention of the multi-scale contexts and the multi-scale features are calculated. The context generation module aligns the multi-scale features with the input image based on the reconstructed optical flow. Specifically, in the context generation module, the multi-scale features are used to compensate the reconstructed optical flow to obtain the multi-scale contexts.​

[0099] Step S40: performing optical flow guided deformable convolution compensation on the multi-scale features according to the reconstructed optical flow and the deformable convolution offset of the multi-scale features, to obtain locally enhanced multi-scale contexts;

[0100] In this embodiment, after obtaining the deformable convolution offset of the multi-scale contexts and the multi-scale features by calculation, the multi-scale contexts are globally enhanced according to the cross attention of the multi-scale contexts and the multi-scale features, to obtain globally enhanced multi-scale contexts.

[0101] Step S50: performing global enhancement on the multi-scale contexts according to the cross attention of the preset minimum scale contexts and the preset minimum scale features, to obtain globally enhanced multi-scale contexts;

[0102] In this embodiment, after obtaining the cross attention of the multi-scale contexts and the multi-scale features by calculation, the multi-scale contexts are globally enhanced according to the cross attention of the multi-scale contexts and the multi-scale features.

[0103] Step S60: fusing the locally enhanced multi-scale contexts and the globally enhanced multi-scale contexts to obtain multi-scale enhanced contexts.

[0104] In this embodiment, after locally and globally enhancing the multi-scale contexts respectively, the multi-scale enhanced contexts are obtained after fusing the locally enhanced multi-scale contexts and the globally enhanced multi-scale contexts, for reference Figure 3 The multi-scale contexts and the multi-scale features are input into the local and global context enhancement module to obtain the locally enhanced multi-scale contexts and the globally enhanced multi-scale contexts.

[0105] Optionally, the method further comprises:

[0106] The to-be-compressed frame and the multi-scale enhanced contexts are input into the conditional encoder and the conditional decoder to obtain reconstructed intermediate features of the to-be-compressed frame.

[0107] The reconstructed intermediate features are input into the frame generator to obtain a reconstructed image and a reconstructed feature of the to-be-compressed frame.

[0108] In an embodiment, the frame to be compressed and the multi-scale enhanced context are input into a conditional encoder and a conditional decoder to obtain a reconstructed intermediate feature of the frame to be compressed, the reconstructed intermediate feature is input into a frame generator to obtain a reconstructed image and a reconstructed feature of the frame to be compressed, the reconstructed image refers to a decoded image frame, that is, an image obtained after decompression, and the reconstructed feature is a feature representing image content; and the conditional encoder and the conditional decoder are used for compressing image data.

[0109] With reference to Figure 3 , Figure 3 The input image x i is the frame to be compressed involved in the embodiment of the application; the frame to be compressed and the multi-scale enhanced context are input into a conditional encoder and a conditional decoder to obtain a reconstructed intermediate feature of the frame to be compressed, the reconstructed intermediate feature is input into a frame generator to obtain a reconstructed frame of the frame to be compressed, and the reconstructed frame includes a reconstructed image and a reconstructed feature; the conditional encoder, the conditional decoder and the frame generator can be constructed by using a self-encoder, and the frame generator is used for restoring pixel values of the frame to be compressed; in addition to improving the quality and efficiency of video compression by using the context enhancement method provided in the application, the reconstructed image and the reconstructed feature are obtained to ensure the compression effect of the video compression model using the context enhancement method provided in the application.

[0110] The application provides a context enhancement method, device, equipment and medium, based on a frame to be compressed and a previously decoded frame, a reconstructed optical flow is obtained; multi-scale feature extraction is performed on a feature of the previously decoded frame to obtain a multi-scale feature, the multi-scale feature includes a preset minimum scale feature; the reconstructed optical flow and the multi-scale feature are input into a context generation module to obtain a multi-scale context, the multi-scale context includes a preset minimum scale context; through a deformable convolution offset of the multi-scale context and the multi-scale feature, and the reconstructed optical flow, a deformable convolution compensation of the multi-scale feature is performed under the guidance of the optical flow to obtain the multi-scale context after local enhancement; through cross attention of the preset minimum scale context and the preset minimum scale feature, global enhancement is performed on the multi-scale context to obtain the multi-scale context after global enhancement; and the multi-scale context after local enhancement and the multi-scale context after global enhancement are fused to obtain a multi-scale enhanced context.

[0111] The context enhancement method in the conventional deep learning video compression technology mainly focuses on the local inter-frame information in the context, and ignores the global information in the context. After complex video content is compressed, important details and motion features in the global information are lost, resulting in that the existing video compression method has low video compression quality when processing complex video content.

[0112] Different from the conventional technology, the application provides a context enhancement method, device, equipment and medium, which first acquires a reconstructed optical flow based on a to-be-compressed frame and a previously decoded frame to reduce redundant data, then acquires a multi-scale feature of the previously decoded frame, and obtains a multi-scale context based on the reconstructed optical flow and the multi-scale feature, the multi-scale context is obtained by extracting the multi-scale feature of the previously decoded frame, the local and global information of the previously decoded frame is fully utilized, and redundant code rate overhead is avoided; then, a deformable convolution offset and cross attention of the multi-scale context and the multi-scale feature are calculated, the local information extraction capability of the deformable convolution offset and the global information extraction capability of the cross attention are used to respectively perform local enhancement and global enhancement on the multi-scale context, a globally enhanced minimum scale context and a locally enhanced multi-scale context are obtained, and the globally enhanced minimum scale context and the locally enhanced multi-scale context are fused to obtain a final multi-scale enhanced context, so that the inter-frame information of the context is focused on, and the global information in the context is retained, thereby solving the problem that the global information is lost when the existing video compression method processes complex video content, resulting in low video compression quality, and improving the video compression performance.

[0113] In addition, the hybrid context generation method provided by the application is a general method, which has a wide application scene and can be integrated into most multi-scale motion compensation video coding frameworks to improve the video compression efficiency.

[0114] Preferably, in the second embodiment, based on the first embodiment, the multi-scale feature further includes a preset medium scale feature and a preset minimum scale feature; the step of inputting the reconstructed optical flow and the multi-scale feature into a context generation module to obtain a multi-scale context includes:

[0115] Bilinear down-sampling the reconstructed optical flow to obtain a preset medium scale reconstructed optical flow and a preset minimum scale reconstructed optical flow;

[0116] Compensating the reconstructed optical flow, the preset medium scale reconstructed optical flow and the preset minimum scale reconstructed optical flow respectively by using the original scale feature, the preset medium scale feature and the preset minimum scale feature to obtain a multi-scale context, the multi-scale context including an original scale context, a preset medium scale context and a preset minimum scale context.

[0117] In an embodiment, the multi-scale features further include preset medium-scale features and preset minimum-scale features; the reconstructed optical flow is bilinearly down-sampled to obtain a preset medium-scale reconstructed optical flow and a preset minimum-scale reconstructed optical flow, bilinear down-sampling is used for scaling an image, and a thumbnail of a specified size is obtained by interpolating the original image; the reconstructed optical flow, the preset medium-scale reconstructed optical flow, and the preset minimum-scale reconstructed optical flow are compensated by using the original-scale features, the preset medium-scale features, and the preset minimum-scale features respectively to obtain multi-scale contexts, the multi-scale contexts include an original-scale context, a preset medium-scale context, and a preset minimum-scale context, wherein the preset medium-scale features are 1 / 2 scale features, the preset minimum-scale features are 1 / 4 scale features, the preset medium-scale reconstructed optical flow is a 1 / 2 scale reconstructed optical flow, the preset minimum-scale reconstructed optical flow is a 1 / 4 scale reconstructed optical flow, the preset medium-scale context is a 1 / 2 scale context, and the preset minimum-scale context is a 1 / 4 scale context, which can be adjusted according to actual conditions; more comprehensive details and textures are provided by using the multi-scale contexts, and the video compression performance is improved.

[0118] Specifically, referring to Figure 5 , Figure 5 a structure diagram of a context generation module is provided for the embodiment of the application, the reconstructed optical flow of the original scale is first bilinearly down-sampled to obtain a 1 / 2 scale reconstructed optical flow and a 1 / 4 scale reconstructed optical flow Then, the reconstructed optical flows of different scales are compensated by using features of different scales to generate multi-scale contexts The multi-scale contexts are as shown in the following formula:

[0119]

[0120]

[0121]

[0122] In the formula, ω represents a warping operator based on the optical flow; represents the context of the original scale; represents the context of the 1 / 2 scale; represents the context of the 1 / 4 scale.

[0123] ​​​​​Preferably, in the third embodiment, based on the second embodiment described above, the multi-scale context further comprises a preset medium-scale context and an original-scale context; and the step of performing optical flow guided deformable convolution compensation on the multi-scale features based on the multi-scale context and the deformable convolution offset of the multi-scale features, and the reconstructed optical flow, to obtain the locally enhanced multi-scale context, comprises:

[0124] performing optical flow guided deformable convolution compensation on the original-scale features based on the original-scale context and the deformable convolution offset of the original-scale features, and the reconstructed optical flow, to obtain a first-scale context;

[0125] performing optical flow guided deformable convolution compensation on the preset medium-scale features based on the preset medium-scale context and the deformable convolution offset of the preset medium-scale features, and the reconstructed optical flow of the preset medium-scale, to obtain a second-scale context;

[0126] performing optical flow guided deformable convolution compensation on the preset minimum-scale features based on the preset minimum-scale context and the deformable convolution offset of the preset minimum-scale features, and the reconstructed optical flow of the preset minimum-scale, to obtain a third-scale context.

[0127] In an embodiment, the multi-scale context further comprises a preset medium-scale context and an original-scale context; a first-scale context is obtained by performing optical flow guided deformable convolution compensation on the original-scale features based on the original-scale context and the deformable convolution offset of the original-scale features, and the reconstructed optical flow, the first-scale context being a locally enhanced original-scale context; a second-scale context is obtained by performing optical flow guided deformable convolution compensation on the preset medium-scale features based on the preset medium-scale context and the deformable convolution offset of the preset medium-scale features, and the reconstructed optical flow of the preset medium-scale, the second-scale context being a locally enhanced preset medium-scale context; and a third-scale context is obtained by performing optical flow guided deformable convolution compensation on the preset minimum-scale features based on the preset minimum-scale context and the deformable convolution offset of the preset minimum-scale features, and the reconstructed optical flow of the preset minimum-scale, the third-scale context being a locally enhanced preset minimum-scale context.

[0128] The deformable convolution offset between the multi-scale context and the previously decoded frame is obtained. Based on the reconstructed optical flow and the deformable convolution offset, the multi-scale context is subjected to optical flow-guided deformable convolution to obtain the locally enhanced multi-scale context. The medium-scale context is preset to be the 1 / 2 scale context, and the minimum-scale context is preset to be the 1 / 4 scale context. The local information of the multi-scale context is enhanced by deformable convolution, which improves the correlation of local detail features and improves the quality of the context without bitrate overhead, thereby improving the video compression efficiency.

[0129] Specifically, refer to Figure 6 , Figure 6 This is a schematic diagram of the local context enhancement module provided in an embodiment of this application. It enhances the context at each scale progressively through multi-scale deformable convolution. Specifically, it first enhances the context at a preset minimum scale. Preset minimum scale features and the reconstructed optical flow at the preset minimum scale As input, to estimate the preset minimum scale features Additional deformable convolution offset

[0130]

[0131] Here, Conv represents a convolutional neural network. Then, based on... and For the preset minimum scale features Flow-guided deformable convolution compensation (FGDC) is performed to obtain the enhanced preset minimum scale context. Expressed as a formula:

[0132]

[0133] Then, to Perform bilinear upsampling by 2x, and... and Cascaded inputs are fed into a deformable convolutional offset estimation network to guide a pre-defined medium-scale offset. The estimate is based on and For preset medium-scale features Optically flow-guided deformable convolution compensation is performed to obtain the enhanced pre-defined mid-scale context.

[0134] Context of the maximum scale The enhancement process is the same as the preset medium-scale process, for bilinear up-sampling by a factor of 2 is performed on the warped features and concatenated as input to the deformable convolution offset estimation network to guide the estimation of the original scale offsets based on and performing optical flow guided deformable convolution compensation on the original scale features to obtain enhanced original scale contexts

[0135] Preferably, in the fourth embodiment, based on the third embodiment described above, the step of fusing the locally enhanced multi-scale contexts and the globally enhanced multi-scale contexts to obtain multi-scale enhanced contexts comprises:

[0136] fusing the third scale context and the globally enhanced preset minimum scale context to obtain a fused context through channel spatial attention;

[0137] obtaining a first residual of the first scale context, a second residual of the second scale context, and a third residual of the fused context through a convolutional neural network;

[0138] adding the first residual, the second residual, and the third residual to the first scale context, the second scale context, and the fused context respectively to obtain multi-scale enhanced contexts.

[0139] In an embodiment, a third scale context and a globally enhanced preset minimum scale context are fused to obtain a fused context through channel spatial attention; a first residual of a first scale context, a second residual of a second scale context, and a third residual of the fused context are obtained through a convolutional neural network; and the first residual, the second residual, and the third residual are added to the first scale context, the second scale context, and the fused context respectively to obtain multi-scale enhanced contexts, wherein the first scale context, the second scale context, and the third scale context respectively refer to a locally enhanced original scale context, a locally enhanced medium scale context, and a locally enhanced minimum scale context; the locally enhanced minimum scale context and the globally enhanced minimum scale context are fused through channel spatial attention to ensure the global enhancement effect; the residuals of the multi-scale contexts are extracted through a convolutional neural network to further strengthen the feature expression capability of a video compression model applying the context enhancement method provided in the present application, improve the video compression quality, and thus improve the video compression efficiency.

[0140] Specifically, assuming that the scales include an original scale, a 1 / 2 scale, and a 1 / 4 scale, with reference to Figure 4 , Figure 4For the structure diagram of the local and global context fusion module involved in the embodiments of the present application, specifically, first, the preset minimum scale context is fused, the local enhanced preset minimum scale context is cascaded with the global enhanced preset minimum scale context , the context is fused through the channel attention mechanism and the spatial attention mechanism in turn, then the convolution network is used to learn the residual of the current scale context, and the residual is added to the current scale context to obtain the final multi-scale multi-scale enhanced context

[0141] Preferably, in the fifth embodiment, based on the above-mentioned first embodiment, the step of obtaining the global enhanced multi-scale context according to the cross attention includes:

[0142] The preset minimum scale context is globally enhanced through the cross attention to obtain the global enhanced preset minimum scale context;

[0143] The obtained global enhanced preset minimum scale context is taken as a new preset minimum scale context, and a new cross attention is obtained according to the new preset minimum scale context, and the step of globally enhancing the preset minimum scale context through the cross attention is continued to be executed for a preset number of times to obtain the preset minimum scale context globally enhanced for a preset number of times.

[0144] In an embodiment, the cross attention of the preset minimum scale context and the preset minimum scale feature is obtained, and the multi-scale context is globally enhanced through the cross attention to obtain the global enhanced multi-scale context, wherein the preset medium scale context is a 1 / 2 scale context, the preset minimum scale context is a 1 / 4 scale context, the local information of the multi-scale context is enhanced based on the deformable convolution, the global information of the multi-scale context is enhanced based on the cross attention mechanism, the quality of the context and the video compression efficiency are improved without code rate overhead, and the cross attention mechanism considers the relevance between different input sequences when calculating the attention weight, and is used to capture the interdependence and interaction between sequences.

[0145] Then, the obtained globally enhanced preset minimum scale context is taken as a new preset minimum scale context, a new cross attention is obtained according to the new preset minimum scale context, the step of globally enhancing the multi-scale context by the cross attention is continuously executed for a preset number of times, and a final globally enhanced preset minimum scale context is obtained; after the output of the first global multi-scale enhanced context is obtained, the global enhancement is continuously executed for a preset number of times, the global enhancement effect is ensured, and a preset minimum scale context after the global enhancement for a preset number of times is obtained; it should be noted that, considering the calculation complexity and the video compression performance, the preset number is set to 4 times, and can be adjusted according to actual conditions.

[0146] Specifically, referring to Figure 7 , Figure 7 , a structure diagram of a global context enhancement module provided by the embodiment of the application is provided. The application adopts an architecture similar to a transformer model to enhance the context of a preset minimum scale, so as to reduce the calculation cost. Specifically, first, the context feature of the preset minimum scale and the preset minimum scale feature are normalized and mapped into a query (Q), a key value (K) and an attribute value (V); then, the correlation between Q and K is calculated to obtain an attention graph (A); the attribute value V is multiplied by the attention graph A, so that the global information is extracted. In addition, the method also adopts a Gated Feedforward Network (GDFN), which suppresses the features with less information through the GDFN, so that more useful information is transmitted in the network architecture, so as to enhance the feature extraction. In the implementation of the method, 4 global enhancement modules are used, and finally a globally enhanced context is generated. Specifically, step 1: input the context of the preset minimum scale and the preset minimum scale feature to the global context enhancement module; step 2: the output of the global context enhancement module is replaced and returned to the input end, and is input again to the global context enhancement module together with , and the step 2 process is repeated 3 times, and finally the globally enhanced context is obtained.

[0147] Referring to Figure 8 , Figure 8The ablation experiment results of the context enhancement method provided by the embodiments of the present application under the commonly used data sets of video compression include UVG, MCL-JCV, HEVC B, C, D and E. The coding configuration is low latency, the intra period is set to 32, and 96 frames of each video sequence are tested. The method without context enhancement (method A) is set as the baseline, and the enhancement is performed on different scales to verify the effectiveness of the method provided by the present application. The experimental results show that the application of the optical flow guided deformable convolution compensation (method B) on the 1 / 4 scale can bring gain on the small motion data set (HEVC E), but no gain is obtained on the data set with complex motion. When the cross attention mechanism is additionally applied on the 1 / 4 scale (method C), the performance on all test data sets is improved. This is because the cross attention mechanism additionally extracts global information compared with the deformable convolution focusing on the local region, and therefore better rate-distortion performance is obtained. The method only applies the cross attention mechanism on the preset minimum scale to save the calculation cost. In addition, the compression efficiency is further improved as the optical flow guided deformable convolution compensation is applied to more scales (methods D and E). These ablation experiments prove the effectiveness of the local and global context enhancement modules provided by the present application.

[0148] Preferably, in the sixth embodiment, based on the first embodiment described above, the step of performing multi-scale feature extraction on the features of the previously decoded frame to obtain multi-scale features comprises:

[0149] inputting the features of the previously decoded frame into a pyramid convolutional neural network to obtain the multi-scale features, wherein the multi-scale features comprise original scale features, preset medium scale features and preset minimum scale features.

[0150] In an embodiment, the features of the previously decoded frame are input into a pyramid convolutional neural network to obtain multi-scale features of the previously decoded frame, wherein the multi-scale features comprise original scale features, preset medium scale features and preset minimum scale features, and the pyramid convolutional neural network is used to extract more comprehensive and rich information from features at different levels. The preset medium scale features are set to 1 / 2 scale features, and the preset minimum scale features are set to 1 / 4 scale features, which can be adjusted according to actual conditions to ensure the accuracy of the video compression model applying the context enhancement method provided by the embodiment and improve the video compression quality and video compression efficiency.

[0151] Preferably, in the seventh embodiment, based on the first embodiment described above, the step of obtaining the reconstructed optical flow based on the to-be-compressed frame and the previously decoded frame comprises:

[0152] inputting the to-be-compressed frame and the previously decoded frame into an optical flow estimation network to obtain an optical flow;

[0153] inputting the optical flow into an optical flow encoder and an optical flow decoder to obtain a reconstructed optical flow.

[0154] In an embodiment, the optical flow encoder and the optical flow decoder are constructed based on a self-encoder structure, and the input image x Figure 3 , Figure 3 inputting the image x i That is, in the embodiments of the present application, the input image and the previously decoded frame are input into the optical flow estimation network to obtain the optical flow, and then the optical flow is input into the optical flow encoder and the optical flow decoder to obtain the reconstructed optical flow; wherein the optical flow encoder and the optical flow decoder are used to realize compression and reconstruction of the optical flow data; the optical flow estimation network is used to generate an optical flow map representing the motion between two images, and to obtain the optical flow based on the frame to be compressed and the previously decoded frame, and to reconstruct the optical flow for compression and transmission after reconstruction, thereby reducing data redundancy and improving video compression efficiency.

[0155] Referring to Figure 9 The present application also provides a context enhancement device, which comprises:

[0156] a reconstruction module M1 configured to obtain a reconstructed optical flow based on a frame to be compressed and a previously decoded frame;

[0157] an extraction module M2 configured to perform multi-scale feature extraction on features of the previously decoded frame to obtain multi-scale features, the multi-scale features comprising preset minimum scale features;

[0158] a context generation module M3 configured to input the reconstructed optical flow and the multi-scale features into the context generation module to obtain multi-scale contexts, the multi-scale contexts comprising preset minimum scale contexts;

[0159] a local enhancement module M4 configured to perform optical flow guided deformable convolution compensation on the multi-scale features by means of deformable convolution offsets of the multi-scale contexts and the multi-scale features, and the reconstructed optical flow, to obtain the multi-scale contexts after local enhancement;

[0160] a global enhancement module M5 configured to perform global enhancement on the multi-scale contexts by means of cross-attention of the preset minimum scale contexts and the preset minimum scale features, to obtain the multi-scale contexts after global enhancement;

[0161] a fusion module M6 configured to fuse the multi-scale contexts after local enhancement and the multi-scale contexts after global enhancement to obtain multi-scale enhanced contexts.

[0162] Optionally, the context generation module is further configured to:

[0163] perform bi-linear down-sampling on the reconstructed optical flow to obtain a reconstructed optical flow of a preset medium scale and a reconstructed optical flow of a preset minimum scale;

[0164] perform compensation on the reconstructed optical flow, the reconstructed optical flow of the preset medium scale and the reconstructed optical flow of the preset minimum scale respectively by using the original scale feature, the preset medium scale feature and the preset minimum scale feature to obtain multi-scale contexts.

[0165] Optionally, the local enhancement module is further configured to:

[0166] perform optical flow guided deformable convolution compensation on the original scale feature by using the original scale context, a deformable convolution offset of the original scale feature and the reconstructed optical flow to obtain a first scale context;

[0167] perform optical flow guided deformable convolution compensation on the preset medium scale feature by using the preset medium scale context, a deformable convolution offset of the preset medium scale feature and the reconstructed optical flow of the preset medium scale to obtain a second scale context;

[0168] perform optical flow guided deformable convolution compensation on the preset minimum scale feature by using the preset minimum scale context, a deformable convolution offset of the preset minimum scale feature and the reconstructed optical flow of the preset minimum scale to obtain a third scale context.

[0169] Optionally, the fusion module is further configured to:

[0170] fuse the third scale context and the preset minimum scale context after global enhancement by using channel space attention to obtain a fusion context;

[0171] obtain a first residual of the first scale context, a second residual of the second scale context and a third residual of the fusion context by using a convolutional neural network;

[0172] add the first residual, the second residual and the third residual to the first scale context, the second scale context and the fusion context respectively to obtain a multi-scale enhanced context.

[0173] Optionally, the global enhancement module is further configured to:

[0174] perform global enhancement on the preset minimum scale context by using the cross attention to obtain the preset minimum scale context after global enhancement;

[0175] The preset minimum scale context obtained by the global enhancement is taken as a new preset minimum scale context, and a new cross attention is obtained according to the new preset minimum scale context, and the step of globally enhancing the preset minimum scale context by the cross attention is continuously performed for a preset number of times to obtain the preset minimum scale context after global enhancement for a preset number of times.

[0176] Optionally, the extraction module is further configured to:

[0177] The features of the previous decoded frame are input into a pyramid convolutional neural network to obtain multi-scale features.

[0178] Optionally, the reconstruction module is further configured to:

[0179] The to-be-compressed frame and the previous decoded frame are input into an optical flow estimation network to obtain an optical flow.

[0180] The optical flow is input into an optical flow encoder and an optical flow decoder to obtain a reconstructed optical flow.

[0181] The context enhancement device provided in the present application adopts the context enhancement method in the above embodiments, and aims to improve the video compression performance. Compared with the conventional technology, the context enhancement device provided in the embodiments of the present application has the same beneficial effects as the context enhancement method provided in the above embodiments, and other technical features in the context enhancement device are the same as the features disclosed in the above embodiments, which will not be repeated here.

[0182] The present application also provides a context enhancement device, which comprises a memory, a processor and a computer program stored in the memory and executable on the processor, and the computer program is configured to implement the steps of the context enhancement method according to any one of the above embodiments.

[0183] The present application also provides a medium, which stores a computer program, and the computer program is executed by a processor to implement the steps of the context enhancement method according to any one of the above embodiments.

[0184] It should be noted that, in this document, the terms "comprising", "including", or any other variant thereof are intended to cover non-exclusive inclusion, so that processes, methods, articles or systems including a series of elements not only include those elements, but also include other elements not explicitly listed, or inherent to such processes, methods, articles or systems. Without more limitations, the element defined by the statement "comprising a" does not exclude the presence of additional identical elements in the process, method, article or system including the element.

[0185] The above application embodiment serial numbers are only for description, and do not represent the advantages and disadvantages of the embodiments.

[0186] Through the above description of the embodiments, those skilled in the art can clearly understand that the above-mentioned embodiment method can be realized by means of software and the necessary general hardware platform, of course, it can also be realized by hardware, but in many cases the former is a better embodiment. Based on such understanding, the technical solutions of the present application can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) as described above, and includes a plurality of instructions for making a terminal device (which can be a mobile phone, computer, server, or network device, etc.) execute the methods described in various embodiments of the present application.

[0187] The above is only the preferred embodiment of the present application, and does not limit the patent scope of the present application, and any equivalent structure or equivalent process transformation using the content of the present application specification and drawings, or direct or indirect application in other related technical fields, are also included in the patent protection scope of the present application.

Claims

1. A method of context enhancement, characterized by, The method comprises: obtaining a reconstructed optical flow based on a frame to be compressed and a previously decoded frame; performing multi-scale feature extraction on features of the previously decoded frame to obtain multi-scale features, the multi-scale features comprising preset minimum scale features; inputting the reconstructed optical flow and the multi-scale features into a context generation module to obtain multi-scale contexts, the multi-scale contexts comprising preset minimum scale contexts; performing optical flow guided deformable convolution compensation on the multi-scale features based on deformable convolution offsets of the multi-scale contexts and the multi-scale features, and the reconstructed optical flow, to obtain locally enhanced multi-scale contexts; performing global enhancement on the multi-scale contexts based on cross attention of the preset minimum scale contexts and the preset minimum scale features, to obtain globally enhanced multi-scale contexts; fusing the locally enhanced multi-scale contexts and the globally enhanced multi-scale contexts to obtain multi-scale enhanced contexts.

2. The context enhancement method of claim 1, wherein, The multi-scale features further comprise preset medium scale features and preset minimum scale features; the step of inputting the reconstructed optical flow and the multi-scale features into a context generation module to obtain multi-scale contexts comprises: performing bilinear down-sampling on the reconstructed optical flow to obtain a preset medium scale reconstructed optical flow and a preset minimum scale reconstructed optical flow; performing compensation on the reconstructed optical flow, the preset medium scale reconstructed optical flow and the preset minimum scale reconstructed optical flow respectively based on original scale features, the preset medium scale features and the preset minimum scale features, to obtain multi-scale contexts.

3. The context enhancement method of claim 2, wherein, The multi-scale contexts further comprise preset medium scale contexts and original scale contexts; the step of performing optical flow guided deformable convolution compensation on the multi-scale features based on deformable convolution offsets of the multi-scale contexts and the multi-scale features, and the reconstructed optical flow, to obtain locally enhanced multi-scale contexts comprises: performing optical flow guided deformable convolution compensation on original scale features based on deformable convolution offsets of the original scale contexts and the original scale features, and the reconstructed optical flow, to obtain first scale contexts; performing optical flow guided deformable convolution compensation on preset medium scale features based on deformable convolution offsets of the preset medium scale contexts and the preset medium scale features, and the preset medium scale reconstructed optical flow, to obtain second scale contexts; performing optical flow guided deformable convolution compensation on preset minimum scale features based on deformable convolution offsets of the preset minimum scale contexts and the preset minimum scale features, and the preset minimum scale reconstructed optical flow, to obtain third scale contexts.

4. The context enhancement method of claim 3, wherein, The step of fusing the locally enhanced multi-scale contexts and the globally enhanced multi-scale contexts to obtain multi-scale enhanced contexts comprises: fusing the third scale contexts and the globally enhanced preset minimum scale contexts based on channel space attention to obtain fused contexts; The first residual of the first scale context, the second residual of the second scale context, and the third residual of the fusion context are obtained through a convolutional neural network; the first residual, the second residual, and the third residual are added to the first scale context, the second scale context, and the fusion context respectively to obtain a multi-scale enhanced context.

5. The context enhancement method of claim 1, wherein, The step of globally enhancing the multi-scale context through the cross-attention of the preset minimum scale context and the preset minimum scale feature to obtain the globally enhanced multi-scale context comprises: The preset minimum scale context is globally enhanced through the cross-attention to obtain a globally enhanced preset minimum scale context; The globally enhanced preset minimum scale context is taken as a new preset minimum scale context, and a new cross-attention is obtained according to the new preset minimum scale context, and the step of globally enhancing the preset minimum scale context through the cross-attention is continuously executed for a preset number of times to obtain the preset minimum scale context after being globally enhanced for a preset number of times.

6. The context enhancement method of claim 1, wherein, The step of performing multi-scale feature extraction on the features of the previously decoded frame to obtain multi-scale features comprises: The features of the previously decoded frame are input into a pyramid convolutional neural network to obtain multi-scale features.

7. The context enhancement method of claim 1, wherein, The step of obtaining a reconstructed optical flow based on the to-be-compressed frame and the previously decoded frame comprises: The to-be-compressed frame and the previously decoded frame are input into an optical flow estimation network to obtain an optical flow; The optical flow is input into an optical flow encoder and an optical flow decoder to obtain a reconstructed optical flow.

8. A context enhancement device, characterized by The context enhancement device comprises: A reconstruction module configured to obtain a reconstructed optical flow based on a to-be-compressed frame and a previously decoded frame; An extraction module configured to perform multi-scale feature extraction on features of the previously decoded frame to obtain multi-scale features, the multi-scale features comprising preset minimum scale features; A context generation module configured to input the reconstructed optical flow and the multi-scale features into the context generation module to obtain a multi-scale context, the multi-scale context comprising a preset minimum scale context; A local enhancement module configured to perform optical flow guided deformable convolution compensation on the multi-scale features through a deformable convolution offset of the multi-scale context and the multi-scale features and the reconstructed optical flow to obtain a locally enhanced multi-scale context; A global enhancement module configured to globally enhance the multi-scale context through cross-attention of the preset minimum scale context and the preset minimum scale feature to obtain a globally enhanced multi-scale context; A fusion module configured to fuse the locally enhanced multi-scale context and the globally enhanced multi-scale context to obtain a multi-scale enhanced context.

9. A context enhancement device, characterized by The context enhancement device comprises a memory, a processor, and a computer program stored on the memory and executable on the processor, the computer program being configured to implement the steps of the context enhancement method according to any one of claims 1 to 7.

10. A computer storage medium, characterized in that, The computer storage medium stores a computer program, and the computer program is executed by the processor to implement the steps of the context enhancement method according to any one of claims 1 to 7.

Images