Method, apparatus, device and program product for video coding

Abstract

One or more embodiments of the present specification relate to a method, an apparatus, a computing device, and a computer program product for video encoding. The method comprises the steps that motion compensation information is acquired based on a current frame and a first historical frame of a target video, and the first historical frame is a previous frame of the current frame; the method further includes obtaining context condition information based on the motion compensation information, the current frame, and a plurality of historical frames, the plurality of historical frames including the first historical frame. Further, the method includes encoding the current frame based on the context condition information.

Description

Technical Field

[0001] The embodiments of the present specification generally relate to the field of computer technology, and more particularly to a method, apparatus, computing device, and computer program product for video encoding. Background Art

[0002] Video encoding refers to the method of converting files in the original video format into files in another video format through compression technology. Video is a continuous sequence of images, consisting of continuous frames, and one frame is an image. Due to the visual persistence effect of the human eye, when the frame sequence is played at a certain rate, people see a video with continuous action. Since the similarity between continuous frames is extremely high, in order to facilitate storage and transmission, the original video needs to be encoded and compressed to remove the redundancy in the spatial and temporal dimensions.

[0003] Traditional video coding technology mainly relies on various algorithms to remove redundant information in video data. These algorithms include intra-frame coding and inter-frame coding. Intra-frame coding mainly uses the spatial correlation within the image for compression, while inter-frame coding uses the temporal correlation of the image sequence for compression. Common traditional video coding methods include H.264, H.265, etc.

[0004] End-to-end coding is an emerging video coding technology that mainly uses deep learning technology to implement various modules of video coding by building neural network models. These neural network models can learn the characteristics of video data and automatically perform encoding and decoding operations. Compared with traditional coding technology, end-to-end coding optimizes the coding process as a whole. It no longer optimizes each module separately, but improves overall performance through joint optimization. Summary of the invention

[0005] In view of this, one or more embodiments of the present specification provide a method, apparatus, computing device and computer program product for video encoding, which can first use the current frame and historical frames for motion encoding, then use the results of motion encoding and multiple historical frames to obtain contextual condition information, and then use the contextual condition information as a reference factor to encode the current frame, thereby being able to abandon fixed algorithms and predefined encoding parameters, and being able to more deeply integrate the temporal and spatial characteristics of video data, thereby improving encoding efficiency and reconstruction quality, and improving the flexibility and adaptability of the video encoding framework.

[0006] In a first aspect of the present specification, a method for video encoding is provided. The method includes acquiring motion compensation information based on a current frame and a first historical frame of a target video, wherein the first historical frame is a frame before the current frame. The method also includes acquiring context condition information based on the motion compensation information, the current frame, and multiple historical frames, wherein the multiple historical frames include the first historical frame. In addition, the method also includes encoding the current frame based on the context condition information.

[0007] In a second aspect of the present specification, a video encoding device is provided. The device includes a motion encoding unit configured to obtain motion compensation information based on a current frame and a first historical frame of a target video, where the first historical frame is a frame before the current frame; a context information acquisition unit configured to obtain context condition information based on the motion compensation information, the current frame and multiple historical frames, where the multiple historical frames include the first historical frame; and a context encoding unit configured to encode the current frame based on the context condition information.

[0008] In a third aspect of the present specification, a computing device is provided, comprising: at least one processing unit; and at least one memory, the at least one memory being coupled to the at least one processing unit and storing instructions for execution by the at least one processing unit, the instructions, when executed by the at least one processing unit, causing the computing device to execute the method as described in the first aspect of the present specification.

[0009] In a fourth aspect of the present specification, a computer program product is provided, comprising machine executable instructions, which, when executed by a device, cause the device to perform the method according to the first aspect of the present specification.

[0010] It should be understood that the contents described in the Summary of the Invention are not intended to limit the key or important features of the embodiments of this specification, nor are they intended to limit the scope of this disclosure. Other features of this specification will become easy to understand through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] The above and other purposes, features and advantages of the various embodiments of the present specification will become more easily understood through the following detailed description with reference to the accompanying drawings. In the accompanying drawings, various embodiments of the present specification will be described in an exemplary and non-limiting manner, wherein:

[0012] Figure 1 A schematic diagram showing an example environment in which some embodiments according to the present specification may be implemented;

[0013] Figure 2 A flowchart of a method for video encoding according to some embodiments of the present specification is shown;

[0014] Figure 3 A schematic diagram showing the structure of a video coding framework according to some embodiments of this specification is shown;

[0015] Figure 4 A schematic diagram showing a process of generating refinement features according to some embodiments of the present specification;

[0016] Figure 5 A schematic diagram showing a process of feature fusion according to some embodiments of the present specification;

[0017] Figure 6 A schematic diagram showing a process of generating context condition information according to some embodiments of the present specification;

[0018] Figure 7A-7B A schematic diagram showing performance indicators of a data set test according to some embodiments of the present specification;

[0019] Figure 8 A block diagram of a video encoding apparatus according to some embodiments of the present specification is shown; and

[0020] Fig. 9 A block diagram of an electronic device according to some embodiments of the present specification is shown.

[0021] Throughout the drawings, the same or similar reference numbers denote the same or similar elements. DETAILED DESCRIPTION

[0022] The embodiments of the present specification will be described in more detail below with reference to the accompanying drawings. Although certain embodiments of the present specification are shown in the accompanying drawings, it should be understood that the present specification can be implemented in various forms and should not be construed as being limited to the embodiments described herein, which are instead provided for a more thorough and complete understanding of the present specification. It should be understood that the drawings and embodiments of the present specification are only for exemplary purposes and are not intended to limit the scope of protection of the present disclosure.

[0023] In the description of the embodiments of this specification, the term "including" and similar terms should be understood as open inclusion, that is, "including but not limited to". The term "based on" should be understood as "based at least in part on". The term "one embodiment" or "the embodiment" should be understood as "at least one embodiment". The terms "first", "second", etc. may refer to different or the same objects. Other explicit and implicit definitions may also be included below.

[0024] As mentioned above, both traditional video coding and end-to-end coding technologies have a certain position in the field of video compression. However, with the increasing demand for compression and transmission of high-resolution and ultra-high-definition videos, although traditional video coding is widely used, it mainly relies on fixed algorithms and predefined coding parameters, lacking sufficient flexibility and adaptability. In addition, the current end-to-end intelligent video coding technology is too dependent on a single type of reference information, or fails to fully consider the multi-dimensional characteristics of video data during feature extraction, resulting in room for improvement in coding efficiency and reconstruction quality.

[0025] To this end, a method for video encoding is provided in an embodiment of the present specification. First, motion encoding is performed using the current frame and historical frames, and then context information is obtained using the result of motion encoding and multiple historical frames. Then, the context information is used as a reference factor to encode the current frame. In this way, fixed algorithms and predefined encoding parameters can be abandoned, and the spatiotemporal characteristics of video data can be integrated at a deeper level, thereby improving encoding efficiency and reconstruction quality, and improving the flexibility and adaptability of video encoding.

[0026] Figure 1 1 shows a schematic diagram of an environment 100 in which various embodiments of the present specification can be implemented. Figure 1 As shown, the environment 100 may include a target video 110 and a video encoding framework 102. The target video 110 may be a raw video stream captured in real time from a video acquisition device (e.g., a camera, a video recorder, etc.). The target video 110 may also be raw video data stored in a storage medium of a local device (e.g., a mobile phone, a computer, etc.).

[0027] like Figure 1 As shown, the video coding framework 102 may include an encoding end 104, a decoding end 106, and a buffer 108. The encoding end 104 may be used to convert the original video data into an encoded data stream, including but not limited to encoder encoding, quantization, and entropy encoding. The decoding end 106 is the inverse process of the encoding end 104, and the encoded data stream may be restored to the original video data by decoder decoding, inverse quantization, entropy decoding, and other means. The buffer 108 may be used to store historical frame data and historical feature data as a reference for subsequent processing (e.g., motion estimation, inter-frame prediction, etc.).

[0028] like Figure 1 As shown, the target video 110 can be input to the encoding end 104 of the video encoding framework 102 to be converted into an encoded data stream. Optionally, before being input to the video encoding framework 102, the target video 110 can also undergo a series of pre-processing operations to improve subsequent encoding efficiency and quality, including but not limited to format conversion, resolution adjustment, denoising and enhancement, frame rate adjustment, etc.

[0029] like Figure 1 As shown, the encoder 104 can first read the historical frame information from the buffer 108 to perform motion estimation on the current frame of the target video 110. Then, the encoder 104 can transmit the result of the motion estimation to the decoder after encoding to obtain motion compensation information. The video encoding framework 102 can calculate the context condition information based on the motion compensation information and the multiple historical frame information in the buffer 108 as the time domain condition for the encoder 104 to encode the current frame of the target video 110. Then, the receiving end 106 can continue to receive the encoded current frame data and decode it to obtain the reconstructed current frame information and store it in the buffer 108 as a reference for subsequent frame encoding.

[0030] It should be understood that the architecture and functions in the example environment 100 are described for exemplary purposes only and do not imply any limitation on the scope of the present description. The embodiments of the present description may also be applied to other environments with different structures and / or functions.

[0031] The following will combine Figures 2 to 7B The process according to the embodiment of this specification is described in detail. For ease of understanding, the specific data mentioned in the following description are exemplary and are not intended to limit the scope of protection of the present disclosure. It should be understood that the embodiments described below may also include additional actions not shown and / or may omit the actions shown, and the scope of the present disclosure is not limited in this respect.

[0032] Figure 2 2 is a schematic flow chart of a method 200 for video encoding according to some embodiments of the present specification. In some embodiments, the method 200 may be performed by, for example, Figure 1 The method 200 is implemented by the video encoding framework 102 shown. It should be understood that the method 200 may further include additional actions not shown and / or may omit the actions shown, and the scope of the present disclosure is not limited in this respect.

[0033] like Figure 2 As shown, in block 210, method 200 may include: obtaining motion compensation information based on a current frame and a first historical frame of the target video, wherein the first historical frame is a frame before the current frame. By referring to the data of the previous frame, a macroblock (or pixel block) matching a certain area in the current frame may be found from the previous frame, and the displacement or motion vector between them may be determined. Then, the motion compensation information may be used to perform motion compensation on the current frame to reduce redundant information in the frame sequence, thereby improving video coding efficiency.

[0034] In some embodiments, the motion compensation information may be a reconstructed motion vector. The video coding framework 102 may first obtain a reconstructed frame of the first historical frame from the buffer, and the reconstructed frame of the first historical frame may be reconstructed by the video coding framework 102 at the previous moment and stored in the buffer. Then, the video coding framework 102 may obtain a motion vector based on the current frame and the reconstructed frame of the first historical frame. The encoding end of the video coding framework 102 may transform the acquired motion vector, and the transformation may include at least one of encoding, quantization, or entropy encoding. Then, the encoding end may transmit the transformed data to the decoding end, and the decoding end may inversely transform the transformed motion vector to obtain the reconstructed motion vector. In this way, it is possible to more accurately use multiple historical frame information for prediction in the subsequent motion compensation stage, thereby reducing the size of the residual data and improving the coding efficiency.

[0035] In block 220, method 200 may include: obtaining contextual condition information based on motion compensation information, a current frame, and a plurality of historical frames, wherein the plurality of historical frames include a first historical frame. By combining information from a plurality of historical frames, prediction accuracy can be improved, thereby improving encoding efficiency in fast motion or complex scenes. Contextual condition information can be used as a temporal condition for encoding, further improving the adaptability of the encoding framework to changes in video content.

[0036] In some embodiments, the multiple historical frames also include a second historical frame, and the second historical frame is the previous frame of the first historical frame. The video coding framework 102 may first obtain from the cache the reconstructed frame of the first historical frame, the reference features of the first historical frame, the second historical frame, and the reference features of the second historical frame stored at the historical moment. Then, the video coding framework 102 may obtain fusion features based on the reconstructed frame of the first historical frame, the reference features of the first historical frame, the second historical frame, and the reference features of the second historical frame. By fusion of features of multiple historical frames, the coding efficiency and accuracy can be further improved. The video coding framework 102 may also obtain contextual condition information based on the fusion features and the reconstructed motion vector as a time domain condition for coding.

[0037] In some embodiments, the above-mentioned fusion features can be obtained through a 3D convolutional neural network. Specifically, the video coding framework 102 can obtain the first feature based on the reconstructed frame of the first historical frame, the reference feature of the first historical frame, the second historical frame, and the reference feature of the second historical frame. Based on the first feature, the video coding framework 102 can obtain a confidence map and a prediction feature. The confidence map can represent the confidence that each pixel or region belongs to a certain category, and the prediction feature can be extracted from the input data to represent certain key attributes of the input data. Based on the confidence map and the prediction feature, for example, by multiplying the two, the video coding framework 102 can obtain a refined feature (which can be called a second feature). Then, the video coding framework 102 can obtain the above-mentioned fusion feature based on the second feature and the reference feature of the first historical frame, for example, through a splicing operation.

[0038] In some embodiments, the video encoding framework 102 may first obtain a third feature based on the reconstructed frame of the first historical frame and the reference feature of the first historical frame, and then obtain a fourth feature based on the reconstructed frame of the second historical frame and the reference feature of the second historical frame, and then fuse the third feature and the fourth feature to obtain a fifth feature. Finally, the video encoding framework 102 may perform a convolution operation on the fifth feature to obtain the first feature.

[0039] In some embodiments, there may be multiple context condition information, and each provides a reference at a different resolution. For example, the context condition information may include first context condition information, second context condition information, and third context condition information. Among them, the video encoding framework 102 can obtain the first context condition information by aligning the corresponding parts of the fused features and the reconstructed motion vector. The video encoding framework 102 can obtain the second context condition information by aligning the corresponding parts of the fused features and the reconstructed motion vector that have been downsampled once. The video encoding framework 102 can also obtain the third context condition information by aligning the corresponding parts of the fused features and the reconstructed motion vector that have been downsampled twice.

[0040] In block 230, method 200 may include: encoding the current frame based on the context condition information. Since the context condition information refers to multiple reference frames and implements multi-dimensional feature extraction, encoding using the context condition information as a time domain condition can improve encoding efficiency and quality. For the encoded data, the video encoding framework 102 can be sent to the receiving end for decoding through a medium such as a network to achieve restoration of the video data.

[0041] In some embodiments, the video encoding framework 102 can obtain the reconstructed frame of the current frame and the reference features of the current frame based on the encoded current frame data, and then store the reconstructed frame of the current frame and the reference features of the current frame in the buffer area as a reference for subsequent frame encoding. After the reconstructed frame of the current frame and the reference features of the current frame are stored in the buffer area, the video encoding framework 102 can also delete the older historical data in the buffer area to save resources.

[0042] In some embodiments, the video encoding framework 102 can transform the encoded current frame data through the encoding end, and the transformation includes at least one of quantization or entropy coding, and then inversely transform the transformed current frame data through the decoding end, and then decode the inversely transformed current frame data based on context condition information, and finally obtain the reconstructed frame of the above-mentioned current frame and the reference features of the current frame based on the decoded current frame data.

[0043] In this way, through the multi-dimensional fusion of motion compensation information and multiple historical frame information, it is possible to abandon fixed algorithms and predefined coding parameters, and more deeply integrate the temporal and spatial characteristics of video data, thereby improving coding efficiency and reconstruction quality, and improving the flexibility and adaptability of video coding.

[0044] Figure 3 FIG. 3 is a schematic diagram showing a structure of a video coding framework 300 according to some embodiments of the present specification. The video coding framework 300 can be regarded as follows: Figure 1 A specific implementation of the video coding framework 102 is shown. Figure 3 As shown, the encoding end of the video encoding framework 300 can first use the reconstructed frame of the previous frame (which can be called the first historical frame) obtained from the buffer area 318 to perform motion estimation 304 on the current frame 302 of the target video to obtain a motion vector that can represent the degree of displacement of the image block. Then, the encoding end of the video encoding framework 300 can use a motion encoder 306 to encode the motion vector to achieve preliminary data compression, and then further compress the encoded motion vector through quantization 308 and entropy coding 310 steps to transmit it to the decoding end of the video encoding framework 300.

[0045] After receiving the motion vector after the compression transformation, the decoding end of the video coding framework 300 can restore the compressed data to obtain the reconstructed motion vector through the corresponding inverse transformation, including entropy decoding 312, inverse quantization 314 and decoding operations implemented by the motion decoder 316. In step 320, the video coding framework 300 can perform multi-reference frame motion compensation 320 based on the obtained reconstructed motion vector and multiple historical frame information obtained from the buffer area 318, including the reconstructed frame of the first historical frame, the reference feature of the first historical frame, the reconstructed frame of the second historical frame and the reference feature of the second historical frame, so as to obtain context condition information 322. The acquisition of the context condition information 322 will be described below. Figure 4-6 Detailed description is given in .

[0046] After obtaining the context condition information 322, the video coding framework 300 can use it as a time domain condition and encode the current frame 302 using the context encoder 324 at the encoding end. After the encoding is completed, the video coding framework 300 can go through a series of transformation and inverse transformation operations, including quantization 326, entropy encoding 328, entropy decoding 330 and inverse quantization 332, and decoded using the context decoder 334, and finally the frame generator 336 uses the decoded data stream to obtain the reconstructed frame 338 and reference features of the current frame, and store them in the buffer 318 together as a reference for subsequent frame encoding.

[0047] The following will refer to Figure 4-Figure 6 The process of obtaining the context condition information 322 is described in detail.

[0048] Figure 4 FIG. 2 is a schematic diagram showing a process of generating refinement features according to some embodiments of the present specification. Figure 4 As shown, the video encoding framework 300 can first perform feature extraction 406 on the reconstructed frame 404 of the first historical frame, and then splice the extracted features with the reference features 402 of the first historical frame obtained from the cache area, and pass through a convolutional layer with residual connection to obtain a spliced ​​feature (which can be called a third feature). Similarly, the video encoding framework 300 can perform feature extraction 412 on the reconstructed frame 408 of the second historical frame, and then splice the extracted features with the reference features 410 of the second historical frame obtained from the cache area, and pass through a convolutional layer with residual connection to obtain a spliced ​​feature (which can be called a fourth feature). Then, the video encoding framework 300 can perform feature fusion 414 on the third feature and the fourth feature to obtain a fused feature. The following will refer to Figure 5 The specific process of feature fusion 414 is described in detail.

[0049] Figure 5 FIG. 2 is a schematic diagram showing a process of feature fusion according to some embodiments of the present specification. Figure 5As shown, the video encoding framework 300 can cross-splice the third feature 502 and the fourth feature 504 to obtain a spliced ​​feature 506 and a spliced ​​feature 508. Then, the spliced ​​feature 506 and the spliced ​​feature 508 can be fused to expand the new dimension to obtain a fused feature 510 (which can be called the fifth feature). In some embodiments, the video encoding framework 300 can perform a 3D convolution on the fifth feature 510 using two convolution kernels of size 3*3*3, and then add the two results to obtain a fused feature 512 (which can be called the first feature) with multi-dimensional information. It is understandable that this specification does not limit the size or depth of the convolution kernel.

[0050] return Figure 4 , the video encoding framework 300 may predict 416 the first feature 512 to obtain a confidence map 418 and a predicted feature 420. The confidence map 418 may indicate the confidence that each pixel or region belongs to a certain category, and the predicted feature may indicate certain key attributes. The confidence map 418 and the predicted feature 420 may be multiplied to obtain a preliminary refined feature (which may be referred to as a second feature), and then concatenated with the reference feature 402 of the first historical frame to obtain a final fused feature 422.

[0051] Figure 6 FIG. 1 is a schematic diagram showing a process of generating context condition information according to some embodiments of the present specification. Figure 6 As shown, after obtaining the fused feature 602 through 3D convolution, the video encoding framework 300 can obtain contextual condition information 622 based on the fused feature 602 and the reconstructed motion vector 606. The fused feature 602 can first be processed by the convolution layer 604, and then input into the calculation module 608 in combination with the reconstructed motion vector 606, and the corresponding parts of the two are aligned using the optical flow method to obtain the first contextual condition information. The first contextual condition information provides the most detailed motion information and details. Among them, the optical flow method can use the changes in the time domain of pixels in the image sequence and the correlation between adjacent frames to find the corresponding relationship between the previous frame and the current frame, thereby calculating the motion information of objects between adjacent frames.

[0052] Then, the fused features 602 may be downsampled once through the downsampling layer 610 and the convolution layer 612, and the reconstructed motion vector 606 may be bilinearly downsampled 614 to reduce the image resolution while retaining key information. The calculation module 608 then aligns the fused features 602 and the reconstructed motion vector 606 that have been downsampled once through the optical flow method to obtain the second contextual condition information. The second contextual condition information provides medium-scale motion information and details.

[0053] Similarly, the fused features 602 after the first downsampling can be downsampled twice through the downsampling layer 616 and the convolution layer 618, and the reconstructed motion vector 606 after the first downsampling can be bilinearly downsampled 620 to further reduce the image resolution. Thus, the calculation module 608 can obtain the third context condition information containing coarser-grained motion information. Thus, the context condition information of three different granularities together constitute a comprehensive description of the object motion, which can provide rich reference information for subsequent encoding.

[0054] In some embodiments, the motion coding stage and the context coding stage can be regarded as a neural network for overall training. During the training process, the convolution layer parameters of the motion coding stage can be trained first, and after the convolution layer parameters of the motion coding stage are fixed, the convolution layer parameters of the context coding stage can be trained. Finally, the two can be jointly back-propagated to simultaneously update the convolution layer parameters and jointly optimize the overall performance. In some embodiments, it is also possible to combine the fine-tuning stage to introduce appropriate parameters for rate-distortion optimization to adjust the bit rate point, thereby effectively controlling the encoded bit rate while maintaining video quality.

[0055] Figure 7A-7B A schematic diagram of performance indicators of data set tests according to some embodiments of the present specification is shown. The High Efficiency Video Coding Common Test Condition (HEVC CTC) sequence dataset is a series of carefully selected video clips that cover a variety of scenes, motion complexity, and image quality to comprehensively evaluate the performance of the encoder under different conditions. These sequences are often used for standardized testing to ensure that the HEVC encoder can perform well in a variety of application scenarios.

[0056] In the HEVC CTC sequence dataset, Class C and D represent different types of video sequences, which differ in content, complexity, encoding difficulty, etc. Specifically, Class C may contain some video content with medium complexity, such as natural scenery, urban landscapes, etc. Class D may contain video content with higher complexity, such as fast-motion scenes, high dynamic range (HDR) videos, etc. Since the video content of Class D is usually more complex, its encoding difficulty may be higher and require more computing resources and time to complete the encoding process. In contrast, the video content of Class C may be easier to encode because its content is relatively simple.

[0057] Fig. 7A and Figure 7BThe coding performance of the technical solution of the present application and some other common coding methods when facing the video sequences of Class C and D in the HEVCCTC sequence data set are respectively shown. Among them, the coding methods used for comparison include deep context video compression-diversified context (DCVC-DC) 704, deep context video compression-hybrid space-time entropy modeling (DCVC-HEM) 706, deep context video compression-temporal context mining (DCVC-TCM) 708, deep context video compression (DCVC) 710 and deep video compression (DCV) 712. It can be seen that the coding method 702 of the present application has good coding efficiency, especially in low bit rate video scenarios.

[0058] Each embodiment in this specification is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the device embodiment and the computing device embodiment, since they are basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0059] Figure 8 FIG. 8 is a schematic block diagram of a video encoding apparatus 800 according to some embodiments of the present specification. Figure 8 As shown, the device 800 includes: a motion encoding unit 802 , a context information acquisition unit 804 and a context encoding unit 806 .

[0060] In some embodiments, the motion encoding unit 802 is configured to obtain motion compensation information based on a current frame and a first historical frame of the target video, where the first historical frame is a frame previous to the current frame; the context information acquisition unit 804 is configured to obtain context condition information based on the motion compensation information, the current frame and multiple historical frames, where the multiple historical frames include the first historical frame; and the context encoding unit 806 is configured to encode the current frame based on the context condition information.

[0061] In some embodiments, the motion compensation information includes a reconstructed motion vector, and the motion encoding unit 802 is further configured to obtain a reconstructed frame of the first historical frame from a cache area; obtain a motion vector based on the current frame and the reconstructed frame of the first historical frame; transform the motion vector, the transformation including at least one of encoding, quantization or entropy coding; and inverse transform the transformed motion vector to obtain a reconstructed motion vector.

[0062] In some embodiments, the multiple historical frames also include a second historical frame, which is the previous frame of the first historical frame, and the context information acquisition unit 804 is also configured to obtain a reconstructed frame of the first historical frame, a reference feature of the first historical frame, a second historical frame, and a reference feature of the second historical frame from a cache area; obtain a fusion feature based on the reconstructed frame of the first historical frame, the reference feature of the first historical frame, the second historical frame, and the reference feature of the second historical frame; and obtain context condition information based on the fusion feature and the reconstructed motion vector.

[0063] In some embodiments, the context information acquisition unit 804 is also configured to acquire a first feature based on a reconstructed frame of the first historical frame, a reference feature of the first historical frame, a second historical frame, and a reference feature of the second historical frame; acquire a confidence map and a predicted feature based on the first feature; acquire a second feature based on the confidence map and the predicted feature; and acquire a fused feature based on the second feature and the reference feature of the first historical frame.

[0064] In some embodiments, the context information acquisition unit 804 is also configured to acquire a third feature based on the reconstructed frame of the first historical frame and the reference feature of the first historical frame; acquire a fourth feature based on the reconstructed frame of the second historical frame and the reference feature of the second historical frame; fuse the third feature and the fourth feature to obtain a fifth feature; and perform a convolution operation on the fifth feature to obtain the first feature.

[0065] In some embodiments, the context condition information includes first context condition information, second context condition information and third context condition information, and the context information acquisition unit 804 is also configured to align the corresponding parts of the fused features and the reconstructed motion vectors to obtain the first context condition information; align the corresponding parts of the fused features and the reconstructed motion vectors that have been downsampled once to obtain the second context condition information; and align the corresponding parts of the fused features and the reconstructed motion vectors that have been downsampled twice to obtain the third context condition information.

[0066] In some embodiments, the device 800 also includes a cache unit configured to obtain a reconstructed frame of the current frame and a reference feature of the current frame based on the encoded current frame data; and store the reconstructed frame of the current frame and the reference feature of the current frame in a cache area.

[0067] In some embodiments, the cache unit is also configured to transform the encoded current frame data, the transformation including at least one of quantization or entropy coding; inversely transform the transformed current frame data; decode the inversely transformed current frame data based on context condition information; and obtain a reconstructed frame of the current frame and a reference feature of the current frame based on the decoded current frame data.

[0068] It should be noted that the reference Figures 1 to 7BFurther actions or steps shown can be performed by Figure 8 For example, the device 800 may include more modules or units to implement the actions or steps described above, or Figure 8 Some of the units or modules shown may be further configured to implement the actions or steps described above, which will not be repeated here.

[0069] Fig. 9 9 shows a schematic block diagram of an example device 900 that can be used to implement some embodiments of the present specification. Fig. 9 As shown, the device 900 includes a computing unit 901, which can perform various appropriate actions and processes according to computer program instructions stored in a read-only memory (ROM) 902 or computer program instructions loaded from a storage unit 906 into a random access memory (RAM) 903. In the RAM 903, various programs and data required for the operation of the device 900 can also be stored. The computing unit 901, the ROM 902, and the RAM 903 are connected to each other via a bus 904. An input / output (I / O) interface 905 is also connected to the bus 904.

[0070] A number of components in the device 900 are connected to the I / O interface 905, including: an input unit 906, such as a keyboard, a mouse, etc.; an output unit 907, such as various types of displays, speakers, etc.; a storage unit 908, such as a disk, an optical disk, etc.; and a communication unit 909, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 909 allows the device 900 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0071] The computing unit 901 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the computing unit 901 include, but are not limited to, a CPU, a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc. The computing unit 901 performs the various methods and processes described above, such as method 200. For example, in some embodiments, method 200 may be implemented as a computer software program, which is tangibly contained in a machine-readable medium, such as a storage unit 908. In some embodiments, part or all of the computer program may be loaded and / or installed on the device 900 via ROM 902 and / or communication unit 909. When the computer program is loaded into RAM 903 and executed by the computing unit 901, one or more steps of the method 200 described above may be performed. Alternatively, in other embodiments, the computing unit 901 may be configured to perform method 200 in any other appropriate manner (e.g., by means of firmware).

[0072] In some embodiments, the methods and processes described above may be implemented as a computer program product. The computer program product may include a computer-readable storage medium carrying computer-readable program instructions for executing various aspects of the present specification.

[0073] A computer-readable storage medium may be a tangible device that can hold and store instructions used by an instruction execution device. A computer-readable storage medium may be, for example, but not limited to, an electrical storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination of the foregoing. More specific examples of computer-readable storage media (a non-exhaustive list) include: a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a static random access memory (SRAM), a portable compact disk read-only memory (CD-ROM), a digital versatile disk (DVD), a memory stick, a floppy disk, a mechanical encoding device, such as a punch card or a raised structure in a groove on which instructions are stored, and any suitable combination of the foregoing. As used herein, a computer-readable storage medium is not to be interpreted as a transient signal per se, such as a radio wave or other freely propagating electromagnetic wave, an electromagnetic wave propagating through a waveguide or other transmission medium (e.g., a light pulse through a fiber optic cable), or an electrical signal transmitted through a wire.

[0074] The computer-readable program instructions described herein can be downloaded from a computer-readable storage medium to each computing / processing device, or downloaded to an external computer or external storage device via a network, such as the Internet, a local area network, a wide area network, and / or a wireless network. The network can include copper transmission cables, optical fiber transmissions, wireless transmissions, routers, firewalls, switches, gateway computers, and / or edge servers. The network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards the computer-readable program instructions for storage in the computer-readable storage medium in each computing / processing device.

[0075] The computer program instructions for performing the operation of each embodiment of this specification can be assembler instructions, instruction set architecture (ISA) instructions, machine instructions, machine-related instructions, microcode, firmware instructions, state setting data or source code or object code written in any combination of one or more programming languages, and the programming languages ​​include object-oriented programming languages, and conventional procedural programming languages. Computer-readable program instructions can be executed completely on the user's computer, partially on the user's computer, executed as an independent software package, partially on the user's computer and partially on the remote computer, or executed completely on the remote computer or server. In the case of a remote computer, the remote computer can be connected to the user's computer by any type of network-including local area network (LAN) or wide area network (WAN), or can be connected to an external computer (for example, using an Internet service provider to connect by the Internet). In certain embodiments, by using the state information of computer-readable program instructions to customize electronic circuits, such as programmable logic circuits, field programmable gate arrays (FPGAs) or programmable logic arrays (PLAs), the electronic circuits can execute computer-readable program instructions, thereby realizing the various aspects of this specification.

[0076] These computer-readable program instructions can be provided to a processing unit of a general-purpose computer, a special-purpose computer, or other programmable data processing device, thereby producing a machine, so that when these instructions are executed by the processing unit of the computer or other programmable data processing device, a device that implements the functions / actions specified in one or more boxes in the flowchart and / or block diagram is generated. These computer-readable program instructions can also be stored in a computer-readable storage medium, and these instructions cause the computer, programmable data processing device, and / or other equipment to work in a specific manner, so that the computer-readable medium storing the instructions includes a manufactured product, which includes instructions for implementing various aspects of the functions / actions specified in one or more boxes in the flowchart and / or block diagram.

[0077] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device so that a series of operating steps are performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions executed on the computer, other programmable data processing apparatus, or other device to implement the functions / actions specified in one or more boxes in the flowchart and / or block diagram.

[0078] The above is a description of a specific embodiment of the specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recorded in the claims can be performed in an order different from that in the embodiments and still achieve the desired results. In addition, the processes depicted in the drawings do not necessarily require the specific order or continuous order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0079] The embodiments of the present specification have been described above, and the above description is exemplary, not exhaustive, and is not limited to the disclosed embodiments. Many modifications and variations will be apparent to those of ordinary skill in the art without departing from the scope and spirit of the described embodiments. The selection of terms used herein is intended to best explain the principles of the embodiments, practical applications, or technical improvements to the technology in the market, or to enable other persons of ordinary skill in the art to understand the embodiments disclosed herein.

[0080] The above are only optional embodiments of the present invention and are not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A method for video encoding, comprising: Acquire motion compensation information based on a current frame and a first historical frame of a target video, where the first historical frame is a frame previous to the current frame; Acquire context condition information based on the motion compensation information, the current frame, and a plurality of historical frames, wherein the plurality of historical frames include the first historical frame; as well as The current frame is encoded based on the context condition information.

2. The method of claim 1 , wherein the motion compensation information comprises a reconstructed motion vector, and obtaining the motion compensation information comprises: Acquire a reconstructed frame of the first historical frame from a buffer area; Obtaining a motion vector based on the current frame and a reconstructed frame of the first historical frame; transforming the motion vector, wherein the transforming comprises at least one of encoding, quantization or entropy encoding; and The transformed motion vector is inversely transformed to obtain the reconstructed motion vector.

3. The method according to claim 2, wherein the plurality of historical frames further include a second historical frame, the second historical frame being a previous frame of the first historical frame, and acquiring the context condition information comprises: Acquire from the buffer a reconstructed frame of the first historical frame, a reference feature of the first historical frame, the second historical frame, and a reference feature of the second historical frame; Acquire fusion features based on the reconstructed frame of the first historical frame, the reference features of the first historical frame, the second historical frame, and the reference features of the second historical frame; and The context condition information is obtained based on the fused feature and the reconstructed motion vector.

4. The method according to claim 3, wherein obtaining the fusion feature comprises: Acquire a first feature based on a reconstructed frame of the first historical frame, a reference feature of the first historical frame, the second historical frame, and a reference feature of the second historical frame; Based on the first feature, obtaining a confidence map and a prediction feature; Based on the confidence map and the predicted feature, obtaining a second feature; and The fusion feature is obtained based on the second feature and the reference feature of the first historical frame.

5. The method according to claim 4, wherein obtaining the first feature comprises: Acquire a third feature based on a reconstructed frame of the first historical frame and a reference feature of the first historical frame; acquiring a fourth feature based on the reconstructed frame of the second historical frame and the reference feature of the second historical frame; Combining the third feature and the fourth feature to obtain a fifth feature; as well as Perform a convolution operation on the fifth feature to obtain the first feature.

6. The method according to claim 3, wherein the context condition information includes first context condition information, second context condition information and third context condition information, and acquiring the context condition information comprises: Aligning the fused feature and corresponding parts of the reconstructed motion vector to obtain the first context condition information; Aligning the fused features and corresponding parts of the reconstructed motion vector that have been downsampled once, respectively, to obtain the second context condition information; as well as The fused features and corresponding parts of the reconstructed motion vectors that have been downsampled twice are aligned to obtain the third context condition information.

7. The method according to claim 1, wherein the method further comprises: Based on the encoded current frame data, obtaining a reconstructed frame of the current frame and a reference feature of the current frame; as well as The reconstructed frame of the current frame and the reference features of the current frame are stored in a buffer.

8. The method according to claim 7, wherein acquiring the reconstructed frame of the current frame and the reference feature of the current frame comprises: Transforming the encoded current frame data, wherein the transformation includes at least one of quantization or entropy coding; Performing an inverse transformation on the transformed current frame data; Based on the context condition information, decoding the inverse transformed current frame data; as well as Based on the decoded current frame data, a reconstructed frame of the current frame and a reference feature of the current frame are obtained.

9. A system for video encoding, comprising: A motion coding unit, configured to obtain motion compensation information based on a current frame and a first historical frame of a target video, wherein the first historical frame is a frame before the current frame; a context information acquisition unit configured to acquire context condition information based on the motion compensation information, the current frame, and a plurality of historical frames, wherein the plurality of historical frames include the first historical frame; as well as The context encoding unit is configured to encode the current frame based on the context condition information.

10. A computing device comprising: at least one processing unit; At least one memory, the at least one memory being coupled to the at least one processing unit and storing instructions for execution by the at least one processing unit, the instructions, when executed by the at least one processing unit, causing the computing device to perform the method as claimed in any one of claims 1 to 8.

11. A computer program product comprising machine executable instructions which, when executed by a device, cause the device to perform the method according to any one of claims 1 to 8.

Images