Video coding method and device, video decoding method and device, electronic equipment and storage medium

Abstract

The invention relates to a video coding method and device, a video decoding method and device, electronic equipment and a storage medium. The video coding method comprises the following steps: acquiring difference information between a plurality of reference frames of a current frame and the current frame; determining weights corresponding to the plurality of reference frames based on the difference information; fusing the coding information of each reference frame in the plurality of reference frames based on the weights corresponding to the plurality of reference frames to obtain fused coding information; and coding the current frame based on the fused coding information to obtain a coding frame of the current frame.

Description

Technical Field

[0001] The present disclosure relates to the field of video processing, and in particular to a video encoding method and apparatus, a video decoding method and apparatus, an electronic device, and a storage medium. Background Art

[0002] Various electronic devices (such as digital televisions, laptop or desktop computers, tablet computers, digital cameras, digital recorders, digital media players, video game consoles, smartphones, video teleconferencing devices, and video streaming devices) support digital video. Electronic devices transmit and receive, or otherwise communicate digital video data over communication networks, and / or store digital video data on storage devices. Due to the limited bandwidth capacity of communication networks and the limited storage resources of storage devices, video codecs may be used to compress video data before transmission or storage according to one or more video codec standards. For example, video codec standards include Versatile Video Codec (VVC), High Efficiency Video Codec (HEVC / H.265), and Advanced Video Codec (AVC / H.264). Video codecs typically employ prediction methods (such as inter-frame prediction and intra-frame prediction) that exploit the inherent redundancy in video data. Video codecs aim to compress video data into a form that uses a lower bit rate while avoiding or minimizing degradation in video quality. Summary of the Invention

[0003] The present disclosure provides a video encoding method and apparatus, a video decoding method and apparatus, an electronic device, and a storage medium, to at least solve the problem of low encoding efficiency caused by the related art ignoring the possible imbalance of reference value of reference frames in bidirectional reference.

[0004] According to a first aspect of an embodiment of the present disclosure, a video encoding method is provided, including: obtaining difference information between a current frame and multiple reference frames of a current frame; determining weights corresponding to each of the multiple reference frames based on the difference information; fusing encoding information of each reference frame in the multiple reference frames based on the weights corresponding to each of the multiple reference frames to obtain fused encoding information; encoding the current frame based on the fused encoding information to obtain an encoded frame of the current frame.

[0005] According to a second aspect of an embodiment of the present disclosure, a video decoding method is provided, including: obtaining difference information between multiple reference frames of a current frame and a coded frame of the current frame; determining weights corresponding to each of the multiple reference frames based on the difference information; fusing decoding information of each reference frame in the multiple reference frames based on the weights corresponding to each of the multiple reference frames to obtain fused decoding information; and decoding the coded frame of the current frame based on the fused decoding information.

[0006] According to a third aspect of an embodiment of the present disclosure, a video encoding device is provided, including: a first acquisition unit, configured to obtain difference information between multiple reference frames of a current frame and the current frame; a first determination unit, configured to determine the weights corresponding to each of the multiple reference frames based on the difference information; a first fusion unit, configured to fuse the encoding information of each reference frame in the multiple reference frames based on the weights corresponding to each of the multiple reference frames to obtain fused encoding information; and an encoding unit, configured to encode the current frame based on the fused encoding information to obtain an encoded frame of the current frame.

[0007] According to a fourth aspect of an embodiment of the present disclosure, a video decoding device is provided, including: a second acquisition unit, configured to obtain difference information between multiple reference frames of a current frame and a coded frame of the current frame; a second determination unit, configured to determine the weights corresponding to each of the multiple reference frames based on the difference information; a second fusion unit, configured to fuse the decoding information of each reference frame in the multiple reference frames based on the weights corresponding to each of the multiple reference frames to obtain fused decoding information; and a decoding unit, configured to decode the coded frame of the current frame based on the fused decoding information. According to a fifth aspect of an embodiment of the present disclosure, an electronic device is provided, comprising: at least one processor; and at least one memory storing computer-executable instructions, wherein the computer-executable instructions, when executed by the at least one processor, prompt the at least one processor to execute the video encoding method and / or video decoding method as described above.

[0008] According to the sixth aspect of an embodiment of the present disclosure, a computer-readable storage medium is provided. When the instructions in the computer-readable storage medium are executed by at least one processor, the at least one processor is prompted to store the bit stream generated by the above-mentioned video encoding method and / or the bit stream decoded by the video decoding method.

[0009] The technical solutions provided by the embodiments of the present disclosure bring at least the following beneficial effects: According to the video encoding method and apparatus, video decoding method and apparatus, electronic device, and storage medium disclosed herein, based on the difference information between the current frame and its multiple reference frames, the encoding information of each reference frame is unevenly fused to encode the current frame, so that the contribution of each reference frame in encoding the current frame can be dynamically adjusted, thereby effectively solving the problem of reference value imbalance that may exist in bidirectional reference video coding, thereby improving end-to-end video coding efficiency. Therefore, the present disclosure solves the problem of low coding efficiency caused by the related art ignoring the possible reference value imbalance of reference frames in bidirectional reference.

[0010] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] The accompanying drawings herein are incorporated into and constitute a part of the specification, illustrate embodiments consistent with the present disclosure, and together with the description are used to explain the principles of the present disclosure, and do not constitute an improper limitation of the present disclosure.

[0012] Figure 1 is a block diagram illustrating an exemplary system for encoding and decoding video blocks according to an exemplary embodiment of the present disclosure; Figure 2 is a block diagram of a video encoder according to an exemplary embodiment of the present disclosure; Figure 3 is a block diagram of a video decoder according to an exemplary embodiment of the present disclosure; Figure 4 is a reference schematic diagram illustrating a bidirectional reference video coding method and a low-latency video coding method according to an exemplary embodiment of the present disclosure; Figure 5 is an end-to-end network structure diagram illustrating two encoding methods of a bidirectional reference mode and a low-latency mode according to an exemplary embodiment of the present disclosure; Figure 6 is a schematic diagram illustrating an imbalance in reference values ​​of two reference frames in a bidirectional coding mode according to an exemplary embodiment of the present disclosure; Figure 7 is a flowchart showing a video encoding method according to an exemplary embodiment of the present disclosure; Figure 8 is a structural diagram of a BRAF module of an encoding end according to an exemplary embodiment of the present disclosure; Figure 9 is a structural diagram of a depth residual block DB with a step size of s according to an exemplary embodiment of the present disclosure; Figure 10 is a flowchart showing a video decoding method according to an exemplary embodiment of the present disclosure; Figure 11 is a structural diagram of a BRAF module at a decoding end according to an exemplary embodiment of the present disclosure; Figure 12 is a structural diagram of a PFA according to an exemplary embodiment of the present disclosure; Figure 13 is a block diagram showing a video encoding apparatus according to an exemplary embodiment of the present disclosure; Figure 14 is a block diagram showing a video decoding device according to an exemplary embodiment of the present disclosure; Figure 15 is a diagram illustrating a computing environment coupled with a user interface according to an exemplary embodiment of the present disclosure. DETAILED DESCRIPTION

[0013] In order to enable ordinary persons in the art to better understand the technical solutions of the present disclosure, the technical solutions in the embodiments of the present disclosure will be clearly and completely described below with reference to the accompanying drawings.

[0014] It should be noted that the terms "first," "second," and the like in the specification and claims of the present disclosure and the above-mentioned drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or precedence. It should be understood that the numbers used in this manner are interchangeable where appropriate so that the embodiments of the present disclosure described herein can be implemented in an order other than those illustrated or described herein. The implementation methods described in the following examples do not represent all implementation methods consistent with the present disclosure. Instead, they are merely examples of devices and methods consistent with certain aspects of the present disclosure as detailed in the appended claims.

[0015] It should be noted that the phrase "at least one of the several items" in this disclosure includes three types of parallel situations: "any one of the several items", "a combination of any multiple of the several items", and "all of the several items". For example, "including at least one of A and B" includes the following three parallel situations: (1) including A; (2) including B; (3) including A and B. For another example, "performing at least one of step 1 and step 2" means the following three parallel situations: (1) performing step 1; (2) performing step 2; and (3) performing both step 1 and step 2.

[0016] Figure 1 1 shows a block diagram of an exemplary system 10 for encoding and decoding video blocks in parallel according to some embodiments of the present disclosure. Figure 1 As shown in , system 10 includes a source device 12 that generates and encodes video data to be later decoded by a destination device 14. Source device 12 and destination device 14 may include any of a wide variety of electronic devices, including a cloud server, a server computer, a desktop or laptop computer, a tablet computer, a smartphone, a set-top box, a digital television, a camera, a display device, a digital media player, a video game console, a video streaming device, etc. In some implementations, source device 12 and destination device 14 are equipped with wireless communication capabilities.

[0017] In some embodiments, target device 14 may receive the encoded video data to be decoded via link 16. Link 16 may include any type of communication medium or device capable of moving the encoded video data from source device 12 to target device 14. In other embodiments, the encoded video data may be sent from output interface 22 to storage device 32. The encoded video data in storage device 32 may then be accessed by target device 14 via input interface 28.

[0018] like Figure 1 , source device 12 includes a video source 18, a video encoder 20, and an output interface 22. Video source 18 may include a source such as a video capture device (e.g., a video camera), a video archive containing previously captured video, a video feed interface for receiving video from a video content provider, and / or a computer graphics system for generating computer graphics data as the source video. Captured, pre-captured, or computer-generated video can be encoded by video encoder 20. Encoded video data can be sent directly to target device 14 via output interface 22 of source device 12. Encoded video data can also (or alternatively) be stored on storage device 32 for later access by target device 14 or other devices for decoding and / or playback.

[0019] Target device 14 includes an input interface 28, a video decoder 30, and a display device 34. Input interface 28 may include a receiver and / or a modem and receives encoded video data via link 16. The encoded video data transmitted via link 16 or provided on storage device 32 may include various syntax elements generated by video encoder 20 for use by video decoder 30 in decoding the video data. Such syntax elements may be included within the encoded video data transmitted over a communication medium, stored on a storage medium, or stored on a file server.

[0020] The video encoder 20 and the video decoder 30 can operate according to proprietary standards or industry standards (e.g., VVC, HEVC, Part 10 of MPEG-4, AVC) or extensions of such standards. It should be understood that the present application is not limited to a specific video encoding / decoding standard and can be applied to other video encoding / decoding standards. It is generally believed that the video encoder 20 of the source device 12 can be configured to encode video data according to any of these current or future standards. Similarly, it is also generally believed that the video decoder 30 of the target device 14 can be configured to decode video data according to any of these current or future standards.

[0021] The video encoder 20 and the video decoder 30 can be implemented as any of a variety of suitable encoder and / or decoder circuits, respectively. When partially implemented in software, the electronic device can store instructions for the software in a suitable non-transitory computer-readable medium and use one or more processors to execute the instructions in hardware to perform the video encoding / decoding operations disclosed in the present disclosure. Each of the video encoder 20 and the video decoder 30 can be included in one or more encoders or decoders, and either encoder or decoder can be integrated as part of a combined encoder / decoder (CODEC) in the corresponding device.

[0022] Figure 2 A block diagram of an exemplary video encoder 20 according to some embodiments described herein is shown. Video encoder 20 can perform intra-frame prediction coding and inter-frame prediction coding on video blocks within a video frame. It should be noted that in the field of video coding and decoding, the term "frame" can be used as a synonym for the term "image" or "picture".

[0023] like Figure 2 As shown in FIG, video encoder 20 includes video data memory 40, prediction processing unit 41, decoded picture buffer (DPB) 64, adder 50, transform processing unit 52, quantization unit 54, and entropy coding unit 56. Prediction processing unit 41 further includes motion estimation unit 42, motion compensation unit 44, segmentation unit 45, intra-prediction processing unit 46, and intra-block copy (BC) unit 48. In some embodiments, video encoder 20 also includes an inverse quantization unit 58 for video block reconstruction, an inverse transform processing unit 60, and adder 62. A loop filter 63, such as a deblocking filter, may be located between adder 62 and DPB 64 to filter block boundaries to remove blocking artifacts from the reconstructed video. In addition to the deblocking filter, another loop filter, such as a sample adaptive offset (SAO) filter, a cross-component sample adaptive offset (CCSAO) filter, and / or an adaptive loop filter (ALF), may also be used to filter the output of adder 62. In some examples, the loop filter may be omitted, and the decoded video block may be provided directly by adder 62 to DPB 64. Video encoder 20 may take the form of a fixed or programmable hardware unit, or may be distributed among one or more of the illustrated fixed or programmable hardware units.

[0024] Video data memory 40 may store video data to be encoded by components of video encoder 20, such as from Figure 1 Video source 18 is shown obtaining video data from video data memory 40. DPB 64 is a buffer that stores reference video data for use by video encoder 20 when encoding the video data.

[0025] like Figure 2 As shown in , after receiving the video data, the segmentation unit 45 within the prediction processing unit 41 segments the video data into video blocks. The prediction processing unit 41 may select one of a plurality of possible prediction coding modes for the current video block based on error results (e.g., coding rate and distortion level). The prediction processing unit 41 may provide the resulting intra-frame prediction coding block or inter-frame prediction coding block to the adder 50 to generate a residual block, and to the adder 62 to reconstruct the coding block for subsequent use as part of a reference frame. The prediction processing unit 41 also provides syntax elements (e.g., motion vectors, intra-frame mode indicators, segmentation information, and other such syntax information) to the entropy coding unit 56.

[0026] To select an appropriate intra-prediction coding mode for the current video block, intra-prediction processing unit 46 within prediction processing unit 41 may perform intra-prediction coding of the current video block relative to one or more neighboring blocks in the same frame as the current block to be coded to provide spatial prediction. Motion estimation unit 42 and motion compensation unit 44 within prediction processing unit 41 perform inter-prediction coding of the current video block relative to one or more prediction blocks in one or more reference frames to provide temporal prediction. In some embodiments, motion estimation unit 42 determines the inter-prediction mode for the current video frame by generating a motion vector based on a predetermined pattern within a sequence of video frames. The motion vector indicates the displacement of a video block within the current video frame relative to a prediction block within a reference video frame. The predetermined pattern may designate video frames in the sequence as P-frames or B-frames. Intra-BC unit 48 may determine a vector (e.g., a block vector) for intra-BC coding in a manner similar to the motion vectors determined by motion estimation unit 42 for inter-prediction, or may utilize motion estimation unit 42 to determine the block vector.

[0027] Regardless of whether the prediction block is from the same frame according to intra-frame prediction or from a different frame according to inter-frame prediction, video encoder 20 may form a residual video block by subtracting pixel values ​​of the prediction block from the pixel values ​​of the current video block being encoded to form pixel difference values.

[0028] Intra-prediction processing unit 46 may encode the current block using various intra-prediction modes, for example, during a separate encoding pass, and select an appropriate intra-prediction mode to use from the tested intra-prediction modes, and then provide information indicating the selected intra-prediction mode for the block to entropy encoding unit 56. Entropy encoding unit 56 may encode the information indicating the selected intra-prediction mode into the bitstream.

[0029] After prediction processing unit 41 determines a prediction block for the current video block via inter-frame prediction or intra-frame prediction, adder 50 forms a residual video block by subtracting the prediction block from the current video block. The residual video data in the residual block may be included in one or more TUs and provided to transform processing unit 52. Transform processing unit 52 uses a transform to transform the residual video data into residual transform coefficients. Transform processing unit 52 may send the resulting transform coefficients to quantization unit 54. Quantization unit 54 quantizes the transform coefficients to further reduce the bit rate.

[0030] After quantization, entropy coding unit 56 entropy encodes the quantized transform coefficients into a video bitstream using, for example, context adaptive variable length coding (CAVLC), context adaptive binary arithmetic coding (CABAC), syntax-based context adaptive binary arithmetic coding (SBAC), probability interval partitioning entropy (PIPE) coding, or another entropy coding method or technique. The encoded bitstream can then be sent to a video bitstream such as Figure 1 The video decoder 30 shown, or archived as Figure 1 The video frame is shown in storage device 32 for later transmission to or retrieval by video decoder 30. Entropy encoding unit 56 may also entropy encode motion vectors and other syntax elements for the current video frame being encoded.

[0031] Inverse quantization unit 58 and inverse transform processing unit 60 apply inverse quantization and inverse transform, respectively, to reconstruct the residual video block in the pixel domain for use in generating a reference block for predicting other video blocks. Adder 62 adds the reconstructed residual block to the motion compensated prediction block generated by motion compensation unit 44 to generate a reference block for storage in DPB 64. The reference block can then be used by intra BC unit 48, motion estimation unit 42, and motion compensation unit 44 as a prediction block for inter-frame prediction of another video block in a subsequent video frame.

[0032] Figure 3 A block diagram of an exemplary video decoder 30 according to some embodiments of the present application is shown. The video decoder 30 includes a video data memory 79, an entropy decoding unit 80, a prediction processing unit 81, an inverse quantization unit 86, an inverse transform processing unit 88, an adder 90, and a DPB 92. The prediction processing unit 81 further includes a motion compensation unit 82, an intra-frame prediction unit 84, and an intra-frame BC unit 85. The video decoder 30 may perform the above-mentioned operations in combination with the above-mentioned operations. Figure 2 The encoding process described with respect to video encoder 20 is essentially the inverse of the decoding process.

[0033] In some examples, implementations of the present disclosure can be dispersed among one or more of the units of video decoder 30. For example, intra BC unit 85 can perform implementations of the present disclosure alone or in combination with other units of video decoder 30, such as motion compensation unit 82, intra prediction unit 84, and entropy decoding unit 80. In some examples, video decoder 30 can not include intra BC unit 85, and the functionality of intra BC unit 85 can be performed by other components of prediction processing unit 81, such as motion compensation unit 82.

[0034] Video data memory 79 can store video data, such as an encoded video bitstream, to be decoded by other components of video decoder 30. During the decoding process, video decoder 30 receives an encoded video bitstream that represents encoded video frames and associated syntax elements. Entropy decoding unit 80 of video decoder 30 entropy decodes the bitstream to generate quantized coefficients, motion vectors, or intra prediction mode indicators, and other syntax elements. Entropy decoding unit 80 then forwards the motion vectors, or intra prediction mode indicators, and other syntax elements to prediction processing unit 81.

[0035] When a video frame is encoded as an intra-predicted (I) frame or an intra coded prediction block in other types of frames, intra prediction unit 84 of prediction processing unit 81 can generate prediction data for a video block of the current video frame based on the intra prediction mode signaled and reference data from previously decoded blocks of the current frame. When a video frame is encoded as an inter-predicted (i.e., B or P) frame, motion compensation unit 82 of prediction processing unit 81 produces one or more prediction blocks for a video block of the current video frame based on the motion vectors and other syntax elements received from entropy decoding unit 80. Each of the prediction blocks can be produced from a reference frame within one of the reference frame lists.

[0036] Motion compensation unit 82 and / or intra BC unit 85 determine the prediction information for a video block of the current video frame by parsing the motion vectors and other syntax elements, and then use the prediction information to produce the prediction block for the current video block being decoded. Motion compensation unit 82 can also perform interpolation using interpolation filters as used by video encoder 20 during encoding of the video block to calculate interpolated values for sub-integer pixels of reference blocks. In this case, motion compensation unit 82 can determine the interpolation filters used by video encoder 20 from the syntax elements received, and use these interpolation filters to produce the prediction block.

[0037] Inverse quantization unit 86 inverse quantizes the quantized transform coefficients provided in the bitstream and entropy decoded by entropy decoding unit 80, using the same quantization parameters that were calculated by video encoder 20 for each video block in the video frame to determine the degree of quantization. Inverse transform processing unit 88 applies an inverse transform (e.g., an inverse DCT, an inverse integer transform, or a conceptually similar inverse transform process) to the transform coefficients to reconstruct the residual block in the pixel domain.

[0038] After motion compensation unit 82 or intra BC unit 85 generates a prediction block for the current video block based on the vectors and other syntax elements, adder 90 reconstructs the decoded video block for the current video block by adding the residual block from inverse transform processing unit 88 to the corresponding prediction block generated by motion compensation unit 82 and intra BC unit 85. A loop filter 91 (e.g., a deblocking filter, SAO filter, CCSAO filter, and / or ALF) may be located between adder 90 and DPB 92 to further process the decoded video block. The decoded video block in a given frame is then stored in DPB 92, which stores reference frames for subsequent motion compensation of the next video block.

[0039] Traditional video coding methods include low-delay (LD) coding and bidirectional reference video coding. The former only refers to the information of the previous frame in a single encoding, while the latter uses the frames before and after the current frame as reference frames for the current frame, and then compresses and encodes the current frame based on the current frame and the corresponding reference frames. The latter still has much room for improvement compared to the former. In addition, the bidirectional reference video coding method often has higher rate-distortion performance and faster overall decoding speed due to better parallel decoding characteristics. Although it requires multiple frames of image cache to be input at one time for encoding and decoding, its performance advantages still have a very wide range of application scenarios, such as on-demand scenarios, surveillance storage video, and other scenarios that are not sensitive to delay.

[0040] In modern video coding technologies, bidirectional reference video coding methods and low-latency video coding methods each have unique characteristics and application scenarios. Figure 4 The bidirectional reference video coding method and the low-latency video coding method are presented in Figure 4 a) in represents a low-latency video coding method, Figure 4 b) in the figure represents a bidirectional reference video coding method. Figure 4 As shown in the figure, in a coding scenario with a Group of Picture (GOP) of 4, an I frame is output by encoding an image without reference to other images, a P frame refers to the previous frame, and a B frame refers to the two frames before and after.

[0041] The bidirectional reference video coding method, often also referred to as random access (RA) and B-frame coding, achieves efficient inter-frame prediction by leveraging information from previous and next frames in the video sequence. The core advantage of this method is that B-frames can refer not only to previous frames but also to later frames when encoding. This bidirectional prediction mechanism enables B-frames to fully utilize the temporal redundancy in the video sequence, significantly improving compression efficiency. However, this coding method also increases the complexity of encoding and decoding, because the decoding of B-frames requires waiting for both the previous and next reference frames to be decoded. However, since B-frame decoding has higher rate-distortion performance (typically saving 20% ​​to 30% of the bit rate compared to the LD setting), the RA mode has very important applications in many scenarios where latency requirements are not high, such as on-demand scenarios, surveillance and storage video, and other scenarios.

[0042] With the development of more and more end-to-end video coding algorithms, research on bidirectional reference end-to-end compression has gradually developed. For example, DCVC-B introduces both inter-reference optical flow and a multi-scale contextual coding framework, as well as a training strategy for bidirectional reference. These designs make DCVC-B the most mature and advanced bidirectional reference compression method. DCVC is a deep learning-based contextual video coding proposed by Microsoft, and its full name is Deep Contextual Video Compression.

[0043] The end-to-end network structure of the two coding methods of bidirectional reference mode and low delay mode is as follows Figure 5 As shown, Figure 5 a) in the figure represents the network structure of low-latency mode. Figure 5 b) in the figure shows the network structure of the bidirectional reference mode. Both have similar main modules of motion estimation, optical flow encoding and decoding, motion compensation and encoding and decoding. The main differences in their structures are as follows: the low-latency mode mainly refers to the reconstructed frame of the previous frame. And the features of the reconstructed frame converted to the feature domain ; The bidirectional reference mode uses the reconstructed frame of the previous reference frame and the reconstructed frame of the subsequent reference frame And the features converted to the feature domain and ,in, k The value of is {1,2,4,8,16} (when GOP is 32). The bidirectional reference mode uses motion compensation for the context features of the previous and next reference frames and then fuses the context features of the two directions ( Figure 5 The obtained fusion context features are used together for conditional encoding and decoding.

[0044] In the bidirectional coding mode, it is necessary to compress motion information from two directions, namely, optical flow in two directions. Optical flow is a feature that describes the motion of pixels in an image sequence. By analyzing the optical flow, the direction and speed of the object in the image can be inferred. For the bidirectional reference mode, in order to make more effective use of motion information, it is necessary to encode the previous and next reference frames. and Respective and current frame The optical flow is used to represent the motion information in two directions, namely the optical flow and . For the original optical flow and Compress and transmit to the decoding end to obtain the reconstructed optical flow and Afterwards, the algorithm will use the optical flow to convert the reconstructed frames of the previous and next reference frames into features on the feature domain. and Motion compensation is performed to supplement the inter-frame information and obtain the features of the predicted frames of the previous and next reference frames converted to the feature domain.

[0045] However, current bidirectional reference video coding often performs equal feature fusion on each reference frame during compression, assuming that multiple reference frames are equally important. This ignores the potential imbalance in reference value among bidirectional reference frames, which can affect coding efficiency. In many cases, each reference frame's contribution to encoding and decoding is not uniform, as one reference frame may be closer to the current frame than another. Figure 6 It shows the problem of unbalanced reference values ​​of two reference frames in bidirectional coding mode, such as Figure 6 As shown in the figure, when encoding the t-th frame, the reference information from the previous frame (i.e., frame t-1) is clearly more valuable for the racehorse's number plate area, because this area is occluded in the subsequent frame. Therefore, the network needs to focus more attention on the previous frame to achieve better compression. This phenomenon is obviously more common in large-span inter-frame encoding and decoding with bidirectional references, because the motion is greater and the uncertainty is greater. Sometimes, a certain area of ​​the current frame cannot be found in either reference frame, so it is necessary to reduce the weight of the reference frame and increase the weight of the current frame's own features.

[0046] In response to the above problems, the present disclosure proposes a video encoding method that can adaptively adjust the weight of the reference frame in the fusion process. Specifically, based on the difference information between the reference frame and the current frame, the weight of each reference frame is determined, and the encoding information of each reference frame is fused based on the weight of each reference frame. Subsequently, the current frame is encoded based on the fused encoding information, so that the contribution of each reference frame in encoding the current frame can be dynamically adjusted, thereby effectively solving the problem of unbalanced reference value of reference frames in bidirectional reference video coding, thereby improving end-to-end video coding efficiency. In addition, a scale information transmission mechanism is designed for multi-scale context, which follows the scale conversion direction of first reducing the scale and then increasing the scale during the encoding and decoding process of the encoded frame, that is, at the encoding end, information is transmitted from large-scale features to small-scale features, and at the decoding end, information is transmitted from small scale to large scale. This mechanism can effectively utilize the information of multi-scale context features, thereby further improving coding efficiency.

[0047] Hereinafter, a video encoding method and apparatus, a video decoding method and apparatus, an electronic device, and a storage medium according to exemplary embodiments of the present disclosure will be described in detail with reference to the accompanying drawings.

[0048] Figure 7 is a flowchart of a video encoding method according to an exemplary embodiment of the present disclosure, as shown in FIG. Figure 7 As shown, the video encoding method includes the following steps: In step S701 , difference information between a plurality of reference frames of the current frame and the current frame is obtained.

[0049] As an example, the above-mentioned difference information can be difference information in the pixel domain, difference information in the feature domain, or difference information such as content information, clarity information, brightness information, etc., and this disclosure does not limit this. Assuming that the actual operation of this disclosure falls on the feature domain, the difference information in step S701 is the difference information in the feature domain. It should be noted that the method of this disclosure can act on the feature domain, or on the pixel domain or other domains, and this disclosure does not limit this.

[0050] As an example, the above-mentioned multiple reference frames include forward reference frames and / or backward reference frames, and may be multiple reference frames in the same direction or multiple reference frames in two directions, which is not limited in the present disclosure.

[0051] In step S702 , weights corresponding to the plurality of reference frames are determined based on the difference information.

[0052] As an example, a smaller weight can be determined for a reference frame that has a larger difference with the current frame, i.e., a reference frame that has a smaller similarity with the current frame, and a larger weight can be determined for a reference frame that has a smaller difference with the current frame, i.e., a reference frame that has a larger similarity with the current frame, without limitation of the present disclosure.

[0053] In step S703, the encoding information of each of the plurality of reference frames is fused based on the respective weights corresponding to the plurality of reference frames, to obtain fused encoding information.

[0054] As an example, the encoding information of each of the plurality of reference frames can be weighted summed based on the respective weights corresponding to the plurality of reference frames, and the weighted sum result is taken as the fused encoding information.

[0055] In step S704, the current frame is encoded based on the fused encoding information, to obtain an encoded frame of the current frame.

[0056] According to an example embodiment of the present disclosure, step S701 can be implemented in the following manner: the prediction frame of each reference frame and the current frame are input into a third deep residual block pre-trained, to extract features of the prediction frame of each reference frame and the current frame; and the extracted features are input into a deep separable convolution pre-trained, to obtain difference information of each reference frame and the current frame. Through this embodiment, the corresponding difference information can be quickly and accurately obtained based on the pre-trained deep residual block and deep separable convolution.

[0057] As an example, the prediction frame of each reference frame can be obtained in the following manner:

[0058] wherein warp represents a warping operation, and respectively represent the prediction frame of the front reference frame and the prediction frame of the rear reference frame, i.e., the motion-compensated features, and this step is to align the current frame with the reference frame in pixels; and respectively represent the reconstructed frame of the front reference frame and the reconstructed frame of the rear reference frame, and respectively represent the reconstructed optical flow of the front reference frame and the reconstructed optical flow of the rear reference frame.

[0059] It should be noted that the above formula is expressed in terms of image frames in the pixel domain, and the difference information can be obtained based on the difference information in the pixel domain and the difference information in the frequency domain. and Calculate the difference information and use the difference information in the pixel domain for subsequent processing; if the actual operation falls on the feature domain, the frame and optical flow in the above formula can be converted to the feature domain to obtain the corresponding features in the feature domain, and the compensation processing of the above formula can be performed to obtain the difference information in the feature domain, and the difference information in the feature domain can be used for subsequent processing; the difference information in the pixel domain can also be converted into the difference information in the feature domain; based on the pixel domain and The difference information on the feature domain is directly obtained, which is not limited in this disclosure.

[0060] As an example, the present disclosure does not limit the training process of the above-mentioned third depth residual block, as long as the features of the predicted frame and the current frame of each reference frame can be extracted; the present disclosure does not limit the training process of the above-mentioned depth separable convolution, as long as the difference information between the reference frame and the current frame can be obtained based on the extracted features of the input.

[0061] According to an exemplary embodiment of the present disclosure, the encoding information of each reference frame in the plurality of reference frames includes encoding information at a predetermined number of scales; wherein step S702 may be implemented as follows: for each scale in the predetermined number of scales, based on difference information, determining a weight of each of the plurality of reference frames at the current scale; wherein step S703 may be implemented as follows: for each scale in the predetermined number of scales, based on the weights of each of the plurality of reference frames at the current scale, fusing the encoding information of each of the plurality of reference frames at the current scale to obtain fused encoding information at the current scale. Through this embodiment, multiple reference frames are fused at multiple scales to obtain richer fused information.

[0062] As an example, when the encoding information of each reference frame in multiple reference frames includes encoding information at a predetermined number of scales, the encoding information at the predetermined number of scales of each reference frame in the previous and next reference frames can be obtained through a multi-scale context feature extraction module.

[0063] Assuming that the actual operation of the present disclosure falls on the feature domain, the features of each reference frame at each scale in a predetermined number of scales are obtained. Specifically, for multiple reference frames, the features of the reconstructed frame on the feature domain of the reference frame are obtained. 、 Input the multi-scale context feature extraction module to obtain the features of the reference frame at each scale in the predetermined number of scales. Assuming the predetermined number is 3, the features of the previous and next reference frames at each scale are and ,in, , , .

[0064] The features of the previous and next reference frames at each scale can be obtained as follows: (1) in, FE represents the multi-scale contextual feature extraction module; 48, 64, and 96 denote the number of channels at three scales, respectively; H and W denote the length and width of the image. These contextual features represent the information predicted from the current frame after the two reference frames have undergone motion compensation. In DCVC-B, these features are fused with the features of the current frame at the corresponding scales during the transformation and inverse transformation processes (i.e., during the encoder and decoder processing). Specifically, the features of the current frame and the two reference frames are concatenated channel by channel and fused through subsequent convolutional layers to achieve encoding and decoding of the current frame.

[0065] As an example, after obtaining the coding information of each reference frame at a predetermined number of scales, for each scale in the predetermined number of scales, the weight of each of the multiple reference frames at the current scale can be determined based on the difference information, and then based on the weights of each of the multiple reference frames at the current scale, the coding information of each of the multiple reference frames at the current scale is fused to obtain the fused coding information at the current scale for use in subsequent encoding of the current frame.

[0066] According to an exemplary embodiment of the present disclosure, determining the weights of each of the multiple reference frames at the current scale based on the difference information may include: obtaining the weight of each reference frame at the current scale based on the difference information and the fused encoded information at the previous scale, wherein the previous scale is an adjacent scale larger than the current scale, and when the current scale is the largest scale among a predetermined number of scales, the fused encoded information at the previous scale is empty. Through this embodiment, information is transmitted from large-scale features to small-scale features at the encoding end, and subsequently, information can be transmitted from small-scale features to large-scale features at the decoding end. This scale information transmission mechanism can effectively utilize information from multi-scale contextual features, thereby further improving rate-distortion performance.

[0067] As an example, the difference information and the fused encoded information at the previous scale can be input into a pre-trained depth residual block to obtain weights for each of the multiple reference frames at the current scale. This disclosure does not limit the training process of the pre-trained depth residual block; as long as the weights for each of the multiple reference frames at the current scale can be obtained based on the difference information and the fused encoded information at the previous scale, the training process will be sufficient.

[0068] As an example, after obtaining the weight of each reference frame at the current scale, the encoded information of each reference frame at the current scale can be weighted and summed based on the weight of each reference frame at the current scale to obtain a weighted sum feature. The weighted sum feature is then input into a pre-trained deep residual block to obtain fused encoded information at the current scale. This disclosure does not limit the training process of the pre-trained deep residual block, as long as the fused encoded information at the current scale can be obtained based on the weighted sum feature.

[0069] According to an exemplary embodiment of the present disclosure, before fusing the coding information of multiple reference frames at the current scale based on the weights of each of the multiple reference frames at the current scale to obtain the fused coding information at the current scale, basic features at the current scale may be obtained based on the difference information and the fused coding information at a previous scale, wherein the previous scale is an adjacent scale larger than the current scale, and when the current scale is the maximum scale among a predetermined number of scales, the fused coding information at the previous scale is empty, and the basic features are used to adjust the influence of the weight of each reference frame at the current scale on the fused coding information at the current scale; wherein, fusing the coding information of multiple reference frames at the current scale based on the weights of each of the multiple reference frames at the current scale to obtain the fused coding information at the current scale may include: fusing the coding information of multiple reference frames at the current scale based on the weights of each of the multiple reference frames at the current scale and the basic features at the current scale to obtain the fused coding information at the current scale.

[0070] Through this embodiment, basic features are introduced for each scale, which not only adjusts the influence of the weight of each reference frame at the current scale on the features of the current frame at the current scale, but also avoids the situation where the features of the corresponding scales of all reference frames cannot accurately provide reference information. For example, when the weights of all reference frames are 0, the generated features of the corresponding scale of the current frame are inaccurate.

[0071] According to an exemplary embodiment of the present disclosure, the weights and basic features of each of the multiple reference frames at the current scale are obtained by inputting the difference information and the fused encoded information at the previous scale into a pre-trained second deep residual block to obtain the weights and basic features of each of the multiple reference frames at the current scale. Through this embodiment, using a pre-trained deep residual block, the weights and basic features can be quickly and conveniently obtained.

[0072] As an example, the difference information and the fused encoded information at the previous scale can be input into a pre-trained depth residual block to obtain the weights and basic features of each of the multiple reference frames at the current scale. This disclosure does not limit the training process of the pre-trained depth residual block, as long as the basic features of the weights of each of the multiple reference frames at the current scale can be obtained based on the difference information and the fused encoded information at the previous scale.

[0073] According to an exemplary embodiment of the present disclosure, based on the weights of each of the multiple reference frames at the current scale and the basic features at the current scale, the encoding information of the multiple reference frames at the current scale is fused to obtain the fused encoding information at the current scale. This may include: based on the weight of each reference frame at the current scale, weighted summing the encoding information of each reference frame at the current scale to obtain a first feature; summing the first feature and the basic features at the current scale to obtain a second feature; and inputting the second feature into a pre-trained first depth residual block to obtain the fused encoding information at the current scale. Through this embodiment, by using weights to perform weighted summing of the reference frames and further summing them with the basic features, it is possible to conveniently and quickly achieve dynamic adjustment of the contribution of each reference frame in generating the current frame.

[0074] As an example, it is still assumed that the actual operation of the present disclosure falls on the feature domain and the predetermined number is 3. At this time, the fused coding information at each scale, that is, the fused feature at each scale is You can obtain it in the following ways:

[0075]

[0076]

[0077] in, and Respectively represent the reference frames before and after the encoding end in i The weights on the scale, Indicates the encoding end i The basic features at each scale, Indicates the encoding end i Depth residual blocks at different scales.

[0078] As an example, the determination of the weight and the generation of the fused feature on the multi-scale are described in combination, and specifically, the following processing is sequentially performed on each scale in the predetermined number of scales in order from large scale to small scale: the base feature on the current scale and the weight of each reference frame on the current scale are obtained based on the difference information and the fused feature on the last scale, wherein the fused feature on the last scale is empty when the current scale is the largest scale in the predetermined number of scales; the fused feature on the current scale is obtained based on the feature of each reference frame on the current scale, the weight and the base feature on the current scale.

[0079] According to the exemplary embodiments of the present disclosure, the fusion operation of the present disclosure can be implemented by a bidirectional reference adaptive fusion (BRAF) module, i.e., the BRAF module performs the operation of fusing the encoding information of each reference frame in the plurality of reference frames based on the difference information of the prediction frame of each reference frame and the current frame. Specifically, the BRAF module obtains the base feature of each reference frame based on the prediction frame of each reference frame and the current frame and ) and the current frame x t The difference in the pixel domain generates the weight information of the reference frame, and the adaptive allocation of the weight is obtained by the multi-scale weight information transmission. Further, in order to avoid the case that the context features of the front and rear reference frames cannot accurately provide reference information, an independent reference, i.e., the base feature, can be provided, and the influence degree of the weights of the front and rear reference frames on the fusion can also be adjusted. , and The dimensions of the three are the same, and , i indicates the scale.

[0080] Generally, the encoding end and the decoding end each contain a respective BRAF module. The BRAF module of the encoding end is introduced first, and the BRAF module of the decoding end will be introduced later: Still assuming that the actual operation of the present disclosure falls in the feature domain, the BRAF module of the encoding end is essentially to fuse the prediction frames of the front and rear reference frames x t and the prediction frame of the current frame , These three images in the pixel domain are used as a guide to determine the weight, and then the is fused based on the determined weight, which can be specifically represented as follows:

[0081] wherein, Indicates the BRAF module on the encoding side.

[0082] Figure 8 The structure of the BRAF module at the encoding end is shown, Figure 8 As shown, Input the pre-trained DB (9,48) and extract the features of the predicted frame of each reference frame and the encoded frame of the current frame ; Then the extracted features are input into the pre-trained deep separable convolution DConv (48,48) to obtain the difference information between each reference frame and the current frame, which can also be called similarity. The difference information is then input into the pre-trained deep residual block DB (48,48×3) to obtain the weights of the previous and next reference frames on the minimum scale. , and basic features ; Then, based on the weight and Features of the previous and next reference frames at the largest scale and Perform weighted summation and add basic features to the weighted summation result , the final summation result is input into the pre-trained deep residual block DB (48,48), and then the fused features at the minimum scale are obtained, that is, the predicted features of the current frame at the minimum scale; the fused features are The fusion process of entering the next scale is similar to the previous one, except that when determining the weight, not only the difference information but also the fused features at the previous scale must be considered. This will not be explained here until the fused features at all scales are obtained.

[0083] It should be noted that Figure 8 In DB(X,Y) that appears in [1], X represents the number of input channels and Y represents the number of output channels. DConv(X,Y) is similar and will not be described here. Figure 8 Appeared in It means downsampling by 2 times; Figure 9 A structure of a depth residual block DB with a step size of s is shown, but this disclosure is not limited to this.

[0084] In getting Afterwards, based on For the current frame x t Encoding can be done as follows:

[0085] in, Enc Indicates the encoder responsible for transforming the current frame. Its specific structural design can refer to DCVC-B; QIt represents the quantization operation, and the subsequent entropy coding and decoding processes are omitted here.

[0086] Figure 10 is a flowchart of a video decoding method according to an exemplary embodiment of the present disclosure, as shown in FIG. Figure 10 As shown, the video decoding method includes the following steps: In step S1001 , difference information between a plurality of reference frames of a current frame and a coding frame of the current frame is obtained.

[0087] As an example, the above-mentioned multiple reference frames include forward reference frames and / or backward reference frames, and may be multiple reference frames in the same direction or multiple reference frames in two directions, which is not limited in the present disclosure.

[0088] In step S1002 , weights corresponding to the plurality of reference frames are determined based on the difference information.

[0089] As an example, for a reference frame that is more different from the current frame, that is, a reference frame that is less similar to the current frame, a smaller weight can be determined, and for a reference frame that is less different from the current frame, that is, a reference frame that is more similar to the current frame, a larger weight can be determined. This disclosure does not limit this.

[0090] In step S1003 , based on the weights corresponding to the multiple reference frames, the decoded information of each reference frame in the multiple reference frames is fused to obtain fused decoded information.

[0091] As an example, based on the weights corresponding to the multiple reference frames, the decoding information of each reference frame in the multiple reference frames may be weighted summed, and the weighted sum result may be used as the fused decoding information.

[0092] In step S1004 , the coded frame of the current frame is decoded based on the fused decoding information.

[0093] Through the above-mentioned video decoding method, the decoding information of each reference frame is fused based on the difference information between the reference frame and the coded frame of the current frame, and then the current frame is decoded based on the fused decoding information, so that the contribution of each reference frame in decoding the coded frame of the current frame can be dynamically adjusted, thereby effectively solving the problem of unbalanced reference value of reference frames in bidirectional reference video coding, thereby improving end-to-end video coding efficiency.

[0094] According to an exemplary embodiment of the present disclosure, step S1001 can be implemented as follows: the predicted frame of each reference frame and the coded frame of the current frame are input into a pre-trained pixel feature alignment module to extract features of the predicted frame of each reference frame and the coded frame of the current frame, wherein the pixel feature alignment module includes a depthwise separable convolution and a depthwise residual block; the extracted features are input into the pre-trained depthwise separable convolution to obtain difference information between the coded frames of each reference frame and the current frame. Through this embodiment, the corresponding difference information can be quickly and accurately obtained based on the pre-trained depthwise residual block and depthwise separable convolution.

[0095] As an example, the method of obtaining the predicted frame of each reference frame is similar to that of the encoding end, which will not be discussed here.

[0096] As an example, the present disclosure does not limit the training process of the above-mentioned pixel feature alignment module, as long as the features of the predicted frame of each reference frame and the encoded frame of the current frame can be extracted; the present disclosure does not limit the training process of the above-mentioned depth-separable convolution, as long as the difference information between the reference frame and the current frame can be obtained based on the extracted features of the input.

[0097] According to an exemplary embodiment of the present disclosure, the decoded information of each reference frame in the plurality of reference frames includes decoded information at a predetermined number of scales. Step 1002 may be implemented as follows: for each scale in the predetermined number of scales, based on the difference information, a weight of each of the plurality of reference frames at the current scale is determined. Step 1002 may be implemented as follows: for each scale in the predetermined number of scales, based on the weights of each of the plurality of reference frames at the current scale, the decoded information of each of the plurality of reference frames at the current scale is fused to obtain fused decoded information at the current scale. This embodiment fuses the plurality of reference frames at the multiple scales to obtain richer fused information.

[0098] As an example, when the decoding information of each reference frame in multiple reference frames includes decoding information at a predetermined number of scales, the decoding information at the predetermined number of scales of each reference frame in the previous and next reference frames can be obtained through a multi-scale context feature extraction module.

[0099] Assuming that the actual operation of the present disclosure falls on the feature domain, the features of each reference frame at each scale in a predetermined number of scales are obtained. Specifically, for each reference frame in the multiple reference frames, the features of the reconstructed frame of the current reference frame on the feature domain are obtained. 、 Input the multi-scale context feature extraction module to obtain the features of the current reference frame at each scale in the predetermined number of scales. Assuming the predetermined number is 3, the features of the previous and next reference frames at each scale are and which can be obtained by formula (1) as above, and will not be discussed here. According to an example embodiment of the present disclosure, determining the weight of each of the plurality of reference frames at the current scale based on the difference information can comprise: obtaining the weight of each of the plurality of reference frames at the current scale based on the difference information and the fused decoded information at the previous scale, wherein the previous scale is an adjacent scale smaller than the current scale, and the fused decoded information at the previous scale is empty when the current scale is the largest scale in the predetermined number of scales. Through this embodiment, the information is conducted from large-scale features to small-scale features at the encoding end, and the information can be conducted from small-scale features to large-scale features at the subsequent decoding end. This scale information conducting mechanism can effectively utilize the information of multi-scale context features, thereby further improving the rate-distortion performance.

[0100] As an example, the difference information and the fused decoded information at the previous scale can be input into a pre-trained deep residual block to obtain the weight of each of the plurality of reference frames at the current scale. The training process of the pre-trained deep residual block is not limited by the present disclosure, as long as the weight of each of the plurality of reference frames at the current scale can be obtained based on the difference information and the fused decoded information at the previous scale.

[0101] As an example, after obtaining the weight of each of the plurality of reference frames at the current scale, the decoded information of each of the plurality of reference frames at the current scale can be weighted and summed based on the weight of each of the plurality of reference frames at the current scale to obtain weighted sum features. Then, the weighted sum features can be input into a pre-trained deep residual block to obtain the fused decoded information at the current scale. The training process of the pre-trained deep residual block is not limited by the present disclosure, as long as the fused decoded information at the current scale can be obtained based on the weighted sum features.

[0102] According to an exemplary embodiment of the present disclosure, before fusing the decoded information of multiple reference frames at the current scale based on the weights of each of the multiple reference frames at the current scale to obtain the fused decoded information at the current scale, basic features at the current scale may be obtained based on the difference information and the fused decoded information at a previous scale, wherein the previous scale is an adjacent scale smaller than the current scale, and when the current scale is the minimum scale among a predetermined number of scales, the fused decoded information at the previous scale is empty, and the basic features are used to adjust the influence of the weight of each reference frame at the current scale on the fused decoded information at the current scale; wherein, fusing the decoded information of multiple reference frames at the current scale based on the weights of each of the multiple reference frames at the current scale to obtain the fused decoded information at the current scale may include: fusing the decoded information of multiple reference frames at the current scale based on the weights of each of the multiple reference frames at the current scale and the basic features at the current scale to obtain the fused decoded information at the current scale.

[0103] Through this embodiment, basic features are introduced for each scale, which not only adjusts the influence of the weight of each reference frame at the current scale on the features of the current frame at the current scale, but also avoids the situation where the features of the corresponding scales of all reference frames cannot accurately provide reference information. For example, when the weights of all reference frames are 0, the generated features of the corresponding scale of the current frame are inaccurate.

[0104] According to an exemplary embodiment of the present disclosure, the weights and basic features of each of the multiple reference frames at the current scale are obtained by inputting the difference information and the fused decoded information at the previous scale into a pre-trained fourth deep residual block to obtain the weights and basic features of each reference frame at the current scale. This embodiment uses a pre-trained deep residual block to quickly and conveniently obtain the weights and basic features.

[0105] As an example, the difference information and the fused decoded information at the previous scale can be input into a pre-trained depth residual block to obtain the weights and basic features of each of the multiple reference frames at the current scale. This disclosure does not limit the training process of the pre-trained depth residual block, as long as the basic features of the weights of each of the multiple reference frames at the current scale can be obtained based on the difference information and the fused decoded information at the previous scale.

[0106] According to an exemplary embodiment of the present disclosure, based on the weights of each of the multiple reference frames at the current scale and the basic features at the current scale, the decoded information of the multiple reference frames at the current scale is fused to obtain the fused decoded information at the current scale, including: based on the weight of each reference frame at the current scale, weighted summing the decoded information of each reference frame at the current scale to obtain a third feature; summing the third feature with the basic features at the current scale to obtain a fourth feature; and inputting the fourth feature into a pre-trained third depth residual block to obtain the fused decoded information at the current scale. Through this embodiment, by using weights to perform weighted summing of the reference frames and further summing them with the basic features, it is possible to conveniently and quickly achieve dynamic adjustment of the contribution of each reference frame in generating the current frame.

[0107] As an example, it is still assumed that the actual operation of the present disclosure falls on the feature domain and the predetermined number is 3. At this time, the decoded information after fusion at each scale, that is, the feature after fusion at each scale is You can obtain it in the following ways:

[0108]

[0109]

[0110] in, and Respectively represent the reference frames before and after the decoding end in i The weights on the scale, Indicates the decoding end i The basic features at each scale, Indicates the first i Depth residual blocks at different scales.

[0111] According to an exemplary embodiment of the present disclosure, the specific fusion operation of the present disclosure can be implemented by a bidirectional reference adaptive fusion (BRAF) module. Generally, the encoding end and the decoding end both include their own BRAF modules. The BRAF module of the decoding end is described below: First, the decoding end parses the code stream to obtain the current frame x t Coded frames Then, still assuming that the actual operation of the present disclosure falls on the feature domain, the BRAF module at the decoding end essentially converts the coded frame and the predicted frames of the previous and next reference frames , The images in these three pixel domains are used as a guide to determine the weights, and then the weights are determined respectively. The fusion can be expressed as follows:

[0112] in, Indicates the BRAF module at the decoding end. Generated sequentially by the BRAF module, this enables the network to better distinguish the weights of multiple reference frames at the decoding end.

[0113] Figure 11 The structure of the BRAF module at the decoding end is shown, Figure 11 As shown, Input the pre-trained pixel feature alignment module (PFA) to extract the features of the predicted frame of each reference frame and the encoded frame of the current frame ; Then the extracted features are input into the pre-trained deep separable convolution DConv (96,96) to obtain the difference information between each reference frame and the current frame, which can also be called similarity. The difference information is then input into the pre-trained deep residual block DB (96,96×3) to obtain the weights of the previous and next reference frames on the minimum scale. , and basic features ; Then, based on the weight and Features of the previous and next reference frames at the largest scale and Perform weighted summation and add basic features to the weighted summation result , the final summation result is input into the pre-trained deep residual block DB (96,96), and then the fused features at the minimum scale are obtained, that is, the predicted features of the current frame at the minimum scale; the fused features are The fusion process of entering the next scale is similar to the previous one, except that when determining the weight, not only the difference information but also the fused features at the previous scale must be considered. This will not be explained here until the fused features at all scales are obtained.

[0114] It should be noted that Figure 11 In DB(X,Y) that appears in [1], X represents the number of input channels and Y represents the number of output channels. DConv(X,Y) is similar and will not be described here. Figure 11 Appeared in It means upsampling by 2 times; Figure 12 A PFA structure is shown. The PFA first Mapped to and In the same dimensional domain (note here is two separate 2-layer deep convolution). Then and the front and back reference frames comparison fusion, obtaining the reference difference guidance ability of After that, the difference information is transmitted in different scales along the encoding direction in the same way as encoding. It should be noted that, Figure 12 UP in the convolution kernel PixelShuflle is up-sampled, and all convolutions are 3x3 by default.

[0115] After obtaining , the encoded frame can be decoded based on , which can be as follows:

[0116] Among them, Dec represents the decoder responsible for transforming the encoded frame, and the specific structure design can refer to DCVC-B; and the subsequent quantization operation, entropy encoding and entropy decoding process is omitted.

[0117] It should be noted that the biggest difference between the BRAF module of the encoding end and the BRAF module of the decoding end is that the BRAF module of the encoding end generates guidance information uses the original image of the current frame, while the BRAF module of the decoding end generates uses the decoded latent feature, i.e. the encoded frame of the current frame.

[0118] In addition, it should be noted that the network structure of the present disclosure is an example, and other network structures can also be used, and the present disclosure does not limit it. The adaptive weight fusion of the present disclosure can act on the feature domain, or on the image domain or other domains, and the present disclosure does not limit it.

[0119] In summary, the present disclosure focuses on bidirectional reference end-to-end video compression encoding, and the related art does not make good use of the information difference of bidirectional reference frames. When compressing and transforming, the reference frames are often fused in equal proportions, which ignores the problem of imbalance in the reference value of the front and back reference frames in bidirectional reference. In order to solve this problem, the present disclosure proposes a bidirectional reference adaptive fusion module, which can adaptively determine the reference proportion of bidirectional reference frames, thereby better improving the compression performance of the network. Specifically, the difference between the current frame and the reference frame is analyzed to generate the weight of each reference frame, and the context information of the reference frame is selectively fused based on the weight, effectively solving the problem of uneven reference frames in bidirectional reference encoding, thereby improving the end-to-end video coding efficiency.

[0120] Figure 13 FIG. 1 is a block diagram of a video encoding device according to an exemplary embodiment of the present disclosure. Figure 13 The device includes a first acquisition unit 1302, a first determination unit 1304, a first fusion unit 1306 and an encoding unit 1308.

[0121] The first acquisition unit 1302 is configured to obtain difference information between multiple reference frames of the current frame and the current frame; the first determination unit 1304 is configured to determine the weights corresponding to each of the multiple reference frames based on the difference information; the first fusion unit 1306 is configured to fuse the encoding information of each reference frame in the multiple reference frames based on the weights corresponding to each of the multiple reference frames to obtain fused encoding information; the encoding unit 1308 is configured to encode the current frame based on the fused encoding information to obtain an encoded frame of the current frame.

[0122] Figure 14 FIG. 1 is a block diagram of a video decoding device according to an exemplary embodiment of the present disclosure. Figure 14 The device includes a second acquiring unit 1402 , a second determining unit 1404 , a second fusion unit 1406 and a decoding unit 1408 .

[0123] The second acquisition unit 1402 is configured to obtain difference information between multiple reference frames of the current frame and the encoded frame of the current frame; the second determination unit 1404 is configured to determine the weights corresponding to each of the multiple reference frames based on the difference information; the second fusion unit 1406 is configured to fuse the decoding information of each reference frame in the multiple reference frames based on the weights corresponding to each of the multiple reference frames to obtain fused decoding information; the decoding unit 1408 is configured to decode the encoded frame of the current frame based on the fused decoding information.

[0124] Figure 15 A computing environment 1510 is shown coupled to a user interface 1550. The computing environment 1510 may be part of a data processing server. The computing environment 1510 includes a processor 1520, a memory 1530, and an input / output (I / O) interface 1540.

[0125] Processor 1520 generally controls the overall operation of computing environment 1510, such as operations associated with display, data acquisition, data communication, and image processing. Processor 1520 may include one or more processors for executing instructions to perform all or some of the steps in the above-described methods. In addition, processor 1520 may include one or more modules that facilitate interaction between processor 1520 and other components. The processor may be a central processing unit (CPU), a microprocessor, a single-chip microcomputer, a graphics processing unit (GPU), etc.

[0126] Memory 1530 is configured to store various types of data to support the operation of computing environment 1510. Memory 1530 may include predetermined software 1532. Examples of such data include instructions for any application or method operating on computing environment 1510, video data sets, image data, and the like. Memory 1530 may be implemented using any type of volatile or non-volatile memory device, or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk, or optical disk.

[0127] I / O interface 1540 provides an interface between processor 1520 and peripheral interface modules (e.g., a keyboard, click wheel, buttons, etc.). Buttons may include, but are not limited to, a home button, a start scan button, and a stop scan button. I / O interface 1540 may be coupled to an encoder and a decoder.

[0128] In an embodiment, a non-transitory computer-readable storage medium including, for example, a plurality of programs in a memory 1530 and / or storing a bitstream generated by the above-mentioned video encoding method or a bitstream to be decoded by the above-mentioned video decoding method is also provided. The plurality of programs can be executed by a processor 1520 in a computing environment 1510 to perform the above-mentioned method. In one example, the plurality of programs can be executed by a processor 1520 in a computing environment 1510 to (for example, from Figure 2 The video encoder 20 in the computing environment 1510 receives a bit stream or data stream including encoded video information (e.g., video blocks representing encoded video frames, and / or one or more associated syntax elements, etc.), and can also be executed by the processor 1520 in the computing environment 1510 to perform the above-mentioned decoding method according to the received bit stream or data stream. In another example, the multiple programs can be executed by the processor 1520 in the computing environment 1510 to perform the above-mentioned encoding method to encode the video information (e.g., video blocks representing video frames, and / or one or more associated syntax elements, etc.) into a bit stream or data stream, and can also be executed by the processor 1520 in the computing environment 1510 to (e.g., to Figure 3 Alternatively, a non-transitory computer-readable storage medium may store the bit stream or data stream generated by the encoder (e.g., Figure 2 The video encoder 20 in FIG. 1 generates a video signal for use by a decoder (eg, Figure 3A bitstream or data stream including encoded video information (e.g., video blocks representing encoded video frames, and / or associated one or more syntax elements, etc.) used by the video decoder 30 in the video decoder 30 when decoding video data. The non-transitory computer-readable storage medium may be, for example, a ROM, a random access memory (RAM), a CD-ROM, a magnetic tape, a floppy disk, an optical data storage device, etc.

[0129] In an embodiment, the computing environment 1510 may be implemented by one or more ASICs, DSPs, digital signal processing devices (DSPDs), programmable logic devices (PLDs), FPGAs, GPUs, controllers, microcontrollers, microprocessors, or other electronic components for performing the above methods.

[0130] According to an embodiment of the present disclosure, an electronic device may be provided, which includes at least one memory and at least one processor, wherein a set of computer-executable instructions is stored in the at least one memory, and when the computer-executable instruction set is executed by the at least one processor, a video encoding method and / or a video decoding method according to an embodiment of the present disclosure is executed.

[0131] As an example, the electronic device may be a PC, a tablet device, a personal digital assistant, a smartphone, or other device capable of executing the above-mentioned instruction set. Here, the electronic device 1000 does not necessarily have to be a single electronic device, but may also be a collection of any devices or circuits capable of executing the above-mentioned instructions (or instruction sets) individually or in combination. The electronic device may also be part of an integrated control system or system manager, or may be configured as a portable electronic device that is interconnected with a local or remote interface (e.g., via wireless transmission). In addition, the electronic device may also include a video display (such as a liquid crystal display) and a user interaction interface (such as a keyboard, mouse, touch input device, etc.). All components of the electronic device may be connected to each other via a bus and / or a network.

[0132] In an embodiment, a method for storing a bitstream is provided, comprising: generating a bitstream according to the video encoding method; and storing the bitstream.

[0133] In an embodiment, a method for sending a bit stream is provided, comprising: generating a bit stream according to the above-mentioned video encoding method; and sending the bit stream.

[0134] According to an embodiment of the present disclosure, a computer program product having instructions for storing a bitstream is also provided, wherein the bitstream includes video data generated by the above-mentioned video encoding method and / or video data decoded by the video decoding method. In an embodiment, a computer program product is also provided, including, for example, multiple programs in a memory 1530, which can be executed by a processor 1520 in a computing environment 1510 to perform the above-mentioned method. For example, the computer program product may include a non-transitory computer-readable storage medium.

[0135] Unless otherwise specifically stated, the order of steps of the method according to the present disclosure is intended to be illustrative only, and the steps of the method according to the present disclosure are not limited to the order specifically described above, but can be changed according to actual circumstances. In addition, at least one of the steps of the method according to the present disclosure can be adjusted, combined, or deleted according to actual needs.

[0136] Other embodiments of the present disclosure will readily occur to those skilled in the art after considering the specification and practicing the invention disclosed herein. This disclosure is intended to cover any variations, uses, or adaptations of the present disclosure that follow the general principles of the present disclosure and include common knowledge or customary techniques in the art not disclosed herein. The description and examples are to be considered as exemplary only, with the true scope and spirit of the present disclosure being indicated by the appended claims.

[0137] It should be understood that the present disclosure is not limited to the exact structures that have been described above and shown in the drawings, and that various modifications and changes can be made without departing from the scope thereof. The scope of the present disclosure is limited only by the appended claims.

Claims

1. A video encoding method, characterized in that: include: Obtaining difference information between a plurality of reference frames of a current frame and the current frame; Determining weights corresponding to the plurality of reference frames based on the difference information; fusing the coding information of each reference frame in the multiple reference frames based on the weights corresponding to the multiple reference frames to obtain fused coding information; The current frame is encoded based on the fused encoding information to obtain an encoded frame of the current frame.

2. The method according to claim 1, wherein The encoding information of each reference frame in the plurality of reference frames includes encoding information at a predetermined number of scales; The step of determining the weights corresponding to the plurality of reference frames based on the difference information includes: For each scale of the predetermined number of scales, determining a weight of each of the plurality of reference frames at the current scale based on the difference information; The fusing of the coding information of each reference frame in the multiple reference frames based on the weights corresponding to the multiple reference frames to obtain the fused coding information includes: For each scale of the predetermined number of scales, encoding information of each of the multiple reference frames at the current scale is fused based on the weights of each of the multiple reference frames at the current scale to obtain fused encoding information at the current scale.

3. The method according to claim 2, wherein The determining, based on the difference information, a weight of each of the plurality of reference frames at the current scale includes: A weight of each reference frame at a current scale is obtained based on the difference information and fused coded information at a previous scale, wherein the previous scale is an adjacent scale larger than the current scale, and when the current scale is a maximum scale among the predetermined number of scales, the fused coded information at the previous scale is empty.

4. The method according to claim 3, wherein Before fusing the coding information of the multiple reference frames at the current scale based on the weights of the multiple reference frames at the current scale to obtain the fused coding information at the current scale, the method further includes: obtaining basic features at the current scale based on the difference information and fused coded information at a previous scale, wherein the previous scale is an adjacent scale larger than the current scale, and when the current scale is a maximum scale among the predetermined number of scales, the fused coded information at the previous scale is empty, and the basic features are used to adjust the influence of the weight of each reference frame at the current scale on the fused coded information at the current scale; The fusing, based on the weights of the multiple reference frames at the current scale, the coded information of each of the multiple reference frames at the current scale to obtain fused coded information at the current scale includes: Based on the weights of the multiple reference frames at the current scale and the basic features at the current scale, the encoding information of the multiple reference frames at the current scale is fused to obtain fused encoding information at the current scale.

5. The method according to claim 4, wherein The fusing, based on the weights of the multiple reference frames at the current scale and the basic features at the current scale, the coded information of the multiple reference frames at the current scale to obtain the fused coded information at the current scale, includes: Based on the weight of each reference frame at the current scale, performing weighted summation on the encoding information of each reference frame at the current scale to obtain a first feature; Summing the first feature and the basic feature at the current scale to obtain a second feature; The second feature is input into a pre-trained first depth residual block to obtain fused coding information at the current scale.

6. The method according to claim 4, wherein The weights of each of the multiple reference frames at the current scale and the basic features at the current scale are obtained in the following manner: The difference information and the fused encoded information at the previous scale are input into a pre-trained second depth residual block to obtain the weights of multiple reference frames at the current scale and the basic features at the current scale.

7. The method according to claim 1, wherein The obtaining difference information between a plurality of reference frames of the current frame and the current frame includes: Inputting the predicted frame of each reference frame and the current frame into a pre-trained third depth residual block, and extracting features of the predicted frame of each reference frame and the current frame; The extracted features are input into a pre-trained depthwise separable convolution to obtain difference information between each reference frame and the current frame.

8. A video decoding method, characterized in that: include: Obtaining difference information between a plurality of reference frames of a current frame and a coding frame of the current frame; Determining weights corresponding to the plurality of reference frames based on the difference information; fusing the decoded information of each reference frame in the plurality of reference frames based on weights corresponding to the respective reference frames to obtain fused decoded information; The coded frame of the current frame is decoded based on the fused decoding information.

9. The method according to claim 8, wherein The decoded information of each reference frame in the plurality of reference frames includes decoded information at a predetermined number of scales; The step of determining the weights corresponding to the plurality of reference frames based on the difference information includes: For each scale of the predetermined number of scales, determining a weight of each of the plurality of reference frames at the current scale based on the difference information; The fusing the decoding information of each reference frame in the multiple reference frames based on the weights corresponding to the multiple reference frames to obtain the fused decoding information includes: For each scale of the predetermined number of scales, decoding information of each of the multiple reference frames at the current scale is fused based on the weights of each of the multiple reference frames at the current scale to obtain fused decoding information at the current scale.

10. The method according to claim 9, wherein The determining, based on the difference information, a weight of each of the plurality of reference frames at the current scale includes: A weight of each reference frame at a current scale is obtained based on the difference information and fused decoded information at a previous scale, wherein the previous scale is an adjacent scale smaller than the current scale, and when the current scale is a maximum scale among the predetermined number of scales, the fused decoded information at the previous scale is empty.

11. The method according to claim 8, wherein The obtaining difference information between a plurality of reference frames of the current frame and the coded frame of the current frame includes: Inputting the predicted frame of each reference frame and the coded frame of the current frame into a pre-trained pixel feature alignment module to extract features of the predicted frame of each reference frame and the coded frame of the current frame, wherein the pixel feature alignment module includes a depthwise separable convolution and a depthwise residual block; The extracted features are input into a pre-trained depthwise separable convolution to obtain difference information between each reference frame and the encoded frame of the current frame.

12. A video encoding device, characterized in that: include: A first acquiring unit is configured to acquire difference information between a plurality of reference frames of a current frame and the current frame; A first determining unit is configured to determine a weight corresponding to each of the plurality of reference frames based on the difference information; A first fusion unit is configured to fuse the coding information of each reference frame in the multiple reference frames based on the weights corresponding to the multiple reference frames to obtain fused coding information; The encoding unit is configured to encode the current frame based on the fused encoding information to obtain an encoded frame of the current frame.

13. A video decoding device, characterized in that: include: a second acquiring unit, configured to acquire difference information between a plurality of reference frames of a current frame and a coding frame of the current frame; A second determining unit is configured to determine a weight corresponding to each of the plurality of reference frames based on the difference information; A second fusion unit is configured to fuse the decoding information of each reference frame in the multiple reference frames based on the weights corresponding to the multiple reference frames to obtain fused decoding information; The decoding unit is configured to decode the coded frame of the current frame based on the fused decoding information.

14. An electronic device, characterized in that: include: at least one processor; at least one memory storing computer-executable instructions, When the computer-executable instructions are executed by the at least one processor, the computer-executable instructions cause the at least one processor to execute the video encoding method according to any one of claims 1 to 8 and / or the video decoding method according to any one of claims 9 to 16.

15. A computer-readable storage medium, characterized in that When the instructions in the computer-readable storage medium are executed by at least one processor, the at least one processor is prompted to store the bit stream generated by the video encoding method according to any one of claims 1 to 8 and / or the bit stream decoded by the video decoding method according to any one of claims 9 to 16.

Images