End-to-end video coding and decoding method and device and electronic equipment

Abstract

The invention provides an end-to-end video encoding and decoding method and device and electronic equipment, and can be applied to the technical field of video decoding. The method comprises the steps that a brightness coding and decoding network is used for coding and decoding the brightness component of the (n + 1) th frame of the video according to the reconstructed brightness component of the nth frame of the video, the reconstructed brightness component of the (n + 1) th frame, the optical flow of the (n + 1) th frame and the brightness propagation characteristics of the (n + 1) th frame are obtained, and n is a positive integer. Coding and decoding the chroma component of the (n + 1) th frame of the video by using a chroma coding and decoding network according to the optical flow of the (n + 1) th frame and the chroma propagation characteristics of the nth frame of the video to obtain the initial chroma characteristics of the (n + 1) th frame; and performing enhancement processing on the initial chrominance feature of the (n + 1) th frame according to the brightness propagation feature of the (n + 1) th frame to obtain a reconstructed chrominance component of the (n + 1) th frame and the chrominance propagation feature of the (n + 1) th frame. And constructing a reconstructed image of the (n + 1) th frame by using a reconstruction network according to the reconstructed brightness component of the (n + 1) th frame and the reconstructed chrominance component of the (n + 1) th frame.

Description

Technical Field

[0001] This disclosure relates to the field of video coding technology, and more specifically, to an end-to-end video coding and decoding method, apparatus, and electronic device. Background Technology

[0002] The purpose of video encoding and decoding is to reduce the amount of data in video storage and transmission, thereby reducing storage space and saving transmission bandwidth. In related technologies, at the encoding end of the video encoding and decoding framework, motion estimation is performed on the current frame using a reference frame to obtain motion vectors, which are then encoded into a motion bitstream. Next, motion compensation is applied to the motion vectors, and the current frame is further processed to generate a residual signal. This residual signal is then converted into a residual bitstream by a residual encoder. Finally, the motion bitstream and the residual bitstream together constitute the bitstream. At the decoding end, the motion bitstream is first decoded to obtain reconstructed motion vectors, which are then used for residual decoding through motion compensation to recover the reconstructed frame.

[0003] In related technologies, video encoding and decoding are performed in a unified color space, which limits the effect of video frame reconstruction and affects the visual experience. Summary of the Invention

[0004] In view of this, the present disclosure provides an end-to-end video encoding and decoding method, apparatus and electronic device.

[0005] One aspect of this disclosure provides an end-to-end video encoding and decoding method, comprising: using a luma encoding and decoding network to encode and decode the luma component of the (n+1)th frame of the video based on the reconstructed luma component of the nth frame, to obtain the reconstructed luma component of the (n+1)th frame, the optical flow of the (n+1)th frame, and the luma propagation features of the (n+1)th frame, where n is a positive integer; using a chroma encoding and decoding network to encode and decode the chroma component of the (n+1)th frame of the video based on the optical flow of the (n+1)th frame and the chroma propagation features of the nth frame, to obtain the initial chroma features of the (n+1)th frame; performing enhancement processing on the initial chroma features of the (n+1)th frame based on the luma propagation features of the (n+1)th frame, to obtain the reconstructed chroma component of the (n+1)th frame and the chroma propagation features of the (n+1)th frame; and using a reconstruction network to construct the reconstructed image of the (n+1)th frame based on the reconstructed luma component and the reconstructed chroma component of the (n+1)th frame.

[0006] According to embodiments of this disclosure, the color fidelity optimization module in the chroma codec network includes an alignment unit, a fusion unit, a feature enhancement unit, and a reconstruction unit. The process of enhancing the initial chroma features of the (n+1)th frame based on the luminance propagation features of the (n+1)th frame using the chroma codec network to obtain the reconstructed chroma components and chroma propagation features of the (n+1)th frame includes: using the alignment unit to perform multimodal feature alignment on the luminance propagation features and the initial chroma features of the (n+1)th frame, obtaining luminance modal features and chroma modal features; using the fusion unit to fuse the luminance modal features and the chroma modal features and perform nonlinear mapping, obtaining a mask matrix; and using the feature enhancement unit to multiply the mask matrix and the initial chroma features of the (n+1)th frame, obtaining the chroma propagation features of the (n+1)th frame. The reconstruction unit is used to convolve the chroma propagation features of the (n+1)th frame according to the first predetermined convolution kernel to obtain the reconstructed chroma components of the (n+1)th frame.

[0007] According to embodiments of this disclosure, the alignment unit includes a downsampling layer, a first convolutional layer, and a second convolutional layer. The process of using the alignment unit to perform multimodal feature alignment on the luminance propagation features and the initial chrominance features of the (n+1)th frame to obtain luminance modal features and chrominance modal features includes: performing bilinear downsampling on the luminance propagation features of the (n+1)th frame using the downsampling layer to obtain downsampled features; performing a convolution operation on the downsampled features using the first convolutional layer with a second predetermined convolutional kernel to obtain the luminance modal features; and performing a convolution operation on the initial chrominance features of the (n+1)th frame using the second convolutional layer with the second predetermined convolutional kernel to obtain the chrominance modal features.

[0008] According to embodiments of this disclosure, the fusion unit includes a stitching layer, a third convolutional layer, and a nonlinear mapping layer. The process of fusing and nonlinearly mapping the luminance modal features and chrominance modal features using the fusion unit to obtain a mask matrix includes: stitching the luminance modal features and chrominance modal features through channels using the stitching layer to obtain stitched features; performing a convolution operation on the stitched features using the third convolutional layer according to a third predetermined convolutional kernel to obtain fused features; and performing a nonlinear mapping on the fused features using the nonlinear mapping layer according to a predetermined nonlinear mapping function to obtain the mask matrix.

[0009] According to embodiments of this disclosure, both the first predetermined convolution kernel and the third predetermined convolution kernel are 3×3 convolution kernels, and the third convolution kernel is a 1×1 convolution kernel.

[0010] According to embodiments of this disclosure, the chroma encoding / decoding network further includes: a downsampling module, a temporal context mining module, a context encoding module, a context entropy encoding module, and a context decoding module. The above-described method of using the chroma encoding / decoding network to encode and decode the chroma components of the (n+1)th frame of the video based on the optical flow of the (n+1)th frame and the chroma propagation features of the nth frame includes: performing bilinear downsampling on the optical flow of the (n+1)th frame using the downsampling module to obtain downsampled optical flow features; performing multi-scale feature extraction on the downsampled optical flow features and the chroma propagation features of the nth frame using the temporal context mining module to obtain multiple context features, wherein the dimensions of the multiple context features are different from each other; encoding the chroma components of the (n+1)th frame using the context encoding module based on the multiple context features to obtain chroma encoded latent variables; and performing context modeling and probability statistics on the multiple context features using the context entropy encoding module to obtain the probability distribution of the latent variables. Using the aforementioned context decoding module, based on the aforementioned multiple context features, the chroma component bitstream obtained from the chroma encoding latent variables and the probability distribution of the latent variables is decoded to obtain the initial chroma features of the aforementioned (n+1)th frame.

[0011] According to embodiments of this disclosure, the luminance component of each frame of the video is a single-channel image luminance component.

[0012] According to embodiments of this disclosure, the residual blocks used in the above-described luminance encoding / decoding network are residual blocks based on a partially convolutional structure.

[0013] Another aspect of this disclosure provides an end-to-end video encoding and decoding apparatus, comprising: a luminance encoding and decoding module, configured to encode and decode the luminance component of the (n+1)th frame of the video based on the reconstructed luminance component of the (n+1)th frame using a luminance encoding and decoding network, to obtain the reconstructed luminance component of the (n+1)th frame, the optical flow of the (n+1)th frame, and the luminance propagation characteristics of the (n+1)th frame, where n is a positive integer; a chrominance encoding and decoding module, configured to encode and decode the chrominance component of the (n+1)th frame of the video based on the optical flow of the (n+1)th frame and the chrominance propagation characteristics of the (n+1)th frame, to obtain the initial chrominance characteristics of the (n+1)th frame; and to enhance the initial chrominance characteristics of the (n+1)th frame based on the luminance propagation characteristics of the (n+1)th frame, to obtain the reconstructed chrominance component and the chrominance propagation characteristics of the (n+1)th frame; and a construction module, configured to construct the reconstructed image of the (n+1)th frame using a reconstruction network based on the reconstructed luminance component and the reconstructed chrominance component of the (n+1)th frame.

[0014] Another aspect of this disclosure provides an electronic device comprising: one or more processors; and a memory for storing one or more programs, wherein, when the one or more programs are executed by the one or more processors, the one or more processors cause the one or more processors to perform the method as described above.

[0015] According to embodiments of this disclosure, by utilizing a luminance coding / decoding network to encode and decode the luminance component of the (n+1)th frame of a video based on the reconstructed luminance component of the nth frame, the reconstructed luminance component, optical flow, and luminance propagation features of the (n+1)th frame are obtained. This enables fine modeling of the luminance components in the video, aligning with the human eye's heightened sensitivity to brightness. By utilizing a chrominance coding / decoding network to encode and decode the chrominance component of the (n+1)th frame based on the optical flow and chrominance propagation features of the nth frame, the initial chrominance features of the (n+1)th frame are obtained. The initial chrominance features of the (n+1)th frame are then enhanced based on the luminance propagation features, resulting in the reconstructed chrominance component and chrominance propagation features of the (n+1)th frame. This utilizes the high-quality luminance structure information within the luminance propagation features to enhance the details of the initial chrominance features, thereby improving visual reproduction. Therefore, this disclosure can independently model the luminance and chrominance components in a video. Through explicit separation and asymmetric design, it can achieve low-cost or high-reliability video encoding and decoding under the premise of controlled bitrate. At the same time, it can prioritize the allocation of more bits and computing power to the luminance component, which is more critical to visual quality, thereby leveraging its perceptual advantages in the YUV420 format to achieve better reconstruction results and bring a better visual experience. Attached Figure Description

[0016] The above and other objects, features and advantages of this disclosure will become clearer from the following description of embodiments with reference to the accompanying drawings, in which:

[0017] Figure 1 A flowchart illustrating an end-to-end video encoding / decoding method according to an embodiment of the present disclosure is shown schematically.

[0018] Figure 2 This illustration schematically shows a network structure diagram used in an end-to-end video encoding / decoding method according to an embodiment of the present disclosure;

[0019] Figure 3 A schematic diagram of the structure of a luminance codec network according to an embodiment of the present disclosure is shown.

[0020] Figure 4 A schematic diagram illustrating the structure of a chroma codec network according to an embodiment of the present disclosure is shown.

[0021] Figure 5This illustration schematically shows a structural diagram of a color fidelity optimization module according to an embodiment of the present disclosure;

[0022] Figure 6 This illustration schematically shows a comparison of reconstructed images from an end-to-end video encoding / decoding method according to embodiments of the present disclosure and related technologies;

[0023] Figure 7 A schematic block diagram illustrating an end-to-end video codec apparatus according to embodiments of the present disclosure is shown; and

[0024] Figure 8 A block diagram of an electronic device suitable for implementing the end-to-end video encoding and decoding method described above, according to embodiments of the present disclosure, is illustrated. Detailed Implementation

[0025] The embodiments of the present disclosure will now be described with reference to the accompanying drawings. However, it should be understood that these descriptions are exemplary only and are not intended to limit the scope of the disclosure. In the following detailed description, numerous specific details are set forth to provide a thorough understanding of the embodiments of the present disclosure for ease of explanation. However, it will be apparent that one or more embodiments may be practiced without these specific details. Furthermore, descriptions of well-known structures and techniques are omitted in the following description to avoid unnecessarily obscuring the concepts of the present disclosure.

[0026] The terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit this disclosure. The terms “comprising,” “including,” etc., as used herein indicate the presence of the stated features, steps, operations, and / or components, but do not exclude the presence or addition of one or more other features, steps, operations, or components.

[0027] All terms used herein (including technical and scientific terms) have the meanings commonly understood by those skilled in the art, unless otherwise defined. It should be noted that the terms used herein are to be interpreted in a manner consistent with the context of this specification, and not in an idealized or overly rigid way.

[0028] When using expressions such as "at least one of A, B and C", they should generally be interpreted in accordance with the meaning that is commonly understood by those skilled in the art (e.g., "a system having at least one of A, B and C" should include, but is not limited to, a system having A alone, a system having B alone, a system having C alone, a system having A and B, a system having A and C, a system having B and C, and / or a system having A, B and C, etc.).

[0029] Traditional video codec frameworks typically consist of modules such as motion estimation, motion information encoding / decoding, motion compensation, and residual encoding / decoding. At the decoding end of the video codec framework, motion estimation is first performed using a reference frame and the current frame to obtain motion vectors, which are then encoded into a motion bitstream. Subsequently, motion compensation is applied to the motion vectors to assist the current frame in generating a residual signal, which is then converted into a residual bitstream by a residual encoder. Finally, the motion bitstream and the residual bitstream together constitute the bitstream. At the decoding end of the video codec framework, the motion bitstream is first decoded to obtain reconstructed motion vectors, which are then used for residual decoding through motion compensation to recover the reconstructed frame.

[0030] In contrast, end-to-end video codec frameworks leverage the feature modeling capabilities and end-to-end optimization characteristics of deep neural networks to replace individual modules in traditional video codec frameworks with learnable neural network modules, and perform overall optimization based on rate-distortion loss. End-to-end video codec frameworks can be categorized into two types based on their inter-frame redundancy elimination methods: one type is based on residual modeling, which calculates the difference between the predicted reference frame and the current frame to obtain the residual signal and encodes it, thereby removing temporal redundancy; the other type is based on conditional modeling, which adaptively learns the temporal correlation between the reference frame and the current frame through the network, directly modeling inter-frame redundancy under conditional input. Compared to residual coding, conditional modeling-based neural video compression frameworks can learn the temporal context more flexibly, and their theoretical entropy lower bound is typically lower, thus showing greater potential for compression efficiency.

[0031] In the human visual system, luminance and chromaticity are perceived by different types of photoreceptor cells, and the human eye has different sensitivities to luminance and chromaticity information, meaning the human eye is more sensitive to luminance. Based on this characteristic, traditional video encoding and decoding typically use the YUV420 format, which compresses the bitrate by reducing the chromaticity sampling rate while prioritizing the preservation of luminance details. In the YUV420 encoding format, Y represents the luminance component, and U and V represent the chromaticity components. At the same bitrate, preserving the clarity of luminance structure is more effective in improving subjective perception than preserving rich colors. However, existing end-to-end video encoding and decoding methods still primarily optimize video frames in the RGB domain. Even if they support the YUV420 format, they first upsample the video frames to the YUV444 format and then use the RGB encoding model, still mixing luminance and chromaticity information, making it difficult to allocate more encoding resources to luminance.

[0032] In related technologies, the implementation methods of using YUV420 format data for end-to-end video encoding and decoding can be categorized into the following two types:

[0033] The first type is the RGB-domain-based video codec model. This model converts both input and output frames to the YUV420 domain for loss optimization. For example, the input and output of an RGB-domain-based video codec model can both be RGB frames, but during the training phase, the original RGB frames and reconstructed RGB frames can be converted to the YUV420 domain, and the weighted distortion loss of the luminance component Y and chrominance components U and V can be calculated separately. In the actual encoding process, the YUV420 video sequence to be encoded is converted frame by frame into RGB data and input into the video codec model. The RGB reconstruction results output by the video codec model are then converted back to YUV420 for subsequent storage or transmission.

[0034] The second category is video encoding / decoding models based on the YUV domain. These models convert both input and output frames to the YUV420 domain for loss optimization. For example, in encoding / decoding methods such as Deep Contextual Video Compression using Diverse Contexts (DCVC-DC), Deep Contextual Video Compression using Feature Modulation (DCVC-FM), or Deep Contextual Video Compression with Real-Time capability (DCVC-RT), the input YUV420 format data is first upsampled to YUV444 format before being input into the video encoding / decoding model for encoding and reconstruction. During the training phase, both the original and reconstructed frames are converted to the YUV420 domain, and the weighted distortion loss of the Y, U, and V components is calculated. In the actual encoding process, the YUV420 format video sequence to be encoded is upsampled frame by frame into YUV444 format, input into the network for encoding and reconstruction, and then the YUV444 format data output by the network is downsampled into YUV420 format for subsequent storage or transmission.

[0035] Although the two methods mentioned above can process YUV420 format video data, they still perform video encoding and decoding in a unified color space and mix luminance and chrominance information. This fails to fully utilize the asymmetric perception characteristics of human eyes regarding luminance and chrominance, thus limiting the reconstruction effect of video frames.

[0036] Following the design principle of prioritizing luminance information over chrominance information, related technologies employ a colorization-based YUV420 video encoding / decoding method to restore color frames in order to further conserve bitrate. In this method, I-frames are encoded and decoded in full color using an image encoder to achieve complete and accurate global color information. P-frames, on the other hand, only encode the luminance component, significantly reducing bitrate by omitting the chrominance component encoding. The chrominance information of P-frames is then generated into reconstructed frames at the decoding end using a colorization algorithm, starting with the I-frame. This method reduces the bitrate overhead of P-frames and improves the reconstruction quality of luminance structure and texture. However, when there is significant motion or rapid scene changes in the video, color drift and inconsistencies in color timing can occur in the moving areas of P-frames. Furthermore, this method relies on colorization processing at the decoding end, requiring additional computational resources and inference time, increasing system complexity. For areas not covered by the I-frame or newly appearing scenes, the generated colors exhibit randomness, potentially leading to unstable visual effects in the reconstructed frames. Therefore, this method remains limited in low-cost or high-reliability video encoding / decoding applications.

[0037] Therefore, while the solutions in related technologies can process YUV420 format video data, the video encoding and decoding schemes in these technologies still suffer from the problem of mixed processing of luminance and chrominance information. That is, they mainly perform modeling and perceptual optimization in a unified color space (such as RGB, or YUV444 obtained by upsampling YUV420), failing to fully utilize the asymmetric perceptual characteristics of human eyes regarding luminance and chrominance. This limits the subjective visual effect of the video frame reconstruction results and affects the visual experience.

[0038] In view of this, based on the separation and asymmetry of human eye in brightness and color processing, this disclosure provides an end-to-end video encoding and decoding method, apparatus and electronic device that can be applied to the field of video coding technology.

[0039] Figure 1 A flowchart illustrating an end-to-end video encoding / decoding method according to an embodiment of the present disclosure is shown.

[0040] like Figure 1 As shown, this end-to-end video encoding and decoding method includes operations S110~S130.

[0041] In operation S110, the luminance encoding and decoding network encodes and decodes the luminance component of the (n+1)th frame of the video based on the reconstructed luminance component of the nth frame, obtaining the reconstructed luminance component of the (n+1)th frame, the optical flow of the (n+1)th frame, and the luminance propagation characteristics of the (n+1)th frame. Here, n is a positive integer.

[0042] The end-to-end video codec method processes video frames in YUV420 format. The nth and (n+1)th frames of the video are adjacent frames. The nth frame is generated earlier than the (n+1)th frame. The nth frame is input to the end-to-end video codec method earlier than the (n+1)th frame.

[0043] A luminance encoding / decoding network is a network used to process the luminance of video frames. It is built based on the human eye's greater sensitivity to luminance information and can be used for encoding, decoding, and feature extraction of luminance components. A video frame is the basic unit that makes up a video; it is essentially a single still image, and multiple video frames are played consecutively to form a video.

[0044] Luminance components represent the brightness and darkness information of a video frame, constituting the image outline and details within the frame. Optical flow is the motion vector field of pixels between adjacent frames in a video, representing the direction and speed of an object's movement within the frame. Luminance propagation characteristics are the rules governing the transmission of luminance information between adjacent frames in a video, representing features such as object motion, lighting changes, or scene occlusion.

[0045] In the case of video encoding and decoding, the reconstructed luminance component of the nth frame and the luminance component of the (n+1)th frame of the video are input into the luminance encoding and decoding network. The luminance encoding and decoding network is used to encode and decode the luminance component of the (n+1)th frame of the video to obtain the reconstructed luminance component of the (n+1)th frame, the optical flow of the (n+1)th frame, and the luminance propagation characteristics of the (n+1)th frame.

[0046] In operation S120, the chroma codec network is used to encode and decode the chroma components of the (n+1)th frame of the video based on the optical flow of the (n+1)th frame and the chroma propagation characteristics of the (n)th frame of the video, to obtain the initial chroma features of the (n+1)th frame; the initial chroma features of the (n+1)th frame are then enhanced based on the luminance propagation characteristics of the (n+1)th frame, to obtain the reconstructed chroma components and the chroma propagation characteristics of the (n+1)th frame.

[0047] Chroma codec networks are used to process the chroma of video frames. They can enhance chroma by utilizing the luminance propagation characteristics output by luminance codec networks. Chroma components represent the color information of video frames and can define the hue and saturation of pixels within a video frame. Chroma propagation characteristics are the rules governing the transmission of color information between adjacent frames in a video, and can represent the characteristics of factors such as lighting or color in the video.

[0048] In video encoding and decoding, the optical flow of frame n+1, the chroma propagation features of frame n, and the chroma components of frame n+1 are input into the chroma encoding / decoding network. The chroma encoding / decoding network encodes and decodes the chroma components of frame n+1 to obtain the initial chroma features of frame n+1. The chroma encoding / decoding network then enhances the initial chroma features of frame n+1 based on the luminance propagation features of frame n+1, thereby obtaining the reconstructed chroma components and chroma propagation features of frame n+1.

[0049] In operation S130, the reconstruction network is used to construct the reconstructed image of frame n+1 based on the reconstructed luminance component and the reconstructed chrominance component of frame n+1.

[0050] A reconstruction network is used to fuse the reconstructed luminance component and the reconstructed chrominance component of frame n+1 and perform image restoration, thereby constructing the reconstructed image of frame n+1. For example, the reconstruction network can be used to perform channel stitching on the reconstructed luminance component and the reconstructed chrominance component of frame n+1 to obtain the reconstructed image of frame n+1.

[0051] According to embodiments of this disclosure, by utilizing a luminance coding / decoding network to encode and decode the luminance component of the (n+1)th frame of a video based on the reconstructed luminance component of the nth frame, the reconstructed luminance component, optical flow, and luminance propagation features of the (n+1)th frame are obtained. This enables fine modeling of the luminance components in the video, aligning with the human eye's heightened sensitivity to brightness. By utilizing a chrominance coding / decoding network to encode and decode the chrominance component of the (n+1)th frame based on the optical flow and chrominance propagation features of the nth frame, the initial chrominance features of the (n+1)th frame are obtained. The initial chrominance features of the (n+1)th frame are then enhanced based on the luminance propagation features, resulting in the reconstructed chrominance component and chrominance propagation features of the (n+1)th frame. This utilizes the high-quality luminance structure information within the luminance propagation features to enhance the details of the initial chrominance features, thereby improving visual reproduction. Therefore, this disclosure can independently model the luminance and chrominance components in a video. Through explicit separation and asymmetric design, it can achieve low-cost or high-reliability video encoding and decoding under the premise of controlled bitrate. At the same time, it can prioritize the allocation of more bits and computing power to the luminance component, which is more critical to visual quality, thereby leveraging its perceptual advantages in the YUV420 format to achieve better reconstruction results and bring a better visual experience.

[0052] The reconstruction network constructs the reconstructed image of frame n+1 based on the reconstructed luminance component and the reconstructed chrominance component of frame n+1. Through explicit separation and asymmetric design, the end-to-end video encoding and decoding method can allocate more bits and computing power to the luminance component, which is more critical to visual quality, while keeping the bit rate under control. This allows it to leverage its perceptual advantages in the YUV420 format and achieve better reconstruction results.

[0053] According to embodiments of this disclosure, a color-separated video compression framework is proposed based on the separation and asymmetry of human eye processing of luminance and chrominance. This framework includes a luminance codec network and a chrominance codec network, capable of independently modeling the luminance and chrominance components in the video. The luminance codec network performs detailed modeling of the luminance component to match the human eye's greater sensitivity to luminance. The chrominance codec network, as a lightweight version of the luminance codec network, efficiently processes the chrominance component with lower complexity. Through explicit separation and asymmetric design, more bits and computing power can be prioritized for the luminance component, which is more critical to visual quality, while maintaining a controlled bitrate. This leverages the perceptual advantages of the YUV420 format to achieve superior reconstruction results.

[0054] Inspired by the asymmetric processing mechanism of brightness and chromaticity in the human visual system, this paper discloses a neural video compression framework based on color separation.

[0055] Figure 2 The diagram illustrates a network structure used in an end-to-end video encoding / decoding method according to an embodiment of the present disclosure.

[0056] like Figure 2 As shown, a video codec network can include a luminance codec network, a chrominance codec network, and a reconstruction network. The reconstruction network, in particular, [is involved in this process]. Figure 2 Not shown in the image.

[0057] like Figure 2 As shown, for any time t, the input frame at time t... Separated into luminance components and chromaticity components The video codec network employs a decoupled design for the luminance component. and chromaticity components Independent encoding. Wherein, the input frame at any time t... Let t be a video frame. The input frame at time t. In the case of the (n+1)th frame, the input frame at time t-1 This is the nth frame. The input frame at time t-1. For the (n+1)th frame, the input frame at time t-2 This is the nth frame. The input frame at time t-2. For the (n+1)th frame, the input frame at time t-3 This is the nth frame.

[0058] Input frame at time t In the case of the (n+1)th frame, the luma encoding / decoding network uses Input: Reconstructed Luminance Components , optical flow and brightness propagation characteristics Subsequently, the chroma codec network in Under the conditions of Encode, and Reconstructed chromaticity components are generated under the following conditions. and chromaticity propagation characteristics .in, and These serve as temporal references for luminance and chrominance, respectively, and are passed to the encoding and decoding process of subsequent frames.

[0059] The following will use the input frame at time t-3. For the nth frame, the input frame at time t-2 Taking the (n+1)th frame as an example, for Figure 2 The process of video frame processing by the video codec network shown will be explained further.

[0060] In constructing the reconstructed image of the (n+1)th frame of the video, the luminance codec network can determine the luminance component based on the reconstructed luminance component of the nth frame. The luminance component of the (n+1)th frame of the video Encode and decode to obtain the reconstructed luminance component of the (n+1)th frame. Optical flow in frame n+1 and the luminance propagation characteristics of the (n+1)th frame .

[0061] Chroma codec networks can utilize the chroma propagation characteristics of the nth frame. and the optical flow of the (n+1)th frame The chroma component of the (n+1)th frame of the video Encode and decode to obtain the initial chroma features of the (n+1)th frame. (exist Figure 2 (Not shown in the image). And based on the luminance propagation characteristics of the (n+1)th frame. Initial chroma features for frame n+1 Enhancement processing is performed to obtain the reconstructed chroma components of the (n+1)th frame. Chroma propagation features of the (n+1)th frame .

[0062] The reconstruction network is connected to the outputs of the chroma encoding / decoding network and the luma encoding / decoding network. The reconstruction network can reconstruct the luma component based on the (n+1)th frame. and the reconstructed chroma component of the (n+1)th frame Construct the reconstructed image of the (n+1)th frame.

[0063] The core of a luma codec network is to conditionally encode and decode the current frame using temporal context features. This disclosure adapts and improves upon the design of uniformly processing RGB three-channel input in related technologies, changing both the input and output channels of the luma codec network from three channels to a single channel. This enables the luma codec network to efficiently model and compress luma signals using a single channel. Therefore, the luma codec network of this disclosure has both a single input and output channel.

[0064] The residual blocks used in the luminance codec network are based on partially convolutional structures to reduce the computational complexity of the video codec network and improve its operating efficiency, thereby reducing computational overhead while maintaining performance.

[0065] For example, the lightweight optical flow estimation module in the luminance codec network of this disclosure can be implemented by replacing the residual blocks in the Luma-SpyNet optical flow estimation module with residual blocks based on partially convolutional structures.

[0066] In the luminance encoding and decoding network, a lightweight optical flow estimation module is used to estimate the optical flow of the luminance component, thereby improving the accuracy and stability of motion estimation in the reconstructed image.

[0067] As one implementation method, the YUV420 color encoding format can be used for video frame encoding to match the asymmetric processing mechanism of luminance and chrominance in the human visual system. The luminance component of each frame of the video can be a single-channel image luminance component, and the reconstructed luminance component of each frame of the video can be a single-channel image reconstructed luminance component, in order to meet the independent encoding requirements of the luminance component under the YUV420 format and reduce unnecessary channel redundancy.

[0068] When using the YUV420 color encoding format for video frame encoding, the luminance component Characterizing the brightness and detail of an image, chromaticity components The brightness component represents the color information of an image. Retaining more brightness information ensures better image quality in the reconstructed image. Therefore, when encoding video frames, the brightness component can be processed separately to better meet the image quality requirements of the human eye.

[0069] The following will combine Figure 3 The structure and function of the luminance coding / decoding network used in the end-to-end video coding / decoding method of this disclosure embodiment will be further described.

[0070] Figure 3 A schematic diagram of the structure of a luminance codec network according to an embodiment of the present disclosure is shown.

[0071] like Figure 3 As shown, the luminance coding / decoding network may include a motion estimation module, a motion coding module, a motion decoding module, a motion entropy coding module, a context coding module, a context entropy coding module, a context decoding module, a temporal context mining module, and a feature and frame buffer module.

[0072] During the motion estimation phase, the motion estimation module calculates the reconstructed luminance components based on the nth frame. and the luminance component of the (n+1)th frame Motion estimation is performed to obtain the initial optical flow of the (n+1)th frame. .

[0073] During the information encoding and processing phase, the motion coding module extracts the initial optical flow of the (n+1)th frame. The motion latent variables are calculated and quantized. The motion entropy coding module converts the quantized motion latent variables into a motion information bitstream. The motion decoding module receives the motion information bitstream and reconstructs it to obtain the optical flow of the (n+1)th frame. .

[0074] In the temporal context feature extraction stage, the temporal context mining module extracts features based on the optical flow of the (n+1)th frame. Luminance propagation characteristics of the nth frame Extracting multiple contextual conditional features , and The dimensions of multiple upper and lower conditional features can be different. The luminance propagation features of the nth frame. It can be extracted from the features and frame buffer module.

[0075] In the luminance component encoding and decoding stage, the context coding module uses multiple contextual condition features. , and As conditional information, the luminance component of the (n+1)th frame Encoding is performed to obtain luminance-encoded latent variables. The context entropy encoding module performs context modeling and probability statistics on multiple contextual conditional features to obtain the conditional latent variable probability distribution. The entropy engine converts the quantized luminance-encoded latent variables into luminance component bitstreams using the conditional latent variable probability distribution. The context decoding module receives the luminance component bitstreams and reconstructs them to obtain the reconstructed luminance components of the (n+1)th frame. and brightness propagation characteristics Among them, the brightness propagation characteristics can be analyzed. Perform a 3×3 convolution operation to obtain the reconstructed luminance components. The feature and frame buffer module stores the reconstructed luminance components. and brightness propagation characteristics This is for use in the encoding process of subsequent frames.

[0076] According to embodiments of this disclosure, the luminance codec network can achieve precise modeling and efficient compression of luminance information while maintaining a compact structure, effectively improving the luminance reconstruction quality and overall subjective visual performance of video encoding.

[0077] According to embodiments of this disclosure, the chroma codec network used in the end-to-end video codec method of this disclosure continues the overall structural framework of the luma codec network, but is lightweighted and structurally optimized for the characteristics of the chroma channel, mainly in the following two aspects: 1) Optical flow multiplexing and lightweight design: Since the luma codec network can provide high-precision optical flow information, this disclosure removes modules related to optical flow estimation and compression in the chroma codec network compared to the luma codec network, and directly reuses the optical flow output by the luma codec network. This avoids redundant calculations and significantly reduces model complexity and computational load; 2) Introduction of color fidelity enhancement mechanism: To further improve the detail and structural consistency of chroma reconstruction, this disclosure introduces a color fidelity optimization module in the decoding stage, utilizing luminance propagation characteristics. The high-quality structural information is used to enhance the details of the initially reconstructed chromaticity features in order to improve the visual reproduction effect.

[0078] According to embodiments of this disclosure, the number of channels for both the input chroma component and the output reconstructed chroma component of the chroma codec network of this disclosure is 2.

[0079] According to embodiments of this disclosure, the chroma codec network of this disclosure multiplexes the optical flow output by the luminance codec network. and brightness propagation characteristics This enables efficient chromaticity modeling guided by luminance information, avoiding redundant calculations and reducing the complexity and computational load of the model.

[0080] Chroma codec networks can be used for chroma component compression encoding of video frames. The following will combine... Figure 4 The structure and function of the chroma codec network used in the end-to-end video codec method of this disclosure embodiment will be further described.

[0081] Figure 4 A schematic diagram of the structure of a chroma codec network according to an embodiment of the present disclosure is shown.

[0082] like Figure 4As shown, the chroma codec network may include a downsampling module, a temporal context mining module, a context encoding module, a context entropy encoding module, a context decoding module, a color fidelity optimization module, and a feature and frame buffer module.

[0083] The chroma codec network can obtain the optical flow of the (n+1)th frame from the luma codec network. And use the downsampling module to measure the optical flow of the (n+1)th frame. Downsampling is performed to match the resolution of the chromaticity components, resulting in downsampled optical flow characteristics.

[0084] According to embodiments of this disclosure, downsampling can be bilinear downsampling. Multiplexing optical flow and downsampling it can reduce the computational load of the chroma codec network while matching the downsampled optical flow features to the resolution of the chroma components.

[0085] The temporal context mining module can analyze downsampled optical flow features and chroma propagation features of the nth frame. Multi-scale feature extraction is performed to obtain multiple contextual features.

[0086] Chromaticity propagation characteristics These can be stored in the feature and frame buffer module. Multiple context features can be... , and In this context, multiple contextual features have different dimensions, for example, The size can be 128*128. The size can be 64*64. The size can be 32*32.

[0087] The context coding module can process the chroma components of the (n+1)th frame based on multiple contextual features. Encode the colors to obtain the chroma-encoded latent variables.

[0088] According to embodiments of this disclosure, the context encoding module can utilize multiple context features to remove temporal and spatial redundancy of the original chroma components while retaining core color features.

[0089] The context entropy encoding module can perform context modeling and probability statistics on multiple context features to obtain the probability distribution of latent variables.

[0090] According to embodiments of this disclosure, the context entropy coding module can mine the statistical regularities of multiple context features, thereby establishing a probability distribution model of the latent variable of chroma coding, and then obtaining the probability distribution model of the latent variable.

[0091] The context decoding module uses multiple context features to decode the chroma component bitstream obtained from the chroma coding latent variables and their probability distributions, thus obtaining the initial chroma features of the (n+1)th frame. Among these methods, the entropy engine can be used to convert the quantized chroma encoding latent variables into chroma component bitstreams based on the probability distribution of latent variables.

[0092] Initial chromaticity features can preserve the basic color information for chromaticity classification, but may contain blurred details or slight misalignments with the luminance structure. To further improve the detail and structural consistency of chromaticity reconstruction, this disclosure incorporates a color fidelity optimization module into the decoding stage of the chromaticity encoding / decoding network.

[0093] The color fidelity optimization module can be based on the initial chromaticity features of the (n+1)th frame. Luminance propagation characteristics of the (n+1)th frame Initial chroma features for frame n+1 Enhancement processing is performed to obtain the reconstructed chroma components of the (n+1)th frame. Chroma propagation features of the (n+1)th frame Finally, the reconstructed chroma components of the (n+1)th frame are... Chroma propagation features of the (n+1)th frame The input feature and frame buffer module is used for the encoding and decoding process of subsequent frames.

[0094] The color fidelity optimization module utilizes high-quality luminance structure information from luminance propagation characteristics to enhance the details of the initial chromaticity features, thereby improving the visual reproduction effect.

[0095] According to embodiments of this disclosure, the chroma codec network, while maintaining a compact structure and high computational efficiency, utilizes luminance propagation characteristics to guide the reconstruction of chroma components, achieving high-fidelity compression and improved visual consistency of chroma components, thereby further optimizing the overall subjective visual effect of video images under the YUV420 format.

[0096] Because luminance coding / decoding networks have high computational precision and are allocated more bitrate resources, luminance propagation characteristics are... It can provide more accurate structural information. This disclosure utilizes this characteristic to construct a color fidelity optimization module, using luminance propagation characteristics as guiding information to correct relatively blurry areas in the chroma channel, thereby achieving luminance-guided chroma enhancement.

[0097] Therefore, in Figure 1The end-to-end video encoding and decoding method shown includes using a chroma encoding and decoding network to encode and decode the chroma components of the (n+1)th frame of the video based on the optical flow of the (n+1)th frame and the chroma propagation features of the nth frame, to obtain the initial chroma features of the (n+1)th frame. This can be achieved by: using a downsampling module to perform bilinear downsampling on the optical flow of the (n+1)th frame to obtain downsampled optical flow features; using a temporal context mining module to perform multi-scale feature extraction on the downsampled optical flow features and the chroma propagation features of the nth frame to obtain multiple context features, wherein the sizes of the multiple context features are different; using a context coding module to encode the chroma components of the (n+1)th frame based on the multiple context features to obtain chroma encoded latent variables; using a context entropy coding module to perform context modeling and probability statistics on the multiple context features to obtain the probability distribution of the latent variables; and using a context decoding module to decode the chroma component bitstream obtained from the chroma encoded latent variables and the probability distribution of the latent variables based on the multiple context features to obtain the initial chroma features of the (n+1)th frame.

[0098] The chroma fidelity optimization module utilizes the high-precision luminance propagation characteristics generated by the luminance codec network to guide and correct the initial chroma features at the structural level, thereby improving the texture detail and structural consistency of the reconstructed chroma. The chroma fidelity optimization module will be explained below.

[0099] Figure 5 A schematic diagram of the structure of a color fidelity optimization module according to an embodiment of the present disclosure is shown.

[0100] like Figure 5 As shown, the color fidelity optimization module may include: an alignment unit, a fusion unit, a feature enhancement unit, and a reconstruction unit. The alignment unit may include: a downsampling layer, a first convolutional layer, and a second convolutional layer. The fusion unit may include: a stitching layer, a third convolutional layer, and a non-linear mapping layer. The reconstruction unit includes a fourth convolutional layer.

[0101] Using a chroma codec network based on the luminance propagation characteristics of the (n+1)th frame Initial chroma features for frame n+1 Enhancement processing is performed to obtain the reconstructed chroma components of the (n+1)th frame. Chroma propagation features of the (n+1)th frame This may include: utilizing the luminance propagation features of the (n+1)th frame using alignment units. and the initial chromaticity features of the (n+1)th frame Multimodal feature alignment is performed to obtain luma modal features and chroma modal features; a fusion unit is used to fuse the luma modal features and chroma modal features and perform nonlinear mapping to obtain a mask matrix; a feature enhancement unit is used to combine the mask matrix with the initial chroma features of the (n+1)th frame. Multiplying them together yields the chroma propagation features of the (n+1)th frame. The reconstruction unit performs convolution on the chroma propagation features of the (n+1)th frame using a first predetermined convolution kernel to obtain the reconstructed chroma components of the (n+1)th frame. The kernel of the fourth convolutional layer is the first predetermined convolutional kernel.

[0102] Multimodal feature alignment can eliminate differences in resolution, dimension, or distribution between luminance propagation features and initial chrominance features, thereby achieving spatial and semantic alignment across modal features. Luminance modal features can preserve core information about the luminance structure, such as the edge contours of the image. Chroma modal features can preserve basic color information and can be aligned with luminance modal features in spatial location or feature dimension for easier subsequent processing.

[0103] The fusion unit can mine the correlation between luma mode features and chroma mode features, thereby generating a mask matrix. The mask matrix can be used to control the enhancement weight of luma mode features on chroma mode features.

[0104] According to embodiments of this disclosure, the color fidelity optimization module can fully utilize the structural prior information provided by the luminance channel to achieve refined compensation of the chrominance channel without significantly increasing the computational burden, thereby significantly improving color consistency and visual fidelity in the video reconstruction process.

[0105] The process of aligning the luminance propagation features and initial chrominance features of frame (n+1) using an alignment unit to obtain luminance modal features and chrominance modal features may include: aligning the luminance propagation features of frame (n+1) using a downsampling layer. Bilinear downsampling is performed to obtain downsampled features; the first convolutional layer performs convolution operation on the downsampled features according to the second predetermined convolutional kernel to obtain luminance modal features; the second convolutional layer performs convolution operation on the initial chroma features of the (n+1)th frame according to the second predetermined convolutional kernel to obtain chroma modal features.

[0106] According to embodiments of this disclosure, bilinear downsampling can adjust the resolution of the luminance propagation features to match that of the chrominance propagation features. A first convolutional layer can be used to extract information related to chrominance enhancement from the luminance propagation features, such as edge structures or texture gradients. A second convolutional layer can be used to extract color information from the initial chrominance features, such as color transitions or color gradients.

[0107] According to embodiments of this disclosure, because the luminance codec network has high computational accuracy and allocates more bitrate resources, the luminance propagation features can provide more accurate luminance information. Using luminance features as guiding information to correct blurry areas in the chroma channel allows for chroma enhancement guided by luminance, thereby improving image reconstruction results.

[0108] The process of fusing and nonlinearly mapping the luminance modal features and chrominance modal features using a fusion unit to obtain a mask matrix may include: using a splicing layer to splice the luminance modal features and chrominance modal features into channels to obtain spliced ​​features; using a third convolutional layer to perform a convolution operation on the spliced ​​features according to a third predetermined convolutional kernel to obtain fused features; and using a nonlinear mapping layer to perform nonlinear mapping on the fused features according to a predetermined nonlinear mapping function to obtain a mask matrix.

[0109] The stitching layer can be used to merge aligned luminance modal features and chrominance modal features in the channel dimension to generate stitched features that contain both luminance modal information and chrominance modal information.

[0110] Using the third convolutional layer to perform convolution operations on the spliced ​​features can enhance the complementary information between the luminance and chrominance spaces, and obtain the fused information of luminance modal features and chrominance modal features.

[0111] The mask matrix can be used to characterize the matching degree between luminance modal features and chrominance modal features, thereby obtaining the enhancement weight of luminance modal features on chrominance modal features.

[0112] The nonlinear mapping function can be the Sigmoid function. The fused features are processed by the Sigmoid function to generate a mask matrix. The mask matrix is ​​applied to the initial chromaticity features. This is enhanced to obtain the chroma propagation features of the (n+1)th frame. The fourth convolutional layer is used to analyze the chromaticity propagation features based on the first predetermined convolutional kernel. Perform a convolution operation to obtain the reconstructed chromaticity components. .

[0113] In one implementation, the first and second predetermined convolution kernels can be 3×3 convolution kernels, and the third convolution kernel can be a 1×1 convolution kernel.

[0114] According to embodiments of this disclosure, the fusion unit obtains spliced ​​features by concatenating the luminance modal features and chrominance modal features through channel splicing, obtains fusion information of luminance modal features and chrominance modal features by convolutional layer, and then obtains fusion features. Finally, the fusion features are nonlinearly mapped to obtain a mask matrix, thereby realizing the joint processing of luminance modal features and chrominance modal features and improving the visual quality of video encoding and decoding.

[0115] According to embodiments of this disclosure, the video encoding / decoding network used in the end-to-end video encoding / decoding method can calculate the encoding / decoding quantization parameter q based on the bitrate index. This encoding / decoding quantization parameter q is multiplied with intermediate features during the calculation process of the context coding module and the context decoding module, thereby affecting the data distribution of the final latent variables and achieving the purpose of variable bitrate.

[0116] The end-to-end video encoding / decoding method employs a decoupled design of luma and chroma encoding / decoding networks, enabling asymmetric resource allocation between them. For example, from a computational resource perspective, the luma encoding / decoding network can account for 84.47% of the overall complexity. Allocating higher complexity to the luma encoding / decoding network allows for more refined encoding of luma information, thus enabling greater computational resource investment in image visual information. From a bitrate allocation perspective, at the four bitrate points from low to high, the bitrate proportions for luma information are 97.1%, 96.6%, 95.9%, and 95.2%, respectively, with an average bitrate proportion of 96.2%. Under the premise of controlled total bitrate, by tilting resources towards luma information, the subjective visual quality and compression efficiency of video encoding / decoding are maximized.

[0117] Finally, the end-to-end video encoding / decoding methods and related technologies provided in this disclosure were tested on multiple publicly available datasets. The related technologies are the Universal Video Coding Test Model (VTM), the DCVC-DC method, and the DCVC-FM method. The multiple publicly available datasets are HEVC_B, HEVC_C, HEVC_D, HEVC_E, MCL-JCV, USTC-TD, and UVG datasets. Here, "ours" represents the end-to-end video encoding / decoding method provided in this disclosure.

[0118] Table 1 shows the video encoding and decoding results of the end-to-end video encoding and decoding method and the video decoding method in related technologies provided in the embodiments of this disclosure on multiple public datasets.

[0119] As shown in Table 1, the end-to-end video encoding and decoding methods and related technologies provided in this disclosure were tested for four metrics: PSNR. YUV420 PSNR Y PSNR U and PSNR V The experimental results. Among them, PSNR YUV420 The peak signal-to-noise ratio (PSNR) represents the overall peak signal-to-noise ratio (PSNR) of the entire YUV420 format image, reflecting the overall image quality. Y Peak signal-to-noise ratio (PSNR), representing the luminance component, reflects the reconstruction quality of image structure, detail, and contrast; UThe peak signal-to-noise ratio (PSNR) represents the chromaticity U component (blue channel), reflecting the accuracy of blue color reproduction; V The peak signal-to-noise ratio (PSNR) represents the chroma V component (red channel), reflecting the red color reproduction accuracy. In the test, no fixed I-frame interval was set, and the test video sequence length was 96 frames. Based on the test results of the VTM method, the percentage reduction in bitrate required by the video encoding / decoding method disclosed herein compared to related video encoding / decoding methods under the same image quality was compared for each dataset and each indicator. Negative values ​​indicate savings, and positive values ​​indicate increases.

[0120] Table 1

[0121]

[0122] As shown in Table 1, the end-to-end video encoding and decoding method provided in this embodiment of the present disclosure achieves an average reduction of 15.53% in luminance component bitrate, 33.61% in U chrominance component bitrate, and 35.4% in V chrominance component bitrate. (This is in the context of PSNR.) YUV420 In terms of metrics, the end-to-end video encoding and decoding method provided in this disclosure outperforms DCVC-DC overall, with higher compression efficiency. Meanwhile, it also achieves better PSNR. Y In terms of metrics, this method achieves state-of-the-art performance, demonstrating that the proposed end-to-end video encoding / decoding method can improve the reconstruction quality of the luminance component. This result verifies the advantages of this disclosure in balancing compression efficiency and visual quality.

[0123] Figure 6 The illustration shows a comparison of reconstructed images from an end-to-end video encoding / decoding method according to embodiments of the present disclosure and related technologies.

[0124] like Figure 6 As shown, at the same bitrate, the end-to-end video encoding and decoding method provided in this disclosure can generate reconstructed images with clearer structure and richer details when reconstructing BQMall and RaceHorses sequences in the HEVC_C dataset with 720P resolution. This improves the subjective perception quality.

[0125] When reconstructing images from a BQMall sequence, the bit rate per pixel (bpp) of the image reconstructed using VTM is 0.0281, the bit rate per pixel (bpp) of the image reconstructed using the DCVC-DC method is 0.0297, the bit rate per pixel (bpp) of the image reconstructed using the DCVC-FM method is 0.0257, and the bit rate per pixel (bpp) of the image reconstructed using the end-to-end video codec method provided in this embodiment is 0.0254. When reconstructing images from a RaceHorses sequence, the bit rate per pixel (bpp) of the image reconstructed using VTM is 0.0512, the bit rate per pixel (bpp) of the image reconstructed using the DCVC-DC method is 0.0534, the bit rate per pixel (bpp) of the image reconstructed using the DCVC-FM method is 0.0481, and the bit rate per pixel (bpp) of the image reconstructed using the end-to-end video codec method provided in this embodiment is 0.0475. Therefore, the bit rate efficiency of the reconstructed image using this disclosure is higher.

[0126] from Figure 6 The reconstructed images shown demonstrate that, in the BQMall sequence, the end-to-end video encoding and decoding method provided in this disclosure preserves the stripe texture of the trousers and clothing, while the contrasting DCVC-FM method exhibits detail loss. In the RaceHorses sequence, the end-to-end video encoding and decoding method provided in this disclosure more accurately restores details such as the horse's eyes and tether, resulting in a more realistic and natural overall image. The comparative results show that, at the same bitrate, related technologies lose some structural information, while the end-to-end video encoding and decoding method provided in this disclosure, through an asymmetric processing mechanism of luminance and chrominance, preserves the fineness of the luminance structure, thereby achieving a better subjective viewing experience.

[0127] Figure 7 A schematic block diagram of an end-to-end video codec apparatus according to an embodiment of the present disclosure is shown.

[0128] like Figure 7 As shown, the video encoding and decoding device 700 includes a luminance encoding and decoding module 710, a chrominance encoding and decoding module 720, and a construction module 730.

[0129] The luminance encoding / decoding module 710 is used to encode and decode the luminance component of the (n+1)th frame of the video based on the reconstructed luminance component of the nth frame of the video using a luminance encoding / decoding network, so as to obtain the reconstructed luminance component of the (n+1)th frame, the optical flow of the (n+1)th frame, and the luminance propagation characteristics of the (n+1)th frame, where n is a positive integer.

[0130] The chroma encoding / decoding module 720 is used to encode and decode the chroma components of the (n+1)th frame of the video based on the optical flow of the (n+1)th frame and the chroma propagation characteristics of the (n)th frame of the video, to obtain the initial chroma features of the (n+1)th frame; and to enhance the initial chroma features of the (n+1)th frame based on the luminance propagation characteristics of the (n+1)th frame, to obtain the reconstructed chroma components and the chroma propagation characteristics of the (n+1)th frame.

[0131] The construction module 730 is used to construct the reconstructed image of the (n+1)th frame using the reconstruction network based on the reconstructed luminance component and the reconstructed chrominance component of the (n+1)th frame.

[0132] According to embodiments of this disclosure, any plurality of the luminance encoding / decoding module 710, chrominance encoding / decoding module 720, and construction module 730 can be combined into one module / unit / subunit, or any one of these modules / units / subunits can be split into multiple modules / units / subunits. Alternatively, at least part of the functionality of one or more of these modules / units / subunits can be combined with at least part of the functionality of other modules / units / subunits and implemented in one module / unit / subunit. According to embodiments of this disclosure, at least one of the luminance encoding / decoding module 710, chrominance encoding / decoding module 720, and construction module 730 can be at least partially implemented as hardware circuitry, such as a field-programmable gate array (FPGA), a programmable logic array (PLA), a system-on-a-chip, a system-on-a-substrate, a system-on-package, an application-specific integrated circuit (ASIC), or any other reasonable means of integrating or packaging the circuitry, or implemented in any one of software, hardware, and firmware methods, or in a suitable combination of any of these methods. Alternatively, at least one of the luminance encoding / decoding module 710, the chrominance encoding / decoding module 720, and the construction module 730 may be at least partially implemented as a computer program module, which can perform corresponding functions when the computer program module is run.

[0133] It should be noted that the video encoding and decoding device part in the embodiments of this disclosure corresponds to the video encoding and decoding method part in the embodiments of this disclosure. For a detailed description of the video encoding and decoding device part, please refer to the video encoding and decoding method part, which will not be repeated here.

[0134] Figure 8 A block diagram of an electronic device suitable for implementing the end-to-end video encoding and decoding method described above, according to embodiments of the present disclosure, is illustrated. Figure 8 The electronic device shown is merely an example and should not be construed as limiting the functionality and scope of the embodiments disclosed herein.

[0135] like Figure 8 As shown, an electronic device 800 according to an embodiment of this disclosure includes a processor 801, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 802 or a program loaded from a storage portion 808 into a random access memory (RAM) 803. The processor 801 may include, for example, a general-purpose microprocessor (e.g., a CPU), an instruction set processor and / or an associated chipset and / or a special-purpose microprocessor (e.g., an application-specific integrated circuit (ASIC)), etc. The processor 801 may also include onboard memory for caching purposes. The processor 801 may include a single processing unit or multiple processing units for performing different actions of the method flow according to an embodiment of this disclosure.

[0136] RAM 803 stores various programs and data required for the operation of electronic device 800. Processor 801, ROM 802, and RAM 803 are interconnected via bus 804. Processor 801 performs various operations of the method flow according to embodiments of the present disclosure by executing programs in ROM 802 and / or RAM 803. It should be noted that the programs may also be stored in one or more memories other than ROM 802 and RAM 803. Processor 801 may also perform various operations of the method flow according to embodiments of the present disclosure by executing programs stored in said one or more memories.

[0137] According to embodiments of this disclosure, the electronic device 800 may further include an input / output (I / O) interface 805, which is also connected to a bus 804. The electronic device 800 may also include one or more of the following components connected to the input / output (I / O) interface 805: an input section 806 including a keyboard, mouse, etc.; an output section 807 including a cathode ray tube (CRT), liquid crystal display (LCD), etc., and a speaker, etc.; a storage section 808 including a hard disk, etc.; and a communication section 809 including a network interface card such as a LAN card, modem, etc. The communication section 809 performs communication processing via a network such as the Internet. A drive 810 is also connected to the input / output (I / O) interface 805 as needed. A removable medium 811, such as a disk, optical disk, magneto-optical disk, semiconductor memory, etc., is installed on the drive 810 as needed so that computer programs read from it can be installed into the storage section 808 as needed.

[0138] According to embodiments of this disclosure, the method flow according to embodiments of this disclosure can be implemented as a computer software program. For example, embodiments of this disclosure include a computer program product comprising a computer program carried on a computer-readable storage medium, the computer program containing program code for performing the methods shown in the flowchart. In such embodiments, the computer program can be downloaded and installed from a network via communication section 809, and / or installed from removable medium 811. When the computer program is executed by processor 801, it performs the functions defined in the system of embodiments of this disclosure. According to embodiments of this disclosure, the systems, devices, apparatuses, modules, units, etc., described above can be implemented by computer program modules.

[0139] This disclosure also provides a computer-readable storage medium, which may be included in the device / apparatus / system described in the above embodiments; or it may exist independently and not assembled into the device / apparatus / system. The computer-readable storage medium carries one or more programs that, when executed, implement the method according to the embodiments of this disclosure.

[0140] According to embodiments of this disclosure, the computer-readable storage medium can be a non-volatile computer-readable storage medium. Examples include, but are not limited to: portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof. In this disclosure, the computer-readable storage medium can be any tangible medium that contains or stores a program that can be used by or in conjunction with an instruction execution system, apparatus, or device.

[0141] For example, according to embodiments of this disclosure, a computer-readable storage medium may include the ROM 802 and / or RAM 803 described above and / or one or more memories other than ROM 802 and RAM 803.

[0142] Embodiments of this disclosure also include a computer program product comprising a computer program containing program code for performing the methods provided in the embodiments of this disclosure. When the computer program product is run on an electronic device, the program code is used to enable the electronic device to implement the methods provided in the embodiments of this disclosure.

[0143] When the computer program is executed by the processor 801, it performs the functions defined in the system / apparatus of this disclosure embodiments. According to embodiments of this disclosure, the systems, apparatuses, modules, units, etc., described above can be implemented by computer program modules.

[0144] In one embodiment, the computer program may rely on a tangible storage medium such as an optical storage device or a magnetic storage device. In another embodiment, the computer program may also be transmitted and distributed in the form of signals over a network medium, and may be downloaded and installed via the communication section 809, and / or installed from a removable medium 811. The program code contained in the computer program can be transmitted using any suitable network medium, including but not limited to: wireless, wired, etc., or any suitable combination thereof.

[0145] According to embodiments of this disclosure, program code for executing the computer programs provided in embodiments of this disclosure can be written in any combination of one or more programming languages. Specifically, these computational programs can be implemented using high-level procedural and / or object-oriented programming languages, and / or assembly / machine languages. Programming languages ​​include, but are not limited to, languages ​​such as Java, C++, Python, "C", or similar programming languages. The program code can execute entirely on a user's computing device, partially on a user's device, partially on a remote computing device, or entirely on a remote computing device or server. In cases involving remote computing devices, the remote computing device can be connected to the user's computing device via any type of network, including a local area network (LAN) or a wide area network (WAN), or it can be connected to an external computing device (e.g., via the Internet using an Internet service provider).

[0146] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in a block diagram or flowchart, and combinations of blocks in a block diagram or flowchart, may be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions. Those skilled in the art will understand that the features recited in the various embodiments and / or claims of this disclosure can be combined and / or combined in various ways, even if such combinations or combinations are not expressly described in this disclosure. In particular, the features described in the various embodiments and / or claims of this disclosure may be combined and / or combined in various ways without departing from the spirit and teachings of this disclosure. All such combinations and / or combinations fall within the scope of this disclosure.

[0147] The embodiments of this disclosure have been described above. However, these embodiments are for illustrative purposes only and are not intended to limit the scope of this disclosure. Although various embodiments have been described above, this does not mean that the measures in the various embodiments cannot be used advantageously in combination. The scope of this disclosure is defined by the appended claims and their equivalents. Various substitutions and modifications can be made by those skilled in the art without departing from the scope of this disclosure, and all such substitutions and modifications should fall within the scope of this disclosure.

Claims

1. An end-to-end video coding method, comprising: encoding and decoding, by using a luminance coding network, a luminance component of an (n+1)th frame of a video according to a reconstructed luminance component of an nth frame of the video, to obtain the reconstructed luminance component of the (n+1)th frame, an optical flow of the (n+1)th frame, and a luminance propagation feature of the (n+1)th frame, wherein n is a positive integer; encoding and decoding, by using a chrominance coding network, a chrominance component of the (n+1)th frame of the video according to the optical flow of the (n+1)th frame and a chrominance propagation feature of the nth frame of the video, to obtain an initial chrominance feature of the (n+1)th frame, and enhancing the initial chrominance feature of the (n+1)th frame according to the luminance propagation feature of the (n+1)th frame, to obtain a reconstructed chrominance component of the (n+1)th frame and a chrominance propagation feature of the (n+1)th frame; constructing, by using a reconstruction network, a reconstructed image of the (n+1)th frame according to the reconstructed luminance component of the (n+1)th frame and the reconstructed chrominance component of the (n+1)th frame.

2. The video coding method of claim 1, wherein, The color fidelity optimization module in the chrominance coding network comprises an alignment unit, a fusion unit, a feature enhancement unit, and a reconstruction unit; enhancing, by using the chrominance coding network, the initial chrominance feature of the (n+1)th frame according to the luminance propagation feature of the (n+1)th frame, to obtain the reconstructed chrominance component of the (n+1)th frame and the chrominance propagation feature of the (n+1)th frame, comprises: aligning, by using the alignment unit, the luminance propagation feature of the (n+1)th frame and the initial chrominance feature of the (n+1)th frame to obtain a luminance modal feature and a chrominance modal feature; fusing and nonlinearly mapping, by using the fusion unit, the luminance modal feature and the chrominance modal feature to obtain a mask matrix; multiplying, by using the feature enhancement unit, the mask matrix and the initial chrominance feature of the (n+1)th frame to obtain the chrominance propagation feature of the (n+1)th frame; convolving, by using the reconstruction unit, the chrominance propagation feature of the (n+1)th frame according to a first predetermined convolution kernel to obtain the reconstructed chrominance component of the (n+1)th frame.

3. The video coding method of claim 2, wherein, The alignment unit comprises a down-sampling layer, a first convolution layer, and a second convolution layer; aligning, by using the alignment unit, the luminance propagation feature of the (n+1)th frame and the initial chrominance feature of the (n+1)th frame to obtain a luminance modal feature and a chrominance modal feature, comprises: performing bilinear down-sampling, by using the down-sampling layer, on the luminance propagation feature of the (n+1)th frame to obtain a down-sampled feature; performing convolution operation, by using the first convolution layer, on the down-sampled feature according to a second predetermined convolution kernel to obtain the luminance modal feature; performing convolution operation, by using the second convolution layer, on the initial chrominance feature of the (n+1)th frame according to the second predetermined convolution kernel to obtain the chrominance modal feature.

4. The video coding method of claim 3, wherein, The fusion unit comprises a concatenation layer, a third convolution layer, and a nonlinear mapping layer; fusing and nonlinearly mapping, by using the fusion unit, the luminance modal feature and the chrominance modal feature to obtain a mask matrix, comprises: performing channel concatenation, by using the concatenation layer, on the luminance modal feature and the chrominance modal feature to obtain a concatenated feature; The third convolutional layer performs a convolution operation on the spliced ​​features according to a third predetermined convolutional kernel to obtain fused features; The mask matrix is ​​obtained by using the nonlinear mapping layer to perform nonlinear mapping on the fused features according to a predetermined nonlinear mapping function.

5. The video coding method of claim 4, wherein, The first and second predetermined convolution kernels are both 3×3 convolution kernels, and the third convolution kernel is a 1×1 convolution kernel.

6. The video coding method of any of claims 2 to 5, wherein, The chroma codec network also includes: a downsampling module, a temporal context mining module, a context encoding module, a context entropy encoding module, and a context decoding module; The process of using a chroma coding / decoding network to encode and decode the chroma components of the (n+1)th frame of the video based on the optical flow of the (n+1)th frame and the chroma propagation characteristics of the nth frame of the video, to obtain the initial chroma characteristics of the (n+1)th frame, includes: The optical flow of the (n+1)th frame is bilinearly downsampled using the downsampling module to obtain the downsampled optical flow features. The temporal context mining module is used to perform multi-scale feature extraction on the downsampled optical flow features and the chroma propagation features of the nth frame to obtain multiple context features, wherein the dimensions of the multiple context features are different from each other; The context coding module is used to encode the chroma components of the (n+1)th frame according to the multiple context features to obtain chroma coding latent variables; The context entropy encoding module is used to perform context modeling and probability statistics on the multiple context features to obtain the probability distribution of latent variables. The context decoding module uses the multiple context features to decode the chroma component bitstream obtained from the chroma encoding latent variables and the probability distribution of the latent variables, thereby obtaining the initial chroma features of the (n+1)th frame.

7. The video coding method of any of claims 1 to 5, wherein, The luminance component of each frame of the video is a single-channel image luminance component.

8. The video coding method of claim 7, wherein, The residual blocks used in the luminance encoding / decoding network are residual blocks based on a partially convolutional structure.

9. An end-to-end video encoding / decoding apparatus, comprising: The luminance encoding / decoding module is used to encode and decode the luminance component of the (n+1)th frame of the video based on the reconstructed luminance component of the nth frame of the video using a luminance encoding / decoding network, so as to obtain the reconstructed luminance component of the (n+1)th frame, the optical flow of the (n+1)th frame, and the luminance propagation characteristics of the (n+1)th frame, where n is a positive integer. The chroma encoding / decoding module is used to encode and decode the chroma components of the (n+1)th frame of the video based on the optical flow of the (n+1)th frame and the chroma propagation characteristics of the (n)th frame of the video, to obtain the initial chroma features of the (n+1)th frame; and to enhance the initial chroma features of the (n+1)th frame based on the luminance propagation characteristics of the (n+1)th frame, to obtain the reconstructed chroma components and the chroma propagation characteristics of the (n+1)th frame. A construction module is used to construct the reconstructed image of the (n+1)th frame using a reconstruction network based on the reconstructed luminance component and the reconstructed chrominance component of the (n+1)th frame.

10. An electronic device, comprising: One or more processors; Memory, used to store one or more programs. wherein the one or more programs, when executed by the one or more processors, cause the one or more processors to carry out the method of any one of claims 1 to 8.

Images