Scalable encoding and decoding method and apparatus

Abstract

To provide a scalable encoding and decoding method and apparatus in the field of artificial intelligence (AI)-based video or picture compression technology.SOLUTION: An encoding method includes the steps of applying a control signal of a first signal component of a video signal to a first feature map of the first signal component to obtain a second feature map of the first signal component, in which the control signal of the first signal component is obtained through learning, applying a control signal of a second signal component of the video signal to a first feature map of the second signal component to obtain a second feature map of the second signal component, in which the control signal of the second signal component is obtained through learning, and obtaining a bitstream of the video signal on the basis of the second feature map of the first signal component and the second feature map of the second signal component. This application can be adapted to picture contents with different color characteristics.SELECTED DRAWING: Figure 7B

Description

[Technical field]

[0001] FIELD OF THE DISCLOSURE Embodiments of the present invention relate to the field of artificial intelligence (AI)-based video or picture compression technology, and in particular to a scalable encoding and decoding method and apparatus. [Background technology]

[0002] Video compression encoding and decoding technologies are widely used in the fields of multimedia services, broadcasting, video communication, storage, and the like, e.g., broadcast digital television, video transmission over the Internet and mobile networks, real-time conversation applications, e.g., video chat and video conferencing, DVDs and Blu-ray discs, video content collection and editing systems, and video cameras for security applications.

[0003] A large amount of video data is required to depict even a short video. This can pose challenges when the data is streamed or otherwise communicated over networks with limited bandwidth capacity. Therefore, video data is typically compressed before being communicated over today's telecommunications networks. The size of the video can also be a concern when the video is stored in a storage device, since memory resources may be limited. Video compression devices often use software and / or hardware to encode video data at the source prior to transmission or storage to reduce the amount of data required to represent a digital video picture. The compressed data is then received by a video decompressor at the destination. With limited network resources and ever-increasing demands for higher video quality, there is a need for improved compression and decompression techniques to improve compression ratios with little to no sacrifice in picture quality.

[0004] In recent years, it has become a gradual trend to apply deep learning to the field of end-to-end picture encoding and decoding technology. In a video encoder and a video decoder using a hybrid architecture, when entropy coding is performed on a feature map, the feature value is assumed to meet a zero-mean Gaussian distribution, and the variance of the Gaussian distribution is estimated by using a hyperprior structure to obtain a probability distribution model of the feature value, and then an arithmetic coding module performs entropy coding on the feature map based on the estimated probability distribution. To enable the decoder side to accurately estimate the probability distribution of the feature map, a module in the hyperprior structure extracts hidden variables of the estimated probability distribution, and the hidden variables are transmitted to the decoder side as side information through quantization and arithmetic coding. In this mechanism, for an input picture in YUV format, the ratio of the bit rates of Y, U, and V components is fixed. However, since pictures have different color characteristics in content, the fixed ratio of the bit rates of Y, U, and V components will cause large distortion in the coded picture. Summary of the Invention

[0005] This application provides a scalable encoding and decoding method and apparatus for accommodating picture contents with different color characteristics.

[0006] In this application, scalable encoding and decoding refers to a video signal being split into a first signal component and a second signal component, or a video signal being split into a first signal component, a second signal component, and a third signal component. The first signal component is a Y component and the second signal component is a UV component, a U component, or a V component. If the second signal component is a U component, the third signal component is a V component. Alternatively, if the second signal component is a V component, the third signal component is a U component.

[0007] According to a first aspect, the application provides an encoding method, comprising: applying a control signal of a first signal component of a video signal to a first feature map of the first signal component to obtain a second feature map of the first signal component, the control signal of the first signal component being obtained through learning, applying a control signal of a second signal component of the video signal to a first feature map of the second signal component to obtain a second feature map of the second signal component, the control signal of the second signal component being obtained through learning, and obtaining a bitstream of the video signal based on the second feature map of the first signal component and the second feature map of the second signal component.

[0008] In one possible implementation, obtaining a bitstream of the video signal based on the second feature map of the first signal component and the second feature map of the second signal component includes performing entropy coding on the second feature map of the first signal component and the second feature map of the second signal component, on the second feature map of the first signal component and the second feature map of the second signal component processed by a neural network, on the second feature map of the first signal component processed by a neural network and the second feature map of the second signal component, or on the second feature map of the first signal component processed by a neural network and the second feature map of the second signal component processed by a neural network to obtain a bitstream of the video signal.

[0009] In one possible implementation, obtaining a bitstream of the video signal based on the second feature map of the first signal component and the second feature map of the second signal component includes performing joint processing on the second feature map of the first signal component and the second feature map of the second signal component to obtain a joint feature map and performing entropy coding on the joint feature map to obtain a bitstream of the video signal; performing joint processing on the second feature map of the first signal component and the second feature map of the second signal component processed by the neural network to obtain a joint feature map and performing entropy coding on the joint feature map to obtain a bitstream of the video signal; performing joint processing on the second feature map of the first signal component processed by the neural network and the second feature map of the second signal component to obtain a joint feature map and performing entropy coding on the joint feature map to obtain a bitstream of the video signal; or performing joint processing on the second feature map of the first signal component processed by the neural network and the second feature map of the second signal component processed by the neural network to obtain a joint feature map and performing entropy coding on the joint feature map to obtain a bitstream of the video signal.

[0010] In one possible implementation, the method includes: obtaining a control signal for a first signal component from N candidate first control signals based on a quality factor of the first signal component, where N is an integer greater than 1, and obtaining a control signal for a second signal component from M candidate second control signals based on a quality factor of the second signal component, where M is an integer greater than 1. N and M may or may not be equal, which is not a limitation in this application.

[0011] In one possible implementation, when the first signal component is a Y component and the second signal component is a UV component, the method includes: deriving a control signal matrix {q y1 ,q y2 ,…,qyi ,…,q yN} and the UV component control signal matrix {q uv1 ,q uv2 ,…,q uvj ,…,q uvM}, where N and M are integers greater than 1, and a control signal q for the first signal component is generated based on an index i of the quality factor of the Y component. yi and based on the index j of the quality factor of the UV component, the control signal q of the second signal component uvj In this case, the bitstream of the video signal includes an index i of a quality factor of the Y component and an index j of a quality factor of the UV component.

[0012] Alternatively, in another possible implementation, when the first signal component is a Y component and the second signal component is a UV component, the method includes deriving a control signal matrix {q c1 ,q c2 ,…,q ci ,…,q cN}, where c is 2 and represents a Y component and a UV component, N is an integer greater than 1, and a control signal q including a first signal component and a second signal component is generated based on an index i of a quality factor of the video signal. ci where the bitstream of the video signal includes an index i of a quality factor of the video signal.

[0013] Alternatively, in yet another possible implementation implemented by using a fully-connected network, if the first signal component is a Y component and the second signal component is a UV component, the method includes using a quality factor of the Y component as an input to the fully-connected network and outputting a control signal for the Y component, and using a quality factor of the UV component as an input to the fully-connected network and outputting a control signal for the UV component, where the bitstream of the video signal includes a quality factor of the Y component and a quality factor of the UV component.

[0014] In one possible implementation, when the second signal component is a U component or a V component, the method further includes applying a control signal of a third signal component of the video signal to the first feature map of the third signal component to obtain a second feature map of the third signal component. The control signal of the third signal component is obtained through learning. When the second signal component is a U component, the third signal component is a V component. Alternatively, when the second signal component is a V component, the third signal component is a U component.

[0015] In one possible implementation, when the first signal component is a Y component and the second signal component is a U component, the third signal component is a V component, and the method further comprises: learning a control signal matrix {q y1 ,q y2 ,…,q yi ,…,q yN} and the U component control signal matrix {q u1 ,q u2 ,…,q uj ,…,q uM} and the V component control signal matrix {q v1 ,q v2 ,…,q vk ,…,q vL}, where N, M, and L are integers greater than 1, and a control signal q for the first signal component is generated based on an index i of the quality factor of the Y component. yi and based on the index j of the quality factor of the U component, the control signal q of the second signal component uj and based on the index k of the quality factor of the V component, the control signal q of the third signal component is obtained. vk In this case, the bitstream of the video signal includes an index i of a quality factor of the Y component, an index j of a quality factor of the U component, and an index k of a quality factor of the V component.

[0016] Alternatively, in another possible implementation, when the first signal component is a Y component and the second signal component is a U component, the third signal component is a V component, and the method further comprises: deriving a control signal matrix {q c1 ,q c2 ,…,q ci ,…,q cN}, where c is 3 and represents a Y component, a U component, and a V component, and N is an integer greater than 1; and based on an index i of a quality factor of the video signal, generate a control signal q including a first signal component, a second signal component, and a third signal component. ci where the bitstream of the video signal includes an index i of a quality factor of the video signal.

[0017] Alternatively, in yet another possible implementation implemented by using a fully-connected network, when the first signal component is a Y component and the second signal component is a U component, the third signal component is a V component, and the method further includes: using a quality factor of the Y component as an input to the fully-connected network and outputting a control signal for the Y component, using a quality factor of the U component as an input to the fully-connected network and outputting a control signal for the U component, and using a quality factor of the V component as an input to the fully-connected network and outputting a control signal for the V component. In this case, the bitstream of the video signal includes a quality factor of the Y component, a quality factor of the U component, and a quality factor of the V component.

[0018] According to a second aspect, the application provides a decoding method, the decoding method including: obtaining a bitstream of a video signal, performing entropy decoding on the bitstream to obtain a feature map of a first signal component of the video signal and a feature map of a second signal component of the video signal, obtaining a reconstruction map of the first signal component based on a response signal of the first signal component and the feature map of the first signal component, the response signal of the first signal component being obtained through learning, obtaining a reconstruction map of the second signal component based on the response signal of the second signal component and the feature map of the second signal component, the response signal of the second signal component being obtained through learning, and reconstructing a video signal based on the reconstruction map of the first signal component and the reconstruction map of the second signal component.

[0019] Note that in this application, the response signal at the decoder side is the same as the control signal at the encoder side. For ease of distinction, the signal at the decoder side is referred to as the response signal and the signal at the encoder side is referred to as the control signal. The response signal at the decoder side includes a response vector or includes a response vector and an offset vector.

[0020] In one possible implementation, the bitstream further includes quality factor information for the first signal component and quality factor information for the second signal component. The quality factor information for the first signal component is a quality factor of the first signal component or an index of the quality factor of the first signal component. The quality factor information for the second signal component is a quality factor of the second signal component or an index of the quality factor of the second signal component. The method further includes obtaining a response signal for the first signal component based on the quality factor information of the first signal component and obtaining a response signal for the second signal component based on the quality factor information of the second signal component.

[0021] If the quality factor information of the first signal component is a quality factor of the first signal component, the value of the quality factor of the first signal component is one of N. Alternatively, if the quality factor information of the first signal component is an index of the quality factor of the first signal component, the value range of the index of the quality factor of the first signal component is from 0 to N-1 or from 1 to N, where N is an integer greater than 1.

[0022] If the quality factor information of the second signal component is a quality factor of the second signal component, the value of the quality factor of the second signal component is one of M. Alternatively, if the quality factor information of the second signal component is an index of the quality factor of the second signal component, the value range of the index of the quality factor of the second signal component is 0 to M-1 or 1 to M, where M is an integer greater than 1.

[0023] In one possible implementation, when the first signal component is a Y component and the second signal component is a UV component, and the bit stream includes an index i of a quality factor of the Y component and an index j of a quality factor of the UV component, the method includes: deriving a response signal matrix {g y1 ,gy2 ,…,g yi ,…,g yN} and the response signal matrix {g uv1 ,g uv2 ,…,g uvj ,…,g uvM}, where N and M are integers greater than 1, and a response signal g of the first signal component is generated based on the quality factor index i of the Y component. yi Based on the index j of the quality factor of the UV component, the response signal g of the second signal component is obtained. uvj This includes obtaining

[0024] Alternatively, in another possible implementation, when the first signal component is a Y component and the second signal component is a UV component, if the bitstream includes an index i of a quality factor of the video signal, the method further comprises: deriving a response signal matrix {g c1 ,g c2 ,…,g ci ,…,g cN}, where c is 2 and represents the Y component and the UV component, and N is an integer greater than 1. Based on the index i of the quality factor of the video signal, generate a response signal g including the first signal component and the second signal component. ci This includes obtaining

[0025] Alternatively, in yet another possible implementation implemented by using a fully connected network, when the first signal component is a Y component and the second signal component is a U component, and the bit stream includes a quality factor of the first signal component and a quality factor of the second signal component, the method includes using the quality factor of the Y component as an input to the fully connected network and outputting a response signal of the Y component, and using the quality factor of the UV components as an input to the fully connected network and outputting a response signal of the UV components.

[0026] In one possible implementation, when the second signal component is a U component or a V component, the method further includes performing entropy decoding on the bitstream to obtain a feature map of a third signal component of the video signal, and obtaining a reconstruction map of the third signal component based on a response signal of the third signal component and the feature map of the third signal component. The response signal of the third signal component is obtained through learning. When the second signal component is a U component, the third signal component is a V component. Alternatively, when the second signal component is a V component, the third signal component is a U component. Reconstructing the video signal includes reconstructing the video signal based on the reconstruction map of the first signal component, the reconstruction map of the second signal component, and the reconstruction map of the third signal component.

[0027] In one possible implementation, when the first signal component is a Y component and the second signal component is a U component, the third signal component is a V component, and the bit stream includes an index i of a quality factor of the Y component, an index j of a quality factor of the U component, and an index k of a quality factor of the V component, the method includes: deriving a response signal matrix {g y1 ,g y2 ,…,g yi ,…,g yN} and the response signal matrix {g u1 ,g u2 ,…,g uj ,…,g uM} and the response signal matrix {g v1 ,g v2 ,…,g vk ,…,g vL}, where N, M, and L are integers greater than 1, and a response signal g of the first signal component is generated based on the quality factor index i of the Y component. yi and based on the index j of the quality factor of the U component, the response signal g of the second signal component uj and based on the quality factor index k of the V component, the response signal g of the third signal component vk This includes obtaining

[0028] Alternatively, in another possible implementation, when the first signal component is a Y component and the second signal component is a U component, the third signal component is a V component, and the bitstream includes an index i of a quality factor of the video signal, the method further comprises: c1 ,g c2 ,…,g ci ,…,g cN}, where c is 3 and represents a Y component, a U component, and a V component, and N is an integer greater than 1; and based on an index i of a quality factor of the video signal, generate a response signal g including a first signal component, a second signal component, and a third signal component. ci This includes obtaining

[0029] Alternatively, in yet another possible implementation implemented by using a fully connected network, when the first signal component is a Y component and the second signal component is a U component, the third signal component is a V component, and the bit stream includes a quality factor of the first signal component, a quality factor of the second signal component, and a quality factor of the third signal component, the method includes using the quality factor of the Y component as an input to the fully connected network and outputting a response signal of the Y component, using the quality factor of the U component as an input to the fully connected network and outputting a response signal of the U component, and using the quality factor of the V component as an input to the fully connected network and outputting a response signal of the V component.

[0030] According to a third aspect, the application provides an encoder including a processing circuit configured to perform a method according to the first aspect and any one of the implementations of the first aspect.

[0031] According to a fourth aspect, the application provides a decoder including a processing circuit configured to perform the method according to the second aspect and any one of the implementations of the second aspect.

[0032] According to a fifth aspect, the application provides a computer program product comprising program code, which, when executed on a computer or processor, is configured to perform a method according to any one of the first and second aspects and implementations of the first and second aspects.

[0033] According to a sixth aspect, the application provides an encoder including one or more processors and a non-transitory computer readable storage medium coupled to the processors and storing a program for execution by the processors, the program configuring the encoder to perform a method according to the first aspect and any one of the implementations of the first aspect when executed by the processor.

[0034] According to a seventh aspect, the application provides a decoder including one or more processors and a non-transitory computer readable storage medium coupled to the processors and storing a program for execution by the processors, the program configuring the decoder to perform a method according to any one of the second aspect and implementations of the second aspect when executed by the processor.

[0035] According to an eighth aspect, the application provides a non-transitory computer readable storage medium comprising program code, which, when executed by a computing device, is used to perform a method according to any one of the first and second aspects and implementations of the first and second aspects.

[0036] According to a ninth aspect, the present invention relates to an encoding device, the device being operable to implement the operations of the method embodiments according to the first aspect and any one of the implementations of the first aspect. The functions may be implemented by hardware or by the hardware executing corresponding software. The hardware or software includes one or more modules corresponding to the above-mentioned functions. In one possible design, the encoding device includes a first control module configured to apply a control signal of a first signal component of a video signal to a first feature map of the first signal component to obtain a second feature map of the first signal component, the control signal of the first signal component being obtained through learning, a second control module configured to apply a control signal of a second signal component of a video signal to a first feature map of the second signal component to obtain a second feature map of the second signal component, the control signal of the second signal component being obtained through learning, and an encoding module configured to obtain a bitstream of the video signal based on the second feature map of the first signal component and the second feature map of the second signal component. These modules may perform corresponding functions of the example method according to the first aspect and any one of the implementations of the first aspect. For details, please refer to the detailed description in the method example, and the details will not be described again here.

[0037] According to a tenth aspect, the present invention relates to a decoding device, having the functionality to implement the operations of the method embodiments according to the second aspect and any one of the implementations of the second aspect. The functionality may be implemented by hardware or by hardware executing corresponding software. The hardware or software comprises one or more modules corresponding to the above mentioned functions. In one possible design, the decoding device includes a decoding module configured to obtain a bitstream of the video signal and perform entropy decoding on the bitstream to obtain a feature map of a first signal component of the video signal and a feature map of a second signal component of the video signal, a first control module configured to obtain a reconstruction map of the first signal component based on a response signal of the first signal component and the feature map of the first signal component, where the response signal of the first signal component is obtained through learning, a second control module configured to obtain a reconstruction map of the second signal component based on a response signal of the second signal component and the feature map of the second signal component, where the response signal of the second signal component is obtained through learning, and a reconstruction module configured to reconstruct the video signal based on the reconstruction map of the first signal component and the reconstruction map of the second signal component. These modules may perform corresponding functions in the example method according to any one of the implementations of the second aspect and the second aspect. For details, please refer to the detailed description in the example method. Details will not be described again here.

[0038] In conventional end-to-end picture coding, in the training and optimization of a particular network, optimization is performed based on fixed weight values ​​of Y, U, and V components. Therefore, the bitrate ratio of Y, U, and V components is fixed. Since different pictures have different color characteristics, the fixed bitrate allocation will cause poor coding performance for some video pictures. In the above-mentioned aspect of this application, the control signal of each signal component is used to control the feature map of the corresponding signal component to support bitrate allocation among Y, U, and V components and adapt to picture contents with different color characteristics.

[0039] The details of one or more embodiments are set forth in the accompanying drawings and the description below. Other features, objects, and advantages will become apparent from the description, the accompanying drawings, and the claims. [Brief description of the drawings]

[0040] The following describes the accompanying drawings used in the embodiments of this application. [Figure 1A] FIG. 1 is a block diagram illustrating an example of a video coding system configured to implement an embodiment of the present invention, the system encoding and decoding video pictures based on deep learning. [Figure 1B] FIG. 2 is a block diagram illustrating another example of a video coding system configured to implement embodiments of the present invention, the system encoding and decoding video pictures based on deep learning. [Figure 1C] FIG. 11 is a block diagram illustrating yet another example of a video coding system configured to implement an embodiment of the present invention, where a video encoder and / or a video decoder encode and decode video pictures based on deep learning. [Diagram 2] FIG. 2 is a block diagram illustrating an example of a video encoder configured to implement an embodiment of the present invention, the video encoder 20 encoding video pictures based on deep learning. [Diagram 3] FIG. 3 is a block diagram illustrating an example of a video decoder configured to implement an embodiment of the present invention, the video decoder 30 decoding video pictures based on deep learning. [Figure 4] 1 is a schematic block diagram of a video coding device configured to implement an embodiment of the present invention; [Diagram 5] 4 is a schematic block diagram of another video coding device configured to implement embodiments of the present invention; [Figure 6]1 is a schematic diagram of the YUV format. [Figure 7A] FIG. 2 is a schematic diagram of a scalable encoding and decoding structure according to an embodiment of the present application; [Figure 7B] 7B is an embodiment of an encoding method based on FIG. 7A. [Figure 7C] 7B is an embodiment of a decoding method based on FIG. 7A. [Figure 7D] FIG. 2 is another schematic diagram of a scalable encoding and decoding structure according to an embodiment of the present application; [Figure 8A] FIG. 2 is a schematic diagram of another scalable encoding and decoding structure according to an embodiment of the present application; [Figure 8B] FIG. 8B is an embodiment of FIG. 8A. [Figure 9A] FIG. 13 is a schematic diagram of yet another scalable encoding and decoding structure according to an embodiment of the present application; [Figure 9B] FIG. 9B is an embodiment of FIG. 9A. [Figure 10] 1 is a schematic diagram of the structure of an encoding device 1000 according to an embodiment of the present application. [Figure 11] 11 is a schematic diagram of the structure of a decoding device 1100 according to an embodiment of the present application. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0041] The embodiments of this application provide an AI-based video picture compression technology, in particular, a scalable encoding and decoding method and apparatus to improve conventional end-to-end hybrid video coding systems.

[0042] Video coding typically refers to the processing of a sequence of pictures, which form a video or video sequence. In the field of video coding, the terms "picture", "frame" and "image" are sometimes used synonymously. Video coding (or, in general, coding) includes two parts: video encoding and video decoding. Video encoding is performed at the source and typically involves processing the original video picture (e.g., by compression) to reduce the amount of data required to represent the video picture (for more efficient storage and / or transmission). Video decoding is performed at the destination and typically involves the inverse process to that of the encoder to reconstruct the video picture. Embodiments referring to "coding" of a video picture (or, in general, a picture) shall be understood to relate to "encoding" or "decoding" of the video picture or the respective video sequence. The combination of the encoding and decoding parts is also called encoding and decoding (CODEC).

[0043] In the case of lossless video coding, the original video picture can be reconstructed. In other words, the reconstructed video picture has the same quality as the original video picture (assuming there is no transmission loss or other data loss during storage or transmission). In the case of lossy video coding, further compression is performed, for example through quantization, to reduce the amount of data required to represent the video picture, and the video picture cannot be completely reconstructed at the decoder. In other words, the quality of the reconstructed video picture is lower or poorer than the quality of the original video picture.

[0044] Some video coding standards use "lossy hybrid video coding" (i.e., spatial and temporal prediction in pixel domain is combined with 2D transform coding that applies quantization in transform domain). Each picture of a video sequence is typically divided into a set of non-overlapping blocks, and coding is typically performed at the block level. Specifically, at an encoder, video is usually processed and coded at the block (video block) level. For example, a predictive block is generated through spatial (intra) prediction and temporal (inter) prediction, the predictive block is subtracted from a current block (a block being processed or to be processed) to obtain a residual block, and the residual block is transformed and quantized in the transform domain to reduce (compress) the amount of data to be transmitted. At a decoder, the inverse processing part to the encoder is applied to the coded or compressed block to reconstruct the current block for representation. Furthermore, the encoder needs to replicate the decoder processing steps so that the encoder and the decoder generate the same predictions (e.g., intra-prediction and inter-prediction) and / or reconstructed pixels for processing or coding a subsequent block.

[0045] In the following embodiment of video coding system 10, encoder 20 and decoder 30 are described with reference to FIGS. 1B-3.

[0046] FIG. 1A is a schematic block diagram showing an example of a coding system. As shown in FIG. 1A, after a video capture device captures a video, a series of pre-processing is performed, and then the processed video is compressed and encoded to obtain an encoded bitstream. The bitstream can be sent by a transmitting module to a receiving module over a transmission network, and can be rendered and displayed after being decoded by a decoder. The video encoded bitstream can also be directly stored.

[0047] 1B is a schematic block diagram of an example coding system 10 that may utilize techniques of this application, such as video coding system 10 (or coding system 10 for short). Video encoder 20 (or encoder 20 for short) and video decoder 30 (or decoder 30 for short) of video coding system 10 represent examples of devices that may be configured to perform techniques according to various examples described in this application.

[0048] 1B, coding system 10 includes a source device 12. Source device 12 is configured to provide coded picture data 21, e.g., coded pictures, to a destination device 14 that is configured to decode the coded picture data 21.

[0049] The source device 12 includes an encoder 20 and may additionally, that is, optionally, include a picture source 16 , a pre-processor (or pre-processing unit) 18 , for example a picture pre-processor, and a communication interface (or communication unit) 22 .

[0050] Picture source 16 may include or be any type of picture capture device, e.g. for capturing real-world pictures, and / or any type of picture generation device, e.g. a computer graphics processor for generating computer animated pictures, or any type of other device for obtaining and / or providing real-world pictures, computer-generated pictures (e.g. screen content, virtual reality (VR) pictures), and / or any combination thereof (e.g. augmented reality (AR) pictures). Picture source may be any type of memory or storage for storing any of the above mentioned pictures.

[0051] The picture (or picture data) 17 may also be referred to as an original picture (or original picture data) 17 to distinguish it from a picture undergoing processing performed by a preprocessor (or a preprocessing unit).

[0052] The pre-processor 18 is configured to receive the (original) picture data 17 and perform pre-processing on the picture data 17 to obtain a pre-processed picture (or pre-processed picture data) 19. The pre-processing performed by the pre-processor 18 may include, for example, cropping, color format conversion (e.g., from RGB to YCbCr), color correction, or noise removal. It may be appreciated that the pre-processing unit 18 may be an optional component.

[0053] A video encoder (or encoder) 20 is configured to receive pre-processed picture data 19 and to provide encoded picture data 21 (as described in further detail below, eg with reference to FIG. 2).

[0054] The communications interface 22 of the source device 12 may be configured to receive the encoded picture data 21 and to transmit the encoded picture data 21 (or any further processed version thereof) via the communications channel 13 to another device, such as the destination device 14 or any other device, for storage or direct reconstruction.

[0055] The destination device 14 includes a decoder 30, and may additionally, that is to say optionally, include a communications interface (or communications unit) 28, a post-processor (or post-processing unit) 32, and a display device .

[0056] The communications interface 28 of the destination device 14 is configured to receive encoded picture data 21 (or any further processed version thereof), for example directly from the source device 12 or from any other source device, for example a storage device such as a coded picture data storage device, and to provide the encoded picture data 21 to a decoder 30.

[0057] The communications interface 22 and the communications interface 28 may be configured to transmit or receive encoded picture data (or encoded data) 21 over a direct communications link, such as a direct wired or wireless connection between the source device 12 and the destination device 14, or over any type of network, such as a wired or wireless network or any combination thereof, or any type of private and public network, or any type of combination thereof.

[0058] The communications interface 22 may be configured, for example, to package the encoded picture data 21 into a suitable format, such as packets, and / or process the encoded picture data through any type of transmission encoding or processing for transmission over a communications link or network.

[0059] Communications interface 28 may be configured in correspondence with communications interface 22, for example, to receive transmitted data and process the transmitted data through any type of corresponding transmission decoding or processing and / or depackaging to obtain encoded picture data 21.

[0060] Communication interface 22 and communication interface 28 may each be configured as a unidirectional communication interface as indicated by the arrows for communication channel 13 in FIG. 1B pointing from source device 12 to destination device 14, or may be configured as a bidirectional communication interface configured to send and receive messages, such as to set up a communication link and / or connection to acknowledge and exchange other information related to a data transmission, such as a coded picture data transmission.

[0061] A video decoder (or decoder) 30 is arranged to receive the encoded picture data 21 and to provide decoded pictures (or decoded picture data) 31 (as described in further detail below, for example with reference to FIG. 3).

[0062] The post-processor 32 is configured to post-process the decoded picture data 31 (also referred to as reconstructed video data), e.g. decoded pictures, to obtain post-processed picture data 33, e.g. post-processed pictures. The post-processing performed by the post-processing unit 32 may comprise, e.g., color format conversion (e.g., from YCbCr to RGB), color correction, cropping, or resampling, or any other processing, e.g. to prepare the decoded picture data 31 for display by a display device 34.

[0063] A display device 34 is configured to receive the post-processed picture data 33 for displaying the picture, e.g., to a user or viewer. The display device 34 can be or include any type of display for presenting the reconstructed picture, e.g., an integrated or external display or monitor. For example, the display can include a liquid crystal display (LCD), an organic light emitting diode (OLED) display, a plasma display, a projector, a micro LED display, a liquid crystal on silicon (LCoS), a digital light processor (DLP), or any other type of display.

[0064] Coding system 10 further includes a training engine 25. Training engine 25 is configured to train encoder 20 or decoder 30 to perform scalable encoding and decoding on reconstructed pictures.

[0065] In this embodiment of the application, the training data includes a training matrix set. The training matrix set includes a pre-filtered luma matrix, a quantization step matrix, and a post-filtered luma matrix of a picture block. A pixel at a corresponding position in the pre-filtered luma matrix corresponds to a pre-filtered luma value of a pixel at a corresponding position in the corresponding picture block. A pixel at a corresponding position in the quantization step matrix corresponds to a luma value of a pixel at a corresponding position in the corresponding picture block. A pixel at a corresponding position in the post-filtered luma matrix corresponds to a post-filtered luma value of a pixel at a corresponding position in the corresponding picture block.

[0066] For example, the matrices in the training matrix set may be input to the training engine 25 in the manner shown in Figures 6a-6c. As shown in Figure 6a, the matrices in the training matrix set are input directly to the training engine 25, and all of the matrices are two-dimensional matrices. As shown in Figure 6b, some or all of the matrices in the training matrix set are selected for combination to obtain a multidimensional matrix, which is then input to the training engine 25. As shown in Figure 6c, some or all of the matrices in the training matrix set are selected for addition (or multiplication) to obtain a two-dimensional matrix, which is then input to the training engine 25.

[0067] The training data can be stored in a database (not shown), and the training engine 25 obtains a target model (which may be, for example, a neural network for scalable encoding and decoding) through training based on the training data. Note that the source of the training data is not limited in this embodiment of the application. For example, the training data may be obtained from the cloud or other locations to perform model training.

[0068] The training engine 25 trains the target model so that the pixels before filtering are close to the original pixel values. In each training process, the small batch size can be 64 pictures, the initial learning rate can be 1e-4, and the step size can be 10. The training data can be data generated by the encoder based on different QP quantization parameter settings. The target model can be used to implement the scalable encoding and decoding method provided in this embodiment of the application. Specifically, a picture or picture block to be reconstructed is input to the target model after related pre-processing to obtain a filtered picture or picture block. The target model in this embodiment of the application can be specifically a filtering network. The target model will be described in detail below with reference to Figures 7A-7D.

[0069] The target model obtained through training by the training engine 25 can be applied to the coding system 10, for example, to the source device 12 (e.g., encoder 20) or the destination device 14 (e.g., decoder 30) shown in FIG. 1B. The training engine 25 may obtain the target model through training on the cloud, and the coding system 10 downloads the target model from the cloud and uses the target model. Alternatively, the training engine 25 may obtain the target model through training on the cloud and use the target model, and the coding system 10 directly obtains the processing result from the cloud.

[0070] 1B depicts source device 12 and destination device 14 as separate devices, an embodiment of the devices may also include source device 12 and destination device 14, or the functionality of source device 12 and destination device 14, i.e., source device 12 or corresponding functionality and destination device 14 or corresponding functionality. In such an embodiment, source device 12 or corresponding functionality and destination device 14 or corresponding functionality may be implemented using the same hardware and / or software, or by separate hardware and / or software, or by any combination thereof.

[0071] In accordance with the description, the presence and (exact) division of different units or functions within source device 12 and / or destination device 14 as shown in FIG. 1B may vary depending on the actual device and application.

[0072] The encoder 20 (e.g., video encoder 20) or the decoder 30 (e.g., video decoder 30), or both the encoder 20 and the decoder 30, may be implemented by processing circuitry as shown in FIG. 1C, such as, for example, one or more microprocessors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), discrete logic, hardware, dedicated video coding processors, or any combination thereof. The encoder 20 may be implemented by processing circuitry 46 to embody various modules as described with respect to the encoder 20 shown in FIG. 2 and / or any other encoder system or subsystem described herein. The decoder 30 may be implemented by processing circuitry 46 to embody various modules as described with respect to the decoder 30 shown in FIG. 3 and / or any other decoder system or subsystem described herein. The processing circuitry may be configured to perform various operations as described below. When some techniques are implemented in software, as shown in Figure 5, an apparatus can perform the techniques of the present invention by storing software instructions in a suitable non-transitory computer-readable storage medium and executing the instructions in hardware by using one or more processors. Either video encoder 20 or video decoder 30 may be integrated as part of a combined encoder / decoder (CODEC) in a single apparatus, as shown in Figure 1C, for example.

[0073] The source device 12 and the destination device 14 may include any of a wide range of devices, including any type of handheld or fixed device, such as, for example, a notebook or laptop computer, a mobile phone, a smart phone, a tablet or tablet computer, a camera, a desktop computer, a set-top box, a television, a display device, a digital media player, a video game console, a video streaming device (e.g., a content service server or a content delivery server), a broadcast receiver device, a broadcast transmitter device, or the like, and may use no operating system or any type of operating system. In some cases, the source device 12 and the destination device 14 may include components for wireless communication. Thus, the source device 12 and the destination device 14 may be wireless communication devices.

[0074] In some cases, the video coding system 10 shown in FIG. 1B is merely an example, and the techniques of this application are applicable to video coding settings (e.g., video encoding or video decoding) that do not necessarily include any data communication between an encoding device and a decoding device. In other examples, data may be retrieved from local memory and streamed over a network, etc. A video encoding device may encode data and store the encoded data in memory, and / or a video decoding device may retrieve data from memory and decode the data. In some examples, encoding and decoding are performed by devices that simply encode data to memory and / or retrieve data from memory and decode data without communicating with each other.

[0075] Figure 1C is an illustrative diagram illustrating an example of a video coding system 40 including the video encoder 20 of Figure 2 and / or the video decoder 30 of Figure 3, according to an example embodiment. The video coding system 40 may include an imaging device 41, a video encoder 20, a video decoder 30 (and / or a video encoder / decoder implemented by processing circuitry 46), an antenna 42, one or more processors 43, one or more memories 44, and / or a display device 45.

[0076] 1C, imaging device 41, antenna 42, processing circuitry 46, video encoder 20, video decoder 30, processor 43, memory 44, and / or display device 45 may be in communication with one another. Video coding system 40 may include only video encoder 20 or only video decoder 30 in different examples.

[0077] In some examples, the antenna 42 may be configured to transmit or receive an encoded bitstream of video data. In some examples, the display device 45 may be configured to present the video data. The processing circuitry 46 may include application-specific integrated circuit (ASIC) logic, a graphics processing unit, a general-purpose processor, or the like. The video coding system 40 may also include an optional processor 43. The optional processor 43 may also include application-specific integrated circuit (ASIC) logic, a graphics processor, a general-purpose processor, or the like. The memory 44 may be any type of memory, such as, for example, a volatile memory (e.g., a static random access memory (SRAM) or a dynamic random access memory (DRAM)) or a non-volatile memory (e.g., a flash memory). In one non-limiting example, the memory 44 may be implemented by a cache memory. In another example, the processing circuitry 46 may include a memory (e.g., a cache) for implementing an image buffer.

[0078] In some examples, video encoder 20 implemented using logic circuitry may include a picture buffer (which may be implemented, for example, by processing circuitry 46 or memory 44) and a graphics processing unit (which may be implemented, for example, by processing circuitry 46). The graphics processing unit may be communicatively coupled to the picture buffer. The graphics processing unit may include video encoder 20 implemented by processing circuitry 46 to embody various modules described with respect to FIG. 2 and / or any other encoder system or subsystem described herein. The logic circuitry may be configured to perform various operations described herein.

[0079] In some examples, the video decoder 30 may be implemented by the processing circuitry 46 in a similar manner to implement the various modules described with reference to the video decoder 30 of FIG. 3 and / or any other decoder system or subsystem described herein. In some examples, the video decoder 30 implemented by using logic circuits may include a picture buffer (which may be implemented by the processing circuitry 46 or memory 44) and a graphics processing unit (which may be implemented, for example, by the processing circuitry 46). The graphics processing unit may be communicatively coupled to the picture buffer. The graphics processing unit may include the video decoder 30 implemented by the processing circuitry 46 to embody the various modules described with reference to FIG. 3 and / or any other decoder system or subsystem described herein.

[0080] In some examples, antenna 42 may be configured to receive an encoded bitstream of video data. As described, the encoded bitstream may include data related to the video frame encoding described herein, such as data related to a coding partition (e.g., transform coefficients or quantized transform coefficients, optional indicators (to be described), and / or data defining the coding partition), indicators, index values, mode selection data, or the like. Video coding system 40 may further include a video decoder 30 coupled to antenna 42 and configured to decode the encoded bitstream. Display device 45 is configured to present the video frames.

[0081] It should be understood that, in this embodiment of the application, for the examples described with reference to video encoder 20, video decoder 30 may be configured to perform an inverse process. With respect to signaling syntax elements, video decoder 30 may be configured to receive and parse such syntax elements and correspondingly decode the associated video data. In some examples, video encoder 20 may entropy encode the syntax elements into the encoded video bitstream. In such examples, video decoder 30 may parse such syntax elements and decode the associated video data accordingly.

[0082] For ease of explanation, embodiments of the present invention are described herein with reference to reference software for Versatile Video Coding (VVC) or High-Efficiency Video Coding (HEVC), e.g., developed by the Joint Collaborative Team on Video Coding (JCT-VC) between the ITU-T Video Coding Experts Group (VCEG) and the ISO / IEC Moving Picture Experts Group (MPEG). As will be appreciated by those skilled in the art, embodiments of the present invention are not limited to HEVC or VVC.

[0083] Encoder and encoding method FIG. 2 is a schematic block diagram illustrating an example of a video encoder 20 configured to implement the technology of this application. In the example of FIG. 2, the video encoder 20 includes an input (or input interface) 201, a residual calculation unit 204, a transform processing unit 206, a quantization unit 208, an inverse quantization unit 210, an inverse transform processing unit 212, a reconstruction unit 214, a loop filter 220, a decoded picture buffer (DPB) 230, a mode selection unit 260, an entropy coding unit 270, and an output (or output interface) 272. The mode selection unit 260 may include an inter prediction unit 244, an intra prediction unit 254, and a division unit 262. The inter prediction unit 244 may include a motion estimation unit and a motion compensation unit (not shown). The video encoder 20 illustrated in FIG. 2 may also be referred to as a hybrid video encoder, or a video encoder based on a hybrid video codec.

[0084] The residual calculation unit 204, the transform processing unit 206, the quantization unit 208, and the mode selection unit 260 form a forward signal path of the encoder 20. The inverse quantization unit 210, the inverse transform processing unit 212, the reconstruction unit 214, the buffer 216, the loop filter 220, the decoded picture buffer (DPB) 230, the inter prediction unit 244, and the intra prediction unit 254 form a backward signal path of the encoder 20. The backward signal path of the encoder 20 corresponds to the signal path of the decoder (see decoder 30 of FIG. 3). The inverse quantization unit 210, the inverse transform processing unit 212, the reconstruction unit 214, the loop filter 220, the decoded picture buffer 230, the inter prediction unit 244, and the intra prediction unit 254 form a “built-in decoder” of the video encoder 20.

[0085] Quantization The quantization unit 208 may be configured to quantize the transform coefficients 207, for example through scalar quantization or vector quantization, to obtain quantized transform coefficients 209. The quantized transform coefficients 209 may also be referred to as quantized residual coefficients 209.

[0086] The quantization process may reduce the bit depth associated with some or all of the transform coefficients 207. For example, an n-bit transform coefficient may be rounded to an m-bit transform coefficient during quantization, where n is greater than m. The degree of quantization may be changed by adjusting a quantization parameter (QP). For example, in scalar quantization, different scales may be applied to achieve finer or coarser quantization. A smaller quantization step corresponds to a finer quantization and a larger quantization step corresponds to a coarser quantization. An appropriate quantization step may be indicated by a quantization parameter (QP). For example, the quantization parameter may be an index to a predefined set of appropriate quantization steps. For example, a smaller quantization parameter may correspond to a finer quantization (smaller quantization step) and a larger quantization parameter may correspond to a coarser quantization (larger quantization step), and vice versa. Quantization may include a division by a quantization step, and the corresponding or inverse dequantization, performed, for example, by the inverse quantization unit 210, may include a multiplication by the quantization step. Some standard embodiments, such as HEVC, may use a quantization parameter to determine the quantization step. In general, the quantization step may be calculated based on the quantization parameter by using a fixed-point approximation of an equation that includes a division. Other scale factors may be introduced for quantization and dequantization to restore the norm of the residual block. The norm of the residual block may be changed due to the scale used for the fixed-point approximation of the equation for the quantization step and the quantization parameter. In one implementation, the scale of the inverse transform may be combined with the scale of the dequantization. Alternatively, a customized quantization table may be used and signaled from the encoder to the decoder, for example, in the bitstream. Quantization is a lossy operation, and a larger quantization step indicates a larger loss.

[0087] In one embodiment, video encoder 20 (and correspondingly, quantization unit 208) may be configured to output a quantization parameter (QP), e.g., directly or after encoding performed by entropy encoding unit 270, or after compression, such that video decoder 30 may receive and apply the quantization parameter for decoding.

[0088] inverse quantization The inverse quantization unit 210 is configured to perform the inverse quantization of the quantization unit 208 on the quantized coefficients, e.g., by applying the inverse of the quantization scheme performed by the quantization unit 208, based on or using the same quantization step as the quantization step of the quantization unit 208, to obtain dequantized coefficients 211. The dequantized coefficients 211, which may also be referred to as dequantized residual coefficients 211, correspond to the transform coefficients 207, but are typically not exactly the same as the transform coefficients due to losses incurred by quantization.

[0089] Reconstruction The reconstruction unit 214 (e.g., adder 214) is configured to add the transformation block 213 (i.e., the reconstructed residual block 213) to the prediction block 265, for example by adding pixel values ​​of the reconstructed residual block 213 and pixel values ​​of the prediction block 265, to obtain a reconstructed block 215 in the pixel domain.

[0090] filtering The loop filter unit 220 (or "loop filter" 220 for short) is configured to filter the reconstructed block 215 to obtain a filtered block 221, or in general, to filter the reconstructed pixels to obtain filtered pixel values. The loop filter unit is configured to, for example, smooth pixel transformations or otherwise improve video quality. The loop filter unit 220 may include one or more loop filters, such as, for example, a deblocking filter, a sample-adaptive offset (SAO) filter, or one or more other filters, such as, for example, an adaptive loop filter (ALF), a noise suppression filter (NSF), or any combination thereof. In one example, the loop filter unit 220 may include a deblocking filter, a SAO filter, and an ALF filter. The order of the filtering process may be a deblocking filter, a SAO filter, and an ALF filter. In another example, a process referred to as luma mapping with chroma scaling (LMCS) (i.e., an adaptive in-loop reshaper) is added. This process is performed before deblocking. In another example, the deblocking filtering process may also be applied to interior sub-block edges, such as affine sub-block edges, ATMVP sub-block edges, sub-block transform (SBT) edges, and intra sub-partition (ISP) edges. Although loop filter unit 220 is illustrated as a loop filter in FIG. 2, in other configurations loop filter unit 220 may be implemented as a post-loop filter. Filtered block 221 may be referred to as filtered reconstruction block 221.

[0091] In one embodiment, video encoder 20 (correspondingly, loop filter unit 220) may be configured to output loop filter parameters (e.g., SAO filter parameters, ALF filter parameters, or LMCS parameters, etc.), e.g., directly or after entropy encoding performed by entropy encoding unit 270, such that decoder 30 can receive and apply the same or different loop filter parameters for decoding.

[0092] Decoder and decoding method 3 shows an example of a video decoder 30 configured to implement the techniques of this application. The video decoder 30 is configured to receive coded picture data 21 (e.g., coded bitstream 21) encoded by, for example, encoder 20 to obtain a decoded picture 331. The coded picture data or bitstream includes information for decoding the coded picture data, e.g., data representing picture blocks of coded video slices (and / or tile groups or tiles), and related syntax elements.

[0093] In the example of Figure 3, the decoder 30 includes an entropy decoding unit 304, an inverse quantization unit 310, an inverse transform processing unit 312, a reconstruction unit 314 (e.g., adder 314), a loop filter 320, a decoded picture buffer (DBP) 330, a mode application unit 360, an inter prediction unit 344, and an intra prediction unit 354. The inter prediction unit 344 may be or include a motion compensation unit. The video decoder 30 may, in some examples, perform a decoding process that is substantially the reverse of the encoding process described with respect to the video encoder 20 of Figure 2.

[0094] As described with respect to encoder 20, inverse quantization unit 210, inverse transform processing unit 212, reconstruction unit 214, loop filter 220, decoded picture buffer (DPB) 230, inter prediction unit 344, and intra prediction unit 354 also form a “built-in decoder” of video encoder 20. Thus, inverse quantization unit 310 may be identical in function to inverse quantization unit 110, inverse transform processing unit 312 may be identical in function to inverse transform processing unit 212, reconstruction unit 314 may be identical in function to reconstruction unit 214, loop filter 320 may be identical in function to loop filter 220, and decoded picture buffer 330 may be identical in function to decoded picture buffer 230. Thus, the description of the respective units and functions of video encoder 20 applies correspondingly to the respective units and functions of video decoder 30.

[0095] inverse quantization Inverse quantization unit 310 may be configured to receive a quantization parameter (QP) (or, in general, information regarding inverse quantization) and quantized coefficients from coded picture data 21 (e.g., by parsing and / or decoding, e.g., by entropy decoding unit 304) and perform inverse quantization on the decoded quantized coefficients 309 based on the quantization parameter to obtain dequantized coefficients 311, which may also be referred to as transform coefficients 311. The inverse quantization process may include determining a degree of quantization based on the quantization parameter calculated by video encoder 20 for each video block in a video slice, as well as determining the degree of inverse quantization to be performed.

[0096] Reconstruction The reconstruction unit 314 (e.g., adder 314) may be configured to add the reconstructed residual block 313 to the prediction block 365, for example by adding pixel values ​​of the reconstructed residual block 313 and pixel values ​​of the prediction block 365, to obtain a reconstructed block 315 in the pixel domain.

[0097] filtering The loop filter unit 320 (either in the coding loop or after the coding loop) is configured to filter the reconstruction block 315 to obtain a filtered block 321, for example, to smooth pixel transformation or improve video quality. The loop filter unit 320 may include one or more loop filters, such as a deblocking filter, a sample-adaptive offset (SAO) filter, or one or more other filters, such as an adaptive loop filter (ALF), a noise suppression filter (NSF), or any combination thereof. In one example, the loop filter unit 320 may include a deblocking filter, an SAO filter, and an ALF filter. The order of the filtering process may be a deblocking filter, an SAO filter, and an ALF filter. In another example, a process referred to as luma mapping with chroma scaling (LMCS) (i.e., an adaptive in-loop reshaper) is added. This process is performed before deblocking. In another example, the deblocking filtering process may also be applied to interior sub-block edges, such as affine sub-block edges, ATMVP sub-block edges, sub-block transform (SBT) edges, and intra sub-partition (ISP) edges. Although loop filter unit 320 is illustrated in FIG. 3 as a loop filter, in other configurations, loop filter unit 320 may be implemented as a post-loop filter.

[0098] The decoder 30 is arranged to output the decoded pictures 331, for example via an output 332, for presentation to or viewing by a user.

[0099] Although the above-mentioned embodiments are mainly described based on video coding, it should be noted that the coding system 10, encoder 20, and decoder 30 embodiments, as well as other embodiments described herein, are also applicable to still image processing or coding, i.e., processing or coding of an individual picture independent of the preceding or successive pictures in video coding. In general, when picture processing is limited to a single picture 17, the inter prediction unit 244 (encoder) and the inter prediction unit 344 (decoder) may be unavailable. For example, all other functions (also referred to as tools or techniques) of the video encoder 20 and the video decoder 30, such as the residual calculation 204 / 304, the transform 206, the quantization 208, the inverse quantization 210 / 310, the (inverse) transform 212 / 312, the partitioning 262 / 362, the intra prediction 254 / 354, and / or the loop filtering 220 / 320, the entropy encoding 270, and the entropy decoding 304, may be equally used for still image processing.

[0100] 4 is a schematic diagram of a video coding device 400 according to an embodiment of the present invention. The video coding device 400 is applicable for implementing the disclosed embodiments described herein. In one embodiment, the video coding device 400 may be a decoder, such as the video decoder 30 of FIG. 1B, or an encoder, such as the video encoder 20 of FIG. 1B.

[0101] The video coding device 400 includes an ingress port 410 (or input port 410) and a receiver unit (Rx) 420 for receiving data, a processor, logic unit, or central processing unit (CPU) 430 (e.g., the processor 430 may be a neural network processing unit 430) for processing the data, a transmitter unit (Tx) 440 and an egress port 450 (or output port 450) for transmitting the data, and a memory 460 for storing the data. The video coding device 400 may also include optical-to-electrical (OE) and electrical-to-optical (EO) components coupled to the ingress port 410, the receiver unit 420, the transmitter unit 440, and the egress port 450 and configured as an egress or ingress for optical or electrical signals.

[0102] The processor 430 is implemented by hardware and software. The processor 430 may be implemented as one or more of a CPU chip, a core (e.g., a multi-core processor), an FPGA, an ASIC, and a DSP. The processor 430 is in communication with the ingress port 410, the receiver unit 420, the transmitter unit 440, the egress port 450, and the memory 460. The processor 430 includes a coding module 470 (e.g., a neural network (NN)-based coding module 470). The coding module 470 implements the above-disclosed embodiments. For example, the coding module 470 implements, processes, prepares, or provides various coding operations. Thus, the inclusion of the coding module 470 provides a substantial improvement to the functionality of the video coding device 400 and affects the switching of the video coding device 400 to different states. Alternatively, the coding module 470 is implemented based on instructions stored in the memory 460 and executed by the processor 430.

[0103] Memory 460 may include one or more disks, tape drives, and solid state drives, and may also be used as an overflow data storage device to store programs when such programs are selected for execution, as well as to store instructions and data retrieved during program execution. Memory 460 may be volatile and / or non-volatile, and may be read-only memory (ROM), random access memory (RAM), ternary content-addressable memory (TCAM), and / or static random-access memory (SRAM).

[0104] 5 is a simplified block diagram of an apparatus 500 according to an example embodiment, which may be used as either or both of the source device 12 and the destination device 14 of FIG.

[0105] The processor 502 in the device 500 may be a central processing unit, or may be any other type of device or devices now existing or later developed capable of controlling or processing information. Although the disclosed implementations may be implemented with a single processor as shown, such as processor 502, greater speed and greater efficiency may be achieved by using two or more processors.

[0106] The memory 504 in the apparatus 500 may, in one implementation, be a read-only memory (ROM) device or a random access memory (RAM) device. Any other suitable type of storage device may be used as the memory 504. The memory 504 may include code and data 506 that may be accessed by the processor 502 via a bus 512. The memory 504 may further include an operating system 508 and application programs 510. The application programs 510 include at least one program that enables the processor 502 to perform the methods described herein. For example, the application programs 510 may include applications 1-N, and may further include a video coding application that performs the methods described herein.

[0107] The device 500 may further include one or more output devices, such as a display 518. The display 518, in one example, may be a touch-sensitive display that combines a display with touch-sensitive elements configured to sense touch input. The display 518 may be coupled to the processor 502 via the bus 512.

[0108] Though depicted here as a single bus, bus 512 of device 500 may include multiple buses. Additionally, secondary storage may be directly coupled to other components of device 500 or accessed over a network, and may include a single integrated unit, such as a memory card, or multiple units, such as multiple memory cards. Device 500 may thus be implemented in a wide variety of configurations.

[0109] DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS An embodiment of the present invention relates to a lossy AI picture coding scheme for color component bit rate allocation and is applicable to the video coding systems, encoders and decoders described in FIGS. 1A-5.

[0110] It should be noted that the method provided in this application is mainly applied to the bitrate allocation process for Y, U and V components, which is mainly controlled by the encoder side. To make the decoder side more adaptive, a corresponding control unit may also be added to the decoder.

[0111] A video picture signal usually includes one luma component and two chroma components. The luma component is usually represented by the symbol Y, and the chroma components are usually represented by the symbol U or V. As shown in (a)-(c) of Figure 6, commonly used YUV formats include the following formats: In Figure 6, the cross marks represent the sampling points of the luma component, and the circles represent the sampling points of each chroma component.

[0112] 4:4:4 format: Chroma components are not downsampled.

[0113] 4:2:2 format: A horizontal downsampling of 2:1 is performed on the chroma components with paired luma components, and no vertical downsampling is performed. For every two U or V sampling points, each scanned row contains four Y sampling points.

[0114] 4:2:0 format: A horizontal downsampling of 2:1 is performed on the chroma components, and a vertical downsampling of 2:1 is performed on the luma components.

[0115] When a video picture is in YUV4:2:0 format, if the luma component of a picture block is a 2N×2N picture block, the chroma component of the picture block is an N×N picture block. In this embodiment of the present invention, the technical solution of the present invention is described using the example of 4:2:0 format. However, it can be understood that the technical solution of the present invention is also applicable to inter-prediction between different components in other video picture formats, such as other YUV formats or RGB formats, in addition to the YUV4:2:0 format. Meanwhile, the current block may be a square block, a non-square rectangular block or a region of other shapes, and the technical solution provided in the embodiment of the present invention is also applicable.

[0116] For ease of explanation, this embodiment of the present invention is described by using a first signal component and a second signal component. If the picture signal includes a luma signal component and a chroma signal component, the first signal component can be a chroma component, and the second signal component can be a luma component. If the picture signal includes three signal components R, G, and B, the first signal component can be any one of the three signal components R, G, and B, and the second signal component can be one of the three signal components R, G, and B different from the first signal component. If the picture signal is decomposed into multiple signal components in other ways, a similar method can be used to designate the first signal component and the second signal component.

[0117] As described below, in this embodiment of the application, the quality factor may be input to a bitrate control module (also referred to as bitrate allocation control module), which generates a control signal for each component's feature map. The control vector of the control signal for each component is multiplied by the corresponding feature map to obtain the quantized feature value, i.e. the feature value to be coded.

[0118] FIG. 7A illustrates an AI picture coding system 700 for YUV bit rate allocation according to an embodiment of the present invention. The AI ​​picture coding system 700 is applicable to video picture encoders and decoders. As shown in FIG. 7A, the picture coding system 700 includes a first signal component processing module (e.g., Y component processing module), a second signal component processing module (e.g., UV component processing module), a bit rate allocation control module, and an entropy encoding module on the encoder side, and a first signal component processing module (Y component processing module 2), a second signal component processing module (e.g., UV component processing module 2), and an entropy decoding module on the decoder side. The picture coding system 700 optionally includes a joint processing module, a joint processing module 2, and a quality response module (also referred to as bit rate allocation control module 2 or bit rate control module 2). In the picture coding system 700, a quality factor of the Y component and a quality factor of the UV component are input to the bit rate allocation control module. The module outputs a control signal which is respectively applied to the feature map of Y component output by the Y component processing module and the feature map of UV component output by the UV component processing module (which may be separately referred to as the first feature map), and outputs a second feature map of each signal component to allocate the bit rate of the Y component and the UV component. Then, a bit stream of the video signal is obtained according to the second feature maps of the signal components. For example, the feature maps output by the Y component processing module and the UV component processing module are directly concatenated together, or the feature maps output by the Y component processing module and the UV component processing module are directly added together to form the feature map finally output by the encoder, and the entropy coding is performed on the finally output feature map.Optionally, the feature maps output by the Y component processing module and the UV component processing module are input to a joint processing module to obtain the feature map finally output by the encoder, and entropy coding is performed on the final output feature map.

[0119] In the architecture shown in FIG. 7A, FIG. 7B is an embodiment of an encoding method. Step 701: Obtain a control signal for a first signal component based on a quality factor of the first signal component. Step 702: Obtain a control signal for a second signal component based on a quality factor of a second signal component. For example, in the embodiment shown in FIG. 7B, the control signal for the first signal component can be obtained from N candidate first control signals based on the quality factor of the first signal component, where N is an integer greater than 1. The control signal for the second signal component can be obtained from M candidate second control signals based on the quality factor of the second signal component, where M is an integer greater than 1. N and M may be equal or unequal. This is not a limitation in this application.

[0120] Step 703: Apply the control signal for the first signal component to a first feature map for the first signal component to obtain a second feature map for the first signal component.Step 704: Apply the control signal for the second signal component to the first feature map for the second signal component to obtain a second feature map for the second signal component.

[0121] For example, in one embodiment, the control signal is generated through training by the network and applied to a feature map output by at least one layer of the network in each of the Y component processing module and the UV component processing module (which may be referred to as a first feature map) to output a second feature map, e.g., the control signal is applied to the feature map output by the last layer of the network.

[0122] Step 705: Obtain a bitstream of the video signal based on the second feature map of the first signal component and the second feature map of the second signal component.

[0123] A control signal may be applied to the first feature map output by any layer of the network in each of the Y component processing module and the UV component processing module, so that processing by the neural network may continue to be performed on the second feature map after the second feature map is output. Correspondingly, obtaining a bitstream of a video signal based on the second feature map of the Y component and the second feature map of the UV component includes: performing entropy coding on the second feature map of the first signal component and the second feature map of the second signal component to obtain a bitstream of the video signal; performing entropy coding on the second feature map of the first signal component and the second feature map of the second signal component processed by the neural network to obtain a bitstream of the video signal; performing entropy coding on the second feature map of the first signal component processed by the neural network and the second feature map of the second signal component to obtain a bitstream of the video signal; or and performing entropy coding on a second feature map of the first signal component processed by the neural network and a second feature map of the second signal component processed by the neural network to obtain a bitstream of the video signal.

[0124] When the picture coding system 700 includes a joint processing module, obtaining a bitstream of the video signal based on the second feature map of the first signal component and the second feature map of the second signal component includes: performing joint processing on a second feature map of the first signal component and a second feature map of the second signal component to obtain a joint feature map, and performing entropy coding on the joint feature map to obtain a bitstream of the video signal; performing joint processing on the second feature map of the first signal component and the second feature map of the second signal component processed by the neural network to obtain a joint feature map, and performing entropy coding on the joint feature map to obtain a bitstream of the video signal; performing joint processing on the second feature map of the first signal component processed by the neural network and the second feature map of the second signal component to obtain a joint feature map, and performing entropy coding on the joint feature map to obtain a bitstream of the video signal; or performing joint processing on the second feature map of the first signal component processed by the neural network and the second feature map of the second signal component processed by the neural network to obtain a joint feature map, and performing entropy coding on the joint feature map to obtain a bitstream of the video signal.

[0125] Details are as follows:

[0126] When the control signal includes control vectors of Y and UV feature maps, the bitrate allocation control module learns N candidate first control signals of the first signal components (e.g., a control vector matrix {q y1 ,q y2 ,…,q yi ,…,q yN}) and M candidate second control signals (e.g., a control vector matrix {q uv1 ,q uv2 ,…,q uvj ,…,q uvM In use, a control signal q for the first signal component is generated based on the index i of the quality factor of the Y component. yi is obtained, and based on the index j of the quality factor of the UV component, the control signal q of the second signal component is uvjis obtained, where N and M are integers greater than 1. Then, each control vector is multiplied by a corresponding feature map for each channel to control the feature maps of different components. In this case, the bitstream of the video signal sent to the decoder side includes an index i of the quality factor of the Y component and an index j of the quality factor of the UV components.

[0127] If the control signal includes a control vector and an offset vector for the Y and UV feature maps, then the control vector q of the first signal component is calculated based on the index i of the quality factor of the Y component as described in the above method. yi and the offset vector b yi is obtained, and based on the index j of the quality factor of the UV component, the control vector q of the second signal component is calculated. uvj and the offset vector b uvi Then, to control the feature maps of different components, each control vector is multiplied by the corresponding feature map for each channel, and a corresponding offset vector is added.

[0128] In another embodiment, the control signal for the Y component and the control signal for the UV component are used as a 2-tuple. The bitrate allocation control module learns N candidate control signals for the video signal (e.g., a control signal matrix {q c1 ,q c2 ,…,q ci ,…,q cN}, where c is 2 and each control vector q ci Then, based on the index i of the quality factor of the video signal, a control signal q ci is obtained. The offset vectors are implemented in a similar manner. Specifically, each offset vector of the video signal includes both the offset vector of the first signal component and the offset vector of the second signal component. In this case, the bitstream of the video signal sent to the decoder side includes the index i of the quality factor of the video signal.

[0129] In yet another embodiment, the quality factor of the Y component and the quality factor of the UV component are used as inputs to a fully connected network, and the control vector q yi and the control vector q uvj To control the feature maps of different components, each control vector is multiplied by the corresponding feature map for each channel. The quality coefficients of the Y component and the quality coefficients of the UV components are used as inputs to the fully connected network, and the offset vector b yi and the offset vector b uvj can be further output. Then, each control vector is multiplied by a corresponding feature map for each channel, and each offset vector is added to a corresponding feature map for each channel to control the feature maps of different components. In this case, the bitstream of the video signal sent to the decoder side includes a quality factor of the Y component and a quality factor of the UV component.

[0130] The decoder side performs entropy decoding on the received bitstream to obtain feature maps, which are decomposed into a Y component feature map and a UV component feature map. Optionally, the feature maps obtained through entropy decoding are first input into joint processing sub-module 2 to obtain a Y component feature map and a UV component feature map.

[0131] The feature map of the Y component and the feature map of the UV component are input to the Y component processing module 2 and the UV component processing module 2, respectively, to output a reconstruction map of the Y component and a reconstruction map of the UV component. Optionally, the quality factor of the Y component and the quality factor of the UV component are input to a quality response module, which outputs a response signal that is applied to the feature map of the Y component output by the Y component processing module and the feature map of the UV component output by the UV component processing module, respectively, to implement an adaptive quality response of the Y component and the UV component. The quality response is sometimes referred to as quality control, or simply as quality response on the decoder side to distinguish it from quality control on the encoder side.

[0132] Specifically, as shown in FIG. 7C as an example, step 711: the decoder side obtains a bitstream of a video signal from the encoder side, and performs entropy decoding on the bitstream to obtain a feature map of a first signal component (e.g., a Y component) of the video signal and a feature map of a second signal component (e.g., a UV component) of the video signal.

[0133] The decoder side further obtains quality factor information of the first signal component and quality factor information of the second signal component from the bitstream. The quality factor information of the first signal component is the quality factor of the first signal component or an index of the quality factor of the first signal component. The quality factor information of the second signal component is the quality factor of the second signal component or an index of the quality factor of the second signal component. Then, a response signal of the first signal component is obtained based on the quality factor information of the first signal component, and a response signal of the second signal component is obtained based on the quality factor information of the second signal component. If the quality factor information of the first signal component is the quality factor of the first signal component, the value of the quality factor of the first signal component is one of N. Alternatively, if the quality factor information of the first signal component is the index of the quality factor of the first signal component, the value range of the index of the quality factor of the first signal component is 0 to N-1 or 1 to N, where N is an integer greater than 1. Similarly, if the quality factor information of the second signal component is the quality factor of the second signal component, the value of the quality factor of the second signal component is one of M. Alternatively, when the quality factor information of the second signal component is an index of the quality factor of the second signal component, the index of the quality factor of the second signal component has a value range of 0 to M-1 or 1 to M, where M is an integer greater than 1.

[0134] If the joint feature map is transmitted from the encoder side, the decoder side further needs to perform entropy decoding on the joint feature map, and obtain the feature map of the first signal component and the feature map of the second signal component through processing by the neural network.

[0135] Step 712: Obtain a response signal of the first signal component based on the quality factor information of the first signal component.Step 713: Obtain a response signal of the second signal component based on the quality factor information of the second signal component.

[0136] In one embodiment, when the bitstream includes an index i of a quality factor of a Y component and an index j of a quality factor of a UV component, the decoder side learns a response signal matrix {g y1 ,g y2 ,…,g yi ,…,g yN} and the response signal matrix {g uv1 ,g uv2 ,…,g uvj ,…,g uvM}, where N and M are integers greater than 1. Optionally, a control signal matrix {q y1 ,q y2 ,…,q yi ,…,q yN}, the response signal matrix {g y1 ,g y2 ,…,g yi ,…,g yN} is obtained, and the control signal matrix {q uv1 ,q uv2 ,…,q uvj ,…,q uvM}, the response signal matrix {g uv1 ,g uv2 ,…,g uvj ,…,g uvM} is obtained, where N and M are integers greater than 1. Based on the index i of the quality factor of the Y component, the response signal g yi Based on the index j of the quality factor of the UV component, the response signal g of the second signal component is obtained. uvj is obtained.

[0137] In another embodiment, when the bitstream includes an index i of a quality factor of a video signal, the decoder side learns through learning a response signal matrix {g c1 ,g c2 ,…,g ci ,…,g cN}, where c is 2 and represents the Y and UV components, and N is an integer greater than 1. Optionally, a control signal matrix {q c1 ,q c2 ,…,q ci ,…,q cN}, the response signal matrix of the video signal, g c1 ,g c2 ,…,g ci ,…,g cN} is obtained, where c is 2 and represents the Y component and the UV component, and N is an integer greater than 1. Based on the index i of the quality factor of the video signal, a response signal g including the first signal component and the second signal component is obtained. ci is obtained.

[0138] In yet another embodiment, when the bitstream includes a quality factor of a first signal component and a quality factor of a second signal component, the decoder side uses the quality factor of the Y component as an input to a fully connected network and outputs a response signal of the Y component, and uses the quality factor of the UV component as an input to the fully connected network and outputs a response signal of the UV component.

[0139] Step 714: Obtain a reconstructed map of the first signal component based on the response signal of the first signal component and the feature map of the first signal component.Step 715: Obtain a reconstructed map of the second signal component based on the response signal of the second signal component and the feature map of the second signal component.

[0140] When the response signal includes a response vector, obtaining a reconstruction map of a first signal component based on the response signal of the first signal component and the feature map of the first signal component includes: The method includes multiplying a response vector of the first signal component by a feature map of the first signal component to obtain a reconstructed map of the first signal component, or multiplying a response vector of the first signal component by a feature map of the first signal component and then obtaining a reconstructed map of the first signal component through processing by a neural network.

[0141] Obtaining a reconstruction map of the second signal component based on the response signal of the second signal component and the feature map of the second signal component, The method includes multiplying a response vector of the second signal component by a feature map of the second signal component to obtain a reconstructed map of the second signal component, or multiplying a response vector of the second signal component by a feature map of the second signal component and then obtaining a reconstructed map of the second signal component through processing by a neural network.

[0142] When the response signal includes a response vector and an offset vector, obtaining a reconstruction map of a first signal component based on the response signal of the first signal component and the feature map of the first signal component includes: The method includes multiplying a response vector of the first signal component by a feature map of the first signal component and then adding an offset vector of the first signal component to obtain a reconstructed map of the first signal component, or multiplying a response vector of the first signal component by a feature map of the first signal component and then adding an offset vector of the first signal component to obtain a reconstructed map of the first signal component through processing by a neural network.

[0143] Obtaining a reconstruction map of the second signal component based on the response signal of the second signal component and the feature map of the second signal component, The method includes multiplying the response vector of the second signal component by a feature map of the second signal component and then adding an offset vector of the second signal component to obtain a reconstructed map of the second signal component, or multiplying the response vector of the second signal component by a feature map of the second signal component and then adding an offset vector of the second signal component to obtain a reconstructed map of the second signal component through processing by a neural network.

[0144] Step 716: Reconstruct the video signal based on the reconstructed map of the first signal component and the reconstructed map of the second signal component.

[0145] FIG. 7D illustrates an AI picture coding system 710 for YUV bit rate allocation according to an embodiment of the present invention. The AI ​​picture coding system 710 is applicable to video picture encoders and decoders. As shown in FIG. 7D, the picture coding system 710 includes a first signal component processing module (e.g., Y component processing module), a second signal component processing module (e.g., U component processing module), a third signal component processing module (e.g., V component processing module), a bit rate allocation control module, and an entropy encoding module on the encoder side, and a first signal component processing module (Y component processing module 2), a second signal component processing module (e.g., U component processing module 2), a third signal component processing module (e.g., V component processing module 2), and an entropy decoding module on the decoder side. The picture coding system 710 optionally includes a joint processing module, a joint processing module 2, and a quality response module (also referred to as bit rate allocation control module 2 or bit rate control module 2). In the picture coding system 710, the quality factor of the Y component, the quality factor of the U component, and the quality factor of the V component are input to a bitrate allocation control module, which outputs a control signal respectively applied to the feature map of the Y component output by the Y component processing module, the feature map of the U component output by the U component processing module, and the feature map of the V component output by the V component processing module (which may be separately referred to as first feature maps), and outputs a second feature map of each signal component to allocate the bitrates of the Y component, the U component, and the V component. Then, a bitstream of the video signal is obtained based on the second feature maps of these signal components.For example, the feature maps output by the Y component processing module, the U component processing module, and the V component processing module are directly concatenated together, or the feature maps output by the Y component processing module, the U component processing module, and the V component processing module are directly added together to form the feature map finally output by the encoder, and entropy coding is performed on the finally output feature map. Optionally, the feature maps output by the Y component processing module, the U component processing module, and the V component processing module are input into a joint processing module to obtain the feature map finally output by the encoder, and entropy coding is performed on the finally output feature map.

[0146] In the embodiment shown in Figure 7D, the control signal of the first signal component can be obtained from N candidate first control signals based on the quality factor of the first signal component (Y component). The control signal of the second signal component can be obtained from M candidate second control signals based on the quality factor of the second signal component (U component). The control signal of the third signal component can be obtained from L candidate third control signals based on the quality factor of the third signal component (V component). N, M, and L are integers greater than 1 and may be equal or unequal. This is not a limitation in this application.

[0147] In the architecture shown in Figure 7D, the encoding and decoding methods are similar to those in Figures 7B and 7C. The details are as follows.

[0148] For example, in one embodiment, the control signal is generated through learning by the network and applied to the feature map (which may be referred to as the first feature map) output by at least one layer of the network in each of the Y component processing module, the U component processing module, and the V component processing module to output the second feature map. For example, the control signal is applied to the feature map output by the last layer of the network. Since the control signal may be applied to the first feature map output by any layer of the network in each of the Y component processing module, the U component processing module, and the V component processing module, processing by the neural network may continue to be performed on the second feature map after the second feature map is output. Correspondingly, obtaining a bitstream of a video signal based on the second feature map of the Y component, the second feature map of the U component, and the second feature map of the V component includes: performing entropy coding on the second feature map of the first signal component, the second feature map of the second signal component, and the second feature map of the third signal component to obtain a bitstream of the video signal; performing entropy coding on the second feature map of the first signal component, the second feature map of the second signal component processed by the neural network, and the second feature map of the third signal component to obtain a bitstream of the video signal; performing entropy coding on the second feature map of the first signal component, the second feature map of the second signal component processed by the neural network, and the second feature map of the third signal component processed by the neural network to obtain a bitstream of the video signal; performing entropy coding on the second feature map of the first signal component, the second feature map of the second signal component, and the second feature map of the third signal component processed by the neural network to obtain a bitstream of the video signal; performing entropy coding on the second feature map of the first signal component processed by the neural network, the second feature map of the second signal component, and the second feature map of the third signal component to obtain a bitstream of the video signal; performing entropy coding on the second feature map of the first signal component processed by the neural network, the second feature map of the second signal component processed by the neural network, and the second feature map of the third signal component to obtain a bitstream of the video signal; performing entropy coding on the second feature map of the first signal component processed by the neural network, the second feature map of the second signal component processed by the neural network, and the second feature map of the third signal component processed by the neural network to obtain a bitstream of the video signal; or and performing entropy coding on the second feature map of the first signal component processed by the neural network, the second feature map of the second signal component, and the second feature map of the third signal component processed by the neural network to obtain a bitstream of the video signal.

[0149] If the picture coding system 710 includes a joint processing module, further joint processing can be performed on the obtained second feature map or a combination with the processed feature map to obtain a joint feature map, and entropy coding is performed on the joint feature map to obtain a bitstream of the video signal.

[0150] Details are as follows:

[0151] When the control signal includes control vectors of Y, U, and V feature maps, the bitrate allocation control module learns N candidate first control signals of the first signal components (e.g., control vector matrix {q y1 ,q y2 ,…,q yi ,…,q yN}) and M candidate second control signals (e.g., a control vector matrix {q u1 ,q u2 ,…,q uj ,…,q uM}) and L candidate third control signals of the third signal components (for example, a control vector matrix {q v1 ,q v2 ,…,q vk ,…,q vL In use, a control signal q for the first signal component is generated based on the index i of the quality factor of the Y component. yi Based on the index j of the quality factor of the U component, a control signal q uj Based on the index k of the quality factor of the V component, a control signal q vk Then, each control vector is multiplied by the corresponding feature map for each channel to control the feature maps of different components. In this case, the bitstream of the video signal sent to the decoder side includes the index i of the quality factor of the Y component, the index j of the quality factor of the U component, and the index k of the quality factor of the V component.

[0152] If the control signal includes control vectors and offset vectors for the Y, U, and V feature maps, then, in use, the control vector q of the first signal component is calculated based on the index i of the quality factor of the Y component, as described in the method above. yi and the offset vector b yi is obtained, and based on the index j of the quality factor of the UV component, the control vector q of the second signal component is calculated. uj and the offset vector b ui is obtained, and based on the index k of the quality factor of the V component, the control vector q of the third signal component is calculated. vk and the offset vector b vi Then, to control the feature maps of different components, each control vector is multiplied by the corresponding feature map for each channel, and a corresponding offset vector is added.

[0153] In another embodiment, the control signal of the Y component, the control signal of the U component, and the control signal of the V component are used as a 3-tuple. The bit rate allocation control module learns N candidate control signals of the video signal (e.g., a control vector matrix {q c1 ,q c2 ,…,q ci ,…,q cN}. In this case, c is 3, and each control vector q ci includes a control vector of the first signal component, a control vector of the second signal component, and a control vector of the third signal component. Then, based on the index i of the quality factor of the video signal, a control signal q including the first signal component, the second signal component, and the third signal component is generated. ci is obtained. The offset vectors are implemented in a similar manner. Specifically, each offset vector of a video signal includes an offset vector of a first signal component, an offset vector of a second signal component, and an offset vector of a third signal component. In this case, the bitstream of the video signal sent to the decoder side includes an index i of the quality factor of the video signal.

[0154] In yet another embodiment, the quality factor of the Y component, the quality factor of the U component, and the quality factor of the V component are used as inputs to a fully connected network, and a control vector q yi , the control vector q uj , and the control vector q vk To control the feature maps of different components, each control vector is multiplied by the corresponding feature map for each channel. The quality factor of the Y component, the quality factor of the U component, and the quality factor of the V component are used as inputs to the fully connected network, and the offset vector b yi , offset vector b uj , and the offset vector b vkcan be further output. Then, each control vector is multiplied by a corresponding feature map for each channel, and each offset vector is added to a corresponding feature map for each channel to control the feature maps of different components. In this case, the bitstream of the video signal sent to the decoder side includes a quality factor of the Y component, a quality factor of the U component, and a quality factor of the V component.

[0155] The decoder side performs entropy decoding on the received bitstream to obtain feature maps, which are decomposed into a Y component feature map, a U component feature map, and a V component feature map. Optionally, the feature maps obtained through entropy decoding are first input into joint processing sub-module 2 to obtain a Y component feature map, a U component feature map, and a V component feature map.

[0156] The feature map of the Y component, the feature map of the U component, and the feature map of the V component are input to a Y component processing module 2, a U component processing module 2, and a V component processing module 2, respectively, to output a reconstructed map of the Y component, a reconstructed map of the U component, and a reconstructed map of the V component. Optionally, the quality factor of the Y component, the quality factor of the U component, and the quality factor of the V component are input to a quality response module, which outputs control signals to be applied to the feature map of the Y component output by the Y component processing module, the feature map of the U component output by the U component processing module, and the feature map of the V component output by the V component processing module, respectively, to implement adaptive quality responses of the Y, U, and V components.

[0157] The response signals are generated in a similar manner to that of the control signals, and for ease of distinction, the signals on the encoder side are referred to as control signals and the signals on the decoder side are referred to as response signals.

[0158] Specifically, the decoder side obtains a bitstream of the video signal from the encoder side, performs entropy decoding on the bitstream to obtain a feature map of a first signal component (e.g., Y component) of the video signal, a feature map of a second signal component (e.g., U component) of the video signal, and a feature map of a third signal component (e.g., V component) of the video signal, obtains a reconstruction map of the first signal component based on a response signal of the first signal component obtained through learning and the feature map of the first signal component, obtains a reconstruction map of the second signal component based on a response signal of the second signal component obtained through learning and the feature map of the second signal component, obtains a reconstruction map of the third signal component based on a response signal of the third signal component obtained through learning and the feature map of the third signal component, and reconstructs the video signal based on the reconstruction map of the first signal component, the reconstruction map of the second signal component, and the reconstruction map of the third signal component.

[0159] The decoder side further obtains quality factor information of the first signal component, quality factor information of the second signal component, and quality factor information of the third signal component from the bitstream. The quality factor information of the first signal component and the quality factor information of the second signal component are the same as those in the embodiment shown in FIG. 7A. Similarly, the quality factor information of the third signal component can be a quality factor of the third signal component or an index of the quality factor of the third signal component. Then, a response signal of the third signal component is obtained based on the quality factor information of the third signal component. If the quality factor information of the third signal component is a quality factor of the third signal component, the value of the quality factor of the third signal component is one of L. Alternatively, if the quality factor information of the third signal component is an index of the quality factor of the third signal component, the value range of the index of the quality factor of the third signal component is 0 to L-1 or 1 to L, where L is an integer greater than 1. L, M, and N may be equal or unequal. This is not a limitation in this application.

[0160] If the joint feature map is transmitted from the encoder side, the decoder side further needs to perform entropy decoding on the joint feature map, and obtain a feature map of the first signal component, a feature map of the second signal component, and a feature map of the third signal component through processing by a neural network.

[0161] In one embodiment, when the bitstream includes an index i of a quality factor of a Y component, an index j of a quality factor of a U component, and an index k of a quality factor of a V component, the decoder side learns a response signal matrix {g y1 ,g y2 ,…,g yi ,…,g yN}, the response signal matrix of the second signal component {g u1 ,g u2 ,…,g uj ,…,g uM}, and the response signal matrix {g v1 ,g v2 ,…,g vk ,…,g vL}, where N, M, and L are integers greater than 1. Optionally, a control signal matrix {q y1 ,q y2 ,…,q yi ,…,q yN}, the response signal matrix {g y1 ,g y2 ,…,g yi ,…,g yN} is obtained, and the control signal matrix {q u1 ,q u2 ,…,q uj ,…,q uM}, the response signal matrix {g u1 ,g u2 ,…,g uj ,…,g uM} is obtained, and the control signal matrix {q v1 ,q v2 ,…,q vk ,…,qvL}, the response signal matrix for the third signal component {g v1 ,g v2 ,…,g vk ,…,g vL} is obtained, where N, M, and L are integers greater than 1. Based on the index i of the quality factor of the Y component, the response signal g yi is obtained. Based on the index j of the quality factor of the U component, the response signal g uj Based on the index k of the quality factor of the V component, the response signal g of the third signal component is obtained. vk is obtained.

[0162] In another embodiment, when the bitstream includes an index i of a quality factor of a video signal, the decoder side learns through learning a response signal matrix {g c1 ,g c2 ,…,g ci ,…,g cN}, where c is 3 and represents the Y, U, and V components, and N is an integer greater than 1. Optionally, a control vector matrix {q c1 ,q c2 ,…,q ci ,…,q cN}, the response signal matrix of the video signal, {g c1 ,g c2 ,…,g ci ,…,g cN} is obtained, where c is 3 and represents the Y component, the U component, and the V component, and N is an integer greater than 1. Based on the index i of the quality factor of the video signal, a response signal g including a first signal component, a second signal component, and a third signal component is obtained. ci is obtained.

[0163] In yet another embodiment, when the bitstream includes a quality factor of a first signal component, a quality factor of a second signal component, and a quality factor of a third signal component, the decoder side uses the quality factor of the Y component as an input to a fully connected network and outputs a response signal of the Y component, uses the quality factor of the U component as an input to the fully connected network and outputs a response signal of the U component, and uses the quality factor of the V component as an input to the fully connected network and outputs a response signal of the V component.

[0164] The reconstruction maps of the first and second signal components are obtained in a similar manner as shown in Fig. 7A, and the details will not be described again here.

[0165] In one embodiment, when the response signal includes a response vector, obtaining a reconstruction map of the third signal component based on the response signal of the third signal component and the feature map of the third signal component includes: The method includes multiplying a response vector of the third signal component by a feature map of the third signal component to obtain a reconstructed map of the third signal component, or multiplying a response vector of the third signal component by a feature map of the third signal component and then obtaining a reconstructed map of the third signal component through processing by a neural network.

[0166] In another embodiment, when the response signal includes a response vector and an offset vector, obtaining a reconstruction map of the third signal component based on the response signal of the third signal component and the feature map of the third signal component includes: The method includes multiplying a response vector of the third signal component by a feature map of the third signal component and then adding an offset vector of the third signal component to obtain a reconstructed map of the third signal component, or multiplying a response vector of the third signal component by a feature map of the third signal component and then adding an offset vector of the third signal component to obtain a reconstructed map of the third signal component through processing by a neural network.

[0167] In the embodiment of Figure 7A, the UV components are processed as a combination, whereas in Figure 7D, the Y, U, and V components can be processed separately or, for example, as a combination of the Y component with the UV component, or another combination of the Y, U, and V components.

[0168] 8A and 8B together provide a specific embodiment. Fig. 8A is an overall block diagram of a technical scheme according to this embodiment. The quality factor of the Y component and the quality factor of the UV component are input to a bitrate allocation control module, which calculates a control vector q yi and the control vector q uvi and allocate the bitrates for the Y and UV components.

[0169] The decoder side inputs the quality factor of the Y component and the quality factor of the UV component to the quality response module, which outputs the control vector g yi and the control vector g uvi to implement the quality gain response for each of the Y and UV components.

[0170] In this embodiment, in order to facilitate understanding of the specific example shown in FIG. 8B, no limitations are imposed on the specific network structures of the Y component processing module, the UV component processing module, the joint processing module, the probability estimation module, the Y component processing module 2, the UV component processing module 2, and the joint processing module 2.

[0171] Step 1: Obtain feature maps of Y and UV components.

[0172] The Y component and the UV component are input into the Y component processing module and the UV component processing module, respectively, and the network outputs the feature maps of the Y component and the UV component. In the example shown in FIG. 8B, the Y component processing module includes two convolution layers and two nonlinear layers. The horizontal and vertical downsampling coefficients in the two convolution layers are 2, and the Y component processing module outputs the feature map of the Y component. The UV component processing module includes two convolution layers and two nonlinear layers. The horizontal and vertical downsampling coefficients in the first convolution layer are 1, that is, no downsampling process is performed. The horizontal and vertical downsampling coefficients in the second convolution layer in the UV component processing module are 2. The UV component processing module outputs the feature map of the UV component. After the above processing by the network, in the YUV420 data format, the width and height of the feature map of the Y component are the same as those of the feature map of the UV component.

[0173] Similar to the processing method of the YUV420 data format, the number of convolution layers and the horizontal and vertical downsampling factors are controlled in the YUV444 data format and the YUV422 data format so that the width and height of the feature map of the Y component are the same as those of the feature map of the UV components.

[0174] Step 2: Input the quality factor of the Y component and the quality factor of the UV component into the bit rate allocation module to obtain the control vector q yi and the control vector q uvi and obtain the control vector q yi and the control vector q uvi multiplying the feature maps of the Y and UV components by each channel to obtain processed feature maps of the Y and UV components, and inputting the summed or concatenated processed feature maps into a joint processing module to output the feature map to be encoded.

[0175] The bitrate allocation module uses the control matrix Q y and Q uvThe quality factor of the Y component and the quality factor of the UV component are the control matrix Q y and Q uv , and use it as the index value of Q y and Q uv From the index of the control vector q yi and the control vector q uvi We obtain the control matrix Q y and Q uv is obtained through network learning. The quality coefficient of the Y component and the quality coefficient of the UV components are arbitrary settings.

[0176] In the example shown in FIG. 8B, the control matrix Q y is a two-dimensional matrix with size K × N, and the control matrix Q uv are two-dimensional matrices with a size of L×M, and each element in these two matrices is a parameter that can be learned by the network. K represents the number of feature maps of the Y component, L represents the number of feature maps of the UV component, N represents N groups of candidate values ​​of the quality factor of the Y component, and M represents M groups of candidate values ​​of the quality factor of the UV component. For example, when N is 4 and M is 4, the candidate values ​​of the quality factor of the Y component are {0.5, 0.7, 0.8, 1.0}, and the candidate values ​​of the quality factor of the UV component are {0.15, 0.2, 0.25, 0.3}.

[0177] Step 3: Input the feature map to be coded into an entropy coding module, and output a bit stream. In the example shown in FIG. 8B, the feature map to be coded is input into a coding feature map hyper-entropy module, and a probability distribution of the symbol to be coded is output. Arithmetic coding is performed based on the probability distribution of the symbol to be coded, and a bit stream is output. In addition, information about the quality factor of the Y component and the quality factor of the UV component are written into the bit stream.

[0178] Information regarding the quality factor of the Y component and the quality factor of the UV components can be expressed in the following three ways and written into the bitstream.

[0179] Method 1: The number and candidate values ​​of the quality factor of the Y component and the number and candidate values ​​of the UV component are predetermined, and the index numbers in the candidate lists of the quality factor of the Y component and the quality factor of the UV component are transmitted to the decoder side. For example, when N is 4 and M is 3, the candidate values ​​of the quality factor of the Y component are {0.5, 0.7, 0.8, 1.0}, and the candidate values ​​of the quality factor of the UV component are {0.15, 0.2, 0.25}. The index number i of the Y component and the index number j of the UV component are written into the bitstream, and the values ​​of i and j are 0, 1, 2, and 3. When i is 1, it indicates that the quality factor of the Y component is 0.7, and when j is 0, it indicates that the quality factor of the UV component is 0.15.

[0180] Method 2: The number of candidate values ​​and the candidate values ​​after the quality factor of the Y component and the quality factor of the UV component are combined are predetermined. For example, the number of candidate values ​​of the combination value of the quality factor of the Y component and the UV component is 6, and the candidate list is {(0.5,0.25),(0.7,0.15),(0.7,0.25),(0.8,0.1),(0.8,0.2), and (1.0,0.2)}. An index number i is written into the bitstream, and the values ​​of i are 0, 1, 2, 3, 4, and 5. When i is 1, it indicates that the quality factor of the Y component and the UV component is (0.7,0.15).

[0181] Method 3: The quality coefficient of the Y component and the quality coefficient of the UV component are directly written into the bitstream and transmitted to the decoder side, for example, (1.0, 0.2) is written into the bitstream.

[0182] Step 4: Input the bitstream into the entropy decoding module, and perform arithmetic decoding to obtain the feature map, the quality factor of the Y component, and the quality factor of the UV component. In the example shown in Figure 8B, the arithmetic decoding is performed based on the probability distribution estimated by the hyper-entropy module.

[0183] Step 5: Obtain a feature map through decoding, input the feature map into the joint processing module 2, output a feature map whose number of channels is M, and divide the feature map whose number of channels is M into a feature map of a Y component whose number of channels is K and a feature map of a UV component whose number of channels is L. In this division scheme, K≦M and L≦M. When K=L=M, it indicates that the feature map of the Y component is the same as the feature map of the UV component, and both are feature maps with M channels. In the example shown in FIG. 8B, the joint processing module 2 includes two convolution layers and one nonlinear layer.

[0184] Step 6: Input the quality coefficients of the Y component and the UV component into the quality response module to obtain the response vector g yi and the response vector g uvi and obtain the response vector g yi and the response vector g uvi multiplying by the feature maps of Y and UV components for each channel to obtain the feature maps of Y and UV components after quality gain. The feature maps of Y and UV components after quality gain are input to the Y component processing module 2 and the UV component processing module 2, respectively, and the reconstructed map of Y component and the reconstructed map of UV component are output.

[0185] The quality response module calculates the response matrix G y and G uv The quality coefficients of the Y component and the UV component obtained through decoding are converted into the response matrix G y and G uv , and G y and G uv from the index of the response vector g yi and the response vector g uvi Get the.

[0186] Response matrix G y and G uv is obtained through network learning. In the example shown in FIG. 8B, the response matrix G yis a two-dimensional matrix with size K × N, and the response matrix G uv are two-dimensional matrices with a size of L × M, and each element in these two matrices is a parameter that can be learned by the network. K represents the number of feature maps of the Y component, L represents the number of feature maps of the UV component, N represents N groups of candidate values ​​of the quality factor of the Y component, and M represents M groups of candidate values ​​of the quality factor of the UV component.

[0187] Optionally, the response matrix G y and G uv are the control matrix Q y and Q uv It is obtained by taking the reciprocal of

[0188] In this embodiment, training and learning are performed on the network module and control matrix parameters in steps 1 to 6. Specifically, in this application, an Adaptive Moment Estimation (Adam) optimization algorithm is used to perform optimization training on the neural network, and the ImageNet dataset is the training dataset. Since this network structure is oriented towards picture coding, the objective of the training optimization is to minimize the bitrate-distortion joint loss function, whose functional formula is as follows:

number

[0189] p(y) represents the probability distribution estimated by the probability estimator, and x y is the original value of the Y component, and x y ' is the reconstruction map of the Y component, and x u is the original value of the U component, and x u ' is the reconstruction map of the U component, and x v is the original value of the V component, and x v ' is the reconstructed map of the V component, and w y is the quality factor of the Y component, and w uis the quality factor of the U component, and w v is the quality factor of the V component. λ is a constant that corresponds to the target bit rate.

[0190] For example, N is 4, and the quality factors of the Y, U, and V components (w y ,w u ,w v The candidate values ​​for w are {(0.5,0.25,0.25),(0.7,0.4,0.4),(0.8,0.1,0.1),(1.0,0.2,0.2)}. During training, the weight group index number i is randomly selected from {0,1,2,3}. Based on i, the weight value group (w yi ,w ui ,w vi ), the control vector q yi , and the control vector q to be learned uvi is determined, and training and learning is performed on the network modules and control matrix parameters based on the training objective.

[0191] If the joint processing module and / or the joint processing module 2 and / or the quality response module are removed from the codec provided in this embodiment, it is still applicable to the other embodiments of this application.

[0192] This embodiment provides a technical solution in which the Y, U, and V components are combined into a Y component and a UV component. For other combinations of Y, U, and V components, such as {YU,V} and {YV,U}, the solution of this application is still applicable.

[0193] Similarly, the UV component can be further split into U and V components for processing, and the solutions of this application can still be applied.

[0194] In conventional end-to-end picture coding, in the learning and optimization of a particular network, optimization is performed based on fixed weight values ​​of Y, U, and V components. Therefore, the ratio of bit rates of Y, U, and V components is fixed. Since different pictures have different color characteristics, fixed bit rate allocation will cause poor coding performance of some video pictures. To implement different bit rate allocation of Y, U, and V components, multiple models may be simply trained based on different weight values ​​of multiple groups of Y, U, and V components. However, this increases the number of models, and training multiple models consumes a large amount of computing resources and time. Compared with the prior art, in this application, a control vector is obtained through learning by a network based on the derived weight values ​​of Y, U, and V components, and distortion control is performed to different degrees on the feature maps of the Y component and UV component based on the control vector to implement bit rate allocation of the Y component and UV component. Therefore, this application has the following advantages.

[0195] (1) Bitrate allocation among Y, U, and V components is supported to accommodate pictures with different color characteristics.

[0196] (2) The time required to train multiple models and the number of new network parameters for the models are reduced.

[0197] Figure 9A and Figure 9B together provide a specific embodiment. In this embodiment, as shown in Figure 9A and Figure 9B, a U component processing module and a V component processing module are used to process U component data and V component data respectively. In this embodiment, the quality factor of Y component, the quality factor of U component and the quality factor of V component are used as input to a bitrate allocation control module, and a control signal is output to process the feature map in any layer in the Y component processing module, the U component processing module and the V component processing module, thereby implementing the bitrate allocation among Y, U and V components. The decoder side uses the quality factor of Y component, the quality factor of U component and the quality factor of V component as input to a quality response module, and a control signal is output to implement the quality gain response to the feature map in any layer in the Y component processing module, the U component processing module and the V component processing module. In this application, in order to facilitate understanding of the specific example shown in FIG. 9A , no limitations are imposed on the specific network structures of the bitrate allocation control module, the quality response module, the Y component processing module, the U component processing module, the joint processing module, the Y component processing module 2, the U component processing module 2, the V component processing module 2, the joint processing module 2, the entropy encoding module, and the entropy decoding module.

[0198] In the example shown in Figure 9B, step 1: input the quality factor of Y component, the quality factor of U component, and the quality factor of V component into a bitrate allocation control module, which includes a fully connected network and outputs a control signal, which is a control vector and an offset vector.

[0199] Step 2: Input the Y, U, and V signals to be coded into the Y component processing module, the U component processing module, and the V component processing module, respectively. For example, in the Y component processing module, the feature map output from each convolutional layer is multiplied by the control vector corresponding to the feature map for each channel, and then the offset vector corresponding to the feature map is added for each channel. The feature map output from the nonlinear layer in the module is multiplied by the control vector corresponding to each channel. The feature map output from each layer of the network in the Y component processing module is processed based on the control signal output by the bit rate control module. The processing of the U and V components is similar to that of the Y component.

[0200] FIG. 9B is a schematic diagram of a specific network structure. The network structures of the Y component processing module, the U component processing module, and the V component processing module are the same except for the first convolution layer. In the YUV422 format, in the first convolution layer in the Y component processing module, the downsampling factor in the horizontal direction and the vertical direction is 2. In the first convolution layer in the U component processing module and the V component processing module, the downsampling factor in the horizontal direction is 1, that is, no downsampling process is performed. In the vertical direction, the downsampling factor is 2. In the YUV420 format, in the first convolution layer in the Y component processing module, the downsampling factor in the horizontal direction and the vertical direction is 2. In the first convolution layer in the U component processing module and the V component processing module, the downsampling factor in the horizontal direction and the vertical direction is 1, that is, no downsampling process is performed.

[0201] Step 3: The feature map of Y component, the feature map of U component, and the feature map of V component are concatenated and stitched to form a feature map to be coded, and the feature map to be coded is input to an entropy coding module, and a bit stream is output. In the example shown in Fig. 9B, the feature map to be coded is input to a coding feature map hyper-entropy module, and a probability distribution of the symbol to be coded is output. Arithmetic coding is performed according to the probability distribution of the symbol to be coded, and a bit stream is output. In addition, information about the quality factor of the Y component, the quality factor of the U component, and the quality factor of the V component is written into the bit stream.

[0202] Step 4: Input the bitstream into an entropy decoding module, and perform arithmetic decoding to obtain feature maps and information about the quality factor of the Y component, the quality factor of the U component, and the quality factor of the V component. In the example shown in Figure 9B, arithmetic decoding is performed based on the probability distribution estimated by the hyper-entropy module.

[0203] Step 5: The feature map obtained through decoding is input to the joint processing module 2, which outputs the feature map.

[0204] Step 6: Input the Y component quality factor, U component quality factor, and V component quality factor into the quality response module to obtain the response vector g yi , response vector g ui , and the response vector g vi Obtain the response vector g yi is multiplied by the feature map output by the second convolutional layer in the Y component processing module 2 for each channel to obtain the feature map after quality gain. The processing process for U and V components is similar. The Y component processing module 2, the U component processing module 2, and the V component processing module 2 output the reconstructed maps of Y, U, and V components.

[0205] The quality factor of the Y component, the quality factor of the U component, and the quality factor of the V component are input to a quality response module, which includes a fully connected network and generates a response vector g yi , response vector g ui , and the response vector g vi Output.

[0206] Optionally, similar to the response vector acquisition method in embodiment 1, the quality response module may obtain a response matrix G y , G u , and G v The quality coefficients of the Y component, the U component, and the V component obtained through decoding are expressed as the response matrix G y , G u , and G v , and the response vector g yi , response vector g ui , and the response vector g vi Obtain the response matrix G y , G u , and G v is obtained through training by the network.

[0207] The network training process is similar to that in the embodiment of Figures 8A and 8B, and the details will not be described again.

[0208] In this embodiment, the control signal is applied to the feature maps output at each layer of the network in the Y component processing module, the U component processing module, and the V component processing module. Optionally, the control signal is applied to only some of the feature maps output by the network in the Y component processing module, the U component processing module, and the V component processing module.

[0209] In this embodiment, the response signal is applied only to feature maps output at intermediate layers of the networks in the Y component processing module, the U component processing module, and the V component processing module. Optionally, a control signal is applied to feature maps output at any one or more layers of the networks in the Y component processing module, the U component processing module, and the V component processing module.

[0210] If the joint processing module 2 and / or the quality response module are removed from the codec provided in this embodiment, the techniques of this application are still applicable.

[0211] If a joint processing module is added to the codec provided in this embodiment, the techniques of this application are still applicable.

[0212] This embodiment provides a technical solution in which the Y, U and V components are processed separately as three components. For other combinations of Y, U and V components, such as {YU,V}, {YV,U} and {Y,UV}, the technical solution of this application is still applicable.

[0213] According to the description in the above embodiment, in this application, in the present invention, the quality coefficients of Y, U and V components are input to a bitrate allocation control module, and the control signal output by the module is applied to the feature maps of different components separately to implement the bitrate allocation of different components. Different components may refer to three components: Y, U and V components, or two components: Y and UV components, or other combinations of Y, U and V components.

[0214] Optionally, the control signal is a control vector q i is generated based on the quality factor of each different component. The weight matrix of the same component {q c1 ,q c2 ,…,q cN} is obtained through learning by the network, c is 2 or 3, and represents the number of different components, and N is the number of candidate values ​​of the quality factor. In use, the control vector q corresponding to each different component is calculated based on the index of the quality factor of each different component. ci is obtained.

[0215] Optionally, the control signal indicates a control vector q and an offset vector b. The quality coefficients of each different component are used as inputs to a fully connected network, which outputs the control vector q and offset vector b corresponding to the different component.

[0216] Thus, embodiments provided in this application include: (1) It can adapt to pictures with different color characteristics and support bitrate allocation among Y, U, and V components based on the control vector; (2) It can reduce the time required to train multiple models and the number of new network parameters for the models.

[0217] FIG. 10 is a schematic diagram of the structure of an encoding device 1000 according to an embodiment of this application. The encoding device may correspond to the video encoder 20. The encoding device 1000 includes a first control module 1001, a second control module 1002, and an encoding module 1003. The first control module 1001 is configured to apply a control signal of a first signal component of the video signal to a first feature map of the first signal component to obtain a second feature map of the first signal component, where the control signal of the first signal component is obtained through learning. The second control module 1002 is configured to apply a control signal of a second signal component of the video signal to a first feature map of the second signal component to obtain a second feature map of the second signal component, where the control signal of the second signal component is obtained through learning. The encoding module 1003 is configured to obtain a bitstream of the video signal based on the second feature map of the first signal component and the second feature map of the second signal component. The encoding device 1000 may further include a bitrate allocation control module described in the above embodiment. The encoding device 1000 is configured to implement the encoding method described in the above embodiment. For detailed functions, please refer to the description in the above embodiment, and the details will not be described again here.

[0218] FIG. 11 is a schematic diagram of the structure of a decoding device 1100 according to an embodiment of this application. The decoding device 1100 may correspond to the video decoder 30. The decoding device 1100 includes a decoding module 1101, a first control module 1102, a second control module 1103, and a reconstruction module 1104. The decoding module 1101 is configured to obtain a bitstream of a video signal, and perform entropy decoding on the bitstream to obtain a feature map of a first signal component of the video signal and a feature map of a second signal component of the video signal. The first control module 1102 is configured to obtain a reconstruction map of the first signal component based on a response signal of the first signal component and the feature map of the first signal component, where the response signal of the first signal component is obtained through learning. The second control module 1103 is configured to obtain a reconstruction map of the second signal component based on a response signal of the second signal component and the feature map of the second signal component, where the response signal of the second signal component is obtained through learning. The reconstruction module 1104 is configured to reconstruct the video signal according to the reconstruction map of the first signal component and the reconstruction map of the second signal component. The decoding device 1100 may further include a quality response module as described in the above embodiment. The decoding device 1100 is configured to implement the decoding method as described in the above embodiment. For detailed functions, please refer to the description in the above embodiment, and the details will not be described again here.

[0219] As one skilled in the art can appreciate, the functions described with reference to the various exemplary logical blocks, modules, and algorithm steps disclosed and described herein can be implemented by hardware, software, firmware, or any combination thereof. When implemented by software, the functions described with reference to the exemplary logical blocks, modules, and steps can be stored on or transmitted on a computer-readable medium as one or more instructions or code and executed by a hardware-based processing unit. A computer-readable medium can include a computer-readable storage medium that corresponds to a tangible medium, such as a data storage medium, or any communication medium that facilitates the transmission of a computer program from one place to another (e.g., according to a communication protocol). Thus, a computer-readable medium can generally correspond to (1) a non-transitory tangible computer-readable storage medium, or (2) a communication medium, such as a signal or carrier wave. A data storage medium can be any available medium that can be accessed by one or more computers or one or more processors to retrieve instructions, code, and / or data structures for implementing the techniques described in this application. A computer program product can include a computer-readable medium.

[0220] By way of example, and not limitation, such computer-readable storage media may include RAM, ROM, EEPROM, CD-ROM or other optical disk storage devices, magnetic disk storage devices or other magnetic storage devices, flash memory, or any other medium that can store the necessary program code in the form of instructions or data structures and that can be accessed by a computer. Also, any connection is properly referred to as a computer-readable medium. For example, if the instructions are transmitted from a website, server, or other remote source over coaxial cable, fiber optic, twisted pair, digital subscriber line (DSL), or wireless technology such as infrared, radio, or microwave, the coaxial cable, fiber optic, twisted pair, DSL, or wireless technology such as infrared, radio, or microwave, is included in the definition of the medium. However, it should be understood that computer-readable storage media and data storage media do not include connections, carrier waves, signals, or other transitory media, and in fact refer to non-transitory, tangible storage media. As used in this specification, disk and disc include compact discs (CDs), laser discs, optical discs, digital versatile discs (DVDs), and Blu-ray discs. Disks usually reproduce data magnetically, while discs reproduce data optically by using a laser. Combinations of the above items should also be included within the scope of computer-readable media.

[0221] The instructions may be executed by one or more processors, such as, for example, one or more digital signal processors (DSPs), general purpose microprocessors, application specific integrated circuits (ASICs), field programmable gate arrays (FPGAs), or other equivalent integrated circuits or discrete logic circuits. Thus, the term "processor" as used herein may refer to the above-mentioned structures, or any other structures that may be adapted to implement the techniques described herein. Furthermore, in some aspects, the functionality described with reference to the exemplary logic blocks, modules, and steps described herein may be provided within dedicated hardware and / or software modules configured for encoding and decoding, or may be incorporated into a combined codec. These techniques may also be fully implemented in one or more circuits or logic elements.

[0222] The techniques in this application may be implemented in a variety of apparatuses or devices, including a wireless handset, an integrated circuit (IC), or a set of ICs (e.g., a chipset). Various components, modules, or units are described in this application to highlight functional aspects of an apparatus configured to perform the disclosed techniques, but do not necessarily require realization by multiple different hardware units. In practice, as described above, the various units may be combined into a codec hardware unit in combination with appropriate software and / or firmware, or may be provided by multiple interoperable hardware units (including one or more processors as described above).

[0223] The above description is merely an example of a specific implementation of this application, and is not intended to limit the scope of protection of this application. Any modifications or replacements that a person skilled in the art can easily think of within the technical scope disclosed in this application shall fall within the scope of protection of this application. Therefore, the scope of protection of this application shall be subject to the scope of protection of the claims.

Claims

1. 1. A method for encoding a video signal, comprising the steps of: encoding quality factor information of a first signal component of a video signal and quality factor information of a second signal component of said video signal into a bitstream; obtaining a control signal for the first signal component based on the quality factor information of the first signal component; obtaining a control signal for the second signal component based on the quality factor information of the second signal component; applying the control signal for the first signal component to a first feature map for the first signal component to obtain a second feature map for the first signal component; applying the control signal for the second signal component to a first feature map for the second signal component to obtain a second feature map for the second signal component; encoding the second feature map of the first signal component and the second feature map of the second signal component into the bitstream; The method according to claim 1,

2. The method further comprises: obtaining the control signal for the first signal component from N candidate first control signals based on the quality factor information of the first signal component, where N is an integer greater than 1; obtaining the control signal for the second signal component from M candidate second control signals based on the quality factor information of the second signal component, where M is an integer greater than 1; 2. The method of claim 1, comprising:

3. The step of encoding the second feature map of the first signal component and the second feature map of the second signal component into the bitstream comprises: performing entropy coding on the second feature map of the first signal component and the second feature map of the second signal component to obtain the bitstream of the video signal; performing entropy coding on the second feature map of the first signal component and the second feature map of the second signal component processed by a neural network to obtain the bitstream of the video signal; performing entropy coding on the second feature map of the first signal component processed by a neural network and the second feature map of the second signal component to obtain the bitstream of the video signal; or performing entropy coding on the second feature map of the first signal component processed by a neural network and the second feature map of the second signal component processed by a neural network to obtain the bitstream of the video signal; The method according to claim 1 or 2, comprising:

4. The step of encoding the second feature map of the first signal component and the second feature map of the second signal component comprises: performing joint processing on the second feature map of the first signal component and the second feature map of the second signal component to obtain a joint feature map, and performing entropy coding on the joint feature map to obtain the bitstream of the video signal; performing joint processing on the second feature map of the first signal component and the second feature map of the second signal component processed by a neural network to obtain a joint feature map, and performing entropy coding on the joint feature map to obtain the bitstream of the video signal; performing joint processing on the second feature map of the first signal component processed by a neural network and the second feature map of the second signal component to obtain a joint feature map, and performing entropy coding on the joint feature map to obtain the bitstream of the video signal; or performing joint processing on the second feature map of the first signal component processed by a neural network and the second feature map of the second signal component processed by a neural network to obtain a joint feature map, and performing entropy coding on the joint feature map to obtain the bitstream of the video signal; The method according to claim 1 or 2, comprising:

5. 5. A method according to claim 1, wherein the first signal component is a Y component and the second signal component is a UV component.

6. When the second signal component is the UV component, the method comprises: Through learning, the Y component control signal matrix {q ｙ１ , q ｙ２ , …, q ｙｉ , …, q ｙＮ } and the control signal matrix {q ｕｖ１ , q ｕｖ２ , …, q ｕｖｊ , …, q ｕｖＭ }, where N and M are integers greater than 1; Based on the index i of the quality factor information of the Y component, the control signal q of the first signal component is ｙｉ and obtaining The control signal q of the second signal component is calculated based on the index j of the quality factor information of the UV component. ｕｖｊ and obtaining 6. The method of claim 5, comprising:

7. The method of claim 6 , wherein the bitstream of the video signal comprises the index i of the quality factor of the Y component and the index j of the quality factor of the UV component.

8. When the second signal component is the UV component, the method comprises: using the quality factor of the Y component as an input to a fully connected network and outputting a control signal for the Y component; using the quality factor of the UV component as an input to the fully connected network and outputting a control signal for the UV component; 6. The method of claim 5, comprising:

9. if the control signal comprises a control vector, applying the control signal for the first signal component to a first feature map for the first signal component to obtain a second feature map for the first signal component, multiplying a control vector of the first signal component by the first feature map of the first signal component to obtain the second feature map of the first signal component; having applying the control signal for the second signal component to a first feature map for the second signal component to obtain a second feature map for the second signal component, multiplying a control vector of the second signal component by the second feature map of the first signal component to obtain the second feature map of the second signal component; 9. The method according to claim 1 , comprising:

10. applying the control signal for the first signal component to a first feature map for the first signal component to obtain a second feature map for the first signal component, if the control signal has a control vector and an offset vector, multiplying a control vector of the first signal component by the first feature map of the first signal component and then adding an offset vector of the first signal component to obtain the second feature map of the first signal component; having applying the control signal for the second signal component to a first feature map for the second signal component to obtain a second feature map for the second signal component, multiplying a control vector of the second signal component by the second feature map of the first signal component and then adding an offset vector of the second signal component to obtain the second feature map of the second signal component; 9. The method according to claim 1 , comprising:

11. the first feature map of the first signal component is obtained by processing the first signal component in at least one convolutional layer and / or in at least one non-linear layer; the first feature map of the second signal components is obtained by processing the second signal components in at least one convolutional layer and / or in at least one nonlinear layer.

11. The method according to any one of claims 1 to 10.

12. the first feature map of the first signal component is obtained by processing the first signal component in two convolutional layers, each having a downsampling factor of 2, and in two nonlinear layers; the first feature map of the second signal component is obtained by processing the second signal component in one convolutional layer without downsampling, in one convolutional layer with a downsampling factor of 2, and in two nonlinear layers. The method of claim 11.

13. 1. A method for decoding a video signal, comprising the steps of: obtaining a bitstream of a video signal; performing entropy decoding on the bitstream to obtain a feature map of a first signal component of the video signal and a feature map of a second signal component of the video signal; obtaining quality factor information of the first signal component and quality factor information of the second signal component from the bitstream; obtaining a response signal of the first signal component based on the quality factor information of the first signal component; obtaining a response signal of the second signal component based on the quality factor information of the second signal component; obtaining a reconstruction map of the first signal component based on the response signal of the first signal component and the feature map of the first signal component; obtaining a reconstruction map of the second signal component based on the response signal of the second signal component and the feature map of the second signal component; reconstructing the video signal based on the reconstruction map of the first signal component and the reconstruction map of the second signal component; The method according to claim 1,

14. The method of claim 13 , wherein the quality factor information of the first signal component is a quality factor of the first signal component and the quality factor information of the second signal component is a quality factor of the second signal component.

15. if the quality factor information of the first signal component is the quality factor of the first signal component, the value of the quality factor of the first signal component is one of N, where N is an integer greater than 1; When the quality factor information of the second signal component is the quality factor of the second signal component, the value of the quality factor of the second signal component is one of M, where M is an integer greater than 1. The method of claim 14.

16. The bitstream includes a joint feature map, the method comprising: performing entropy decoding on the joint feature map to obtain the feature map of the first signal component and the feature map of the second signal component through processing by a neural network; 16. The method according to any one of claims 13 to 15, comprising:

17. 17. A method according to any one of claims 13 to 16, wherein the first signal component is a Y component and the second signal component is a UV component.

18. If the second signal component is the UV component, and the bitstream comprises an index i of a quality factor of the Y component and an index j of a quality factor of the UV component, the method further comprising: Through learning, the response signal matrix {g ｙ１ , g ｙ２ , …, g ｙｉ , …, g ｙＮ } and the response signal matrix {g ｕｖ１ , g ｕｖ２ , …, g ｕｖｊ , …, g ｕｖＭ }, where N and M are integers greater than 1; Based on the index i of the quality factor of the Y component, a response signal g of the first signal component is ｙｉ and obtaining Based on the index j of the quality factor of the UV component, a response signal g of the second signal component is ｕｖｊ and obtaining 20. The method of claim 17, comprising:

19. When the response signal has a response vector, the step of obtaining a reconstruction map of the first signal component based on the response signal of the first signal component and the feature map of the first signal component includes: multiplying the response vector of the first signal component by the feature map of the first signal component to obtain the reconstructed map of the first signal component, or multiplying the response vector of the first signal component by the feature map of the first signal component and then obtaining the reconstructed map of the first signal component through processing by a neural network; having The step of obtaining a reconstruction map of the second signal component based on the response signal of the second signal component and the feature map of the second signal component includes: multiplying the response vector of the second signal component by the feature map of the second signal component to obtain the reconstructed map of the second signal component, or multiplying the response vector of the second signal component by the feature map of the second signal component and then obtaining the reconstructed map of the second signal component through processing by a neural network; having 19. The method according to any one of claims 13 to 18.

20. When the response signal has a response vector and an offset vector, the step of obtaining a reconstruction map of the first signal component based on the response signal of the first signal component and the feature map of the first signal component includes: multiplying the response vector of the first signal component by the feature map of the first signal component and then adding an offset vector of the first signal component to obtain the reconstructed map of the first signal component, or multiplying the response vector of the first signal component by the feature map of the first signal component and adding the offset vector of the first signal component and then obtaining the reconstructed map of the first signal component through processing by a neural network; having The step of obtaining a reconstruction map of the second signal component based on the response signal of the second signal component and the feature map of the second signal component includes: multiplying the response vector of the second signal component by the feature map of the second signal component and then adding an offset vector of the second signal component to obtain the reconstructed map of the second signal component, or multiplying the response vector of the second signal component by the feature map of the second signal component and adding the offset vector of the second signal component and then obtaining the reconstructed map of the second signal component through processing by a neural network; having 19. The method according to any one of claims 13 to 18.

21. An encoder having processing circuitry configured to carry out a method according to any one of claims 1 to 12.

22. A decoder comprising processing circuitry configured to carry out a method according to any one of claims 13 to 20.

23. 1. An encoder comprising: one or more processors; a non-transitory computer readable storage medium coupled to the processor and storing a program for execution by the processor, the program configuring the encoder to perform the method of any one of claims 1 to 12 when executed by the processor; An encoder having

24. A decoder comprising: one or more processors; a non-transitory computer readable storage medium coupled to the processor and storing a program for execution by the processor, the program configuring the decoder to perform the method of any one of claims 13 to 20 when executed by the processor; and A decoder having

25. A non-transitory computer readable storage medium having a program code, the program code being used to perform the method according to any one of claims 1 to 12 or the method according to any one of claims 13 to 20 when the program code is executed by a computing device.

26. 1. A device for storing a bitstream, comprising: a receiver configured to receive one or more bitstreams; one or more storage media configured to store the one or more bitstreams; having Each of the one or more bitstreams comprises: quality factor information of a first signal component of a video signal and quality factor information of a second signal component of the video signal; encoded data of a second feature map of the first signal component and encoded data of a second feature map of the second signal component; having the second feature map of the first signal component is obtained by applying a control signal of the first signal component to the first feature map of the first signal component, the second feature map of the second signal component is obtained by applying a control signal of the second signal component to the first feature map of the second signal component, the control signal of the first signal component is obtained based on the quality factor information of the first signal component, and the control signal of the second signal component is obtained based on the quality factor information of the second signal component. device.

Images