Layered encoding and decoding methods and apparatus
By using a layered encoding and decoding method, control signals and feature maps are generated for different components of the video signal, which solves the problem of poor encoding performance caused by fixed bit rate allocation and achieves more efficient video compression and better image quality.
Patent Information
- Application Number
- CN202110071775.8
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2021-01-19
- Publication Date
- 2026-11-17
- Estimated Expiration
- 2041-01-19
AI Technical Summary
Existing video compression technologies suffer from poor encoding performance when processing images with different color characteristics due to fixed bitrate allocation, especially since the fixed bitrate of the YUV components cannot adapt to the color characteristics of different image content.
A layered encoding and decoding method is adopted to divide the video signal into multiple signal components. By learning, control signals and feature maps of each signal component are generated, and entropy encoding is performed to adapt to image content with different color characteristics.
It achieves dynamic adjustment of bitrate allocation based on the color characteristics of image content, improving encoding efficiency and image quality while reducing encoding distortion.
Smart Images

Figure CN114827622B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of video or image compression technology based on artificial intelligence (AI), and in particular to a method and apparatus for layered encoding and decoding. Background Technology
[0002] Video compression and encoding / decoding technologies are widely used in multimedia services, broadcasting, video communication, and storage. Examples include broadcast digital television, video transmission over the Internet and mobile networks, real-time conversational applications such as video chat and video conferencing, DVD and Blu-ray discs, video content capture and editing systems, and security applications for portable cameras.
[0003] Even in shorter videos, a large amount of video data needs to be described, which can be challenging when the data needs to be sent over bandwidth-constrained networks or transmitted otherwise. Therefore, video data is typically compressed before transmission over modern telecommunications networks. Video size can also be an issue when storing video on storage devices due to potentially limited memory resources. Video compression devices typically use software and / or hardware at the source side to encode video data before transmission or storage, thereby reducing the amount of data needed to represent the digital video image. The compressed data is then received at the destination by video decompression equipment. Given limited network resources and the growing demand for higher video quality, there is a need for improved compression and decompression techniques that can increase compression ratios with minimal impact on image quality.
[0004] In recent years, applying deep learning to end-to-end image encoding and decoding technology has gradually become a trend. In video encoders and decoders using hybrid architectures, when entropy encoding feature maps, it is assumed that the feature values follow a zero-mean Gaussian distribution. The super-prior structure estimates the variance of the Gaussian distribution to obtain a probability distribution model of the feature values. The arithmetic coding module then performs entropy encoding on the feature maps based on the estimated probability distribution. To ensure that the decoding end can also accurately estimate the probability distribution of the feature maps, the module in the super-prior structure extracts the latent variables of the estimated probability distribution. These latent variables are passed to the decoding end as side information via quantization and arithmetic coding. Under this mechanism, when the input image is in YUV format, the bitrate ratio of the Y, U, and V components is fixed. However, due to the different color characteristics of the image content, a fixed bitrate for the YUV components can lead to significant image distortion in the encoded image. Summary of the Invention
[0005] This application provides a layered encoding and decoding method and apparatus that can adapt to image content with different color characteristics.
[0006] In this application, layered encoding and decoding refers to dividing a video signal into a first signal component and a second signal component; or dividing a video signal into a first signal component, a second signal component, and a third signal component. The first signal component is a Y component, and the second signal component is a UV component, a U component, or a V component. When the second signal component is a U component, the third signal component is a V component, or when the second signal component is a V component, the third signal component is a U component.
[0007] In a first aspect, this application provides an encoding method. The encoding method includes: applying a control signal of a first signal component of the video signal to a first feature map of the first signal component to obtain a second feature map of the first signal component, wherein the control signal of the first signal component is obtained through learning; applying a control signal of a second signal component of the video signal to the first feature map of the second signal component to obtain a second feature map of the second signal component, wherein the control signal of the second signal component is obtained through learning; and obtaining a bitstream of the video signal based on the second feature map of the first signal component and the second feature map of the second signal component.
[0008] In one possible implementation, obtaining the bitstream of the video signal based on the second feature map of the first signal component and the second feature map of the second signal component includes: performing entropy coding on the second feature map of the first signal component and the second feature map of the second signal component, or on the second feature map of the first signal component and the second feature map of the second signal component processed by a neural network, or on the second feature map of the first signal component and the second feature map of the second signal component processed by a neural network, or on the second feature map of the first signal component and the second feature map of the second signal component processed by a neural network, to obtain the bitstream of the video signal.
[0009] In one possible implementation, obtaining the bitstream of the video signal based on the second feature map of the first signal component and the second feature map of the second signal component includes: performing joint processing on the second feature map of the first signal component and the second feature map of the second signal component to obtain a joint feature map, and performing entropy coding on the joint feature map to obtain the bitstream of the video signal; or, performing joint processing on the second feature map of the first signal component and the second feature map of the second signal component processed by a neural network to obtain a joint feature map, and performing entropy coding on the joint feature map to obtain the bitstream of the video signal; or, performing joint processing on the second feature map of the first signal component processed by a neural network and the second feature map of the second signal component processed by a neural network to obtain a joint feature map, and performing entropy coding on the joint feature map to obtain the bitstream of the video signal; or, performing joint processing on the second feature map of the first signal component processed by a neural network and the second feature map of the second signal component processed by a neural network to obtain a joint feature map, and performing entropy coding on the joint feature map to obtain the bitstream of the video signal.
[0010] In one possible implementation, a control signal for the first signal component is obtained from N candidate first control signals based on the quality factor of the first signal component, where N is an integer greater than 1; and a control signal for the second signal component is obtained from M candidate second control signals based on the quality factor of the second signal component, where M is an integer greater than 1. N and M may be equal or unequal, and this application does not limit this.
[0011] In one possible implementation, when the first signal component is a Y component and the second signal component is a UV component, the control signal matrix {q} of the Y component is generated by learning. y1 q y2 ,...q yi ...q yN} and the control signal matrix {q} of the UV components uv1 q uv2 ,...q uvj ,...q uvM}, where N and M are integers greater than 1; the control signal q of the first signal component is obtained according to the index i of the quality factor of the Y component. yi The control signal q of the second signal component is obtained based on the index j of the quality factor of the UV component. uvj At this time, the video signal bitstream includes the index i of the quality factor of the Y component and the index j of the quality factor of the UV component. Or...
[0012] In another possible implementation, when the first signal component is the Y component and the second signal component is the UV component, the control signal matrix {q} for generating the video signal is learned. c1 q c2 ,...q ci ...q cN}, where c is 2 representing the Y and UV components, and N is an integer greater than 1; a control signal q containing the first signal component and the second signal component is obtained according to the index i of the quality factor of the video signal. ci At this time, the bitstream of the video signal includes the index i of the quality factor of the video signal. Or
[0013] In another possible implementation, when the first signal component is a Y component and the second signal component is a UV component, a fully connected network is used. The quality factor of the Y component is used as the input to the fully connected network, and the control signal for the Y component is output. Similarly, the quality factor of the UV component is used as the input to the fully connected network, and the control signal for the UV component is output. In this case, the video signal bitstream includes the quality factors of both the Y and UV components.
[0014] In one possible implementation, when the second signal component is a U component or a V component, the method further includes: applying a control signal of the third signal component of the video signal to a first feature map of the third signal component to obtain a second feature map of the third signal component, wherein the control signal of the third signal component is obtained through learning, wherein when the second signal component is a U component, the third signal component is a V component, or when the second signal component is a V component, the third signal component is a U component.
[0015] In one possible implementation, when the first signal component is a Y component and the second signal component is a U component, the third signal component is a V component, and the control signal matrix {q} of the Y component is generated through learning. y1 q y2 ,...q yi ...q yN The control signal matrix {q} of the U component u1 q u2 ,...q uj ..., q uM}, the control signal matrix {q} of the V component v1 q v2 , ..., q vk ...q vL}, where N, M, and L are integers greater than 1; the control signal q of the first signal component is obtained according to the index i of the quality factor of the Y component. yi The control signal q of the second signal component is obtained based on the index j of the quality factor of the U component. uj ; and the control signal q of the third signal component is obtained according to the index k of the quality factor of the V component. vk At this time, the video signal bitstream includes the index i of the quality factor of the Y component, the index j of the quality factor of the U component, and the index k of the quality factor of the V component. Or
[0016] In another possible implementation, when the first signal component is a Y component and the second signal component is a U component, the third signal component is a V component, and the control signal matrix {q} of the video signal is generated through learning. c1 q c2 ,...q ci ...q cN}, where c is 3 representing the Y component, U component, and V component, and N is an integer greater than 1; a control signal q containing the first signal component, the second signal component, and the third signal component is obtained according to the index i of the quality factor of the video signal. ci At this time, the bitstream of the video signal includes the index i of the quality factor of the video signal. Or
[0017] In another possible implementation, when the first signal component is a Y component, the second signal component is a U component, and the third signal component is a V component, implemented through a fully connected network, the quality factor of the Y component is used as the input to the fully connected network to output a control signal for the Y component; the quality factor of the U component is used as the input to the fully connected network to output a control signal for the U component; and the quality factor of the V component is used as the input to the fully connected network to output a control signal for the V component. In this case, the video signal bitstream includes the quality factors of the Y component, the U component, and the V component.
[0018] Secondly, this application provides a decoding method. The decoding method includes: obtaining a bitstream of the video signal; performing entropy decoding on the bitstream to obtain a feature map of a first signal component and a feature map of a second signal component of the video signal; obtaining a reconstructed map of the first signal component based on a response signal of the first signal component and the feature map of the first signal component, wherein the response signal of the first signal component is obtained through learning; obtaining a reconstructed map of the second signal component based on the response signal of the second signal component and the feature map of the second signal component, wherein the response signal of the second signal component is obtained through learning; and reconstructing the video signal based on the reconstructed map of the first signal component and the reconstructed map of the second signal component.
[0019] It should be noted that, in this application, the response signal at the decoding end is similar to the control signal at the encoding end. For the sake of distinction, it is referred to as the response signal at the decoding end and the control signal at the encoding end. The response signal at the decoding end includes a response vector, or includes both a response vector and an offset vector.
[0020] In one possible implementation, the bitstream further includes quality factor information for the first signal component and quality factor information for the second signal component, wherein the quality factor information for the first signal component is either the quality factor of the first signal component or an index of the quality factor of the first signal component, and the quality factor information for the second signal component is either the quality factor of the second signal component or an index of the quality factor of the second signal component; the response signal of the first signal component is obtained through the quality factor information of the first signal component; and the response signal of the second signal component is obtained through the quality factor information of the second signal component.
[0021] Wherein, when the quality factor information of the first signal component is the quality factor of the first signal component, the quality factor of the first signal component takes one of N values; when the quality factor information of the first signal component is the index of the quality factor of the first signal component, the index of the quality factor of the first signal component takes a value range of 0 to N-1 or 1 to N, where N is an integer greater than 1.
[0022] When the quality factor information of the second signal component is the quality factor of the second signal component, the quality factor of the second signal component takes one of M values; when the quality factor information of the second signal component is the index of the quality factor of the second signal component, the index of the quality factor of the second signal component takes a value ranging from 0 to M-1 or from 1 to M, where M is an integer greater than 1.
[0023] In one possible implementation, when the first signal component is a Y component and the second signal component is a UV component, if the bitstream includes the index i of the quality factor of the Y component and the index j of the quality factor of the UV component, the response signal matrix {g} of the first signal component is generated through learning. y1 g y2 , ...g yi ...g yN}, and the response signal matrix {g} of the second signal component uv1 g uv2 , ...g uvj , ...g uvM}, where N and M are integers greater than 1; the response signal g of the first signal component is obtained according to the index i of the quality factor of the Y component. yi The response signal g of the second signal component is obtained based on the index j of the quality factor of the UV component. uvj .or
[0024] In another possible implementation, when the first signal component is the Y component and the second signal component is the UV component, if the bitstream includes the index i of the quality factor of the video signal, the response signal matrix {g} of the video signal is generated through learning. c1 g c2 , ...g ci ...g cN}, where c is 2 representing the Y component and UV component, and N is an integer greater than 1; the response signal g containing the first signal component and the second signal component is obtained according to the index i of the quality factor of the video signal. ci .
[0025] In another possible implementation, when the first signal component is a Y component and the second signal component is a UV component, if the bitstream includes the quality factors of the first signal component and the second signal component, implemented through a fully connected network, then the quality factor of the Y component is used as the input of the fully connected network, and the response signal of the Y component is output; the quality factor of the UV component is used as the input of the fully connected network, and the response signal of the UV component is output.
[0026] In one possible implementation, when the second signal component is a U component or a V component, the method further includes: performing entropy decoding on the bitstream to obtain a feature map of a third signal component of the video signal; and obtaining a reconstructed map of the third signal component based on the response signal of the third signal component and the feature map of the third signal component, wherein the response signal of the third signal component is obtained through learning, wherein when the second signal component is a U component, the third signal component is a V component, or when the second signal component is a V component, the third signal component is a U component. Then, reconstructing the video signal includes: reconstructing the video signal based on the reconstructed map of the first signal component, the reconstructed map of the second signal component, and the reconstructed map of the third signal component.
[0027] In one possible implementation, when the first signal component is a Y component, the second signal component is a U component, and the third signal component is a V component, if the bitstream includes the index i of the quality factor of the Y component, the index j of the quality factor of the U component, and the index k of the quality factor of the V component, the response signal matrix {g} of the first signal component is generated through learning. y1 g y2 , ...g yi ...g yN}, the response signal matrix of the second signal component {g u1 g u2 , ...g uj ..., g uM The response signal matrix of the third signal component {g} v1 g v2 , ..., g vk ...g vL}, where N, M, and L are integers greater than 1; the response signal g of the first signal component is obtained according to the index i of the quality factor of the Y component. yi The response signal g of the second signal component is obtained based on the index j of the quality factor of the U component. uj ; and obtain the response signal g of the third signal component according to the index k of the quality factor of the V component. vk .or
[0028] In another possible implementation, when the first signal component is the Y component, the second signal component is the U component, and the third signal component is the V component, if the bitstream includes the index i of the quality factor of the video signal, the response signal matrix {g} of the video signal is generated through learning. c1 g c2 , ...g ci ...g cN}, where c is 3 representing the Y component, U component, and V component, and N is an integer greater than 1; based on the index i of the quality factor of the video signal, a response signal g containing the first signal component, the second signal component, and the third signal component is obtained. ci .or
[0029] In another possible implementation, when the first signal component is a Y component, the second signal component is a U component, and the third signal component is a V component, if the bitstream includes the quality factors of the first signal component, the second signal component, and the third signal component, and is implemented through a fully connected network, then the quality factor of the Y component is used as the input of the fully connected network to output the response signal of the Y component; the quality factor of the U component is used as the input of the fully connected network to output the response signal of the U component; and the quality factor of the V component is used as the input of the fully connected network to output the response signal of the V component.
[0030] Thirdly, this application provides an encoder including processing circuitry for performing the method according to the first aspect and any one of the first aspects described above.
[0031] Fourthly, this application provides a decoder, including processing circuitry, for performing the methods described in the second aspect and any one of the second aspects above.
[0032] Fifthly, this application provides a computer program product, including program code, which, when executed on a computer or processor, is used to perform the methods described in the first aspect and any one of the first aspects, and the second aspect and any one of the second aspects.
[0033] In a sixth aspect, this application provides an encoder comprising: one or more processors; a non-transitory computer-readable storage medium coupled to the processors and storing a program executed by the processors, wherein the program, when executed by the processors, causes the decoder to perform the method described in the first aspect and any one of the first aspects.
[0034] In a seventh aspect, this application provides a decoder comprising: one or more processors; a non-transitory computer-readable storage medium coupled to the processor and storing a program executed by the processor, wherein the program, when executed by the processor, causes the encoder to perform the method described in the second aspect and any one of the second aspects.
[0035] Eighthly, this application provides a non-transitory computer-readable storage medium including program code, which, when executed by a computer device, is used to perform the methods described in the first aspect and any one of the first aspects, and the second aspect and any one of the second aspects.
[0036] Ninthly, the present invention relates to an encoding apparatus having the function of implementing the behavior in the method embodiments of the first aspect or any of the first aspects described above. The function can be implemented in hardware or by hardware executing corresponding software. The hardware or software includes one or more modules corresponding to the above-described functions. In one possible design, the encoding apparatus includes: a first control module for applying a control signal of a first signal component of the video signal to a first feature map of the first signal component to obtain a second feature map of the first signal component, wherein the control signal of the first signal component is obtained through learning; a second control module for applying a control signal of a second signal component of the video signal to the first feature map of the second signal component to obtain a second feature map of the second signal component, wherein the control signal of the second signal component is obtained through learning; and an encoding module for obtaining the bitstream of the video signal based on the second feature map of the first signal component and the second feature map of the second signal component. These modules can perform the corresponding functions in the method embodiments of the first aspect or any of the first aspects described above, as detailed in the method embodiments, and will not be repeated here.
[0037] Tenthly, the present invention relates to a decoding apparatus having the functionality to implement the behavior described in the method embodiments of the second aspect or any of the second aspects above. The functionality can be implemented in hardware or by hardware executing corresponding software. The hardware or software includes one or more modules corresponding to the above-described functionality. In one possible design, the decoding apparatus includes: a decoding module for obtaining a bitstream of the video signal and performing entropy decoding on the bitstream to obtain a feature map of a first signal component and a feature map of a second signal component of the video signal; a first control module for obtaining a reconstructed map of the first signal component based on the response signal of the first signal component and the feature map of the first signal component, wherein the response signal of the first signal component is obtained through learning; a second control module for obtaining a reconstructed map of the second signal component based on the response signal of the second signal component and the feature map of the second signal component, wherein the response signal of the second signal component is obtained through learning; and a reconstruction module for reconstructing the video signal based on the reconstructed map of the first signal component and the reconstructed map of the second signal component. These modules can perform the corresponding functions in the method embodiments of the second aspect or any of the second aspects above, as detailed in the method embodiments, and will not be repeated here.
[0038] In existing end-to-end image coding, when optimizing a specific network, the Y, U, and V component weights are fixed, resulting in a fixed bitrate for the YUV components. Since different images have different color characteristics, this fixed bitrate allocation leads to poor coding performance for some video image content. This application addresses this by controlling the feature maps of each signal component separately using control signals, thereby supporting bitrate allocation between YUV components and adapting to image content with different color characteristics.
[0039] One or more embodiments will be described in detail in the accompanying drawings and the following description. Other features, objects, and advantages will be apparent from the description, drawings, and claims. Attached Figure Description
[0040] The accompanying drawings used in the embodiments of this application are described below.
[0041] Figure 1A This is a block diagram of an example video decoding system for implementing embodiments of the present invention, wherein the system is based on deep learning to encode or decode video images;
[0042] Figure 1B A block diagram of another example of a video decoding system for implementing embodiments of the present invention, wherein the system is based on deep learning to encode or decode video images;
[0043] Figure 1C A block diagram of yet another example of a video decoding system for implementing embodiments of the present invention, wherein the video encoder and / or video decoder encodes or decodes video images based on deep learning;
[0044] Figure 2 This is a block diagram of an example video encoder 20 for implementing embodiments of the present invention, wherein the video encoder 20 encodes video images based on deep learning;
[0045] Figure 3 This is a block diagram illustrating an example of a video decoder for implementing an embodiment of the present invention, wherein the video decoder 30 decodes video images based on deep learning;
[0046] Figure 4 This is a schematic block diagram of a video decoding apparatus for implementing embodiments of the present invention;
[0047] Figure 5 This is a schematic block diagram of a video decoding apparatus for implementing embodiments of the present invention;
[0048] Figure 6 This is a schematic diagram in YUV format;
[0049] Figure 7AThis is a schematic diagram of the layered encoding / decoding structure provided in the embodiments of this application;
[0050] Figure 7B Based on Figure 7A An embodiment of the encoding method;
[0051] Figure 7C Based on Figure 7A An embodiment of the decoding method;
[0052] Figure 7D This is another schematic diagram of the layered encoding / decoding structure provided in the embodiments of this application;
[0053] Figure 8A This is a schematic diagram of the layered encoding / decoding structure provided in the embodiments of this application;
[0054] Figure 8B yes Figure 8A One embodiment;
[0055] Figure 9A This is a schematic diagram of the layered encoding / decoding structure provided in the embodiments of this application;
[0056] Figure 9B yes Figure 9A One embodiment;
[0057] Figure 10 This is a schematic diagram showing the structure of an encoding device 1000 according to an embodiment of this application;
[0058] Figure 11 This is a schematic diagram showing the structure of a decoding device 1100 according to an embodiment of this application. Detailed Implementation
[0059] This application provides an AI-based video image compression technology, specifically a method and apparatus for layered encoding and decoding, to improve traditional end-to-end hybrid video encoding and decoding systems.
[0060] Video coding generally refers to the processing of image sequences that form a video or video sequence. In the field of video coding, the terms "picture," "frame," or "image" can be used synonymously. Video coding (or commonly referred to as encoding) comprises two parts: video encoding and video decoding. Video encoding is performed on the source side and typically involves processing (e.g., compressing) the raw video image to reduce the amount of data required to represent that video image (thus enabling more efficient storage and / or transmission). Video decoding is performed on the destination side and typically involves inverse processing relative to the encoder to reconstruct the video image. The "encoding" of the video image (or commonly referred to as an image) mentioned in the embodiments should be understood as the "encoding" or "decoding" of the video image or video sequence. The encoding and decoding parts are also collectively referred to as codec (encoding and decoding, CODEC).
[0061] In lossless video coding, the original video image can be reconstructed, meaning the reconstructed video image has the same quality as the original (assuming no transmission loss or other data loss during storage or transmission). In lossy video coding, further compression is performed through quantization to reduce the amount of data required to represent the video image, and the decoder cannot completely reconstruct the video image, meaning the quality of the reconstructed video image is lower or worse than the quality of the original video image.
[0062] Several video coding standards fall under the category of "lossy hybrid video coding and decoding" (i.e., combining spatial and temporal prediction in the pixel domain with 2D transform coding in the transform domain for applying quantization). Each image in a video sequence is typically segmented into a set of non-overlapping blocks, usually encoded at the block level. In other words, the encoder typically processes and encodes the video at the block (video block) level, for example, generating prediction blocks through spatial (intra-frame) and temporal (inter-frame) prediction; subtracting the prediction blocks from the current block (the block currently being processed / to be processed) to obtain residual blocks; transforming and quantizing the residual blocks in the transform domain to reduce the amount of data to be transmitted (compressed), while the decoder applies the inverse processing relative to the encoder to the encoded or compressed blocks to reconstruct the current block for representation. Additionally, the encoder needs to repeat the decoder's processing steps so that the encoder and decoder generate the same predictions (e.g., intra-frame and inter-frame predictions) and / or reconstruct pixels for processing, i.e., encoding subsequent blocks.
[0063] In the following embodiment of the decoding system 10, the encoder 20 and decoder 30 are based on Figures 1B to 3 Describe it.
[0064] Figure 1A A schematic block diagram of an exemplary decoding system, such as Figure 1AAs shown, after video is captured using a video capture device, it undergoes a series of preprocessing steps, followed by compression and encoding to obtain an encoded bitstream. The bitstream is then transmitted via a transmission network to a receiving module using a sending module. After decoding by a decoder, it can be rendered and displayed. Alternatively, the encoded video bitstream can also be directly stored.
[0065] Figure 1B This is a schematic block diagram of an exemplary decoding system 10, such as a video decoding system 10 (or simply decoding system 10) that can utilize the technology of this application. The video encoder 20 (or simply encoder 20) and video decoder 30 (or simply decoder 30) in the video decoding system 10 represent devices, etc., that can be used to perform various technologies according to the various examples described in this application.
[0066] like Figure 1B As shown, the decoding system 10 includes a source device 12, which provides encoded image data 21, such as encoded images, to a destination device 14 for decoding the encoded image data 21.
[0067] The source device 12 includes an encoder 20, and optionally may include an image source 16, a preprocessor (or preprocessing unit) 18 such as an image preprocessor, and a communication interface (or communication unit) 22.
[0068] Image source 16 may include or may be any type of image capture device for capturing real-world images, and / or any type of image generation device, such as a computer graphics processor for generating computer animation images or any type of device for acquiring and / or providing real-world images, computer-generated images (e.g., screen content, virtual reality (VR) images, and / or any combination thereof (e.g., augmented reality (AR) images). The image source may be any type of memory or storage device storing any of the images described above.
[0069] To distinguish the processing performed by the preprocessor (or preprocessing unit) 18, the image (or image data) 17 may also be referred to as the raw image (or raw image data) 17.
[0070] The preprocessor 18 receives (raw) image data 17 and preprocesses the image data 17 to obtain a preprocessed image (preprocessed image data) 19. For example, the preprocessing performed by the preprocessor 18 may include cropping, color format conversion (e.g., from RGB to YCbCr), color correction, or noise reduction. It is understood that the preprocessing unit 18 may be an optional component.
[0071] Video encoder (or encoder) 20 is used to receive preprocessed image data 19 and provide encoded image data 21 (hereinafter referred to as...) Figure 2 (and so on, for further description).
[0072] The communication interface 22 in the source device 12 can be used to receive encoded image data 21 and send the encoded image data 21 (or other arbitrarily processed version) to another device such as the destination device 14 or any other device via the communication channel 13 for storage or direct reconstruction.
[0073] The target device 14 includes a decoder 30, and optionally may include a communication interface (or communication unit) 28, a post-processor (or post-processing unit) 32 and a display device 34.
[0074] The communication interface 28 in the destination device 14 is used to receive encoded image data 21 (or other processed versions) directly from the source device 12 or from any other source device such as a storage device, for example, the storage device is an encoded image data storage device, and to provide the encoded image data 21 to the decoder 30.
[0075] Communication interfaces 22 and 28 can be used to send or receive encoded image data (or encoded data 21) through a direct communication link between source device 12 and destination device 14, such as a direct wired or wireless connection, or through any type of network, such as a wired network, a wireless network or any combination thereof, any type of private network and public network or any combination thereof.
[0076] For example, the communication interface 22 can be used to encapsulate the encoded image data 21 into a suitable format such as a message, and / or process the encoded image data using any type of transmission encoding or processing, so as to transmit it on a communication link or communication network.
[0077] Communication interface 28 corresponds to communication interface 22. For example, it can be used to receive transmitted data and process the transmitted data using any type of corresponding transmission decoding or processing and / or decapsulation to obtain encoded image data 21.
[0078] Both communication interface 22 and communication interface 28 can be configured as follows: Figure 1B The arrow pointing from the source device 12 to the corresponding communication channel 13 of the destination device 14 indicates a one-way or two-way communication interface, which can be used to send and receive messages, establish connections, acknowledge and exchange any other information related to the communication link and / or data transmission, such as encoded image data transmission, etc.
[0079] Video decoder (or decoder) 30 is used to receive encoded image data 21 and provide decoded image (or decoded image data) 31 (hereinafter referred to as...). Figure 3 (and so on, for further description).
[0080] The post-processor 32 is used to post-process the decoded image data 31 (also known as the reconstructed image data) to obtain post-processed image data 33. The post-processing performed by the post-processing unit 32 may include, for example, color format conversion (e.g., from YCbCr to RGB), color adjustment, trimming or resampling, or any other processing to generate the decoded image data 31 for display by the display device 34, etc.
[0081] Display device 34 is used to receive post-processed image data 33 to display the image to a user or viewer. Display device 34 can be or includes any type of display for representing the reconstructed image, such as an integrated or external display screen or monitor. For example, the display screen may include a liquid crystal display (LCD), an organic light emitting diode (OLED) display, a plasma display, a projector, a micro LED display, a liquid crystal on silicon (LCoS), a digital light processor (DLP), or any other type of display screen.
[0082] The decoding system 10 also includes a training engine 25, which is used to train the encoder 20 or the decoder 30 to perform hierarchical coding processing on the reconstructed image.
[0083] In this embodiment, the training data includes a training matrix set, which includes a pre-filter brightness matrix, a quantization step size matrix, and a post-filter brightness matrix for an image patch. Pixels at corresponding positions in the pre-filter brightness matrix correspond to the pre-filter brightness values of pixels at corresponding positions in the corresponding image patch. Pixels at corresponding positions in the quantization step size matrix correspond to the quantization step size values corresponding to the brightness values of pixels at corresponding positions in the corresponding image patch. Pixels at corresponding positions in the post-filter brightness matrix correspond to the post-filter brightness values of pixels at corresponding positions in the corresponding image patch.
[0084] Multiple matrices in the training matrix set can, for example, be... Figure 6 Input the training engine 25 as shown in steps a through 6c. Figure 6 As shown in Figure a, multiple matrices from the training matrix set are directly input into the training engine 25; these matrices are all two-dimensional matrices. Figure 6 As shown in b, a portion or all of multiple matrices in the training matrix set are selected and merged to obtain a multidimensional matrix, which is then input into the training engine 25. For example... Figure 6 As shown in c, select some or all of the matrices in the training matrix set, add (or multiply) them to obtain a two-dimensional matrix, and then input the two-dimensional matrix into the training engine 25.
[0085] The training data can be stored in a database (not shown), and the training engine 25 trains the target model based on the training data (e.g., a neural network for hierarchical encoding and decoding). It should be noted that this application embodiment does not limit the source of the training data; for example, the training data can be obtained from the cloud or other places for model training.
[0086] The training engine 25 trains the target model so that the unfiltered pixels approximate the original pixel values. Each training process can use a mini-batch size of 64 images and an initial learning rate of 1e-4, following a step size of 10. The training data can be data generated by the encoder under different QP quantization parameter settings. The target model can be used to implement the hierarchical encoding and decoding method provided in this application embodiment, that is, after the reconstructed image or image patch is preprocessed, it is input into the target model to obtain the filtered image or image patch. The target model in this application embodiment can specifically be a filtering network, which will be discussed below in conjunction with... Figures 7A-7D Provide a detailed description of the target model.
[0087] The target model trained by training engine 25 can be applied to decoding system 10, for example, applied to... Figure 1B The source device 12 (e.g., encoder 20) or the destination device 14 (e.g., decoder 30) is shown. The training engine 25 can be trained in the cloud to obtain the target model, and then the decoding system 10 can download and use the target model from the cloud; or, the training engine 25 can be trained in the cloud to obtain the target model and use the target model, and the decoding system 10 can directly obtain the processing result from the cloud.
[0088] although Figure 1B The source device 12 and destination device 14 are shown as independent devices, but device embodiments may also include both source device 12 and destination device 14, or the functions of both source device 12 and destination device 14, that is, simultaneously including source device 12 or its corresponding functions and destination device 14 or its corresponding functions. In these embodiments, source device 12 or its corresponding functions and destination device 14 or its corresponding functions may be implemented using the same hardware and / or software, or by separate hardware and / or software, or any combination thereof.
[0089] According to the description, Figure 1B The presence and (accurate) division of different units or functions in the source device 12 and / or destination device 14 shown may vary depending on the actual device and application.
[0090] Encoder 20 (e.g., video encoder 20) or decoder 30 (e.g., video decoder 30) or both can be transmitted via, for example, Figure 1C The processing circuitry shown can be implemented as, for example, one or more microprocessors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), discrete logic, hardware, video encoding dedicated processors, or any combination thereof. Encoder 20 can be implemented via processing circuitry 46 to include reference... Figure 2 Encoder 20 refers to various modules discussed herein and / or any other encoder system or subsystem described herein. Decoder 30 may be implemented via processing circuitry 46 to include references. Figure 3 Decoder 30 may include various modules discussed herein and / or any other decoder system or subsystem described herein. The processing circuitry 46 may be used to perform various operations discussed below. Figure 5 As shown, if some of the technology is implemented in software, the device can store the software instructions in a suitable non-transitory computer-readable storage medium and execute the instructions in hardware using one or more processors, thereby performing the technology of the present invention. One of the video encoder 20 and video decoder 30 can be integrated into a single device as part of a combined codec (encoder / decoder, CODEC), such as... Figure 1C As shown.
[0091] Source device 12 and destination device 14 may include any of a variety of devices, including any type of handheld or fixed device, such as a laptop or notebook computer, mobile phone, smartphone, tablet or tablet computer, camera, desktop computer, set-top box, television, display device, digital media player, video game console, video streaming device (e.g., content service server or content distribution server), broadcast receiving device, broadcast transmitting device, etc., and may or may not use an operating system of any type. In some cases, source device 12 and destination device 14 may be equipped with components for wireless communication. Therefore, source device 12 and destination device 14 may be wireless communication devices.
[0092] In some cases, Figure 1BThe video decoding system 10 shown is merely exemplary, and the technology provided in this application can be applied to video encoding setups (e.g., video encoding or video decoding) that do not necessarily include any data communication between the encoding and decoding devices. In other examples, data is retrieved from local memory, sent over a network, etc. A video encoding device may encode data and store it in memory, and / or a video decoding device may retrieve data from memory and decode it. In some examples, encoding and decoding are performed by devices that do not communicate with each other but simply encode data into memory and / or retrieve and decode data from memory.
[0093] Figure 1C It is a inclusion according to an exemplary embodiment Figure 2 Video encoder 20 and / or Figure 3 An illustrative diagram illustrating an example of a video decoding system 40 with a video decoder 30. The video decoding system 40 may include an imaging device 41, a video encoder 20, a video decoder 30 (and / or a video encoder / decoder implemented by processing circuitry 46), an antenna 42, one or more processors 43, one or more memory storage devices 44, and / or a display device 45.
[0094] like Figure 1C As shown, the imaging device 41, antenna 42, processing circuitry 46, video encoder 20, video decoder 30, processor 43, memory storage 44, and / or display device 45 are capable of communicating with each other. In different instances, the video decoding system 40 may contain only the video encoder 20 or only the video decoder 30.
[0095] In some instances, antenna 42 can be used to transmit or receive encoded bitstreams of video data. Additionally, in some instances, display device 45 can be used to present video data. Processing circuitry 46 can include application-specific integrated circuit (ASIC) logic, a graphics processor, a general-purpose processor, etc. Video decoding system 40 can also include an optional processor 43, which similarly can include application-specific integrated circuit (ASIC) logic, a graphics processor, a general-purpose processor, etc. Furthermore, memory storage 44 can be any type of memory, such as volatile memory (e.g., static random access memory (SRAM), dynamic random access memory (DRAM), etc.) or non-volatile memory (e.g., flash memory, etc.). In a non-limiting example, memory storage 44 can be implemented using high-speed cache memory. In other instances, processing circuitry 46 can include memory (e.g., cache, etc.) for implementing image buffers, etc.
[0096] In some instances, the video encoder 20 implemented via logic circuitry may include (e.g., implemented via processing circuitry 46 or memory storage 44) an image buffer and (e.g., implemented via processing circuitry 46) a graphics processing unit. The graphics processing unit may be communicatively coupled to the image buffer. The graphics processing unit may include the video encoder 20 implemented via processing circuitry 46 to implement reference... Figure 2 And / or any other encoder system or subsystem described herein, and the various modules discussed herein. Logic circuits may be used to perform the various operations discussed herein.
[0097] In some instances, the video decoder 30 may be implemented in a similar manner via processing circuitry 46 to implement the various modules discussed in the video decoder 30 with reference to FIG3 and / or any other decoder system or subsystem described herein. In some instances, the logic-circuit-implemented video decoder 30 may include (implemented via processing circuitry 46 or memory storage 44) an image buffer and (e.g., implemented via processing circuitry 46) a graphics processing unit. The graphics processing unit may be communicatively coupled to the image buffer. The graphics processing unit may include the video decoder 30 implemented via processing circuitry 46 to implement the various modules discussed in the video decoder 30 with reference to FIG3 and / or any other decoder system or subsystem described herein. Figure 3 And / or the various modules discussed in any other decoder system or subsystem described herein.
[0098] In some instances, antenna 42 can be used to receive encoded bitstreams of video data. As discussed herein, the encoded bitstream may contain data related to encoded video frames, indicators, index values, mode selection data, etc., such as data related to code segmentation (e.g., transform coefficients or quantized transform coefficients, optional indicators, and / or data defining code segmentation). Video decoding system 40 may also include a video decoder 30 coupled to antenna 42 for decoding the encoded bitstream. Display device 45 is used to display the video frames.
[0099] It should be understood that, for the examples described with reference to video encoder 20 in this application embodiment, video decoder 30 can be used to perform the reverse process. Regarding signaling syntax elements, video decoder 30 can be used to receive and parse such syntax elements, and accordingly decode the associated video data. In some examples, video encoder 20 can entropy-encode syntax elements into an encoded video bitstream. In such instances, video decoder 30 can parse such syntax elements and accordingly decode the associated video data.
[0100] For ease of description, embodiments of the present invention are described with reference to Versatile video coding (VVC) reference software or High-Efficiency Video Coding (HEVC) developed by the Joint Collaboration Team on Video Coding (JCT-VC) of the ITU-T Video Coding Experts Group (VCEG) and the ISO / IEC Motion Picture Experts Group (MPEG). Those skilled in the art will understand that embodiments of the present invention are not limited to HEVC or VVC.
[0101] Encoders and Encoding Methods
[0102] Figure 2 This is a schematic block diagram illustrating an example of a video encoder 20 used to implement the technology of this application. Figure 2In the example, the video encoder 20 includes an input terminal (or input interface) 201, a residual calculation unit 204, a transform processing unit 206, a quantization unit 208, an inverse quantization unit 210, an inverse transform processing unit 212, a reconstruction unit 214, a loop filter 220, a decoded picture buffer (DPB) 230, a mode selection unit 260, an entropy coding unit 270, and an output terminal (or output interface) 272. The mode selection unit 260 may include an inter-frame prediction unit 244, an intra-frame prediction unit 254, and a segmentation unit 262. The inter-frame prediction unit 244 may include a motion estimation unit and a motion compensation unit (not shown). Figure 2 The video encoder 20 shown can also be called a hybrid video encoder or a video encoder based on a hybrid video codec.
[0103] The residual calculation unit 204, transform processing unit 206, quantization unit 208, and mode selection unit 260 constitute the forward signal path of encoder 20, while the inverse quantization unit 210, inverse transform processing unit 212, reconstruction unit 214, buffer 216, loop filter 220, decoded picture buffer (DPB) 230, inter-frame prediction unit 244, and intra-frame prediction unit 254 constitute the backward signal path of encoder 20. The backward signal path of encoder 20 corresponds to the signal path of decoder (see [link to decoder]). Figure 3 The decoder 30 in the video encoder 20 consists of an inverse quantization unit 210, an inverse transform processing unit 212, a reconstruction unit 214, a loop filter 220, a decoded image buffer 230, an inter-frame prediction unit 244, and an intra-frame prediction unit 254.
[0104] Quantification
[0105] Quantization unit 208 is used to quantize the transform coefficients 207 by, for example, scalar quantization or vector quantization, to obtain quantized transform coefficients 209. Quantized transform coefficients 209 can also be called quantized residual coefficients 209.
[0106] The quantization process can reduce the bit depth associated with some or all of the transform coefficients 207. For example, n-bit transform coefficients can be rounded down to m-bit transform coefficients during quantization, where n is greater than m. The degree of quantization can be modified by adjusting the quantization parameter (QP). For example, for scalar quantization, different scales can be applied to achieve finer or coarser quantization. Smaller quantization steps correspond to finer quantization, while larger quantization steps correspond to coarser quantization. The appropriate quantization step size can be indicated by the quantization parameter (QP). For example, the quantization parameter can be an index to a predefined set of appropriate quantization steps. For example, a smaller quantization parameter can correspond to fine quantization (smaller quantization step size), a larger quantization parameter can correspond to coarse quantization (larger quantization step size), and vice versa. Quantization may include division by the quantization step size, while corresponding or inverse dequantization performed by the dequantization unit 210, etc., may include multiplication by the quantization step size. Embodiments of some HEVC standards, for example, can be used to determine the quantization step size using the quantization parameter. In general, the quantization step size can be calculated from the quantization parameter using a fixed-point approximation of an equation involving division. Additional scaling factors can be introduced for quantization and dequantization to recover the norm of the residual block, which may have been modified by the scaling used in the fixed-point approximation of the equations used for the quantization step size and quantization parameters. In one exemplary implementation, the scaling of the inverse transform and dequantization can be combined. Alternatively, a custom quantization table can be used and indicated from the encoder to the decoder in the bitstream, etc. Quantization is a lossy operation, where the loss increases with the quantization step size.
[0107] In one embodiment, the video encoder 20 (correspondingly, the quantization unit 208) can be used to output the quantization parameter (QP), for example, directly outputting it or outputting it after being encoded or compressed by the entropy encoding unit 270, for example, so that the video decoder 30 can receive it and use it for decoding.
[0108] Inverse Quantization
[0109] The dequantization unit 210 is used to perform dequantization on the quantization coefficients by the quantization unit 208 to obtain the dequantization coefficients 211. For example, it performs a dequantization scheme based on or using the same quantization step size as the quantization unit 208 to perform the quantization scheme performed by the quantization unit 208. The dequantization coefficients 211 can also be called dequantization residual coefficients 211, corresponding to the transform coefficients 207. However, due to the loss caused by quantization, the dequantization coefficients 211 are usually not exactly the same as the transform coefficients.
[0110] reconstruction
[0111] The reconstruction unit 214 (e.g., summer 214) is used to add the transform block 213 (i.e., the reconstruction residual block 213) to the prediction block 265 to obtain the reconstruction block 215 in the pixel domain, for example, by adding the pixel values of the reconstruction residual block 213 and the pixel values of the prediction block 265.
[0112] Filtering
[0113] Loop filter unit 220 (or simply "loop filter" 220) is used to filter the reconstructed block 215 to obtain the filtered block 221, or typically to filter the reconstructed pixels to obtain filtered pixel values. For example, the loop filter unit is used to smoothly perform pixel transformations or improve video quality. Loop filter unit 220 may include one or more loop filters, such as a deblocking filter, a sample-adaptive offset (SAO) filter, or one or more other filters, such as an adaptive loop filter (ALF), a noise suppression filter (NSF), or any combination thereof. For example, loop filter unit 220 may include a deblocking filter, a SAO filter, and an ALF filter. The filtering process may be performed in the order of deblocking filter, SAO filter, and ALF filter. As another example, a process called luma mapping with chromascaling (LMCS) (i.e., an adaptive in-loop shaper) is added. This process is performed before deblocking. For example, the deblocking filtering process can also be applied to internal sub-block edges, such as affine sub-block edges, ATMVP sub-block edges, sub-block transform (SBT) edges, and intra sub-partition (ISP) edges. Although the loop filter unit 220 in... Figure 2 The loop filter is shown in the diagram, but in other configurations, the loop filter unit 220 can be implemented as a post-loop filter. The filter block 221 can also be called the filter reconstruction block 221.
[0114] In one embodiment, the video encoder 20 (correspondingly, the loop filter unit 220) can be used to output loop filter parameters (e.g., SAO filter parameters, ALF filter parameters, or LMCS parameters), for example, directly outputting or outputting after entropy encoding by the entropy encoding unit 270, for example, enabling the decoder 30 to receive and decode using the same or different loop filter parameters.
[0115] Decoder and Decoding Method
[0116] Figure 3An exemplary video decoder 30 for implementing the technology of this application is shown. The video decoder 30 is used to receive encoded image data 21 (e.g., encoded bitstream 21) encoded by encoder 20, for example, to obtain a decoded image 331. The encoded image data or bitstream includes information for decoding the encoded image data, such as data representing image blocks (and / or groups or blocks of encoded video clips) and associated syntax elements.
[0117] exist Figure 3 In the example, decoder 30 includes an entropy decoding unit 304, an inverse quantization unit 310, an inverse transform processing unit 312, a reconstruction unit 314 (e.g., a summer 314), a loop filter 320, a decoded image buffer (DBP) 330, a mode application unit 360, an inter-frame prediction unit 344, and an intra-frame prediction unit 354. The inter-frame prediction unit 344 may be or include a motion compensation unit. In some examples, video decoder 30 may perform substantially the same functions as the referenced unit. Figure 2 The video encoder 100 describes the encoding process as the opposite of the decoding process.
[0118] As described in encoder 20, the inverse quantization unit 210, inverse transform processing unit 212, reconstruction unit 214, loop filter 220, decoded image buffer DPB 230, inter-frame prediction unit 344, and intra-frame prediction unit 354 also constitute the "built-in decoder" of video encoder 20. Correspondingly, inverse quantization unit 310 can be functionally identical to inverse quantization unit 110, inverse transform processing unit 312 can be functionally identical to inverse transform processing unit 122, reconstruction unit 314 can be functionally identical to reconstruction unit 214, loop filter 320 can be functionally identical to loop filter 220, and decoded image buffer 330 can be functionally identical to decoded image buffer 230. Therefore, the explanation of the corresponding units and functions of video encoder 20 is correspondingly applicable to the corresponding units and functions of video decoder 30.
[0119] Inverse Quantization
[0120] The dequantization unit 310 can be used to receive quantization parameters (QP) (or generally information related to dequantization) and quantization coefficients from encoded image data 21 (e.g., parsed and / or decoded by the entropy decoding unit 304), and dequantize the decoded quantization coefficients 309 based on the quantization parameters to obtain dequantization coefficients 311, which may also be referred to as transform coefficients 311. The dequantization process may include using the quantization parameters calculated by the video encoder 20 for each video block in the video slice to determine the degree of quantization, and also to determine the degree of dequantization to be performed.
[0121] reconstruction
[0122] The reconstruction unit 314 (e.g., summer 314) is used to add the reconstruction residual block 313 to the prediction block 365 to obtain the reconstruction block 315 in the pixel domain, for example, by adding the pixel values of the reconstruction residual block 313 and the pixel values of the prediction block 365.
[0123] Filtering
[0124] Loop filter unit 320 (in or after the encoding loop) is used to filter the reconstructed block 315 to obtain filtered block 321, thereby facilitating pixel transformation or improving video quality. Loop filter unit 320 may include one or more loop filters, such as a deblocking filter, a sample-adaptive offset (SAO) filter, or one or more other filters, such as an adaptive loop filter (ALF), a noise suppression filter (NSF), or any combination thereof. For example, loop filter unit 320 may include a deblocking filter, a SAO filter, and an ALF filter. The filtering process may be performed in the order of deblocking filter, SAO filter, and ALF filter. As another example, a process called luma mapping with chroma scaling (LMCS) (i.e., an adaptive in-loop shaper) may be added. This process is performed before deblocking. For example, the deblocking filtering process can also be applied to internal sub-block edges, such as affine sub-block edges, ATMVP sub-block edges, sub-block transform (SBT) edges, and intra sub-partition (ISP) edges. Although the loop filter unit 320 in... Figure 3 The loop filter is shown in the diagram, but in other configurations, the loop filter unit 320 can be implemented as a post-loop filter.
[0125] The decoder 30 is used to output the decoded image 311 through the output terminal 312, etc., for display to the user or for the user to view.
[0126] Although the above embodiments primarily describe video encoding and decoding, it should be noted that embodiments of the decoding system 10, encoder 20, and decoder 30, as well as other embodiments described herein, can also be used for still image processing or encoding and decoding, i.e., the processing or encoding and decoding of a single image independent of any previous or consecutive images in video encoding and decoding. Generally, if image processing is limited to a single image 17, the inter-frame prediction unit 244 (encoder) and inter-frame prediction unit 344 (decoder) may not be available. All other functions (also referred to as tools or techniques) of the video encoder 20 and video decoder 30 can also be used for still image processing, such as residual calculation 204 / 304, transform 206, quantization 208, inverse quantization 210 / 310, (inverse) transform 212 / 312, segmentation 262 / 362, intra-frame prediction 254 / 354 and / or loop filtering 220 / 320, entropy coding 270, and entropy decoding 304.
[0127] Figure 4 This is a schematic diagram of a video decoding device 400 provided for an embodiment of the present invention. The video decoding device 400 is suitable for implementing the disclosed embodiments described herein. In one embodiment, the video decoding device 400 may be a decoder, for example... Figure 1B The video decoder 30 in the text can also be an encoder, for example... Figure 1B The video encoder 20 in the middle.
[0128] The video decoding device 400 includes: an input port 410 (or input port 410) and a receiver unit (Rx) 420 for receiving data; a processor, logic unit, or central processing unit (CPU) 430 for processing data; for example, the processor 430 may be a neural network processor 430; a transmitter unit (Tx) 440 and an output port 450 (or output port 450) for transmitting data; and a memory 460 for storing data. The video decoding device 400 may also include optical-to-electrical (OE) components and electro-optical (EO) components coupled to the input port 410, receiver unit 420, transmitter unit 440, and output port 450 for the entry or exit of optical or electrical signals.
[0129] Processor 430 is implemented in both hardware and software. Processor 430 may be implemented as one or more processor chips, cores (e.g., multi-core processors), FPGAs, ASICs, and DSPs. Processor 430 communicates with ingress port 410, receiver 420, transmitter 440, egress port 450, and memory 460. Processor 430 includes a decoding module 470 (e.g., a neural network (NN) based decoding module 470). Decoding module 470 implements the embodiments disclosed above. For example, decoding module 470 performs, processes, prepares, or provides various encoding operations. Therefore, decoding module 470 provides a substantial improvement to the functionality of video decoding device 400 and affects the switching of video decoding device 400 to different states. Alternatively, decoding module 470 may be implemented with instructions stored in memory 460 and executed by processor 430.
[0130] Memory 460 includes one or more disks, tape drives, and solid-state drives, which can be used as overflow data storage devices to store programs when an executable program is selected, and to store instructions and data read during program execution. Memory 460 can be volatile and / or non-volatile, and can be read-only memory (ROM), random access memory (RAM), ternary content-addressable memory (TCAM), and / or static random-access memory (SRAM).
[0131] Figure 5 A simplified block diagram of the apparatus 500 provided for an exemplary embodiment; the apparatus 500 can be used as... Figure 1B The source device 12 and the destination device 14 are either or both of them.
[0132] The processor 502 in device 500 can be a central processing unit. Alternatively, processor 502 can be any other type of device or multiple devices, existing or to be developed in the future, capable of manipulating or processing information. While the disclosed implementation can be implemented using a single processor, such as processor 502 as shown in the figure, using more than one processor is faster and more efficient.
[0133] In one implementation, the memory 504 in device 500 may be a read-only memory (ROM) device or a random access memory (RAM) device. Any other suitable type of storage device may be used as memory 504. Memory 504 may include code and data 506 accessed by processor 502 via bus 512. Memory 504 may also include an operating system 508 and an application program 510, which includes at least one program that allows processor 502 to perform the methods described herein. For example, application program 510 may include applications 1 to N, and also includes a video decoding application that performs the methods described herein.
[0134] The device 500 may also include one or more output devices, such as a display 518. In one example, the display 518 may be a touch-sensitive display that combines a display with a touch-sensitive element that can be used to sense touch input. The display 518 may be coupled to the processor 502 via a bus 512.
[0135] Although bus 512 in device 500 is described herein as a single bus, bus 512 may include multiple buses. Furthermore, auxiliary storage may be directly coupled to other components of device 500 or accessed via a network, and may include a single integrated unit such as a memory card or multiple units such as multiple memory cards. Therefore, device 500 can have a wide variety of configurations.
[0136] This invention relates to a lossy AI image coding scheme oriented towards color component bitrate allocation, which can be applied to Figures 1A to 1B. Figure 5 The video decoding system, encoder, and decoder described.
[0137] It should be noted that the method provided in this application is mainly used for the YUV component bitrate allocation process, which is mainly controlled by the encoding end. In order to make the decoding end more compatible, a corresponding control unit can also be added to the decoder.
[0138] Video image signals typically include one luminance component and two chrominance components. The luminance component is usually represented by the symbol Y, and the chrominance components are usually represented by the symbols U and V. As shown in Figures 7(a) to (c), commonly used YUV formats include the following formats, where the crosses in Figure 7 represent luminance component sampling points, and the circles represent chrominance component sampling points:
[0139] 4:4:4 format: indicates that the chroma components are not downsampled;
[0140] 4:2:2 format: indicates that the chroma component is downsampled horizontally at a ratio of 2:1 to the luminance component, with no vertical downsampling. For every two U or V sample points, each scan line contains four Y sample points;
[0141] 4:2:0 format: indicates that the chroma component is downsampled horizontally by a ratio of 2:1 and vertically by a ratio of 2:1 relative to the luminance component.
[0142] When a video image uses the YUV4:2:0 format, if the luminance component of an image block is a 2Nx2N block, then the chrominance component of the image block is an NxN block. This embodiment of the invention will use the 4:2:0 format as an example to explain the technical solution of the invention. However, it is understood that the technical solution of the invention, in addition to being used for the YUV4:2:0 format, can also be used for other YUV formats, or for mutual prediction between different components in other video image formats, such as the RGB format. Furthermore, the current block can be a square block, a non-square rectangular block, or other shaped regions; the technical solution provided in this embodiment of the invention is equally applicable.
[0143] For ease of description, the embodiments of the present invention use the terms "first signal component" and "second signal component". If the image signal includes a luminance signal component and a chrominance signal component, then the first signal component can be the chrominance component and the second signal component can be the luminance component; if the image signal includes three signal components R, G, and B, then the first signal component can be any one of the three signal components R, G, and B, and the second signal component can be one of the three signal components R, G, and B that is different from the first signal component; if the image signal is decomposed into multiple signal components in other ways, then a similar method can be used to define the first signal component and the second signal component.
[0144] As described in detail below, in this embodiment of the application, a quality factor can be input to the bitrate control module (or bitrate allocation control module), which generates control signals for each component feature map; the control vector of each component control signal is multiplied by the corresponding feature map to obtain the quantized feature value to be encoded.
[0145] like Figure 7A As shown, the AI image decoding system 700 for YUV bitrate allocation provided in this embodiment of the invention can be used for video image encoders and decoders. Figure 7AAs shown, the image decoding system 700 includes a first signal component processing module (e.g., a Y component processing module), a second signal component processing module (e.g., a UV component processing module), a bitrate allocation control module, and an entropy coding module at the encoding end; and a first signal component processing module (Y component processing module 2), a second signal component processing module (e.g., a UV component processing module 2), and an entropy decoding module at the decoding end. The image decoding system 700 optionally includes a joint processing module, a joint processing module 2, and a quality response module (also referred to as a bitrate allocation control module 2 or a bitrate control module 2). The Y component quality factor and UV component quality factor in the image decoding system 700 are input to the bitrate allocation control module. This module outputs control signals that act on the Y component feature map of the Y component processing module and the UV component feature map of the UV component processing module (which can be referred to as the first feature map), respectively, and outputs the second feature map of each signal component, thereby realizing the bitrate allocation of Y and UV. Subsequently, the bitstream of the video signal is obtained based on the second feature map of each signal component. For example, the feature maps output by the Y component processing module and the UV component processing module can be directly concatenated together, or the feature maps output by the Y component processing module and the UV component processing module can be directly added together to form the final feature map output by the encoder, and entropy encoding is performed on this feature map. Optionally, the feature maps output by the Y component processing module and the UV component processing module can be input into the joint processing module to obtain the final feature map output by the encoder, and entropy encoding is performed on this feature map.
[0146] exist Figure 7A In the architecture shown, Figure 7B This is an embodiment of an encoding method. Step 701: Obtain the control signal for the first signal component based on the quality factor of the first signal component. Step 702: Obtain the control signal for the second signal component based on the quality factor of the second signal component. For example, Figure 7B The illustrated embodiment can obtain a control signal for the first signal component from N candidate first control signals based on the quality factor of the first signal component, where N is an integer greater than 1; and obtain a control signal for the second signal component from M candidate second control signals based on the quality factor of the second signal component, where M is an integer greater than 1. N and M may be equal or unequal, and this application does not limit this.
[0147] Step 703: Apply the control signal of the first signal component to the first feature map of the first signal component to obtain the second feature map of the first signal component. Step 704: Apply the control signal of the second signal component to the first feature map of the second signal component to obtain the second feature map of the second signal component.
[0148] For example, in one embodiment, the control signal is generated by network learning and applied to the feature map (which may be called the first feature map) of at least one layer of the network in each of the Y component processing module and the UV component processing module, and then outputs the second feature map. For example, it is applied to the output of the last layer of the network.
[0149] Step 705: Obtain the bitstream of the video signal based on the second feature map of the first signal component and the second feature map of the second signal component.
[0150] Since the control signal can be applied to the first feature map of any layer in the network of each module in the Y component processing module and the UV component processing module, after outputting the second feature map, neural network processing can continue on the second feature map. Accordingly, based on the second feature map of the Y component and the second feature map of the UV component, the bitstream of the video signal is obtained as follows:
[0151] Entropy coding is performed on the second feature map of the first signal component and the second feature map of the second signal component to obtain the bitstream of the video signal; or,
[0152] Entropy coding is performed on the second feature map of the first signal component and the second feature map of the second signal component processed by the neural network to obtain the bitstream of the video signal; or,
[0153] Entropy coding is performed on the second feature map of the first signal component and the second feature map of the second signal component, which have been processed by the neural network, to obtain the bitstream of the video signal; or,
[0154] Entropy coding is performed on the second feature map of the first signal component processed by the neural network and the second feature map of the second signal component processed by the neural network to obtain the bitstream of the video signal.
[0155] When the image decoding system 700 includes a joint processing module, obtaining the bitstream of the video signal based on the second feature map of the first signal component and the second feature map of the second signal component includes:
[0156] The second feature map of the first signal component and the second feature map of the second signal component are jointly processed to obtain a joint feature map. Entropy coding is then performed on the joint feature map to obtain the bitstream of the video signal; or,
[0157] The second feature map of the first signal component and the second feature map of the second signal component processed by the neural network are jointly processed to obtain a joint feature map. Entropy coding is then performed on the joint feature map to obtain the bitstream of the video signal; or,
[0158] The second feature map of the first signal component and the second feature map of the second signal component, which have been processed by the neural network, are jointly processed to obtain a joint feature map. Entropy coding is then performed on the joint feature map to obtain the bitstream of the video signal; or,
[0159] The second feature map of the first signal component processed by the neural network and the second feature map of the second signal component processed by the neural network are jointly processed to obtain a joint feature map. The joint feature map is then entropy encoded to obtain the bitstream of the video signal.
[0160] The specific explanation is as follows:
[0161] When the control signal includes the control vector of the Y and UV feature maps: the rate allocation control module learns and generates N candidate first control signals for the first signal component (such as the control vector matrix {q}). y1 q y2 ,...q yi ...q yN}), the M candidate second control signals of the second signal component (such as the control vector matrix {q) uv1 q uv2 ,...q uvj ,...q uvM When in use, the control signal q of the first signal component is obtained according to the index i of the quality factor of the Y component. yi The control signal q of the second signal component is obtained based on the index j of the quality factor of the UV component. uvj Where N and M are integers greater than 1. Then, the control vector controls the feature maps of different components by multiplying them channel-by-channel with the corresponding feature maps. At this point, the video signal bitstream sent to the decoding end includes the index i of the quality factor of the Y component and the index j of the quality factor of the UV component.
[0162] When the control signal includes control vectors and offset vectors of Y and UV feature maps: as described above, the control vector q of the first signal component is obtained according to the index i of the quality factor of the Y component. yi Offset vector b yi The control vector q of the second signal component is obtained based on the index j of the quality factor of the UV component. uvj Offset vector b uvi Then, the control vector is multiplied channel by channel with the corresponding feature map and then added with the corresponding offset vector to control the feature map of different components.
[0163] In another embodiment, the control signals for the Y component and the UV component are used as a pair, and the bitrate allocation control module learns to generate N candidate control signals for the video signal (such as the control vector matrix {q}). c1 q c2 ,...q ci ...q cN}), at this time, c is 2, and each control vector q ci The signal includes both the control vector of the first signal component and the control component of the second signal component. Then, the control signal q, containing both the first and second signal components, is obtained through the index i of the video signal's quality factor. ci The offset vector is implemented similarly, meaning that each offset vector of the video signal includes both the offset components of the first and second signal components. The bitstream of the video signal sent to the decoding end then includes the index i of the quality factor of the video signal.
[0164] In another embodiment, the Y component quality factor and the UV component quality factor are used as inputs to a fully connected network, and the output is a control vector q. yi Control vector q uvj By multiplying the control vector channel-by-channel with the corresponding feature map, the control of different component feature maps is achieved. The Y-component quality factor and UV-component quality factor serve as inputs to the fully connected network and can also output an offset vector b. yi Offset vector b uvj Then, by multiplying the control vector with the corresponding feature map channel by channel and adding the offset vector with the corresponding feature map channel by channel, the control of different component feature maps is achieved. At this time, the bitstream of the video signal sent to the decoding end includes the quality factors of the Y component and the UV components.
[0165] The decoding end performs entropy decoding on the received bitstream to obtain a feature map, which is decomposed into a Y component feature map and a UV component feature map. Optionally, the feature map obtained from entropy decoding is first input into the joint processing submodule 2 to obtain the Y component feature map and the UV component feature map.
[0166] The Y-component feature map and UV-component feature map are input to Y-component processing module 2, and UV-component processing module 2 outputs the Y-component reconstructed map and UV-component reconstructed map, respectively. Optionally, the Y-component quality factor and UV-component quality factor are input to the quality response module. The response signal output by this module is applied to the Y-component feature map of the Y-component processing module and the UV-component feature map of the UV-component processing module, respectively, thereby achieving adaptive quality response for the Y and UV components. Quality response can also be called quality control, but for the purpose of distinguishing it from the quality control at the encoding end, it is called quality response at the decoding end.
[0167] Specifically, with Figure 7C For example, in step 711, the decoding end obtains the bitstream of the video signal from the encoding end, and performs entropy decoding on the bitstream to obtain the feature map of the first signal component (such as the Y component) and the feature map of the second signal component (such as the UV component) of the video signal.
[0168] The decoding end also obtains the quality factor information of the first signal component and the quality factor information of the second signal component from the bitstream. The quality factor information of the first signal component is either the quality factor of the first signal component or its index, and the quality factor information of the second signal component is either the quality factor of the second signal component or its index. Then, the response signal of the first signal component is obtained using the quality factor information of the first signal component; the response signal of the second signal component is obtained using the quality factor information of the second signal component. When the quality factor information of the first signal component is the quality factor of the first signal component, the quality factor of the first signal component takes one of N values; when the quality factor information of the first signal component is the index of the quality factor of the first signal component, the index of the quality factor of the first signal component ranges from 0 to N-1 or from 1 to N, where N is an integer greater than 1. Similarly, when the quality factor information of the second signal component is the quality factor of the second signal component, the quality factor of the second signal component takes one of M values; when the quality factor information of the second signal component is the index of the quality factor of the second signal component, the index of the quality factor of the second signal component takes a value range of 0 to M-1 or 1 to M, where M is an integer greater than 1.
[0169] When the encoding end transmits a joint feature map, the decoding end also needs to perform entropy decoding on the joint feature map and process it through a neural network to obtain the feature map of the first signal component and the feature map of the second signal component.
[0170] Step 712: Obtain the response signal of the first signal component using the quality factor information of the first signal component. Step 713: Obtain the response signal of the second signal component using the quality factor information of the second signal component.
[0171] In one embodiment, if the bitstream includes the index i of the quality factor of the Y component and the index j of the quality factor of the UV component, then the decoding end needs to generate the response signal matrix {g} of the first signal component through learning. y1 g y2 , ...g yi ...g yN}, and the response signal matrix {g} of the second signal component uv1 g uv2 , ...g uvj , ...g uvM}, where N and M are integers greater than 1; optionally, the control signal matrix {q} of the first signal component at the encoding end is used. y1 q y2 ,...q yi ...q yN Taking the reciprocal yields the response signal matrix {g} of the first signal component. y1 g y2 , ...g yi ...g yN}, the control signal matrix {q} of the first signal component at the encoding end uv1 q uv2 ,...q uvi ...q uvN Taking the reciprocal yields the response signal matrix {g} of the second signal component. uv1 g uv2 , ...g uvj , ...g uvM}, where N and M are integers greater than 1; the response signal g of the first signal component is obtained according to the index i of the quality factor of the Y component. yi The response signal g of the second signal component is obtained based on the index j of the quality factor of the UV component. uvj .
[0172] In another embodiment, if the bitstream includes an index i of the quality factor of the video signal, the decoder needs to learn and generate the response signal matrix {g} of the video signal. c1 g c2 , ...g ci ...g cN}, where c is 2 representing the Y and UV components, and N is an integer greater than 1; optionally, the control signal matrix {q} of the video signal at the encoding end is used. c1 q c2 ,...q ci ...q cN Taking the reciprocal, we obtain the response signal matrix {g} of the video signal. c1 g c2 , ...g ci ...g cN}, where c is 2 representing the Y and UV components, and N is an integer greater than 1; the response signal g containing the first signal component and the second signal component is obtained according to the index i of the quality factor of the video signal. ci .
[0173] In another embodiment, if the bitstream includes the quality factor of the first signal component and the quality factor of the second signal component, then the decoding end uses the quality factor of the Y component as the input of a fully connected network and outputs the response signal of the Y component; and uses the quality factor of the UV component as the input of a fully connected network and outputs the response signal of the UV component.
[0174] Step 714: Obtain a reconstructed image of the first signal component based on the response signal of the first signal component and the feature map of the first signal component. Step 715: Obtain a reconstructed image of the second signal component based on the response signal of the second signal component and the feature map of the second signal component.
[0175] When the response signal includes a response vector, obtaining the reconstructed map of the first signal component based on the response signal of the first signal component and the feature map of the first signal component includes:
[0176] The reconstructed image of the first signal component is obtained by multiplying the response vector of the first signal component with the feature map of the first signal component; or the reconstructed image of the first signal component is obtained by multiplying the response vector of the first signal component with the feature map of the first signal component and then processing it through a neural network.
[0177] Based on the response signal of the second signal component and the feature map of the second signal component, the reconstructed map of the second signal component is obtained, including:
[0178] The response vector of the second signal component is multiplied by the feature map of the second signal component to obtain the reconstructed map of the second signal component; and the response vector of the second signal component is multiplied by the feature map of the second signal component and then processed by a neural network to obtain the reconstructed map of the second signal component.
[0179] When the response signal includes a response vector and an offset vector, obtaining the reconstructed map of the first signal component based on the response signal of the first signal component and the feature map of the first signal component includes:
[0180] The reconstructed image of the first signal component is obtained by multiplying the response vector of the first signal component with the feature map of the first signal component and then adding it to the offset vector of the first signal component; or by multiplying the response vector of the first signal component with the feature map of the first signal component and then adding it to the offset vector of the first signal component, and then processing it through a neural network.
[0181] Based on the response signal of the second signal component and the feature map of the second signal component, the reconstructed map of the second signal component is obtained, including:
[0182] The response vector of the second signal component is multiplied by the feature map of the second signal component, and then added to the offset vector of the second signal component to obtain the reconstructed map of the second signal component; or the response vector of the second signal component is multiplied by the feature map of the second signal component, and then added to the offset vector of the second signal component, and then processed by a neural network to obtain the reconstructed map of the second signal component.
[0183] Step 716: Reconstruct the video signal based on the reconstruction map of the first signal component and the reconstruction map of the second signal component.
[0184] like Figure 7D As shown, the AI image decoding system 710 for YUV bitrate allocation provided in this embodiment of the invention can be used in video image encoders and decoders. Figure 7D As shown, the image decoding system 710 includes a first signal component processing module (e.g., a Y component processing module), a second signal component processing module (e.g., a U component processing module), a third signal component processing module (e.g., a V component processing module), a code rate allocation control module, and an entropy coding module at the encoding end; and a first signal component processing module (Y component processing module 2), a second signal component processing module (e.g., a U component processing module 2), a third signal component processing module (e.g., a V component processing module 2), and an entropy decoding module at the decoding end. The image decoding system 710 optionally includes a joint processing module, a joint processing module 2, and a quality response module (also called a code rate allocation control module 2 or a code rate control module 2). The Y component quality factor, U component quality factor, and V component quality factor in the image decoding system 710 are input to the code rate allocation control module. This module outputs control signals that act on the Y component feature map of the Y component processing module, the U component feature map of the U component processing module, and the V component feature map of the V component processing module (which can be called the first feature map), respectively, and outputs the second feature map of each signal component, thereby realizing the code rate allocation of Y, U, and V. Subsequently, the video signal bitstream is obtained based on the second feature maps of each signal component. For example, the feature maps output by the Y component processing module, U component processing module, and V component processing module can be directly concatenated together, or the feature maps output by the Y component processing module, U component processing module, and V component processing module can be directly added together to form the final output feature map of the encoder, which is then entropy encoded. Optionally, the feature maps output by the Y component processing module, U component processing module, and V component processing module can be input into a joint processing module to obtain the final output feature map of the encoder, which is then entropy encoded.
[0185] Figure 7D The illustrated embodiment can obtain a control signal for the first signal component (Y component) from N candidate first control signals based on the quality factor of the first signal component (Y component); obtain a control signal for the second signal component (U component) from M candidate second control signals based on the quality factor of the second signal component (U component); and obtain a control signal for the third signal component (V component) from L candidate third control signals based on the quality factor of the third signal component (V component). N, M, and L are integers greater than 1 and may be equal or unequal; this application does not limit their equality.
[0186] exist Figure 7D In the architecture shown, the encoding and decoding methods are... Figure 7B and 7C Similarly, the explanation is as follows:
[0187] For example, in one embodiment, the control signal is generated by network learning and acts on the feature map (which can be called the first feature map) of at least one layer of the network in each of the Y component processing module, U component processing module, and V component processing module, and then outputs the second feature map. For example, it acts on the output of the last layer of the network. Since the control signal can act on the first feature map of any layer of the network in each of the Y component processing module, U component processing module, and V component processing module, after outputting the second feature map, neural network processing can continue to be performed on the second feature map. Accordingly, obtaining the bitstream of the video signal based on the second feature map of the Y component, the second feature map of the U component, and the second feature map of the V component includes:
[0188] Entropy coding is performed on the second feature map of the first signal component, the second feature map of the second signal component, and the second feature map of the third signal component to obtain the bitstream of the video signal; or,
[0189] Entropy coding is performed on the second feature map of the first signal component, the second feature map of the second signal component processed by the neural network, and the second feature map of the third signal component to obtain the bitstream of the video signal; or,
[0190] Entropy coding is performed on the second feature map of the first signal component, the second feature map of the second signal component processed by the neural network, and the second feature map of the third signal component processed by the neural network to obtain the bitstream of the video signal; or,
[0191] Entropy coding is performed on the second feature map of the first signal component, the second feature map of the second signal component, and the second feature map of the third signal component processed by the neural network to obtain the bitstream of the video signal; or,
[0192] Entropy coding is performed on the second feature map of the first signal component, the second feature map of the second signal component, and the second feature map of the third signal component, which have been processed by the neural network, to obtain the bitstream of the video signal; or,
[0193] Entropy coding is performed on the second feature map of the first signal component processed by the neural network, the second feature map of the second signal component processed by the neural network, and the second feature map of the third signal component to obtain the bitstream of the video signal; or
[0194] Entropy coding is performed on the second feature map of the first signal component processed by the neural network, the second feature map of the second signal component processed by the neural network, and the second feature map of the third signal component processed by the neural network to obtain the bitstream of the video signal; or,
[0195] Entropy coding is performed on the second feature map of the first signal component processed by the neural network, the second feature map of the second signal component, and the second feature map of the third signal component processed by the neural network to obtain the bitstream of the video signal.
[0196] When the image decoding system 710 includes a joint processing module, it can also perform joint processing on the aforementioned obtained second feature map or the processed feature map and combination thereof to obtain a joint feature map, and perform entropy coding on the joint feature map to obtain the bitstream of the video signal.
[0197] The specific explanation is as follows:
[0198] When the control signal includes the control vector of the Y, U, and V feature maps: the code rate allocation control module learns and generates N candidate first control signals for the first signal component (such as the control vector matrix {q}). y1 q y2 ,...q yi ...q yN}), the M candidate second control signals of the second signal component (such as the control vector matrix {q) u1 q u2 ,...q uj ..., q uM}) and the L candidate third control signals (such as the control vector matrix {q) of the third signal component v1 q v2 , ..., q vk ...q vL When in use, the control signal q of the first signal component is obtained according to the index i of the Y component quality factor. yiThe control signal q of the second signal component is obtained based on the index j of the quality factor of the U component. uj ; and the control signal q of the third signal component is obtained according to the index k of the quality factor of the V component. vk Then, the control vector controls the feature maps of different components by multiplying them channel by channel with the corresponding feature maps. At this time, the bitstream of the video signal sent to the decoding end includes the index i of the quality factor of the Y component, the index j of the quality factor of the U component, and the index k of the quality factor of the V component.
[0199] When the control signal includes the control vector and offset vector of the Y, U, and V feature maps: as described above, the control vector q of the first signal component is obtained according to the index i of the Y component quality factor. yi Offset vector b yi The control vector q of the second signal component is obtained based on the index j of the quality factor of the U component. uj Offset vector b ui ; and obtain the control vector q of the third signal component based on the index k of the quality factor of the V component. vk Offset vector b vi Then, the control vector is multiplied channel by channel with the corresponding feature map and then added with the corresponding offset vector to control the feature map of different components.
[0200] In another embodiment, the control signals of the Y component, U component, and V component are used as a triplet, and the bitrate allocation control module learns to generate N candidate control signals of the video signal (such as the control vector matrix {q}). c1 q c2 ,...q ci ...q cN At this point, c is 3, and each control vector q ci The signal includes the control vector of the first signal component, the control vector of the second signal component, and the control vector of the third signal component. Then, the control signal q, containing the first, second, and third signal components, is obtained through the index i of the video signal's quality factor. ci The offset vector is implemented similarly, meaning that each offset vector of the video signal includes offset components of the first signal component, the second signal component, and the third signal component. The bitstream of the video signal sent to the decoding end then includes the index i of the quality factor of the video signal.
[0201] In another embodiment, the Y-component quality factor, U-component quality factor, and V-component quality factor are used as inputs to a fully connected network, and the output is a control vector q. yiControl vector q uj and control vector q vk By multiplying the control vector channel-by-channel with the corresponding feature map, the control of different component feature maps is achieved. The Y-component quality factor, U-component quality factor, and V-component quality factor serve as inputs to the fully connected network and can also output an offset vector b. yi Offset vector b uj and offset vector b vk Then, by multiplying the control vector with the corresponding feature map channel by channel, and adding the offset vector with the corresponding feature map channel by channel, the control of different component feature maps is achieved. At this time, the video signal bitstream sent to the decoding end includes the quality factors of the Y component, the U component, and the V component.
[0202] The decoding end performs entropy decoding on the received bitstream to obtain a feature map, which is decomposed into Y component feature maps, U component feature maps, and V component feature maps. Optionally, the feature map obtained from entropy decoding is first input into the joint processing submodule 2 to obtain the Y component feature map, U component feature map, and V component feature map.
[0203] The Y component feature map, U component feature map, and V component feature map are respectively input into Y component processing module 2, U component feature map 3, and V component feature map 4. Figure 2 The V component processing module 2 outputs reconstructed Y-component, U-component, and V-component maps. Optionally, the Y-component quality factor, U-component quality factor, and V-component quality factor are input to the quality response module. This module outputs control signals that act on the Y-component feature maps of the Y-component processing module, the U-component feature maps of the U-component processing module, and the V-component feature maps of the V-component processing module, respectively, thereby achieving adaptive quality response for the Y, U, and V components.
[0204] The response signal is generated in a similar way to the control signal, but for ease of distinction, the encoding end is called the control signal and the decoding end is called the response signal.
[0205] Specifically, the decoding end obtains the video signal bitstream from the encoding end; performs entropy decoding on the bitstream to obtain feature maps of a first signal component (e.g., Y component), a second signal component (e.g., U component), and a third signal component (e.g., V component); obtains a reconstructed map of the first signal component based on the response signal of the first signal component and its feature map, wherein the response signal of the first signal component is obtained through learning; obtains a reconstructed map of the second signal component based on the response signal of the second signal component and its feature map, wherein the response signal of the second signal component is obtained through learning; obtains a reconstructed map of the third signal component based on the response signal of the third signal component and its feature map, wherein the response signal of the third signal component is obtained through learning; and reconstructs the video signal based on the reconstructed maps of the first, second, and third signal components.
[0206] The decoding end also obtains the quality factor information of the first signal component, the second signal component, and the third signal component from the bitstream. The quality factor information of the first and second signal components is... Figure 7A The implementation is similar. Similarly, the quality factor information of the third signal component can also be the quality factor of the third signal component or an index of the quality factor of the third signal component. Then, the response signal of the third signal component is obtained through the quality factor information of the third signal component. When the quality factor information of the third signal component is the quality factor of the third signal component, the quality factor of the third signal component takes one of L values; when the quality factor information of the third signal component is an index of the quality factor of the third signal component, the index of the quality factor of the first signal component ranges from 0 to L-1 or from 1 to L, where L is an integer greater than 1. L, M, and N can be equal or unequal, and this application does not limit this.
[0207] When the encoding end transmits a joint feature map, the decoding end also needs to perform entropy decoding on the joint feature map and process it through a neural network to obtain the feature map of the first signal component, the feature map of the second signal component, and the feature map of the third signal component.
[0208] In one embodiment, if the bitstream includes the index i of the quality factor of the Y component, the index j of the quality factor of the U component, and the index k of the quality factor of the V component, then the decoding end needs to generate the response signal matrix {g} of the first signal component through learning. y1 gy2 , ...g yi ...g yN}, the response signal matrix of the second signal component {g u1 g u2 , ...g uj ..., g uM The response signal matrix of the third signal component {g} v1 g v2 , ..., g vk ...g vL}, where N, M, and L are integers greater than 1; optionally, the control signal matrix {q} of the first signal component at the encoding end is used. y1 q y2 ,...q yi ...q yN Taking the reciprocal yields the response signal matrix {g} of the first signal component. y1 g y2 , ...g yi ...g yN}, the control signal matrix {q} of the second signal component at the encoding end u1 q u2 ,...q ui ...q uM Taking the reciprocal yields the response signal matrix {g} of the second signal component. u1 g u2 , ...g uj , ...g uM}, and the control signal matrix {q} of the third signal component at the encoding end. v1 q v2 ,...q vi ...q vL} Taking the reciprocal yields the response signal matrix {g} of the third signal component. v1 g v2 , ..., g vk ...g vL}, where N, M, and L are integers greater than 1; the response signal g of the first signal component is obtained according to the index i of the quality factor of the Y component. yi The response signal g of the second signal component is obtained based on the index j of the quality factor of the U component. uj ; and obtain the response signal g of the third signal component according to the index k of the quality factor of the V component. vk .
[0209] In another embodiment, if the bitstream includes an index i of the quality factor of the video signal, the decoder needs to learn and generate the response signal matrix {g} of the video signal.c1 g c2 , ...g ci ...g cN}, where c is 3 representing the Y, U, and V components, and N is an integer greater than 1; optionally, the control signal matrix {q} of the video signal at the encoding end is used. c1 q c2 ,...q ci ...q cN Taking the reciprocal, we obtain the response signal matrix {g} of the video signal. c1 g c2 , ...g ci ...g cN}, where c is 3 representing the Y component, U component, and V component, and N is an integer greater than 1; based on the index i of the quality factor of the video signal, a response signal g containing the first signal component, the second signal component, and the third signal component is obtained. ci .
[0210] In another embodiment, if the bitstream includes the quality factors of the first signal component, the second signal component, and the third signal component, then the decoding end uses the quality factor of the Y component as the input of a fully connected network and outputs the response signal of the Y component; uses the quality factor of the U component as the input of a fully connected network and outputs the response signal of the U component; and uses the quality factor of the V component as the input of a fully connected network and outputs the response signal of the V component.
[0211] The reconstructed images of the first and second signal components are as follows: Figure 7A Similarly, this will not be elaborated upon here.
[0212] In one embodiment, when the response signal includes a response vector, obtaining a reconstructed map of the third signal component based on the response signal of the third signal component and the feature map of the third signal component includes:
[0213] The reconstructed image of the third signal component is obtained by multiplying the response vector of the third signal component with the feature map of the third signal component; or the reconstructed image of the third signal component is obtained by multiplying the response vector of the third signal component with the feature map of the third signal component and then processing it through a neural network.
[0214] In another embodiment, when the response signal includes a response vector and an offset vector, obtaining the reconstructed map of the third signal component based on the response signal of the third signal component and the feature map of the third signal component includes:
[0215] The reconstructed image of the third signal component is obtained by multiplying the response vector of the third signal component with the feature map of the third signal component and then adding it to the offset vector of the third signal component; or by multiplying the response vector of the third signal component with the feature map of the third signal component and then adding it to the offset vector of the third signal component, and then processing it through a neural network.
[0216] Example 7A combines and merges the UV components, while 7D processes the Y, U, and V components individually. Alternatively, they can be combined and merged, for example, into two components, Y and UV, or other combinations of Y, U, and V.
[0217] Figure 8A and 8B A specific embodiment is given. The overall block diagram of the technical solution of this embodiment is as follows: Figure 8A As shown, the Y component quality factor and UV component quality factor are input to the bitrate allocation control module, which outputs a control vector q. yi Control vector q uvi The bitrate allocation of YUV components is achieved by applying the Y component feature map output by the Y component processing module and the UV component feature map output by the UV component processing module, respectively.
[0218] The decoder inputs the Y component quality factor and UV component quality factor to the quality response module, which outputs the control vector g. yi Control vector g uvi It is applied to the Y component feature map and the UV component feature map respectively, thereby realizing the respective quality gain responses of the Y and UV components.
[0219] This embodiment does not impose constraints on the specific network structures of the Y component processing module, UV component processing module, joint processing module, probability estimation module, Y component processing module 2, UV component processing module 2, and joint processing module 2, for ease of understanding. Figure 8B Provide a specific example.
[0220] The first step is to obtain the feature maps of the Y and UV components:
[0221] The Y and UV components are input into the Y component processing module and the UV component processing module, respectively, and the network outputs feature maps of the Y and UV components. Figure 8BFor example, the Y component processing module contains two convolutional layers and two nonlinear layers. The horizontal and vertical downsampling factors in both convolutional layers are both 2. The Y component processing module outputs a Y component feature map. The UV component processing module contains two convolutional layers and two nonlinear layers. The first convolutional layer has a horizontal and vertical downsampling factor of 1, meaning no downsampling is performed. The second convolutional layer in the UV component processing module has a horizontal and vertical downsampling factor of 2. The UV component processing module outputs a UV component feature map. After the above network processing, for the YUV420 data format, the Y component feature map and the UV component feature map have the same width and height.
[0222] Similar to the processing method for YUV420 data format, for data formats such as YUV444 and YUV422, the width and height of the Y component feature map are made the same as those of the UV component feature map by controlling the number of convolutional layers and the downsampling factors in the horizontal and vertical directions.
[0223] The second step involves inputting the Y component quality factor and UV component quality factor into the rate allocation module to obtain the control vector q. yi Control vector q uvi Control vector q yi Control vector q uvi The feature maps of the Y and UV components are multiplied channel by channel to obtain the processed feature maps of the Y and UV components. The processed feature maps of the Y and UV components are added together or concatenated, and then input into the joint processing module to output the feature map to be encoded.
[0224] The bitrate allocation module consists of the control matrix Q. y and Q uv Composition, with the Y component quality factor and UV component quality factor as the control matrix Q y and Q uv The index value from Q y and Q uv The index yields the control vector q. yi Control vector q uvi Among them, the control matrix Q y and Q uv This was obtained through online learning. The Y component quality factor and UV component quality factor are arbitrary values that can be set.
[0225] by Figure 8B For example, the control matrix Q y Given a two-dimensional matrix of size K x N, the control matrix Q uvLet K be a two-dimensional matrix of size L x M, where each element of the two matrices represents a parameter that the network can learn. Here, K represents the number of Y component feature maps, L represents the number of UV component feature maps, N represents N sets of candidate Y component quality factors, and M represents M sets of candidate UV component quality factors. For example, with N = 4 and M = 4, the possible values for the Y component quality factor are {0.5, 0.7, 0.8, 1.0}, and the possible values for the UV component quality factor are {0.15, 0.2, 0.25, 0.3}.
[0226] The third step involves inputting the feature map to be encoded into the entropy encoding module, which then outputs a bitstream. Figure 8B For example, the feature map to be encoded is input into the Hyper Entropy module, which outputs the probability distribution of the symbol to be encoded. Arithmetic encoding is performed based on the probability distribution of the symbol to be encoded, and the bitstream is output. Simultaneously, the Y component quality factor and UV component quality factor information are written into the bitstream.
[0227] The Y component quality factor and UV component quality factor information can be expressed and written into the bitstream in the following three ways:
[0228] Option 1: Predefine the number and values of candidate quality factors for the Y and UV components, and pass the index numbers of the Y and UV component quality factors in their respective candidate lists to the decoder. For example, with N = 4 and M = 3, the selectable values for the Y component quality factor are {0.5, 0.7, 0.8, 1.0}, and the selectable values for the UV component quality factor are {0.15, 0.2, 0.25}. Write the Y component index number i and the UV component index number j into the bitstream. The values of i and j are 0, 1, 2, and 3. When i is 1, it indicates that the Y component quality factor is 0.7; when j is 0, it indicates that the UV component quality factor is 0.15.
[0229] Option 2: Predefine the number of candidate values and the candidate values for the combination of Y and UV component quality factors. For example, the number of candidate values for the combination of Y and UV component quality factors is 6, and the candidate list is {(0.5, 0.25), (0.7, 0.15)}.
[0230] (0.7,0.25), (0.8,0.1), (0.8,0.2), (1.0,0.2)}, write the index number i into the bit stream, i takes the value 0, 1, 2, 3, 4, 5. When i is 1, it means that the quality factor of Y, U, V components is (0.7,0.15).
[0231] Option 3: Directly write the Y component quality factor and UV component quality factor into the bitstream and pass it to the decoding end. For example, write (1.0, 0.2) into the bitstream.
[0232] The fourth step involves inputting the bitstream into the entropy decoding module, where arithmetic decoding is performed to obtain the feature map and the quality factors of the Y and UV components. Figure 8B For example, arithmetic decoding is performed based on the probability distribution estimated by the Hyper Entropy module.
[0233] The fifth step involves decoding the feature map and inputting it into the joint processing module 2. The module outputs a feature map with M channels, which is then split into a Y component feature map with K channels and a UV component feature map with L channels. The splitting scheme ensures that K ≤ M and L ≤ M. Specifically, when K = L = M, it means that the Y classification feature map and the UV component feature map are identical, both being feature maps with M channels. Figure 8B For example, the joint processing module 2 contains two convolutional layers and one nonlinear layer.
[0234] Step 6: Input the Y component quality factor and UV component quality factor into the quality response module to obtain the response vector g. yi Response vector g uvi Response vector g yi Response vector g uvi The Y and UV component feature maps are multiplied channel by channel to obtain the Y and UV component feature maps after quality gain. The Y and UV component feature maps after quality gain are input into Y component processing module 2 and UV component processing module 2, respectively, and output Y component reconstruction map and UV component reconstruction map.
[0235] The quality response module consists of the response matrix G y and G uv The solution is composed of the decoded Y component quality factor and UV component quality factor as the response matrix G. y and G uv The index value from G y and G uv The index yields the response vector g. yi Response vector g uvi .
[0236] Wherein, the response matrix G y and G uv Learned through online resources. Figure 8B For example, the response matrix G y Given a two-dimensional matrix of size K x N, the response matrix G uv Let K be a two-dimensional matrix of size L x M, where each element of the two matrices represents a parameter that the network can learn. Here, K represents the number of Y component feature maps, L represents the number of UV component feature maps, N represents N sets of candidate Y component quality factors, and M represents M sets of candidate UV component quality factors.
[0237] Optionally, for the control matrix Q yand Q uv Taking the reciprocals of each gives the response matrix G. y and G uv .
[0238] This embodiment trains and learns the network modules and control matrix parameters involved in steps 1 to 7 above. Specifically, this application uses the Adaptive Moment Estimation (Adam) optimization algorithm to optimize and train the neural network, and the ImageNet dataset is used as the training dataset. Since this network structure is geared towards image encoding, the training optimization objective is to minimize the bitrate-distortion joint loss function, the expression of which is:
[0239]
[0240]
[0241] Where p(y) represents the probability distribution estimated by the probability estimator, x y The original values of the Y component, x y ′ is the reconstructed image of the Y component, x u x is the original value of the U component. u ′ is the U component reconstruction image, x v x represents the original value of the V component. v ′ is the V component reconstruction image, w y w is the quality factor of the Y component. u w is the quality factor of the U component. v λ is the quality factor for the V component. λ is a constant, matching the target bit rate.
[0242] Taking N=4 as an example, the quality factors (w) of the Y, U, and V components y w u w v The possible values are {(0.5,0.25,0.25), (0.7,0.4,0.4), (0.8,0.1,0.1), (1.0,0.2,0.2)}. During network training, the weight regroup index i is randomly selected from {0,1,2,3}, and the weight value group (w) is determined based on i. yi w ui w vi ) and the control vector q to be learned yi Control vector q uvi The network modules and control matrix parameters are trained and learned according to the optimization objective.
[0243] When the joint processing module or / and joint processing module 2 or / and quality response module are removed from the codec mentioned in this embodiment, it still applies to other embodiments of this application.
[0244] This embodiment provides a technical solution when the Y, U, and V components are combined into two components, Y and UV. The solution approach of this application is still applicable to other combinations of Y, U, and V, such as {YU, V} and {YV, U}.
[0245] Similarly, the solution proposed in this application remains applicable by further decomposing the UV components into U and V components for separate processing.
[0246] In existing end-to-end image coding, when optimizing a specific network, the Y, U, and V component weights are fixed, resulting in a fixed bitrate for the YUV components. Since different images have different color characteristics, a fixed bitrate allocation leads to poor coding performance for some video image content. While it's possible to achieve different bitrate allocations for the YUV components by training multiple models with different Y, U, and V component weights, this increases the number of models and consumes significant computational resources and time. Compared to existing technologies, this application derives control vectors learned by the network based on the Y, U, and V component weights. These control vectors are then used to apply different degrees of distortion control to the feature maps of the Y and UV components, achieving bitrate allocation for the Y and UV components. Therefore, this application has the following advantages:
[0247] 1) Supports bitrate allocation between YUV components to adapt to image content with different color characteristics.
[0248] 2) Reduce the time spent training multiple models, and also reduce the number of new network parameters added to the models.
[0249] Figure 9A and 9B A specific implementation example is given. Figure 9A and 9B Based on the previous embodiment, this embodiment uses a U-component processing module and a V-component processing module to process the U-component and V-component data respectively. In this embodiment, the Y-component quality factor, U-component quality factor, and V-component quality factor are used as inputs to the rate allocation control module, and the output control signal processes any layer of feature maps in the Y-component processing module, U-component processing module, and V-component processing module to achieve rate allocation among the Y, U, and V components. At the decoding end, the Y-component quality factor, U-component quality factor, and V-component quality factor are used as inputs to the quality response module, and the output control signal performs quality gain response on any layer of feature maps in the Y-component processing module, U-component processing module, and V-component processing module. This application does not impose constraints on the specific network structure of the rate allocation control module, quality response module, Y-component processing module, U-component processing module, joint processing module, Y-component processing module 2, U-component processing module 2, V-component processing module 2, joint processing module 2, entropy coding module, and entropy decoding module, for ease of understanding. Figure 9A Provide a specific example.
[0250] by Figure 9B For example, in the first step, the Y component quality factor, U component quality factor, and V component quality factor are input to the code rate allocation control module. This module consists of a fully connected network and outputs control signals: control vector and offset vector.
[0251] The second step involves inputting the Y, U, and V signals to be encoded into the Y component processing module, U component processing module, and V component processing module, respectively. Taking the Y component processing module as an example, the feature map output by each convolutional layer in this module is multiplied channel-by-channel by its corresponding control vector, and then added channel-by-channel by its corresponding offset vector. The feature maps output by the nonlinear layers in the module are multiplied channel-by-channel by their corresponding control vectors. The output of each layer in the Y component processing module is processed by the control signal from the rate control module. The processing methods for the U and V components are similar to those for the Y component.
[0252] Figure 9B A schematic diagram of a specific network structure is given. The network structures of the Y-component processing module, U-component processing module, and V-component processing module are identical except for the first convolutional layer. For YUV422 format, the horizontal and vertical downsampling factors of the first convolutional layer in the Y-component processing module are both 2. In the U-component and V-component processing modules, the horizontal downsampling factor of the first convolutional layer is 1 (no downsampling operation), and the vertical downsampling factor is 2. For YUV420 format, the horizontal and vertical downsampling factors of the first convolutional layer in the Y-component processing module are both 2. In the U-component and V-component processing modules, the horizontal and vertical downsampling factors of the first convolutional layer are both 1 (no downsampling operation).
[0253] The third step involves concatenating and stitching the Y-component feature maps, U-component feature maps, and V-component feature maps together to form the feature map to be encoded. This map is then input into the entropy encoding module, and the output is a bitstream. Figure 9B For example, the feature map to be encoded is input into the Hyper Entropy module, which outputs the probability distribution of the symbol to be encoded. Arithmetic encoding is performed based on the probability distribution of the symbol to be encoded, and the bitstream is output. Simultaneously, the Y component quality factor, U component quality factor, and U component quality factor information are written into the bitstream.
[0254] The fourth step involves inputting the bitstream into the entropy decoding module, where arithmetic decoding is performed to obtain the feature map and the quality factors of the Y, U, and V components. Figure 9B For example, arithmetic decoding is performed based on the probability distribution estimated by the Hyper Entropy module.
[0255] The sixth step is to input the decoded feature map into the joint processing module 2, and output the feature map.
[0256] Step 7: Input the Y-component quality factor, U-component quality factor, and V-component quality factor into the quality response module to obtain the response vector g. yi Response vector g ui Response vector g vi Response vector g yi The feature map output from the second convolutional layer in Y component processing module 2 is multiplied channel by channel to obtain the feature map after quality gain. The processing of U and V components is similar. Y component processing module 2, U component processing module 2, and V component processing module 2 output the reconstructed maps of Y, U, and V components.
[0257] The Y-component quality factor, U-component quality factor, and V-component quality factor are input into the quality response module, which consists of a fully connected network. The module outputs a response vector g. yi Response vector g ui Response vector g vi .
[0258] Optionally, similar to the response vector acquisition method in Embodiment 1, the quality response module consists of the response matrix G. y G u and G v The decoded Y-component quality factors, U-component quality factors, and V-component quality factors are used as the response matrix G. y G u and G v The index value from G y G u and G v The index yields the response vector g. yi Response vector g ui Response vector g vi Among them, the response matrix G y G u and G v I learned this through the internet.
[0259] Network training process and Figure 8A and 8B The implementation examples are similar and will not be repeated here.
[0260] In this embodiment, the control signal acts on the output of each network layer in the Y component processing module, U component processing module, and V component processing module. Optionally, the control signal acts only on the output of some networks in the Y component processing module, U component processing module, and V component processing module.
[0261] In this embodiment, the response signal only applies to the output of the middle layer of the network in the Y component processing module, U component processing module, and V component processing module. Optionally, the control signal applies to the output of any one or more layers of the network in the Y component processing module, U component processing module, and V component processing module.
[0262] The technology of this application is still applicable when the codec removes the joint processing module 2 and / or the quality response module mentioned in this embodiment.
[0263] The technology of this application is still applicable when a joint processing module is added to the codec as mentioned in this embodiment.
[0264] This embodiment provides a technical solution for processing Y, U, and V as three components separately. The solution approach of this application is still applicable to other combinations of Y, U, and V, such as {YU, V}, {YV, U}, and {Y, UV}.
[0265] As described in the preceding embodiments, the quality factor input rate allocation control module for different YUV components in this application of the present invention outputs control signals that act on the feature maps of different components, thereby realizing the rate allocation of different components. The different components may refer to three components (Y, U, and V), two components (Y and UV), or other combinations of Y, U, and V.
[0266] Optionally, the control signal refers to the control vector q. i Based on the quality factors of different components, the network learns to obtain a weight matrix {q} with equal components. c1 q c2 , ..., q cN}, where c is 2 or 3 representing the number of different components, and N is the number of candidate quality factor values. In use, the control vector q corresponding to each component is obtained based on its quality factor index. ci .
[0267] Optionally, the control signals refer to the control vector q and the offset vector b. The quality factors of different components are used as inputs to the fully connected network, and the corresponding control vector q and offset vector b are output for different components.
[0268] Therefore, the embodiments provided in this application are capable of:
[0269] 1) Adapt to image content with different color characteristics and support bitrate allocation between YUV components through control vectors.
[0270] 2) Reduce the time spent training multiple models, and also reduce the number of new network parameters added to the models.
[0271] Figure 10This is a schematic diagram illustrating the structure of an encoding device 1000 according to an embodiment of this application. This encoding device can correspond to a video encoder 20. The encoding device 1000 includes a first control module 1001, a first control module 1002, and an encoding module 1003. The first control module 1001 is used to apply a control signal of a first signal component of the video signal to a first feature map of the first signal component to obtain a second feature map of the first signal component, wherein the control signal of the first signal component is obtained through learning. The second control module 1002 is used to apply a control signal of a second signal component of the video signal to the first feature map of the second signal component to obtain a second feature map of the second signal component, wherein the control signal of the second signal component is obtained through learning. The encoding module 1003 is used to obtain the bitstream of the video signal based on the second feature map of the first signal component and the second feature map of the second signal component. The encoding device 1000 may also include the bitrate allocation control module described in the preceding embodiments. The encoding device 1000 is used to implement the encoding method described in the foregoing embodiments; detailed functions are described in the preceding embodiments and will not be repeated here.
[0272] Figure 11 This is a schematic diagram illustrating the structure of a decoding apparatus 1100 according to an embodiment of this application. The decoding apparatus 1100 may correspond to a video decoder 30. The decoding apparatus 1100 includes a decoding module 1101, a first control module 1102, a first control module 1103, and a reconstruction module 1104. Specifically, the decoding module 1101 is used to obtain the bitstream of the video signal and perform entropy decoding on the bitstream to obtain feature maps of a first signal component and a second signal component of the video signal; the first control module 1102 is used to obtain a reconstructed map of the first signal component based on the response signal of the first signal component and the feature map of the first signal component, wherein the response signal of the first signal component is obtained through learning; the second control module 1103 is used to obtain a reconstructed map of the second signal component based on the response signal of the second signal component and the feature map of the second signal component, wherein the response signal of the second signal component is obtained through learning; and the reconstruction module 1104 is used to reconstruct the video signal based on the reconstructed map of the first signal component and the reconstructed map of the second signal component. The decoding apparatus 1100 may also include the quality response module described in the previous embodiments. The decoding device 1100 is used to implement the decoding method described in the foregoing embodiments. For detailed functions, please refer to the description of the previous embodiments, which will not be repeated here.
[0273] Those skilled in the art will appreciate that the functionality described in conjunction with the various illustrative logic blocks, modules, and algorithmic steps disclosed herein can be implemented in hardware, software, firmware, or any combination thereof. If implemented in software, the functionality described by the various illustrative logic blocks, modules, and steps can be stored or transmitted as one or more instructions or codes on a computer-readable medium and executed by a hardware-based processing unit. The computer-readable medium may comprise a computer-readable storage medium, which corresponds to a tangible medium, such as a data storage medium, or a communication medium that includes any medium facilitating the transfer of a computer program from one place to another (e.g., according to a communication protocol). In this way, the computer-readable medium may substantially correspond to (1) a non-transitory tangible computer-readable storage medium, or (2) a communication medium, such as a signal or carrier wave. The data storage medium may be any available medium accessible by one or more computers or one or more processors to retrieve instructions, code, and / or data structures for implementing the techniques described in this application. A computer program product may comprise a computer-readable medium.
[0274] By way of example and not limitation, such computer-readable storage media may include RAM, ROM, EEPROM, CD-ROM or other optical disc storage devices, magnetic disk storage devices or other magnetic storage devices, flash memory, or any other media that can be used to store desired program code in the form of instructions or data structures and is accessible by a computer. Furthermore, any connection is properly referred to as computer-readable media. For example, if instructions are transmitted from a website, server, or other remote source using coaxial cable, fiber optic cable, twisted pair, digital subscriber line (DSL), or wireless technologies such as infrared, radio, and microwave, then coaxial cable, fiber optic cable, twisted pair, DSL, or wireless technologies such as infrared, radio, and microwave are included in the definition of media. However, it should be understood that the computer-readable storage media and data storage media do not include connections, carrier waves, signals, or other temporary media, but are specifically addressed to non-temporary tangible storage media. As used herein, disks and optical discs include compact optical discs (CDs), laser optical discs, optical discs, digital versatile optical discs (DVDs), and Blu-ray discs, where disks typically reproduce data magnetically, while optical discs reproduce data optically using lasers. The combination of the above items should also be included in the scope of computer-readable media.
[0275] Instructions can be executed by one or more processors, such as digital signal processors (DSPs), general-purpose microprocessors, application-specific integrated circuits (ASICs), field-programmable arrays (FPGAs), or other equivalent integrated or discrete logic circuits. Therefore, the term "processor" as used herein can refer to any of the foregoing structures or any other structures suitable for implementing the techniques described herein. Furthermore, in some aspects, the functionality described in the various illustrative logic blocks, modules, and steps described herein can be provided within dedicated hardware and / or software modules configured for encoding and decoding, or incorporated into combined codecs. Moreover, the techniques can be fully implemented within one or more circuit or logic elements.
[0276] The technology of this application can be implemented in a wide variety of devices or apparatuses, including wireless handheld devices, integrated circuits (ICs), or a set of ICs (e.g., chipsets). The various components, modules, or units described in this application are intended to emphasize functional aspects of the apparatus for performing the disclosed technology, but do not necessarily need to be implemented by different hardware units. In fact, as described above, the various units can be combined with suitable software and / or firmware within a codec hardware unit, or provided via interoperable hardware units (containing one or more processors as described above).
[0277] The above description is merely an exemplary embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.
Claims
1. A method of encoding a video signal, characterized by, include: Based on the quality factor of the first signal component of the video signal, the control signal of the first signal component is obtained; Based on the quality factor of the second signal component of the video signal, the control signal of the second signal component is obtained; The control signal of the first signal component is applied to the first feature map of the first signal component to obtain the second feature map of the first signal component, wherein the control signal of the first signal component is obtained by learning. The control signal of the second signal component is applied to the first feature map of the second signal component to obtain the second feature map of the second signal component, wherein the control signal of the second signal component is obtained by learning. as well as The video signal bitstream is obtained based on the second feature map of the first signal component and the second feature map of the second signal component.
2. The method as described in claim 1, characterized in that, The step of obtaining the control signal for the first signal component based on the quality factor of the first signal component of the video signal includes: Based on the quality factor of the first signal component, the control signal of the first signal component is obtained from N candidate first control signals, where N is an integer greater than 1; The step of obtaining the control signal for the second signal component based on the quality factor of the second signal component of the video signal includes: The control signal of the second signal component is obtained from M candidate second control signals based on the quality factor of the second signal component, where M is an integer greater than 1.
3. The method of claim 1 or 2, wherein, Obtaining the bitstream of the video signal based on the second feature map of the first signal component and the second feature map of the second signal component includes: Entropy coding is performed on the second feature map of the first signal component and the second feature map of the second signal component to obtain the bitstream of the video signal; or, Entropy coding is performed on the second feature map of the first signal component and the second feature map of the second signal component processed by the neural network to obtain the bitstream of the video signal; or, Entropy coding is performed on the second feature map of the first signal component and the second feature map of the second signal component, which have been processed by the neural network, to obtain the bitstream of the video signal; or, Entropy coding is performed on the second feature map of the first signal component processed by the neural network and the second feature map of the second signal component processed by the neural network to obtain the bitstream of the video signal.
4. The method as described in claim 1 or 2, characterized in that, Obtaining the bitstream of the video signal based on the second feature map of the first signal component and the second feature map of the second signal component includes: The second feature map of the first signal component and the second feature map of the second signal component are jointly processed to obtain a joint feature map. Entropy coding is then performed on the joint feature map to obtain the bitstream of the video signal; or, The second feature map of the first signal component and the second feature map of the second signal component processed by the neural network are jointly processed to obtain a joint feature map. Entropy coding is then performed on the joint feature map to obtain the bitstream of the video signal; or, The second feature map of the first signal component and the second feature map of the second signal component, which have been processed by the neural network, are jointly processed to obtain a joint feature map. Entropy coding is then performed on the joint feature map to obtain the bitstream of the video signal; or, The second feature map of the first signal component processed by the neural network and the second feature map of the second signal component processed by the neural network are jointly processed to obtain a joint feature map. The joint feature map is then entropy encoded to obtain the bitstream of the video signal.
5. The method according to any one of claims 1-4, characterized in that, The first signal component is the Y component, and the second signal component is the UV component, U component, or V component.
6. The method as described in claim 5, characterized in that, When the second signal component is a UV component, the method includes: The quality factor of the Y component is used as the input of a fully connected network, and the control signal of the Y component is output. The quality factor of the UV component is used as the input to a fully connected network, and the control signal of the UV component is output.
7. The method as described in claim 6, characterized in that, The video signal bitstream includes the quality factor of the Y component and the quality factor of the UV component.
8. The method as described in claim 5, characterized in that, When the second signal component is a U component or a V component, the method further includes: The control signal of the third signal component of the video signal is applied to the first feature map of the third signal component to obtain the second feature map of the third signal component. The control signal of the third signal component is obtained by learning. When the second signal component is a U component, the third signal component is a V component, or when the second signal component is a V component, the third signal component is a U component.
9. The method as described in claim 8, characterized in that, The method includes: The quality factor of the Y component is used as the input of the fully connected network, and the control signal of the Y component is output. The quality factor of the U component is used as the input to the fully connected network, and the control signal of the U component is output; and The quality factor of the V component is used as the input of the fully connected network, and the control signal of the V component is output.
10. The method as described in claim 9, characterized in that, The video signal bitstream includes the quality factor of the Y component, the quality factor of the U component, and the quality factor of the V component.
11. The method according to any one of claims 1-10, characterized in that, When the control signal includes a control vector, the method includes: The control vector of the first signal component is multiplied by the first feature map of the first signal component to obtain the second feature map of the first signal component. The control vector of the second signal component is multiplied by the second feature map of the first signal component to obtain the second feature map of the second signal component.
12. The method according to any one of claims 1-10, characterized in that, When the control signal includes a control vector and an offset vector, the method includes: The control vector of the first signal component is multiplied by the first feature map of the first signal component, and then the offset vector of the first signal component is added to obtain the second feature map of the first signal component. The control vector of the second signal component is multiplied by the second feature map of the first signal component, and then the offset vector of the second signal component is added to obtain the second feature map of the second signal component.
13. The method according to any one of claims 1-12, characterized in that, The first feature map of the first signal component is obtained by the first signal component through at least one layer of convolution and / or at least one layer of nonlinear processing; The first feature map of the second signal component is obtained by the second signal component through at least one layer of convolution and / or at least one layer of nonlinear processing.
14. The method as described in claim 13, characterized in that, The first feature map of the first signal component is obtained by passing the first signal component through two convolutional layers with a downsampling factor of 2 and two nonlinear layers. The first feature map of the second signal component is obtained by passing the second signal component through one convolutional layer without downsampling, one convolutional layer with a downsampling factor of 2, and two nonlinear layers.
15. A method for decoding a video signal, characterized in that, include: Obtain the bitstream of the video signal; Entropy decoding is performed on the bitstream to obtain feature maps of a first signal component and a second signal component of the video signal; the bitstream also includes quality factor information of the first signal component and quality factor information of the second signal component, wherein the quality factor information of the first signal component is the quality factor of the first signal component and the quality factor information of the second signal component is the quality factor of the second signal component. The response signal of the first signal component is obtained by using the quality factor information of the first signal component; The response signal of the second signal component is obtained by using the quality factor information of the second signal component; A reconstructed map of the first signal component is obtained based on the response signal of the first signal component and the feature map of the first signal component, wherein the response signal of the first signal component is obtained through learning. A reconstructed map of the second signal component is obtained based on the response signal of the second signal component and the feature map of the second signal component, wherein the response signal of the second signal component is obtained through learning; as well as The video signal is reconstructed based on the reconstruction map of the first signal component and the reconstruction map of the second signal component.
16. The method as described in claim 15, characterized in that, When the quality factor information of the first signal component is the quality factor of the first signal component, the quality factor of the first signal component takes one of N values; where N is an integer greater than 1. When the quality factor information of the second signal component is the quality factor of the second signal component, the quality factor of the second signal component takes one of M values; where M is an integer greater than 1.
17. The method as described in claim 15 or 16, characterized in that, The bitstream includes a joint feature map. The joint feature map is entropy decoded and then processed by a neural network to obtain the feature map of the first signal component and the feature map of the second signal component.
18. The method as described in any one of claims 15-17, characterized in that, The first signal component is the Y component, and the second signal component is the UV component, U component, or V component.
19. The method as described in claim 18, characterized in that, When the second signal component is a UV component, if the bitstream includes the quality factor of the first signal component and the quality factor of the second signal component, the method includes: The quality factor of the Y component is used as the input of the fully connected network, and the response signal of the Y component is output. The quality factor of the UV component is used as the input to a fully connected network, and the response signal of the UV component is output.
20. The method as described in claim 18, characterized in that, When the second signal component is a U component or a V component, the method further includes: Entropy decoding is performed on the bitstream to obtain a feature map of the third signal component of the video signal; Based on the response signal of the third signal component and the feature map of the third signal component, a reconstructed map of the third signal component is obtained, wherein the response signal of the third signal component is obtained through learning, wherein when the second signal component is a U component, the third signal component is a V component, or when the second signal component is a V component, the third signal component is a U component; The reconstruction of the video signal includes: The video signal is reconstructed based on the reconstructed map of the first signal component, the reconstructed map of the second signal component, and the reconstructed map of the third signal component.
21. The method as described in claim 20, characterized in that, The bitstream also includes quality factor information of the third signal component, wherein the quality factor information of the third signal component is the quality factor of the third signal component, and the quality factor of the third signal component takes one of L values, where L is an integer greater than 1; The response signal of the third signal component is obtained by using the quality factor information of the third signal component.
22. The method as described in claim 20, characterized in that, If the bitstream includes the quality factor of the first signal component, the quality factor of the second signal component, and the quality factor of the third signal component, the method includes: The quality factor of the Y component is used as the input of the fully connected network, and the response signal of the Y component is output. The quality factor of the U component is used as the input of the fully connected network, and the response signal of the U component is output. The quality factor of the V component is used as the input to a fully connected network, and the response signal of the V component is output.
23. The method according to any one of claims 15-22, characterized in that, When the response signal includes a response vector, obtaining the reconstructed map of the first signal component based on the response signal of the first signal component and the feature map of the first signal component includes: Multiply the response vector of the first signal component with the feature map of the first signal component to obtain the reconstructed map of the first signal component; or multiply the response vector of the first signal component with the feature map of the first signal component and then process it through a neural network to obtain the reconstructed map of the first signal component. Based on the response signal of the second signal component and the feature map of the second signal component, the reconstructed map of the second signal component is obtained, including: The response vector of the second signal component is multiplied by the feature map of the second signal component to obtain the reconstructed map of the second signal component; or the response vector of the second signal component is multiplied by the feature map of the second signal component and then processed by a neural network to obtain the reconstructed map of the second signal component.
24. The method according to any one of claims 15-22, characterized in that, When the response signal includes a response vector and an offset vector, obtaining the reconstructed map of the first signal component based on the response signal of the first signal component and the feature map of the first signal component includes: The response vector of the first signal component is multiplied by the feature map of the first signal component, and then added to the offset vector of the first signal component to obtain the reconstructed map of the first signal component; or the response vector of the first signal component is multiplied by the feature map of the first signal component, and then added to the offset vector of the first signal component, and then processed by a neural network to obtain the reconstructed map of the first signal component. Based on the response signal of the second signal component and the feature map of the second signal component, the reconstructed map of the second signal component is obtained, including: The response vector of the second signal component is multiplied by the feature map of the second signal component, and then added to the offset vector of the second signal component to obtain the reconstructed map of the second signal component; or the response vector of the second signal component is multiplied by the feature map of the second signal component, and then added to the offset vector of the second signal component, and then processed by a neural network to obtain the reconstructed map of the second signal component.
25. An encoder, characterized in that, Includes processing circuitry for performing the method according to any one of claims 1-14.
26. A decoder, characterized in that, Includes processing circuitry for performing the method according to any one of claims 15-24.
27. A computer program product, characterized in that, Includes program code, which, when executed on a computer or processor, is used to perform the method according to any one of claims 1-14 or 15-24.
28. An encoder, characterized in that, include: One or more processors; A non-transitory computer-readable storage medium coupled to the processor and storing a program executed by the processor, wherein the program, when executed by the processor, causes the encoder to perform the method of any one of claims 1-14.
29. A decoder, characterized in that, include: One or more processors; A non-transitory computer-readable storage medium coupled to the processor and storing a program executed by the processor, wherein the program, when executed by the processor, causes the decoder to perform the method of any one of claims 15-24.
30. A non-transitory computer-readable storage medium, characterized in that, Includes program code, which, when executed by a computer device, is used to perform the method according to any one of claims 1-14 or any one of claims 15-24.
Citation Information
Patent Citations
Multi-viewpoint video signal coding method based on vision
CN102158712A
