Picture encoding method and apparatus, and picture decoding method and apparatus

Abstract

Picture encoding methods and apparatuses, as well as picture decoding methods and apparatuses, are provided to provide encoding and decoding schemes in the fields of artificial intelligence and picture compression, thereby meeting the requirements of different application scenarios. According to the encoding and decoding methods provided herein, the encoder and decoder networks used may be determined based on profile information (or identification information). That is, the codec may select corresponding profile information based on the capabilities of the decoding device in order to select or indicate different encoder and decoder networks. In this way, the network may have the ability to adapt not only to terminals with low computing power but also to terminals with higher computing power.

Description

[Technical Field]

[0001] Book The application concerns, in particular, the fields of picture compression technology and artificial intelligence technology. Picture encoding method and apparatus, and picture decoding method and apparatus Regarding. [Background technology]

[0002] Many consumer applications (such as news, social, and shopping networking applications) require picture decoding to be completed on terminal devices with low computing power (such as mobile phones, personal PCs, and televisions). In some other industrial applications, picture decoding requires terminal devices with higher computing power. discrete This process is acceptable to complete on a GPU workstation (such as one equipped with a graphics card), and higher requirements are imposed on picture compression ratios.

[0003] Current neural network-based picture encoding and decoding methods typically have a fixed network structure and cannot meet the requirements of different application scenarios. [Overview of the project] [Problems that the invention aims to solve]

[0004] Embodiments of the present application provide an encoding and decoding scheme, Picture encoding method and apparatus, and picture decoding method and apparatus This provides the ability to meet the requirements of different application scenarios. [Means for solving the problem]

[0005] According to the first aspect, one embodiment of the present application provides a picture encoding method, the method: The step involves encoding identification information indicating the decoder network to be used into the bitstream, The identification information is a first value, indicating that the decoder network used to decode the bitstream to obtain the picture to be processed is the first decoder network, or The aforementioned identification information is a second value, indicating that the decoder network used to decode the bitstream to obtain the picture to be processed is the second decoder network, and the processing resources required by the first decoder network are higher than those required by the second decoder network; The step of transmitting the bitstream Includes.

[0006] The aforementioned identification information may also be called profile information (Profile ID).

[0007] According to the aforementioned solution in this embodiment of the present application, the transmitting end instructs the receiving end to use a certain network structure, thereby allowing different network structures to implement different decoding performances, thereby improving the flexibility of the decoder side. Users can flexibly balance latency and compression performance by adjusting the computing power of their encoder and decoder networks based on their scenarios.

[0008] In one possible implementation, the first decoder network and the second decoder network are entirely different decoder networks, or they share a portion of a subnet, or the second decoder network is a subnet of the first decoder network.

[0009] If the second decoder network is a subnet of the first decoder network, it can be understood that when the identification information is a second value, some network layers within the first decoder network are skipped, i.e., the process of decoding is performed by using the second decoder network.

[0010] In one possible implementation, the method is: The step of obtaining the aforementioned identification information; When the identification information is the first value, the residual information obtained by encoding the picture to be processed using the first encoder network is encoded into the bitstream, or When the identification information is the second value, the process involves encoding the residual information obtained by encoding the picture to be processed using the second encoder network into the bitstream; The processing resources required by the first encoder network are higher than those required by the second encoder network.

[0011] In the aforementioned solution, for bitstreams generated using different AI encoder networks, the decoder can select a different decoder network structure based on the bitstream content to perform decoding. This provides high flexibility to the codec. Users can flexibly balance latency and compression performance by adjusting the computing power of their encoder and decoder networks based on their scenarios.

[0012] In one possible implementation, the first encoder network and the second encoder network are two different encoder networks, or the first encoder network and the second encoder network share a part of a subnet, or the second encoder network is a subnet of the first encoder network.

[0013] In one possible implementation, the first encoder network includes a first feature extraction network, an autoregressive network, a side information extraction network, and a probability estimation network. The residual information obtained by encoding the target processing picture by using the first encoder network is: The Extracting a three-dimensional feature map of the target processing picture by using a first feature extraction network, where the three-dimensional feature map includes a plurality of feature elements; Extracting side information of the feature elements to be encoded from the three-dimensional feature map by using the side information extraction network; Estimating a first probability distribution average of the feature elements to be encoded by using the probability estimation network based on the side information; Inputting the encoded feature elements and the first probability distribution average into the autoregressive network to obtain a second probability distribution average of the feature elements to be encoded; Obtaining residual information of the feature elements to be encoded based on the feature elements to be encoded and the second probability distribution average of the feature elements to be encoded. Including.

[0014] In one possible implementation, the first encoder network includes a second feature extraction network, a side information extraction network, and a probability estimation network. The residual information obtained by encoding the picture to be processed using the first encoder network is: The second step involves extracting a 3D feature map of the picture to be processed by using a second feature extraction network, wherein the 3D feature map includes multiple feature elements; By using the aforementioned side information extraction network, the process involves extracting side information of the feature elements to be encoded from the 3D feature map; Based on the aforementioned side information, the step of estimating the mean of the probability distribution of the feature element to be encoded by using the aforementioned probability estimation network; A step of obtaining residual information of the feature element to be encoded based on the feature element to be encoded and the mean of the probability distribution. Includes.

[0015] In one possible implementation, the second feature extraction network is a subnet of the first feature extraction network, or the second and first feature extraction networks are two completely different subnets.

[0016] In one possible implementation, the method is: The process further includes encoding the side information into the bitstream.

[0017] In one possible implementation, the identification information is located in the header of the bitstream.

[0018] According to a second aspect, one embodiment of the present application provides a picture decoding method, the method of which: The stage of receiving the bitstream; The steps include: decoding the bitstream to obtain identification information indicating the decoder network to be used; When the identification information is a first value, the bitstream is decoded using the first decoder network to obtain the picture to be processed, or If the identification information is a second value, the process involves decoding the bitstream and obtaining the picture to be processed by using a second decoder network. The processing resources required by the first decoder network are higher than those required by the second decoder network.

[0019] In one possible implementation, the first decoder network and the second decoder network are entirely different decoder networks, or they share a portion of a subnet, or the second decoder network is a subnet of the first decoder network.

[0020] In one possible implementation, the first decoder network includes an entropy decoder network, a probability estimation network, an autoregressive network, and a first picture reconstruction network. The step of decoding the bitstream using the first decoder network to obtain the picture to be processed is: The steps include: decoding the bitstream using an entropy decoder network to obtain side information of a 3D feature map of the picture to be processed, wherein the 3D feature map includes multiple feature elements; Based on the aforementioned side information, the step of estimating the first probability distribution mean of the feature element to be decoded by using the aforementioned probability estimation network; The steps include determining a second probability distribution mean of the decoded feature element by using the autoregressive network based on the first probability distribution mean and the decoded feature element; The steps include: decoding the bitstream using the entropy decoder network based on the second probability distribution mean to obtain residual information of the feature element to be decoded; and obtaining the feature element to be decoded based on the residual information and the second probability distribution mean; The process involves restoring the target picture using a first picture restoration network based on a three-dimensional feature map obtained through decoding. Includes.

[0021] In one possible implementation, the second decoder network includes the entropy decoder network, the probability estimation network, and the second picture reconstruction network. The step of decoding the bitstream using the second decoder network to obtain the picture to be processed is: The steps include: decoding the bitstream using an entropy decoder network to obtain side information of a 3D feature map of the picture to be processed, wherein the 3D feature map includes multiple feature elements; Based on the aforementioned side information, the step of estimating the first probability distribution mean of the feature element to be decoded by using the aforementioned probability estimation network; The steps include: decoding the bitstream using the entropy decoder network based on the first probability distribution mean to obtain residual information of the feature element to be decoded; and obtaining the feature element to be decoded based on the residual information and the first probability distribution mean; The steps include: restoring the picture to be processed by using a second picture restoration network based on a three-dimensional feature map obtained through decoding; Includes.

[0022] In one possible implementation, the second picture restoration network is a subnet of the first picture restoration network, or the picture restoration network and the first picture restoration network share a portion of the subnet, or the second picture restoration network and the first picture restoration network are two different networks.

[0023] According to a third aspect, one embodiment of the present application provides a picture encoding device including a memory and a video encoder, where The memory is configured to store video data, and the video data includes the picture to be processed. The video encoder is configured to encode identification information indicating the decoder network to be used into a bitstream, where The identification information is a first value, indicating that the decoder network used to decode the bitstream and obtain the picture to be processed is the first decoder network, or The identification information is a second value, indicating that the decoder network used to decode the bitstream and obtain the picture to be processed is the second decoder network, and the processing resources required by the first decoder network are higher than those required by the second decoder network.

[0024] According to a fourth aspect, one embodiment of the present application provides a picture decoding device including a memory and a video decoder, where The memory is configured to store video data in bitstream format, and the video data includes the picture to be processed. The video decoder decodes the bitstream to obtain identification information indicating the decoder network to be used. If the identification information is a first value, the bitstream is decoded using the first decoder network to obtain the picture to be processed, or If the identification information is a second value, the system is configured to decode the bitstream and obtain the picture to be processed by using a second decoder network, and the processing resources required by the first decoder network are higher than those required by the second decoder network.

[0025] According to a fifth aspect, one embodiment of the present application provides a video decoding device comprising coupled non-volatile memory and a processor, the processor calling program code stored in memory to perform the method described in any implementation of the second aspect.

[0026] According to the sixth aspect, one embodiment of the present application provides a video encoding device comprising coupled non-volatile memory and a processor, the processor calling program code stored in the memory to perform the methods described in any implementation of the first or seventeenth aspect.

[0027] According to the seventh aspect, one embodiment of the present application provides a computer-readable storage medium that stores program code. , Program ·code When this is executed on a computer, the computer is made capable of performing the method according to any implementation of the second aspect.

[0028] According to the eighth aspect, one embodiment of the present application provides a computer-readable storage medium that stores program code. , Program ·code When executed on a computer, the computer is enabled to perform a method according to any implementation of the first aspect or the seventeenth aspect.

[0029] According to the ninth aspect, one embodiment of the present application provides a computer-readable storage medium which is decoded by one or more processors in accordance with the method of any implementation of the second aspect. Obtained through doing Store the video bitstream.

[0030] According to the tenth aspect, one embodiment of the present application provides a computer-readable storage medium that stores a video bitstream obtained by encoding by one or more processors in accordance with a method according to an implementation of either the first aspect or the seventeenth aspect.

[0031] According to the eleventh aspect, one embodiment of the present application provides a computer-readable storage medium which stores a bitstream, the bitstream containing identification information, where The identification information is a first value, indicating that the decoder network used to decode the bitstream and obtain the picture to be processed is the first decoder network, or The identification information is a second value, indicating that the decoder network used to decode the bitstream and obtain the picture to be processed is the second decoder network, and the processing resources required by the first decoder network are higher than those required by the second decoder network.

[0032] According to the twelfth aspect, one embodiment of the present application provides an encoded bitstream, the encoded bitstream comprising a plurality of syntax elements, the plurality of syntax elements comprising identification information indicating a decoder network used to decode the bitstream to obtain a picture to be processed.

[0033] According to the thirteenth aspect, one embodiment of the present application provides a video encoder configured to encode a picture to be processed. For example, the video encoder may implement the method according to the first aspect or the seventeenth aspect.

[0034] According to a fourteenth aspect, one embodiment of the present application provides a video decoder configured to decode a bitstream to obtain a picture to be processed. For example, a video encoder may implement the method according to the second aspect.

[0035] According to the fifteenth aspect, one embodiment of the present application provides an encoder network, which is: It includes a first feature extraction network, a second feature extraction network, a quantization network, an autoregressive network, a side information extraction network, and a probabilistic estimation network. If the identifier indicating the encoder network to be used is a first value, the first feature extraction network extracts a 3D feature map of the picture being processed; or, if the identifier is a second value, the first feature extraction network extracts a 3D feature map of the picture being processed; The side information extraction network is the 3D feature map Pu Then extract the side information of the picture to be processed; The aforementioned probability estimation network estimates the mean of the first probability distribution of the feature element to be encoded based on the side information. When the identification information indicating the encoder network to be used is the first value, the encoded feature element and the first probability distribution mean are input to the autoregressive network to obtain a second probability distribution mean of the feature element to be encoded, and residual information of the feature element to be encoded is obtained based on the feature element to be encoded and the second probability distribution mean of the feature element to be encoded, or When the identification information indicating the encoder network to be used is the second value, residual information of the feature element to be encoded is obtained based on the feature element to be encoded and the mean of the first probability distribution.

[0036] In one possible implementation, the second feature extraction network is a subnet of the first feature extraction network, or the second and first feature extraction networks are two completely different subnets.

[0037] According to the sixteenth aspect, one embodiment of the present application provides a decoder network, the decoder network is: It includes an entropy decoder network, a probability estimation network, an autoregressive network, a first picture reconstruction network, and a second picture reconstruction network. The entropy decoder network decodes the bitstream to obtain side information and identification information of the 3D feature map of the picture to be processed, and the 3D feature map includes multiple feature elements. The aforementioned probability estimation network estimates the mean of the first probability distribution of the feature element to be decoded based on the side information. If the identification information is a first value, the autoregressive network determines a second probability distribution mean of the feature element to be decoded based on the first probability distribution mean and the decoded feature element, the entropy decoder network decodes the bitstream and obtains residual information of the feature element to be decoded based on the second probability distribution mean, obtains the feature element to be decoded based on the residual information and the second probability distribution mean, the first picture reconstruction network reconstructs the picture to be processed based on the 3D feature map obtained through decoding, or If the identification information is a second value, the entropy decoder network decodes the bitstream to obtain residual information of the feature element to be decoded based on the first probability distribution mean, and obtains the feature element to be decoded based on the residual information and the first probability distribution mean; the second picture reconstruction network reconstructs the picture to be processed based on the three-dimensional feature map obtained through decoding.

[0038] In one possible implementation, the second picture restoration network is a subnet of the first picture restoration network, or the second and first picture restoration networks share a portion of the subnet, or the second and first picture restoration networks are two distinct networks.

[0039] According to aspect 17, one embodiment of the present application provides a picture encoding method, the method of which: The stage of obtaining identification information; If the identification information is a first value, the residual information obtained by encoding the picture to be processed based on (or by using) a first encoder network is encoded into a bitstream, or If the aforementioned identification information is a second value, the steps include encoding the residual information obtained by encoding the picture to be processed based on (or by using) the second encoder network into a bitstream. The processing resources required by the first encoder network are higher than those required by the second encoder network.

[0040] In one possible implementation, the method is: The process further includes encoding the identification information into the bitstream.

[0041] In one possible implementation, the identification information further indicates the decoder network used to decode the bitstream and obtain the picture to be processed, where The identification information is a first value, indicating that the decoder network used to decode the bitstream and obtain the picture to be processed is the first decoder network, or The identification information is a second value, indicating that the decoder network used to decode the bitstream and obtain the picture to be processed is the second decoder network, and the processing resources required by the first decoder network are higher than those required by the second decoder network.

[0042] The aforementioned identification information may also be called profile information (Profile ID).

[0043] In one possible implementation, the first decoder network and the second decoder network are entirely different decoder networks, or they share a portion of a subnet, or the second decoder network is a subnet of the first decoder network.

[0044] In one possible implementation, the first encoder network and the second encoder network are two different encoder networks, or the first encoder network and the second encoder network share a portion of a subnet, or the second encoder network is a subnet of the first encoder network.

[0045] In one possible implementation, the first encoder network includes a first feature extraction network, an autoregressive network, a side information extraction network, and a probability estimation network. The residual information obtained by encoding the picture to be processed using the first encoder network is: The The first step involves extracting a 3D feature map of the picture to be processed by using a feature extraction network, wherein the 3D feature map contains multiple feature elements; By using the aforementioned side information extraction network, the process involves extracting side information of the feature elements to be encoded from the 3D feature map; Based on the aforementioned side information, the step of estimating the first probability distribution mean of the feature element to be encoded by using the probability estimation network; The steps include: inputting the encoded feature elements and the first probability distribution mean into the autoregressive network to obtain a second probability distribution mean of the feature elements to be encoded; A step of obtaining residual information of the feature element to be encoded based on the feature element to be encoded and the second probability distribution mean of the feature element to be encoded. Includes.

[0046] In one possible implementation, the first encoder network includes a second feature extraction network, a side information extraction network, and a probability estimation network. The residual information obtained by encoding the picture to be processed using the first encoder network is: The second step involves extracting a 3D feature map of the picture to be processed by using a second feature extraction network, wherein the 3D feature map includes multiple feature elements; By using the aforementioned side information extraction network, the process involves extracting side information of the feature elements to be encoded from the 3D feature map; The steps include: estimating the mean of the probability distribution of the feature element to be encoded by using the probability estimation network based on the aforementioned side information; A step of obtaining residual information of the feature element to be encoded based on the feature element to be encoded and the mean of the probability distribution. Includes.

[0047] In one possible implementation, the second feature extraction network is a subnet of the first feature extraction network, or the second and first feature extraction networks are two completely different subnets.

[0048] In one possible implementation, the method is: The process further includes encoding the side information into the bitstream.

[0049] In this application, based on the implementations provided in the aforementioned aspects, these implementations may be further combined to provide more implementations. [Brief explanation of the drawing]

[0050] [Figure 1] This is an illustrative block diagram of a coding system according to one embodiment of the present invention.

[0051] [Figure 2] This is a diagram showing the structure of a convolutional neural network according to one embodiment of the present invention.

[0052] [Figure 3] This is a diagram of a deep learning-based video encoder and decoder network according to one embodiment of the present invention.

[0053] [Figure 4] This figure shows the structure of a deep learning-based end-to-end video encoder and decoder network according to one embodiment of the present invention.

[0054] [Figure 5] This is a schematic flowchart of an encoding and decoding method according to one embodiment of the present invention.

[0055] [Figure 6A] This is a diagram showing the structure of a first encoder network according to one embodiment of the present invention.

[0056] [Figure 6B] This is a diagram showing the structure of a second encoder network according to one embodiment of the present invention.

[0057] [Figure 7A] This is a diagram of an encoding process according to one embodiment of the present invention.

[0058] [Figure 7B] This is a diagram of another encoding process according to one embodiment of the present invention.

[0059] [Figure 8] This is a diagram showing the structure of a decoder network according to one embodiment of the present invention.

[0060] [Figure 9A] This is a diagram of a possible decoding process using a first decoder network according to one embodiment of the present invention.

[0061] [Figure 9B] This is a diagram of a possible decoding process using a second decoder network according to one embodiment of the present invention.

[0062] [Figure 10A] This is a diagram showing the execution process of an encoder network according to one embodiment of the present invention.

[0063] [Figure 10B] This is a diagram showing the execution process of a decoder network according to one embodiment of the present invention.

[0064] [Figure 11A]Figures 11A and 11B show the structure of an encoder network according to one embodiment of the present invention. [Figure 11B] Figures 11A and 11B show the structure of an encoder network according to one embodiment of the present invention.

[0065] [Figure 12] This is a diagram showing the structure of a ResAU 3x3 network without tanh according to one embodiment of the present invention.

[0066] [Figure 13] This is a diagram of an RNAB structure according to one embodiment of the present invention.

[0067] [Figure 14] This is a diagram showing the structure of the residual block layer according to one embodiment of the present invention.

[0068] [Figure 15] This is a diagram showing the network structure of a hyperdecoder network according to one embodiment of the present invention.

[0069] [Figure 16] This is a diagram showing the network structure of a hyperscale decoder network according to one embodiment of the present invention.

[0070] [Figure 17A] Figures 17A and 17B show the execution process of a decoder network according to one embodiment of the present invention. [Figure 17B] Figures 17A and 17B show the execution process of a decoder network according to one embodiment of the present invention.

[0071] [Figure 18] This is a diagram showing the network structure of a LightResBlock according to one embodiment of the present invention.

[0072] [Figure 19]This is a diagram showing the structure of a decoder network according to Example 2, based on one embodiment of the present invention.

[0073] [Figure 20A] This is a diagram showing the execution process of an encoder network according to Example 3, based on one embodiment of the present invention.

[0074] [Figure 20B] This is a diagram showing the structure of an encoder and decoder network according to Example 3, based on one embodiment of the present invention.

[0075] [Figure 21A] Figures 21A and 21B show the structure of an encoder network according to Example 3, according to one embodiment of the present application. [Figure 21B] Figures 21A and 21B show the structure of an encoder network according to Example 3, according to one embodiment of the present application.

[0076] [Figure 22A] Figures 22A and 22B show the structure of a decoder network according to Example 3, according to one embodiment of the present application. [Figure 22B] Figures 22A and 22B show the structure of a decoder network according to Example 3, according to one embodiment of the present application. [Modes for carrying out the invention]

[0077] The embodiments of the present application will be described below with reference to the accompanying drawings of embodiments thereof. The following description will refer, as illustrative, to the accompanying drawings which form part of the present disclosure and illustrate specific aspects of embodiments of the present application or specific aspects in which embodiments of the present application may be used. It should be understood that embodiments of the present application may be used in other aspects and may include structural or logical modifications not shown in the accompanying drawings. Therefore, the following detailed description should not be constrained to mean restrictively, and the scope of the present application is defined by the accompanying claims. For example, it should be understood that the disclosed content of a described method may apply to a corresponding device or system for performing the method, and vice versa. For example, if one or more specific method steps are described, the corresponding device may include one or more units, such as functional units, for performing the one or more method steps described, even if such one or more units are not explicitly described or shown in the accompanying drawings (e.g., one unit performing the one or more steps; or multiple units, each performing one or more of the multiple steps). In addition, for example, if a particular device is described based on one or more units such as a functional unit, the corresponding method may include one step for performing the function of one or more units (for example, one step for performing the function of one or more units; or there may be multiple steps, each for performing the function of one or more of the multiple units), even if such one or more steps are not explicitly described or illustrated in the accompanying drawings. Furthermore, it should be understood that the exemplary embodiments and / or aspect features described herein may be combined with each other unless otherwise explicitly stated.

[0078] The technical solutions in the embodiments of this application are applicable not only to existing video encoding standards (e.g., standards such as H.264 and HEVC) but may also be applicable to future video encoding standards (e.g., the H.266 standard). The terms used in the implementation of this application are used solely to describe specific embodiments of this application and are not intended to limit this application. Below, we will first briefly describe some relevant concepts in the embodiments of this application.

[0079] The picture decoding and encoding methods provided in embodiments of the present application can be applied to the video encoding field and the picture encoding field. Specifically, the decoding and encoding methods can be applied to album management, human-computer interaction, video compression or transmission, and picture compression or transmission scenarios.

[0080] encoding methodExamples are used in which the decoding method is applied to an end-to-end video-picture encoding and decoding system. An end-to-end video-picture encoding and decoding system comprises two parts: picture encoding and picture decoding. Picture encoding is determined at the source and typically involves processing the original video picture (e.g., compressing) to reduce the amount of data required to represent the video picture (for more efficient storage and / or transmission). Picture decoding is determined at the destination and typically involves inverse processing of the encoder to reconstruct the picture. Current neural network-based picture encoding and decoding schemes typically have a fixed network structure, for example, the encoding and decoding model in JPEG AI VM1.0. If the network structure is adapted to the capabilities of a terminal with low computing power, the compression efficiency of the encoding scheme will be reduced to some extent. If the network structure is adapted to the computing power of a device with high computing power, the network cannot run on a device with low computing power. In an end-to-end video-picture encoding and decoding system, the encoding provided in this application method By using the and decoding method, the encoder and decoder network to be used may be determined based on profile information. Profile information may also be called identification information or network identifier, or may have another name. This is not particularly limited in the embodiments of the present application. Profile information indicates the decoder network used. That is, the codec may select corresponding profile information based on the capabilities of the decoding device in order to select or indicate different encoder and decoder networks. In this way, the network may have the capability to adapt not only to terminals with lower computing power, but also to terminals with higher computing power.

[0081] Video encoding and decoding generally refer to the processing of picture sequences that make up a video or video sequence. In the field of video encoding and decoding, the terms "picture," "frame," and "image" can be used interchangeably.

[0082] Figure 1 is an exemplary block diagram of a coding system according to one embodiment of the present application, for example, a video coding system 10 (or simply coding system 10) on which the technology of the present application can be utilized. The video encoder 20 (or simply encoder 20) and video decoder 30 (or simply decoder 30) of the video coding system 10 represent examples of devices that may be configured to perform the technology based on the various examples described herein.

[0083] As shown in Figure 1, the coding system 10 includes a source device 12. The source device 12 is configured to provide encoded picture data 21, such as an encoded picture, to a destination device 14 configured to decode the encoded picture data 21.

[0084] The source device 12 includes an encoder 20 and optionally includes a picture source 16, a preprocessor (or preprocessing unit) 18 such as a picture preprocessor, and a communication interface (or communication unit) 22.

[0085] The picture source 16 may include, or may include, any type of picture capturing device configured to capture real-world pictures, and / or any type of picture generating device, such as a computer graphics processing unit configured to generate computer-animated pictures, or any type of device configured to acquire and / or provide real-world pictures, computer-generated pictures (e.g., screen content, virtual reality (VR) pictures, and / or any combination thereof (e.g., augmented reality (AR) pictures). The picture source may also include any type of memory or storage for storing any of the aforementioned pictures.

[0086] To distinguish the processing performed by the preprocessor (or preprocessing unit) 18, the picture (or picture data) 17 is sometimes referred to as the original picture (or original picture data) 17.

[0087] The preprocessor 18 is configured to receive the original picture data 17, preprocess the original picture data 17, and obtain a preprocessed picture (or preprocessed picture data) 19. For example, the preprocessing performed by the preprocessor 18 may include cropping, color format conversion (e.g., RGB to YCbCr), color correction, or denoising. It can be understood that the preprocessing unit 18 may be an optional component.

[0088] The video encoder (or encoder) 20 is configured to receive pre-processed picture data 19 and provide encoded picture data 21.

[0089] The communication interface 22 of the source device 12 may be configured to receive the encoded picture data 21 and send the encoded picture data 21 (or any other processed version) to another device, such as the destination device 14 or any other device, via the communication channel 13 for storage or direct reconstruction.

[0090] The source device 12 may further include memory (not shown in Figure 1). The memory may be configured to store at least one of the following data: original picture data 17, preprocessed picture (or preprocessed picture data) 19, and encoded picture data 21.

[0091] The destination device 14 includes a decoder 30 and may optionally include a communication interface (or communication unit) 28, a post-processor (or post-processing unit) 32, and a display device 34.

[0092] The communication interface 28 of the destination device 14 is configured to directly receive encoded picture data 21 (or any other processed version) from the source device 12 or any other source device, such as an encoded picture data storage device, and to provide the encoded picture data 21 to the decoder 30.

[0093] Communication interfaces 22 and 28 may be configured to transmit or receive encoded picture data (or encoded data) 21 via a direct communication link between the source device 12 and the destination device 14, for example, via a direct wired or wireless connection, or via any type of network, for example, a wired network, a wireless network, or any combination thereof, or any type of private network, a public network, or any combination thereof.

[0094] For example, the communication interface 22 may be configured to process the encoded picture data 21 by packaging it into an appropriate format such as a packet for transmission over a communication link or communication network, and / or by using any type of transmit encoding or processing.

[0095] Communication interface 28 corresponds to communication interface 22 and may be configured to receive transmitted data and process the transmitted data by using any type of corresponding transmit decoding or processing and / or depackaging to obtain encoded picture data 21.

[0096] Communication interfaces 22 and 28 can be configured as one-way or two-way communication interfaces, respectively, as indicated by the arrows of the communication channel 13 pointing from the source device 12 to the destination device 14 in Figure 1, and can be configured to send and receive messages, establish connections, and confirm and exchange any other information related to data transmission, such as communication links and / or the transmission of encoded picture data.

[0097] The video decoder (or decoder) 30 receives the encoded picture data 21 and decodes the picture data (or decoded Ta De It is configured to provide (data) 31.

[0098] The post-processor 32 is configured to post-process the decoded picture data 31 (also called reconstructed picture data), such as the decoded picture, to obtain post-processed picture data 33, such as the post-processed picture. For example, post-processing performed by the post-processing unit 32 may include color format conversion (e.g., YCbCr to RGB), color correction, cropping, or resampling, or any other processing to generate the decoded picture data 31 for display by a display device 34 or the like.

[0099] The display device 34 is configured to receive the post-processed picture data 33 and display the picture to a user, observer, etc. The display device 34 may be any type of display for representing the reconstructed picture, such as an integrated or external display or monitor, or may include one. For example, the display may include a liquid crystal display (LCD), an organic light-emitting diode (OLED) display, a plasma display, a projector, a microLED display, a liquid crystal on silicon (LCoS), a digital light processor (DLP), or any other type of display.

[0100] The destination device 14 may further include memory (not shown in Figure 1). The memory may be configured to store at least one of the following data: encoded picture data 21, decoded picture data 31, and post-processed picture data 33.

[0101] The coding system 10 further includes a training engine 25. The training engine 25 is configured to process an input picture, picture region, or picture block, obtain a feature map of the input picture, picture region, or picture block, obtain an estimated probability distribution of the feature map, and train the encoder 20 to encode the feature map based on the estimated probability distribution.

[0102] The training engine 25 is further configured to acquire an estimated probability distribution of the bitstream, decode the bitstream based on the estimated probability distribution to obtain a feature map, and reconstruct the feature map to obtain a reconstructed picture.

[0103] As shown in Figure 1, the source device 12 and the destination device 14 are separate devices. However, the device embodiment may include both the source device 12 and the destination device 14, or include the functionality of both the source device 12 and the destination device 14, that is, it may include both the source device 12 or its corresponding functionality and the destination device 14 or its corresponding functionality. In these embodiments, the source device 12 or its corresponding functionality and the destination device 14 or its corresponding functionality may be implemented by the same hardware and / or software, by separate hardware and / or software, or by any combination thereof.

[0104] Based on the description, it will be apparent to those skilled in the art that the presence and (exact) division of different units or functions of the source device 12 and / or destination device 14 shown in Figure 1 may vary depending on the actual device and application.

[0105] In recent years, the application of deep learning to the field of video encoding and decoding has gradually become a trend. Deep learning involves multi-layer learning with different abstraction layers using machine learning algorithms. Deep learning-based video encoding and decoding is sometimes called AI video encoding and decoding, or neural network-based video encoding and decoding. Embodiments of this application relate to the application of neural networks. For ease of understanding, some nouns or terms used in embodiments of this application are first explained below. These nouns or terms are also used as part of the content of the invention.

[0106] (1) Artificial neural network (ANN):

[0107] Artificial neural networks, also known as neural networks (NNs), are dynamic systems that are manually established and use directed graphs as their topological structure. Artificial neural networks process information using continuous or discontinuous inputs as state responses, and are information processing systems that mimic the structure and function of the human brain. After decades of development, artificial neural networks have achieved great success and are widely used in many fields, including pattern recognition, automated control, signal processing, decision support, artificial intelligence, and scientific computing. Generally, a network includes an input layer, a hidden layer, and an output layer. In this application, neural networks may include multiple types, such as deep neural networks (DNNs), convolutional neural networks (CNNs), recurrent neural networks (RNNs), residual networks, neural networks using transformer models, or other types of neural networks. Several types of neural networks are described below as examples.

[0108] (2) Convolutional neural networks:

[0109] A convolutional neural network (CNN) is a deep neural network with a convolutional structure and is a deep learning architecture. A deep learning architecture means that multi-layer learning is performed at different abstraction layers by using machine learning algorithms. As a deep learning architecture, a CNN is a feed-forward artificial neural network, where each neuron in the feed-forward artificial neural network processes the data input to the neuron.

[0110] As shown in Figure 2, a convolutional neural network (CNN) 100 may include an input layer 110, a convolutional / pooling layer 120 (the pooling layer is optional), and a neural network layer 130. As shown in Figure 2, the convolutional / pooling layer 120 may include, for example, layers 121-126. In one implementation, layer 121 is a convolutional layer, layer 122 is a pooling layer, layer 123 is a convolutional layer, layer 124 is a pooling layer, layer 125 is a convolutional layer, and layer 126 is a pooling layer. In another implementation, layers 121 and 122 are convolutional layers, layer 123 is a pooling layer, layers 124 and 125 are convolutional layers, and layer 126 is a pooling layer. That is, the output of a convolutional layer may be used as input to a subsequent pooling layer, or as input to another convolutional layer to continue performing convolution operations. Convolutional layer 121 is used as an example. Convolutional layer 121 may contain multiple convolution operators, which are also called convolution kernels. Convolution operators may essentially be weight matrices, which are usually predefined. Picture processing is used as an example. Different weight matrices are used to extract different features in a picture. For example, one weight matrix is ​​used to extract edge information in the picture, another weight matrix is ​​used to extract specific colors in the picture, and yet another weight matrix is ​​used to blur unwanted noise in the picture.

[0111] The weight values ​​in these weight matrices need to be obtained through extensive training in actual applications. Each weight matrix formed by the weight values ​​obtained through training can be used to extract information from the input data in order to help the convolutional neural network 100 make correct predictions.

[0112] If a convolutional neural network 100 has multiple convolutional layers, typically a large number of general features are extracted in the initial convolutional layer (e.g., 121). General features are sometimes called low-level features. As the depth of the convolutional neural network 100 increases, the features extracted in later convolutional layers (e.g., 126) become more complex, for example, higher-level semantic features. Higher semantic features are more applicable to the problem to be solved.

[0113] Pooling layer:

[0114] Since the amount of training parameters often needs to be reduced, pooling layers often need to be introduced periodically after convolutional layers. For layers 121-126 shown in Figure 2, one pooling layer may follow one convolutional layer, or one or more pooling layers may follow multiple convolutional layers. During picture processing, pooling layers are used solely to reduce the spatial size of the picture. Pooling layers include an average pooling operator and / or a max pooling operator, which can perform sampling on the input picture to obtain a picture with a smaller size. The average pooling operator can calculate the pixel values ​​within a picture within a specific range to generate an average value. The max pooling operator can use the largest pixel within a specific range as the max pooling result. In addition, the operators in the pooling layer must be associated with the size of the picture, as must the size of the weight matrix in the convolutional layer. The size of the processed picture output from the pooling layer may be smaller than the size of the picture input to the pooling layer. Each pixel in the picture output from the pooling layer represents the average or maximum value of the corresponding sub-region of the picture input to the pooling layer.

[0115] After processing is performed in the convolutional / pooling layer 120, the convolutional neural network 100 is still unable to output the required output information. This is because, as mentioned above, the convolutional / pooling layer 120 only extracts features and reduces the parameters that the input picture provides. However, in order to generate the final output information (required type information or other relevant information), the convolutional neural network 100 needs to use the neural network layer 130 to generate an output of one required type or a group of required types. Thus, the neural network layer 130 may include multiple hidden layers (131, 132, ..., and 13n shown in Figure 2) and an output layer 140. The parameters included in the multiple hidden layers may be obtained through pre-training based on relevant training data for a particular task type. For example, task types may include picture recognition, picture classification, and super-resolution picture reconstruction.

[0116] Following multiple hidden layers within the neural network layer 130 is the output layer 140, which is the final layer of the entire convolutional neural network 100. The output layer 140 has a loss function similar to the classification cross-entropy, which is specifically used to calculate the prediction error. Once the forward propagation of the entire convolutional neural network 100 (for example, the propagation from layer 110 to layer 140 in Figure 2 is forward propagation) is complete, backpropagation (for example, the propagation from layer 140 to layer 110 in Figure 2 is backpropagation) is initiated to update the weight values ​​and deviations of the aforementioned layers, reduce the loss of the convolutional neural network 100, and reduce the error between the ideal result and the result output by the convolutional neural network 100 through the output layer.

[0117] It should be noted that the convolutional neural network 100 shown in Figure 2 is used only as an example of a convolutional neural network. In a particular application, the convolutional neural network may alternatively exist in the form of another network model, for example, multiple parallel convolutional / pooling layers, and all extracted features are input to the neural network layer 130 for processing.

[0118] (3) Loss function:

[0119] In the process of training a neural network, it is expected that the output of the neural network will be as close as possible to the value that actually needs to be predicted. Therefore, the network's current prediction may be compared to the expected target value, and then the weight vectors of each layer of the neural network are updated based on the difference between the current prediction and the target value (of course, before the first update, there is usually an initialization process, i.e., a process of pre-configuring the parameters for each layer of the neural network). For example, if the network's prediction is large, the weight vectors are adjusted to decrease the prediction, and this adjustment continues until the neural network can predict the expected target value. Therefore, it is necessary to pre-define "how to obtain the difference between the prediction and the target value through comparison." This is the loss function or objective function. The loss function and objective function are important formulas that measure the difference between the prediction and the target value. The loss function is used as an example. A larger output value (loss) of the loss function indicates a larger difference. Therefore, training a neural network is the process of minimizing the loss as much as possible.

[0120] (4) Linear motion:

[0121] Linearity refers to a proportional and linear relationship between quantities, and mathematically, it can be understood as a function whose first derivative is constant. Linear operations can be, but are not limited to, addition, null operations, identity operations, convolution operations, layer normalization (LN) operations, and pooling operations. Linear operations are also called linear mappings. Linear mappings must satisfy two conditions: homogeneity and additivity. If either condition is not met, a nonlinear mapping occurs.

[0122] Homogeneity means f(ax) = af(x), and additivity means f(x+y) = f(x) + f(y). For example, f(x) = ax is linear. Note that x, a, and f(x) here do not need to be scalars; they can be vectors or matrices, forming a linear space of any dimension. When x and f(x) are n-dimensional vectors, if a is a constant, it is equivalent to satisfying homogeneity, or if a is a matrix, it is equivalent to satisfying additivity. Relatively speaking, the graph of a function that is a straight line does not necessarily conform to a linear mapping. For example, f(x) = ax + b does not satisfy homogeneity or additivity, and therefore belongs to a nonlinear mapping.

[0123] In the embodiments of this application, a combination of multiple linear operations may be called a linear operation, and each linear operation included in the linear operation may be called a sub-linear operation.

[0124] (5) Attention-grabbing models:

[0125] An attention model is a neural network that uses an attention mechanism. In deep learning, the attention mechanism can be broadly defined as a weight vector that describes importance, and elements are predicted or inferred by using this weight vector. For example, for pixels in a picture or words in a sentence, the correlation between a target element and other elements can be quantitatively estimated by using the attention vector, and the weighted sum of the attention vectors is used as an approximation of the target.

[0126] The attention mechanism in deep learning mimics the attention mechanism of the human brain. For example, when a person looks at a picture, their eyes can see the entire picture, but when a person observes the picture closely, their eyes focus on only a part of the picture, and at this time, the human brain focuses on this small pattern. In other words, when a person carefully observes a picture, the brain's attention to the picture as a whole is unbalanced and distinguished by specific weights. This is the core idea of ​​the attention mechanism.

[0127] Simply put, the human visual processing system typically selectively focuses on certain parts of a picture, ignoring other irrelevant information, thereby facilitating human perception. Similarly, in deep learning attention mechanisms, certain parts of an input may be more relevant than others in certain problems related to language, speech, or vision. Therefore, by using an attention mechanism in an attention model, the attention model can dynamically focus only on the parts of the input that help it effectively perform the task at hand.

[0128] (6) Self-attention network:

[0129] A self-attention network is a neural network that uses a self-attention mechanism. The self-attention mechanism is an extension of the attention mechanism. In essence, the self-attention mechanism is an attention mechanism that correlates different positions within a single sequence to compute a representation of the same sequence. Self-attention mechanisms can play a crucial role in machine reading, abstract summarization, or picture description generation. For example, self-attention networks are applied to natural language processing. A self-attention network processes input data of arbitrary length, generates new feature representations of the input data, and then translates these feature representations into target words. In a self-attention network, the self-attention network layer uses the attention mechanism to capture the relationships between all other words, thereby generating a new feature representation for each word. The advantage of a self-attention network is that the attention mechanism can directly capture the relationships between all words in a sentence without considering word positions.

[0130] Figure 3 shows a deep learning-based video encoder and decoder network (or system) according to one embodiment of the present invention. 3 This is illustrated by using entropy encoding and decoding as an example. This network includes a feature extraction module, a feature quantization module, an entropy encoding module, an entropy decoding module, a feature dequantization module, and a feature decoding (or picture reconstruction) module.

[0131] On the encoder side, the original picture (or the picture to be compressed) is input to the feature extraction module, which outputs an extracted 3D feature map of the original picture by stacking multiple convolutional layers with reference to a nonlinear mapping activation function. The feature quantization module quantizes the floating-point feature values ​​in the 3D feature map to obtain a quantized feature map. Entropy encoding is performed on the quantized 3D feature map to obtain a bitstream.

[0132] On the decoder side, the entropy decode module parses the bitstream to obtain a quantized 3D feature map. The feature dequantization module dequantizes the integer feature values ​​within the quantized feature map to obtain a dequantized feature map. After the feature map dequantized by the feature decode module is reconstructed, the reconstructed picture is obtained.

[0133] Entropy encoding is an encoding method in which no information is lost during the encoding process, according to the principle of entropy. Entropy encoding is used to obtain encoded data that can be output by an output terminal in the form of an encoded bitstream, etc., by applying an entropy encoding algorithm or scheme to quantized coefficients and other syntactic elements, thereby allowing a decoder, etc., to receive and use the parameters used for decoding. The encoded bitstream can be sent to a decoder or stored in memory for later transmission or retrieval by a decoder. Entropy encoding algorithms or schemes include variable length coding (VLC) schemes and context-adaptive VLC schemes. CAVLCThis includes, but is not limited to, arithmetic coding schemes, binarization algorithms, context adaptive binary arithmetic coding (CABAC), syntax-based context-adaptive binary arithmetic coding (SBAC), probability interval partitioning entropy (PIPE) coding, or other entropy coding methods or techniques.

[0134] Alternatively, this network may not include feature quantization and feature dequantization modules. In this case, the network is feature element A series of operations may be performed directly on the feature map where the values ​​are floating-point numbers. Alternatively, integerization may be performed on the network so that all feature values ​​in the feature map output by the feature extraction module are integers.

[0135] After the picture to be processed (or compressed) passes through the feature extraction module and the feature quantization module, a quantized 3D feature map is obtained. When processing each feature value in the quantized 3D feature map, the entropy encoding module may estimate the probability distribution of the feature values ​​by using the processed feature values ​​in their neighborhood as context, obtain the probability distribution of the feature values, and then perform subsequent encoding based on this probability distribution to obtain an encoded bitstream.

[0136] Figure 4 is a diagram illustrating the structure of a deep learning-based end-to-end video encoder and decoder network according to one embodiment of the present invention. Figure 4 is illustrated by using entropy encoding and decoding as an example. The neural network includes a feature extraction module, a quantization module, a side information extraction module, an entropy encoding module, an entropy decoding module, a probability estimation module, and a reconstruction module. Module This may be an autoencoder (AE), and entropy decoding Module It may also be an autodecoder (AD).

[0137] On the encoder side, the original picture x is input to the feature extraction module, which outputs a feature map y of the original picture. The feature map y is input to the quantization module, which outputs a quantized feature map, and the quantized feature map is input to the entropy encoding module. The feature map y is also input to the side information extraction module, which outputs side information z. The side information z is input to the quantization module, which outputs quantized side information. The quantized side information passes through the entropy encoding module to obtain a bitstream of the side information, and then passes through the entropy decoding module to obtain decoded side information. The decoded side information is input to the probability estimation module. The probability estimation module outputs the probability distribution of each feature element [x][y][i] in the quantized feature map, and inputs the probability distribution of each feature element to the entropy encoding module. The entropy encoding module performs entropy encoding on each input feature element based on the probability distribution of each feature element to obtain a hyperplier bitstream.

[0138] Side information z is feature information represented as a 3D feature map. The number of feature elements in the 3D feature map is less than the number of feature elements in the feature map y.

[0139] On the decoder side, the entropy decode module parses the bitstream of side information to obtain the side information and inputs it to the probability estimation module. The probability estimation module outputs the probability distribution of each feature element [x][y][i] within the symbol to be decoded. The probability distribution of each feature element [x][y][i] is input to the entropy decode module. The entropy decode module performs entropy decoding on each feature element based on its probability distribution to obtain a decoded feature map. The decoded feature map is input to the reconstruction module, which outputs the reconstructed picture.

[0140] In addition, some variational autoencoders (VAEs) use their probability estimation modules to further estimate the probability distribution of the current feature element by using encoded or decoded feature elements surrounding the current feature element.

[0141] Please note that the network structures shown in Figures 3 and 4 are for illustrative purposes only. The modules included in the network and the structure of the modules are not limited to the embodiments of this application.

[0142] In some possible scenarios, an autoregressive module may be added to further improve the accuracy of the mean. The autoregressive module may further obtain the probability distribution used to obtain the residuals, based on the mean and quantized feature map output by the probability distribution module.

[0143] The encoding and decoding methods provided in embodiments of the present application are described in detail below. Figure 5 is a schematic flowchart of an encoding and decoding method according to one embodiment of the present application. The method process may be performed by two electronic devices or by one electronic device. For example, when the method process is performed by two electronic devices, one electronic device includes an encoder configured to perform an encoding operation, and the other electronic device includes a decoder configured to perform a decoding operation. When the method process is performed by one electronic device, the electronic device may include an encoder and a decoder. The method may be performed by an electronic device by calling a neural network model. The method process is described as a series of operations. It should be understood that the method process may be performed in various sequences and / or simultaneously, and is not limited to the execution sequence shown in Figure 5.

[0144] 501: The encoder encodes identification information into the bitstream that indicates the decoder network being used.

[0145] The identifier is either a first value indicating that the decoder network used to decode the bitstream and obtain the picture to be processed is a first decoder network, or a second value indicating that the decoder network used to decode the bitstream and obtain the picture to be processed is a second decoder network. It can also be understood that the identifier is a first value and the encoder performs the encoding operation on the picture to be processed by using a first encoder network corresponding to the first decoder network, or the identifier is a second value and the encoder performs the encoding operation on the picture to be processed by using a second encoder network corresponding to the second decoder network. The identifier may be called profile information (Profile ID), or network information, network identifier, or by any other name. This is not limited to this embodiment of the present application. In other words, the identifier indicates the processing that needs to be supported by the decoder. For example, the general_profile_idc syntax element in the H.265 standard. In one example, the first value may be 0 and the second value may be 1, or the first value may be 1 and the second value may be 0. The first and second values ​​may be other values ​​instead. The processing resources (or computing power) required by the first decoder network are higher than those required by the second decoder network. Processing resources (computing power) may include memory resources, processor resources, etc. In some embodiments, it may be understood that the decoding rates (or decompression efficiencies) of the first decoder network and the second decoder network are different. For example, the decoding rate of the first decoder network is higher than that of the second decoder network, or the quality of the picture restored by the first decoder network is different from the quality of the picture restored by the second decoder network.For example, the quality of the picture restored by the first decoder network is higher than the quality of the picture restored by the second decoder network.

[0146] 502: The encoder transmits the bitstream.

[0147] 503: The decoder decodes the received bitstream to obtain identification information indicating the decoder network to be used.

[0148] 504: If the identification information is a first value, the bitstream is decoded using the first decoder network to obtain the picture to be processed; or, if the identification information is a second value, the bitstream is decoded using the second decoder network to obtain the picture to be processed.

[0149] In one possible implementation, the first decoder network and the second decoder network are entirely different decoder networks, or they share a portion of a subnet, or the second decoder network is a subnet of the first decoder network.

[0150] In some embodiments, the identification information may further include other values ​​indicating different decoder networks. It can be understood that multiple different decoder networks may be indicated by multiple different values. For example, the identification information may be a third value indicating that the decoder network used is a third decoder network. The decoding rate of the third decoder network is different from that of the first decoder network (or second decoder network). In some embodiments, the decoding rate of the third decoder network is higher than that of the second decoder network, and the decoding rate of the second decoder network is higher than that of the first decoder network. In some other embodiments, the decoding rate of the third decoder network is between the decoding rate of the first decoder network and the decoding rate of the second decoder network. Here, only three decoder networks are used as examples. The number of decoder networks is not particularly limited in this embodiment of the present application.

[0151] It can be understood that a higher picture decoding rate results in a shorter picture decoding delay.

[0152] In another example, the quality of the picture restored by the third decoder network differs from the quality of the picture restored by the first decoder network (or the second decoder network).

[0153] The quality of the picture restored by the third decoder network is higher than that of the picture restored by the second decoder network, and the quality of the picture restored by the second decoder network is higher than that of the picture restored by the first decoder network. In some other embodiments, the quality of the picture restored by the third decoder network is between the quality of the picture restored by the first decoder network and the quality of the picture restored by the second decoder network.

[0154] In some scenarios, if a third decoder network is further included, the third decoder network is distinct from the first decoder network (and the second decoder network). For example, the third decoder network, the second decoder network, and the first decoder network are three different decoder networks, or the third decoder network and the second decoder network (or the first decoder network) share a portion of the subnet, or the third decoder network is a subnet of the second decoder network (or the first decoder network).

[0155] The fact that the first decoder network and the second decoder network share a portion of the subnet can be understood as the first decoder network reusing a portion of the subnet of the second decoder network. For example, the first decoder network includes networks A, B, and C, and the second decoder network includes networks D, B, and C, and the two decoder networks share network B. Therefore, when the first decoder network is used, data is input to network A, the output of network A is input to network B, and the output of network B is input to network C. When the second decoder network is used, data is input to network D, the output of network D is input to network B, and the output of network B is input to network C.

[0156] In another example, the first decoder network is a subnet of the second decoder network. For example, the first decoder network includes networks A1, A2, and A3. The second decoder network includes networks A1 and A3. When the first decoder network is used, data is input to network A1, the output of network A1 is input to network A2, and the output of network A2 is input to network A3. When the second decoder network is used, it can be understood that when data is input to network A1, the output of network A1 is not input to network A2, but skips network A2 and is input to network A3.

[0157] In another possible implementation, when performing encoding, the encoder may use different encoder networks based on different values ​​of the identifier. Alternatively, after encoding the picture to be processed into a bitstream by using an encoder network, the encoder may encode the identifier of the decoder network corresponding to the encoder network used into the bitstream. It can be understood that the identifier indicates both the decoder network used and the encoder network used. If the identifier is a first value, the residual information obtained by encoding the picture to be processed using a first encoder network is encoded into the bitstream; or if the identifier is a second value, the residual information obtained by encoding the picture to be processed using a second encoder network is encoded into the bitstream, where the processing resources (or computing power) required by the first encoder network are higher than those required by the second encoder network. Note that the first encoder network and the first decoder network may be a pair of networks, where the first encoder network is used for encoding and then the first decoder network is used for decoding, and the second encoder network and the second decoder network are a pair of networks, where the second encoder network is used for encoding and then the second decoder network is used for decoding.

[0158] In some embodiments, the first encoder network and the second encoder network are two different encoder networks, or the first encoder network and the second encoder network share a portion of a subnet, or the first encoder network is a subnet of the second encoder network.

[0159] The identifier may have other values, and different values ​​indicate different encoder networks to be used. For example, the identifier is a third value, the encoder network to be used is a third encoder network, and the decoder network to be used is a third decoder network. Note that the third encoder network and the third decoder network are a pair of networks, and the third encoder network is used for encoding, and then the third decoder network is used for decoding. The encoding rate of the third encoder network is different from the encoding rate of the first encoder network (or second encoder network). In some embodiments, the encoding rate of the third encoder network is higher than the encoding rate of the second encoder network, and the encoding rate of the second encoder network is higher than the encoding rate of the first encoder network. In some other embodiments, the encoding rate of the third encoder network is between the encoding rate of the first encoder network and the encoding rate of the second encoder network.

[0160] It can be understood that a higher picture encoding rate results in shorter picture encoding latency.

[0161] In another example, the quality of the picture restored by the third encoder network differs from the quality of the picture restored by the first encoder network (or the second encoder network).

[0162] The quality of the picture restored by the third encoder network is higher than that of the picture restored by the second encoder network, and the quality of the picture restored by the second encoder network is higher than that of the picture restored by the first encoder network. In some other embodiments, the quality of the picture restored by the third encoder network is between the quality of the picture restored by the first encoder network and the quality of the picture restored by the second encoder network.

[0163] In some scenarios, if a third encoder network is further included, the third encoder network is distinct from the first encoder network (and the second encoder network). For example, the third encoder network, the second encoder network, and the first encoder network are three different encoder networks, or the third encoder network and the second encoder network (or the first encoder network) share a portion of a subnet, or the third encoder network is a subnet of the second encoder network (or the first encoder network).

[0164] For example, the first encoder network includes a feature extraction module, a quantization module, a side information extraction module, an entropy encoding module, and a probability estimation module. The second encoder network also includes a feature extraction module, a quantization module, a side information extraction module, an entropy encoding module, and a probability estimation module. In some embodiments, the network structure used for at least one module in the first encoder network is different from the network structure used for at least one module in the second encoder network. For example, the network structure of the probability estimation module in the first encoder network is different from the network structure of the probability estimation module in the second encoder network. In another example, the network structure of the feature extraction module in the first encoder network is different from the network structure of the feature extraction module in the second encoder network.

[0165] For example, a feature extraction module in a first encoder network is called the first feature extraction module, and a feature extraction module in a second encoder network is called the second feature extraction module. Note that for each module belonging to a neural network, "module" can also be called "network." For example, a feature extraction module may be called a feature extraction network. In another example, a quantization module may be called a quantization network. The network structure of the feature extraction module in the first encoder network differs from the network structure of the feature extraction module in the second encoder network if the second feature extraction network is a subnet of the first feature extraction network, or if the second feature extraction network and the first feature extraction network are two completely different subnets, or if the second feature extraction network and the first feature extraction network share one or more subnets. The difference between the network structure of the probability estimation module in the first encoder network and the network structure of the probability estimation module in the second encoder network can be that the second feature extraction network is a subnet of the first feature extraction network, or that the second feature extraction network and the first feature extraction network are two completely different subnets, or that the second feature extraction network and the first feature extraction network share one or more subnets.

[0166] For an example, please refer to Figure 6A. The network structure of the first encoder network is as follows: The first encoder network includes a first feature extraction network 610, a quantization network 620, an autoregressive network 630, a side information extraction network 640, and a probability estimation network 650.

[0167] Furthermore, residual information obtained by encoding the picture to be processed using the first encoder network can be implemented in the following implementation. Figure 7A is a diagram of a possible process for encoding residual information.

[0168] 701a: A 3D feature map of the picture to be processed is extracted using the first feature extraction network 610.

[0169] 702a: Quantize the 3D picture features using the quantization network 620 to obtain a quantized 3D feature map.

[0170] 703a: By using the side information extraction network 640, side information of the picture to be processed is extracted from the edges in the 3D feature map.

[0171] 704a: Estimate the first probability distribution mean of the picture to be processed by using the probability estimation network 650 based on the side information.

[0172] 705a: The quantized 3D feature map and the first probability distribution information are input to the autoregressive network 630 to obtain the mean of the second probability distribution.

[0173] 706a: Residual information is obtained based on the 3D feature map of the picture to be processed and the mean of a second probability distribution.

[0174] The first encoder network may further include an entropy encoder network 660, which can encode residual and side information into the bitstream.

[0175] In some scenarios, it may be understood that side information is encoded in bitstream 1, residual information in bitstream 2, and then bitstream 1 and bitstream 2 are combined into a single bitstream. In other scenarios, side information and residual information may be encoded into a single bitstream.

[0176] For another example, see Figure 6B. The network structure of the second encoder network is as follows: The second encoder network includes a second feature extraction network 611, a side information extraction network 640, and a probability estimation network 650.

[0177] Furthermore, residual information obtained by encoding the picture to be processed using a second encoder network can be implemented in the following implementation. Figure 7B shows a possible process for encoding residual information.

[0178] 701b: A 3D feature map of the picture to be processed is extracted by using the second feature extraction network 611.

[0179] 702b: Side information is extracted from the 3D feature map by using the side information extraction network 640.

[0180] 703b: Estimate the mean of the probability distribution of the picture to be processed by using the probability estimation network 650 based on side information.

[0181] 704b: Picture to be processed 3 Residual information is obtained based on dimensional feature maps and the mean of the probability distribution.

[0182] The second encoder network may further include an entropy encoder network 660. The entropy encoder network 660 can encode residual and side information into bitstreams. In some scenarios, it may be understood that the side information is encoded into bitstream 1, the residual information into bitstream 2, and then bitstreams 1 and 2 are combined into a single bitstream. In other scenarios, the side information and residual information may be encoded into a single bitstream.

[0183] The identification information in this embodiment of the present application may be located in the bitstream header. In some scenarios, the identification information may alternatively be added to the suffix of the bitstream file. For example, different identification information may correspond to different suffixes. For example, the header may include information such as picture length and width, picture format, and profile ID. This information must be stored in an agreed sequence. The specific storage sequence is not particularly limited in this application.

[0184] For example, the bitstream header may contain one or more of the following parameter information: profile information (profile ID), picture height (H) and width (W), tile position and size in latent space, control flags for each tool, scaling factors for the first and second components, model index (model Idx): learnable model index, and bitrate control parameter β. v It includes the rate control parameter, which is the rate control parameter β of the first-order component. Y , the rate control parameter β of the second-order component UV This includes, among others.

[0185] For example, the parameters in the bitstream header may be encoded by using a fixed bit length.

[0186] Next, we will explain the parameter information.

[0187] W represents the width of the input picture. For example, W may range from 1 pixel to 8192 pixels.

[0188] H represents the height of the input picture. For example, H may range from 1 pixel to 8192 pixels.

[0189] The `format` parameter represents the data format of the input picture, such as YUV420, YUV444, or sRGB.

[0190] `bit_depth` represents the bit depth of the input picture, for example, 8 and 10.

[0191] β is a parameter representing the quality level of the variable rate. The primary and secondary components may have different β values. Therefore, the primary component is beta_luma(β Y ) is expressed as, and the secondary component is beta_chroma(β UV It is expressed as β. Y The value of is between 0 and 1 and may be expressed as a 16-bit fixed-point number. Y represents luminance (or luma). UV represents chrominance (chroma). (A parameter indicating the quality level for a variable rate. The primary and secondary components may have different betas, and thus the primary (beta_luma) and secondary (beta_chroma) are signaled. Value β v It lies between 0 and 1 and is signaled as a 16-bit fixed-point number.

[0192] Color transformation information (color_transform_info): By default, the signal's encoded representation is YUV Bt.709 (full range). However, custom color transformations are also supported. In this case, 12 coefficients (transform matrix and offset) are used and can be encoded as a fixed-point number with 8-bit resolution.

[0193] Tile information: Represents the decoder tile size and overlap for luminance, and for chrominance. Inter-Channel Correlation Information filter (ICCI) tile size and overlap. Generally, tiles have a square shape. However, tiles may be smaller at the right or bottom picture boundary, so tiles at the right or boundary may be non-rectangular. (Decoder tile size and overlap for lumens, decoder tile size and overlap for chromens. ICCI tile size and overlap. Tiles have a square shape except at the right or bottom picture boundary, where tiles may be smaller and non-rectangular.)

[0194] Skip Mode Enable Flag (SkipMode_enable_flag): Indicates whether Skip Mode is used for picture encoding. (Signaled per picture to indicate whether Skip Mode is used).

[0195] RVS Enable Flag (RVS_enable_flag): Indicates whether to use the Residual and Variance Scale (RVS) to encode each picture.

[0196] LSBS Enable Flag (LSBS_enable_flag): Indicates whether to use Latent Scale Before Synthesis (LSBS) to encode each picture.

[0197] ICIC Enable Flag (ICIC_enable_flag): Indicates whether to use the Inter-Channel Correlation Information filter (ICCI) to encode each picture.

[0198] numThreads: A 16-bit unsigned integer that specifies the number of samples to be processed in parallel. (A 16-bit unsigned integer that specifies the number of samples to be processed in parallel).

[0199] Note that in some scenarios, different values ​​of the identification information (profile information) may indicate only different decoder networks used. In some other scenarios, different encoder networks are used based solely on different values ​​of the identification information. In some other scenarios, different values ​​of the identification information indicate different encoder and decoder networks used.

[0200] For a possible example, see Figure 8. The first decoder network (or second decoder network) may include an entropy decoder network 810, a probability estimation network 820, and a picture reconstruction network. The picture reconstruction network may also be called the reconstruction network, or may have another name, which is not particularly limited in this embodiment of the present application. On the decoder side, the entropy decoder network decodes the bitstream to obtain side information and residual information of the picture to be processed. The network structure of at least one network in the first decoder network is different from the network structure of at least one network in the second decoder network, for example, the picture reconstruction network or the probability estimation network. For example, the network structure of the picture reconstruction network in the first decoder network is different from the network structure of the picture reconstruction network in the second decoder network. For ease of distinction, the picture reconstruction network in the first decoder network is referred to as the first picture reconstruction network 831, and the picture reconstruction network in the second decoder network is referred to as the second picture reconstruction network 832. The first picture restoration network 831 differs from the second picture restoration network 832 in that, for example, the second picture restoration network 832 is a subnet of the first picture restoration network 831, or the second picture restoration network 832 and the first picture restoration network 831 share a portion of a subnet, or the second picture restoration network 832 and the first picture restoration network 831 are two different networks. The first decoder network further includes an autoregressive network 840.

[0201] The decoding process will be described with reference to example structures of the first and second decoder networks.

[0202] Figure 9A is a diagram of a possible decoding process using the first decoder network.

[0203] 901a: By using the entropy decoder network 810, the bitstream is decoded to obtain side information of the 3D feature map of the picture to be processed. Here, the 3D feature map contains multiple feature elements.

[0204] 902a: Based on the side information, the mean of the first probability distribution of the feature element to be decoded is estimated using the probability estimation network 820.

[0205] 903a: Based on the first probability distribution mean and the decoded feature element, the second probability distribution mean of the decoded feature element is determined by using the autoregressive network 840.

[0206] 904a: The bitstream is decoded using the entropy decoder network 810 based on a second probability distribution mean to obtain residual information of the feature element to be decoded, and the feature element to be decoded is obtained based on the residual information and the second probability distribution mean.

[0207] 905a: Based on the 3D feature map obtained through decoding, the picture to be processed is restored using the first picture restoration network 831.

[0208] Figure 9B is a diagram of a possible decoding process using a second decoder network.

[0209] 901b: By using the entropy decoder network 810, the bitstream is decoded to obtain side information of the 3D feature map of the picture to be processed, and the 3D feature map contains multiple feature elements.

[0210] 902b: Estimate the first probability distribution mean of the feature element to be decoded by using the probability estimation network 820 based on side information.

[0211] 903b: The bitstream is decoded using the entropy decoder network 810 based on the mean of a first probability distribution to obtain residual information of the feature element to be decoded, and the feature element to be decoded is obtained based on the residual information and the mean of the first probability distribution.

[0212] 904b: Based on the 3D feature map obtained through decoding, the picture to be processed is restored using a second picture restoration network 832.

[0213] In some possible implementations, the probabilistic estimation network in the encoder network (including the first and second encoder networks) may be the same as the probabilistic estimation network used in the decoder network. [Examples]

[0214] The following describes the solutions in embodiments of the present invention with reference to specific examples. The following examples are illustrated by using an end-to-end picture encoding and decoding process as an example.

[0215] Example 1:

[0216] Figures 10A and 10B illustrate the execution processes of an encoder network and a decoder network according to embodiments of the present invention. The encoder and decoder networks are dynamically adjusted based on a profile ID. The encoder network is described with reference to the aforementioned network structure in Figures 6A and 6B. The first feature extraction network (module) 610 includes encoder network submodules 1 to 3. Encoder network submodules 1 to 3 extract features from the picture to be processed, gradually converting the picture from pixel regions to feature regions, thereby making the picture easier to compress. The second feature extraction module in the second encoder network includes encoder network submodule 1 and encoder network submodule 3.

[0217] Correspondingly, the decoder network is described with reference to the network structure shown in Figure 8. Decoder network submodules 1→2→3 or 1→2→4 gradually reconstruct the 3D feature map into a picture. The difference between decoder network submodule 3 and decoder network submodule 4 lies in their structure. For example, decoder network submodule 3 is a lightweight module suitable for terminal devices with low computing power, and is characterized by faster execution but poorer picture reconstruction quality than decoder network submodule 4, while decoder network submodule 4 is a module suitable for devices with high computing power, and is characterized by slower execution but better picture reconstruction quality than decoder network submodule 3. In Figure 10B, an example is used in which the first picture reconstruction network of the first decoder network includes decoder network submodules 1, 2, and 4, and an example is used in which the second picture reconstruction network of the second decoder network includes decoder network submodules 1-3.

[0218] Figure 10A shows the encoding process according to Example 1. Specifically, the implementation process on the encoder side is as follows.

[0219] Step 1: Calculate and output picture features y by using the Feature Extraction module. During the calculation, the execution or skipping of several network submodules is selected based on the profile ID. If the profile ID is 0, encoder network submodule 2 is executed, i.e., encoding is performed by using the first encoder network. If the profile ID is 1, encoder network submodule 2 is skipped, i.e., encoding is performed by using the second encoder network. In some scenarios, encoder network submodule 2 may be skipped when the profile ID is 1, or encoder network submodule 2 may be executed when the profile ID is 0. For example, picture features y may also be called feature map y, or 3D feature map y. After feature extraction is performed on the picture to be encoded using the Feature Extraction module to obtain picture features y, the picture features y may be further quantized, which may be understood as processing the floating-point feature values ​​(e.g., truncating) to obtain integer feature values, thereby obtaining the quantized feature map ^y (for convenience, y with ^ is sometimes written this way).

[0220] Stage 2: The picture features y calculated in Stage 1 are input to the side information extraction network (module) to extract side information z; z is quantized to obtain ^z, and ^z is compressed to create bitstream 1. Note that the side information extraction module is not mandatory. In some possible scenarios, after feature extraction is performed on the original picture, quantization and compression (or encoding) are performed directly to generate the bitstream.

[0221] In some embodiments, a quantized feature map ^y may be input to a side information extraction network, which may output quantized side information ^z. The side information extraction module may be implemented by a neural network. Specific neural network structures will be illustrated later with examples and will not be described in detail here. Side information ^z may also be understood as a feature map ^z obtained by performing further feature extraction on the quantized feature map ^y, where the number of feature elements in ^z is less than the number of feature elements in feature map ^y.

[0222] In some scenarios, the encoder network (first encoder network and second encoder network) may further include a quantization network configured to perform quantization operations on picture features y. In some other scenarios, the side information extraction network may have the capability to perform quantization operations, thereby enabling the side information extraction network to perform quantization operations on picture features y.

[0223] Step 3: Obtain the probability distribution of picture feature y from the side information. If the profile ID is 1, the side information is input to the probability estimation network. The probability estimation network (also called the probability estimation module) includes feature probability distribution modules A and B that predict the mean and variance information of picture feature y. Feature probability distribution modules A and B may also be called feature map probability distribution estimation modules A and B, or by other names. This is not limited to this embodiment of the present application. If the profile ID is 0, the side information is input to feature probability distribution modules A and B. Feature probability distribution module B outputs the variance information of picture feature y. The output of feature probability distribution module A and the quantized picture feature y need to be sent to the autoregressive module to generate the mean of picture feature y. In some scenarios, feature probability distribution modules A and B may be combined into a single module, i.e., their functions are performed by a single module.

[0224] For example, the probabilistic estimation network may use a Gaussian single model (GSM), an asymmetric Gaussian model, a Gaussian mixture model (GMM), or a Laplace distribution model. Alternatively, the probabilistic estimation network may be a deep learning-based network, such as a recurrent neural network (RNN) or a convolutional neural network (CNN), but this is not limited herein.

[0225] Step 4: Referencing the probability distribution information (mean, variance) of the picture feature y obtained in Step 3, calculate the residual information r = y - mean of the picture feature y relative to the mean, and perform entropy encoding on the quantized residual information ^r to obtain the compressed bitstream 2. The residual information r is sometimes called the residual feature map r. Therefore, the quantized residual information ^r is sometimes called the quantized residual feature map ^r, or simply the quantized residual feature ^r.

[0226] Step 5: Combine bitstream 1 and bitstream 2 into a single bitstream and write the profile ID to the bitstream, for example, in the bitstream header information.

[0227] It should be noted that the encoding stages in stages 2, 4, and 5 on the encoder side may be combined. In stage 2, the side information ^z is not encoded and is not written to the bitstream. Instead, after the quantized residual information ^r is obtained in stage 4, the quantized residual information ^r and the side information ^z are encoded (e.g., entropy encoded) and written to the bitstream.

[0228] Figure 10B shows the decoding process according to Example 1. Specifically, the implementation process on the decoder side is as follows.

[0229] Step 1: By using an entropy decoder network (module), parse the bitstream to obtain profile ID information, for example, obtain the profile ID from the header of the bitstream. The profile ID is profile information within the bitstream and indicates the processing that needs to be supported by the decoder. For example, it is the general_profile_idc syntax element in the H.265 standard. The profile ID may be an integer (of course, the profile ID may not be an integer, and this is not particularly limited in this application). The profile information may be understood to indicate the processing that needs to be supported by the decoder or the different networks that the profile information needs to be used by the decoder.

[0230] Step 2: By using an entropy decoder network, decode the bitstream (for example, bitstream 1) to obtain side information. For example, the bitstream 1 may be decoded through an asymmetric numeric system (ANS) / arithmetic decoding to obtain the quantized side information ^z.

[0231] Step 3: Obtain the probability distribution of the feature map ^y from the side information ^z by using a probability estimation network (module). When the profile ID = 1, the side information ^z is input into a probability estimation module (or called a probability estimation network), and probability estimation is performed for each feature element ^y[x][y][i] in the feature map ^y to be decoded, and the probability distribution of the feature element ^y[x][y][i] is obtained. It is assumed that the feature element ^y[x][y][i] follows a Gaussian distribution with mean μ[x][y][i] and variance σ [x][y][i]. Here, the mean μ[x][y][i] may be used as the predicted value of the feature element ^y[x][y][i]. When the profile ID = 0, the side information ^z is input into a probability estimation module (or called a probability estimation network), and probability estimation is performed for each feature element ^y[x][y][i] in the feature map ^y to be decoded, and the probability distribution of the feature element ^y[x][y][i] is obtained. Then, the predicted value of the current feature element to be decoded is obtained based on the autoregressive network by using the information of the decoded feature element and the mean output by the probability estimation network.

[0232] The parameters x, y, i in the feature element ^y[x][y][i] are all positive integers, and the coordinates (x, y, i) represent the position of the current feature element to be decoded. Specifically, the coordinates (x, y, i) represent the position of the current feature element to be decoded relative to the feature element at the upper left vertex in the current 3D feature map. This step can be specifically implemented by a probability estimation module. The probability estimation method used on the decoder side may be the same as and corresponding to the probability estimation method used on the encoder side, that is, the structure of the probability estimation module on the decoder side may be the same as the structure of the probability estimation module on the encoder side, and the details are not described here.

[0233] Bitstream 2 may be understood as a bitstream transformed from multiple matrices y, and decoding is the process of reconstructing multiple matrices y from the bitstream. Reconstructing y means sequentially reconstructing elements. For example, in a 10x10 matrix, the elements of the matrix are reconstructed one by one in order from left to right and from top to bottom. When the element in the 7th row and 8th column is reconstructed, the elements preceding that element (i.e., all elements whose horizontal coordinate is less than 7 and whose vertical coordinate is less than 8) may be called the context of that element, or may be understood as decoded context information.

[0234] Step 4: By using an entropy decoder network, the bitstream is continued to parse by using the Gaussian distribution of the mean μ and variance σ of each feature element in the quantized feature map ^y obtained in Step 3 to obtain a quantized residual feature map ^r, and further, a quantized feature map ^y = ^r + μ is obtained based on ^r and μ.

[0235] One possible implementation for parsing a bitstream and obtaining a feature map ^r is as follows:

[0236] The probability P(k) of having the value k of the feature element ^r[x][y][i] to be decoded is obtained based on a probability distribution (e.g., a Gaussian distribution with mean 0 and variance σ), the bitstream is parsed, and the feature element ^r[x][y][i] is obtained through ANS decoding / arithmetic decoding based on P(k). k may be any integer, for example, 0, 1, 2, or 3.

[0237] Step 5: Reconstruct the picture from the quantized ^y by using the picture reconstruction network. In the process of running the picture reconstruction network (or reconstruction network), after decoder network submodule 1 and decoder network submodule 2 have been executed, decoder network submodule 3 and decoder network submodule 4 are selected based on the profile ID value. If the profile ID is 1, decoder network submodule 3 is selected, i.e., the second decoder network is used; or if the profile ID is 0, decoder network submodule 4 is selected, i.e., the first decoder network is used.

[0238] The following describes the structure of each subnetwork in the encoder network described above (including the first and second encoder networks), with reference to specific examples. Figures 11A and 11B illustrate the execution process of the encoder network. Note that Figures 11A and 11B are merely examples and do not constitute a limitation on the specific structure of the encoder network.

[0239] Refer to Figures 11A and 11B. Encoder network submodule 1 includes multiple layers, which are, respectively, a padding layer 1_1, a convolutional layer 1_11, a residual activation function (ResAU) layer 1_21, padding 1_2, a convolutional layer 1_12, a residual activation function layer 1_22, and padding 1_3. In this embodiment of the present application, a convolution with a convolutional size of K×K, a number of output channels of M, and a stride of N may be represented as Conv M×K×K SN. In Figures 11A and 11B, an example is used in which convolutional layers 1_11 and 1_12 use Conv12 28×3×3 S2.

[0240] For example, a padding layer can use zero padding (constant padding) (default is padding 0), reflect padding, replicated padding, and circular padding. For example, padding1_1, padding1_2, and padding1_3 all use replicated padding, making the length and width of the input tensor even by padding using replicated padding (padding using the nearest element). For example, if the input tensor has a length of 5 and a width of 6, the padding layer pads the elements in the length direction to change the length to an even number, 6. However, the width of the input tensor is 6, which is an even number. Therefore, no padding operation is performed in the width direction.

[0241] The residual activation function layers 1_21 and 1_22 are primarily used as activation functions and can further provide an attention mechanism. As an example, the residual activation function layers 1_21 and 1_22 can use a ResAU 3×3 no-tanh network. The ResAU 3×3 no-tanh network may use the structure shown in Figure 12. The residual activation function layer 1_21 (and residual activation function layer 1_22) includes activation function layer 2_1 and convolutional layer 2_1. In Figure 12...

number

[0242] The encoder network submodule 2 may use a residual non-local attention block (RNAB) configured to provide an attention mechanism, for example, to provide global or local attention information in space. For example, Figure 13 shows a possible structure of an RNAB. The RNAB uses a network structure that includes multiple residual block (RB) layers, multiple convolutional layers, deconvolutional layers, and an activation function layer (for example, a sigmoid function is used). By using the RNAB, global or local attention information in space is extracted.

[0243] In one example, the residual block layer may use the network structure shown in Figure 14. The RB layer may include a convolutional layer 4_1, an activation function layer 4_11, and a convolutional layer 4_2. In some embodiments, the activation function layer 4_11 may use a Leaky ReLU function. The main line of the residual block layer in Figure 14 performs the operation of inputting features into a 3x3 convolutional layer 4_1 to obtain a feature matrix, then outputting the feature matrix by using an activation function, and then adding the result obtained by using a 3x3 convolutional layer 4_2 to the input features.

[0244] Refer to Figures 11A and 11B. Encoder network submodule 3 includes convolutional layer 5_1, residual activation function layer 5_11, padding layer 5_12, convolutional layer 5_2, and convolutional layer 5_3. In Figures 11A and 11B, convolutional layers 5_1 and 5_2 may use conv 128×3×3 S2. Convolutional layer 5_3 uses conv 128×1×1 S1. Residual activation function layer 5_1 1 For example, a network structure like the one shown in Figure 12 can be used, which is a ResAU 3x3 without tanh.

[0245] The quantization network may include a rounding layer 6_11 configured to perform a quantization operation, sometimes called a rounding operation, and return a rounded floating-point value. See Figures 11A and 11B for some embodiments. The quantization network may further include a Gunit layer 6_1 and an invGunit layer 6_21. The Gunit layer 6_1 and the invGunit layer 6_21 are configured to perform bitrate matching so that the encoder network has bitrate adjustment capability. In one example, the Gunit layer 6_1 and the invGunit layer 6_21 may use the Gain and Inverse Gain structure described in Non-Patent Literature 1. [Non-Patent Document 1] Ze Cui, Jing Wang et al., "G-VAE: A CONTINUOUSLY VARIABLE RATE DEEP IMAGE COMPRESSION FRAMEWORK"

[0246] In FIGS. 11A and 11B, the side information extraction module may include a hyper encoder net and a rounding layer. The hyper encoder net may also be referred to as a hyper encoder network. The function of the hyper encoder net is to extract side information z using the input quantized picture feature y. In FIGS. 11A and 11B, the autoregressive network may include a context model net and a prediction fusion net. The context model net is an autoregressive process. The context model net refers to the prediction fusion net and predicts the expected value of the encoding target element ^y[:,i,j] using information about the encoded element ^y[:,i',j'] (i'≦i, j'<j-i-j'). For example, in a certain implementation process, mask convolution (mask conv) may be used for implementation. The prediction fusion net is configured to receive information about the encoded element extracted by the context model net and the side information extracted by the hyper decoder and predict the expected value (or predicted value) of the encoding target element. The expected value of the encoding target element may be understood as the average of the predicted probability distribution.

[0247] In some possible implementation scenarios, the feature probability estimation module A may use a Hyper Decoder Net. The feature probability estimation module B may use a Hyper Scale Decoder Net. Of course, another network structure may be alternatively used, and any network capable of performing probability estimation for side information is applicable to the present application. In one example, the feature probability estimation module A can use the network structure of the hyper decoder network shown in FIG. 15. The feature probability estimation module A includes a convolutional layer 7_1, a deconvolutional layer 7_11, a Crop layer 7_21, an activation function layer 7_31, a convolutional layer 7_2, a deconvolutional layer 7_12, a Crop layer 7_22, an activation function layer 7_32, a convolutional layer 7_3, and an activation function layer 7_33.

[0248] The Crop layer 7_21 and the Crop layer 7_22 are configured to perform a crop operation on the input tensor. The crop operation may be represented as Crop(H out ,W out ,d,s d ). Here, H out , W out are the length and width of the picture finally output by the decoder network (or may be understood as the size of the picture input by the encoder network, and the size information may be obtained from the header of the bitstream), and s d is the stride information of the deconvolution operation. In one example, s d =2. d represents the depth of the deconvolutional layer. The Crop layer inputs a tensor with a size of [C, s d h d , s d w d and outputs a tensor with a size of [C, h d-1 , w d-1 . h d =ceil(h d-1 / s d );wd = ceil(w d-1 / s d )、h0 = H out 、w0 = W out It is. In one example, in FIGS. 11A and 11B, an example where the activation function layers 7_31, 7_32, and 7_33 use the LeakyRelu function is used. In FIGS. 11A and 11B, the convolutional layer 7_1 uses conv 128×1×1 S1, the deconvolutional layer 7_11 uses DConv 128×4×4 S2, the convolutional layer 7_2 uses conv 128×3×3 S1, the deconvolutional layer 7_12 uses DConv 128×4×4 S2, and the convolutional layer 7_3 uses conv 128×3×3 S1.

[0249] In another example, the feature probability estimation module B may use the network structure of the hyper-scale decoder network shown in FIG. 16. The feature probability estimation module B includes a deconvolutional layer 7_13, a Crop layer 7_23, an activation function layer 7_34, a convolutional layer 7_4, an activation function layer 7_34, a deconvolutional layer 7_14, a Crop layer 7_24, an activation function layer 7_35, a convolutional layer 7_5, and a Gunit layer 7_6. In FIGS. 11A and 11B, the deconvolutional layer 7_13 uses DConv 128×4×4 S2, the activation function layer 7_34 uses the LeakyRelu function, the convolutional layer 7_4 uses conv 128×3×3 S1, the deconvolutional layer 7_14 uses DConv 128×4×4 S2, the activation function layer 7_35 uses the LeakyRelu function, and the convolutional layer 7_5 uses conv ******** S1.

[0250] In FIGS. 11A and 11B, a lossless encoder is used in the entropy encoder network, and the function of the lossless encoder is to convert the feature to be encoded into a bitstream.

[0251] The following describes the structure of each subnetwork of the decoder network described above (including the first and second decoder networks) with reference to specific examples. Figures 17A and 17B illustrate the execution process of the decoder network. Note that Figures 17A and 17B are merely examples and do not constitute a limitation on the specific structure of the decoder network. In Figures 17A and 17B, a reversible decoder is used in the entropy decoder network, and the function of the reversible decoder is to reconstruct the features of the bitstream to be decoded. The probabilistic estimation network within the decoder network may use the same structure as the encoder network. See Figures 11A and 11B for details. Decoder network submodule 1 includes an invGunit layer, a LightResBlock, a deconvolution layer 8_1, a crop layer 8_11, and a residual activation function layer 8_21. Decoder network submodule 2 includes a deconvolution layer 8_2, a crop layer 8_12, and a residual activation function layer 8_22. The residual activation layers 8_21 and 8_22 may use a ResAU structure. The deconvolution layer 8_1 may use a Dconv 96×4×4 S2, and the deconvolution layer 8_2 may use a Dconv 64×4×4 S2.

[0252] Decoder network submodule 3 may include a convolutional layer 8_31, a residual activation function layer 8_23, a convolutional layer 8_32, a PxlShuffleS4, and a crop layer 8_13. Decoder network submodule 4 may include a deconvolutional layer 8_3, an RNAB, a crop layer 8_14, a residual activation function layer 8_24, a deconvolutional layer 8_4, and a crop layer 8_15. For example, the RNAB may use the network structure shown in Figure 13.

[0253] For an example, see Figure 18 for the network structure of the LightResBlock. PxlShuffleS4: represents a 4x upsampling pixel shuffle operation.

[0254] Example 2:

[0255] The encoder network used in Example 2 is the same as that used in Example 1, and the execution process is also the same. Details are not described here. The decoder network in Example 2 is different from the decoder network in Example 1. In Example 2, the second decoder network in the decoder network is a subnet of the first decoder network, or the second picture restoration network in the second decoder network is a subnet of the structure of the first picture restoration network in the first decoder network. See Figure 19. The second picture restoration network includes decoder network submodule 5 and decoder network submodule 7, and the first picture restoration network includes decoder network submodules 5-7.

[0256] Unlike Example 1, in Example 2, if the profile IDs are different, the decoder does not make a selection between Decoder Network Submodule 3 and Decoder Network Submodule 4, but rather chooses whether to skip the Decoder Network Submodule or not. If Profile ID = 1, Decoder Network Submodule 6 is skipped. If Profile ID = 0, Decoder Network Submodule 6 is executed.

[0257] Example 3:

[0258] Example 3 uses an example where the feature extraction networks of the two encoder networks are two different networks, and the picture reconstruction networks of the two decoder networks are two different networks. See Figure 20A. The encoder network includes a first feature extraction network, a second feature extraction network, a quantization network, an autoregressive network, a side information extraction network, a probability estimation network, and an entropy encoder network. Correspondingly, see Figure 20B. The decoder network includes an autoregressive network, a side information extraction network, a probability estimation network, a first picture reconstruction network, a second picture reconstruction network, and an entropy decoder network.

[0259] As shown in Figure 20A, the difference between the encoder implementation process and Example 1 lies in step 1. In the process of calculating the input picture feature y, different feature extraction networks are selected based on the profile ID. If profile ID = 0, the first feature extraction network is selected. If profile ID = 1, the second feature extraction network is selected.

[0260] Similarly, the difference between the decoder implementation process and the implementation process of Example 1 lies in step 5, namely, the picture is restored from ^y by using the picture restoration network. When the decoder network is running, a picture restoration network of a different structure is selected based on the profile ID. If profile ID = 0, the first picture restoration network is selected. If profile ID = 1, the second picture restoration network is selected.

[0261] The following describes the structure of each subnetwork of the encoder network described above (including the first and second encoder networks) with reference to specific examples. Figures 21A and 21B illustrate the execution process of the encoder network. Note that Figures 21A and 21B are merely examples and do not constitute a limitation on the specific structure of the encoder network. In Figures 21A and 21B, the first feature extraction network includes a padding layer 1_1, a convolutional layer 1_11, a residual activation function (ResAU) layer 1_21, padding 1_2, a convolutional layer 1_12, a residual activation function layer 1_22, padding 1_3, RNAB, a convolutional layer 5_1, a residual activation function layer 5_11, a padding layer 5_12, a convolutional layer 5_2, and a convolutional layer 5_3. The second feature extraction network includes a padding layer 1_1, a convolutional layer 1_11, a residual activation function (ResAU) layer 1_21, padding 1_2, a convolutional layer 1_12, a residual activation function layer 1_22, padding 1_3, a convolutional layer 5_1, a residual activation function layer 5_11, a padding layer 5_12, a convolutional layer 5_2, and a convolutional layer 5_3. For a description of the aforementioned layers, please refer to the relevant descriptions in the embodiments corresponding to Figures 11A and 11B. Details are not described here. For the structure of the other networks in Figures 21A and 21B, please refer to the description in Example 1. Details are not described here.

[0262] Figures 22A and 22B are diagrams illustrating the execution process of a decoder network. Figures 22A and 22B are merely examples. DeIt should be noted that this does not constitute a limitation on the specific structure of the CODA network. The first picture reconstruction network includes an invGunit layer, a LightResBlock, a deconvolution layer 8_1, a crop layer 8_11, a residual activation function layer 8_21, a deconvolution layer 8_2, a crop layer 8_12, a residual activation function layer 8_22, a deconvolution layer 8_3, an RNAB, a crop layer 8_14, a residual activation function layer 8_24, a deconvolution layer 8_4, and a crop layer 8_15. The second picture restoration network includes an invGunit layer, a LightResBlock layer, a deconvolution layer 8_1, a crop layer 8_11, a residual activation function layer 8_21, a deconvolution layer 8_2, a crop layer 8_12, a residual activation function layer 8_22, a convolution layer 8_31, a residual activation function layer 8_23, a convolution layer 8_32, a PxlShuffleS4 layer, and a crop layer 8_13. For a description of the aforementioned layers, please refer to the relevant descriptions in the embodiments corresponding to Figures 11A and 11B. Details are not described herein. For the structure of the other networks in Figures 22A and 22B, please refer to the description of Embodiment 1. Details are not described here.

[0263] Please note that the network structure described above is used only as an example, and the specific network structure is not particularly limited. Any network structure capable of implementing the corresponding functions is applicable to this application.

[0264] In addition, it should be noted that the aforementioned submodule-level tuning and network-wide tuning performed on the encoder and decoder networks based on profile IDs can be flexibly combined. For example, in some possible cases, the encoder side may run a dynamic computation graph on the cloud side using a framework such as PyTorch and tune encoder network submodules based on profile IDs, while the decoder side may run a static computation graph on the device side and switch the entire decoder network based on profile information.

[0265] In the embodiments of this application, the encoder transmits network structure information, which allows the decoder to adjust the decoder network using the bitstream content. This solution has the following advantages: 1. For bitstreams generated using different AI encoder networks, the decoder can select different decoder network structures to perform decoding using the bitstream content. This provides high flexibility to the codec. Users can flexibly balance latency and compression performance by adjusting the computing power of their encoder and decoder networks based on their scenarios. 2. According to one usage scenario, Users can dynamically select and adjust several decoder network modules based on their profile ID, or switch between different decoder networks.

[0266] For the sake of brevity, it will be readily apparent to those skilled in the art that, for specific operating processes of the communication system described above, we refer to the corresponding processes in the method embodiments described above. Further details are not described herein.

[0267] One embodiment of the present application provides a computer-readable medium configured to store a computer program. The computer program includes instructions used to perform the method steps in an embodiment of the method corresponding to Figure 5.

[0268] Those skilled in the art should understand that embodiments of the present application may be provided as methods, systems, or computer program products. Accordingly, the present application may be provided in the form of hardware-only embodiments, software-only embodiments, or embodiments having a combination of software and hardware. Furthermore, the present application may be provided in the form of a computer program product implemented on one or more computer-readable storage media (including, but not limited to, disk memory, optical memory, etc.) containing computer-readable program code.

[0269] This application has been described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of this application. It should be understood that computer program instructions may be used to implement each process and / or each block in the flowcharts and / or block diagrams, as well as combinations of processes and / or blocks in the flowcharts and / or block diagrams. These computer program instructions may be provided to a processor of a general-purpose computer, a dedicated computer, an embedded processor, or any other programmable data processing device to generate a machine, thereby, the instructions executed by the processor of the computer or any other programmable data processing device, to generate a device for implementing a particular function in one or more processes in the flowchart and / or one or more blocks in the block diagram.

[0270] It will be apparent to those skilled in the art that various modifications and variations can be made to this application without departing from its scope. This application is intended to encompass these modifications and variations to the extent that they fall within the scope of protection defined by the following claims and their equivalent art.

Claims

1. Picture encoding method: The step involves encoding identification information indicating the decoder network to be used into a bitstream, The identification information is a first value, indicating that the decoder network used to decode the bitstream to obtain the picture to be processed is the first decoder network, or The aforementioned identification information is a second value, indicating that the decoder network used to decode the bitstream to obtain the picture to be processed is the second decoder network, and the processing resources required by the first decoder network are higher than those required by the second decoder network; The step of transmitting the bitstream Methods that include...

2. The method according to claim 1, wherein the first decoder network and the second decoder network are completely different decoder networks, or the first decoder network and the second decoder network share a portion of a subnet, or the second decoder network is a subnet of the first decoder network.

3. The method in question is: The step of obtaining the aforementioned identification information; When the identification information is the first value, the residual information obtained by encoding the picture to be processed based on the first encoder network is encoded into the bitstream, or When the identification information is the second value, the process involves encoding the residual information obtained by encoding the picture to be processed based on the second encoder network into the bitstream. The processing resources required by the first encoder network are higher than those required by the second encoder network. The method according to claim 1 or 2.

4. The method according to claim 3, wherein the first encoder network and the second encoder network are two different encoder networks, or the first encoder network and the second encoder network share a portion of a subnet, or the second encoder network is a subnet of the first encoder network.

5. The first encoder network includes a first feature extraction network, an autoregressive network, a side information extraction network, and a probability estimation network. The residual information obtained by encoding the picture to be processed using the first encoder network is: A step of extracting a three-dimensional feature map of the picture to be processed by using the first feature extraction network, wherein the three-dimensional feature map includes multiple feature elements; The steps include: extracting side information of the feature element to be encoded from the 3D feature map using the aforementioned side information extraction network; Based on the aforementioned side information, the step of estimating the first probability distribution mean of the feature element to be encoded by using the probability estimation network; The steps include: inputting the encoded feature elements and the first probability distribution mean into the autoregressive network to obtain a second probability distribution mean of the feature elements to be encoded; A step of obtaining residual information of the feature element to be encoded based on the feature element to be encoded and the second probability distribution mean of the feature element to be encoded. The method according to claim 4, including the method described in claim 4.

6. The second encoder network includes a second feature extraction network, a side information extraction network, and a probability estimation network. The residual information obtained by encoding the picture to be processed using the second encoder network is: The step of extracting a three-dimensional feature map of the picture to be processed by using the second feature extraction network, wherein the three-dimensional feature map includes a plurality of feature elements; The steps include: extracting side information of the feature element to be encoded from the 3D feature map using the aforementioned side information extraction network; Based on the aforementioned side information, the step of estimating the mean of the probability distribution of the feature element to be encoded by using the aforementioned probability estimation network; A step of obtaining residual information of the feature element to be encoded based on the feature element to be encoded and the mean of the probability distribution. The method according to claim 5, including the method described in claim 5.

7. The method according to claim 6, wherein the second feature extraction network is a subnet of the first feature extraction network, or the second feature extraction network and the first feature extraction network are two completely different subnets.

8. The method in question is: The step further includes encoding the side information into the bitstream, The method according to any one of claims 5 to 7.

9. The method according to any one of claims 1 to 8, wherein the identification information is located in the header of the bitstream.

10. Picture decoding method: The stage of receiving the bitstream; The steps include: decoding the bitstream to obtain identification information indicating the decoder network to be used; When the aforementioned identification information is a first value, the bitstream is decoded using the first decoder network to obtain the picture to be processed, or When the aforementioned identification information is a second value, the process involves decoding the bitstream and obtaining the picture to be processed by using a second decoder network. A method comprising the above, wherein the processing resources required by the first decoder network are higher than the processing resources required by the second decoder network.

11. The method according to claim 10, wherein the first decoder network and the second decoder network are completely different decoder networks, or the first decoder network and the second decoder network share a portion of a subnet, or the second decoder network is a subnet of the first decoder network.

12. The first decoder network includes an entropy decoder network, a probability estimation network, an autoregressive network, and a first picture reconstruction network. The step of decoding the bitstream using the first decoder network to obtain the picture to be processed is: The steps include: decoding the bitstream using the entropy decoder network to obtain side information of a three-dimensional feature map of the picture to be processed, wherein the three-dimensional feature map includes a plurality of feature elements; Based on the aforementioned side information, the step of estimating the first probability distribution mean of the feature element to be decoded by using the aforementioned probability estimation network; The steps include: determining a second probability distribution mean of the decoded feature element by using the autoregressive network based on the first probability distribution mean and the decoded feature element; The steps include: decoding the bitstream using the entropy decoder network based on the second probability distribution mean to obtain residual information of the feature element to be decoded, and obtaining the feature element to be decoded based on the residual information and the second probability distribution mean; The steps include: restoring the picture to be processed by using the first picture restoration network based on the three-dimensional feature map obtained through decoding; The method according to claim 10 or 11, including the method described in claim 10 or 11.

13. The second decoder network includes the entropy decoder network, the probability estimation network, and the second picture reconstruction network. The step of decoding the bitstream using the second decoder network to obtain the picture to be processed is: The steps include: decoding the bitstream using the entropy decoder network to obtain side information of a three-dimensional feature map of the picture to be processed, wherein the three-dimensional feature map includes a plurality of feature elements; Based on the aforementioned side information, the step of estimating the first probability distribution mean of the feature element to be decoded by using the aforementioned probability estimation network; The steps include: decoding the bitstream using the entropy decoder network based on the first probability distribution mean to obtain residual information of the feature element to be decoded, and obtaining the feature element to be decoded based on the residual information and the first probability distribution mean; The steps include: restoring the picture to be processed by using the second picture restoration network based on the three-dimensional feature map obtained through decoding; The method according to claim 12, including the method described in claim 12.

14. The method according to claim 13, wherein the second picture restoration network is a subnet of the first picture restoration network, or the picture restoration network and the first picture restoration network share a portion of a subnet, or the second picture restoration network and the first picture restoration network are two different networks.

15. A picture encoding device having memory and a video encoder, The memory is configured to store video data, and the video data includes the picture to be processed. The video encoder is configured to encode identification information indicating the decoder network to be used into a bitstream, where The identification information is a first value, indicating that the decoder network used to decode the bitstream and obtain the picture to be processed is the first decoder network, or The aforementioned identification information is a second value, indicating that the decoder network used to decode the bitstream and obtain the picture to be processed is the second decoder network, and the processing resources required by the first decoder network are higher than the processing resources required by the second decoder network. Device.

16. A picture decoding device having memory and a video decoder, The memory is configured to store video data in the form of a bitstream, and the video data includes the picture to be processed. The video decoder decodes the bitstream to obtain identification information indicating the decoder network to be used. If the identification information is a first value, the bitstream is decoded using the first decoder network to obtain the picture to be processed, or If the identification information is a second value, the system is configured to decode the bitstream and obtain the picture to be processed by using a second decoder network, wherein the processing resources required by the first decoder network are higher than those required by the second decoder network. Device.

17. A video decoding device having coupled non-volatile memory and a processor, wherein the processor invokes program code stored in the memory to perform the method according to any one of claims 10 to 14.

18. A video encoding device having coupled non-volatile memory and a processor, wherein the processor calls program code stored in the memory to perform the method according to any one of claims 1 to 14.

19. A computer-readable storage medium, wherein the computer-readable storage medium stores program code, and when the computer program is executed on the computer, the computer is enabled to perform the method according to any one of claims 10 to 14.

20. A computer-readable storage medium, wherein the computer-readable storage medium stores program code, and when the computer program is executed on the computer, the computer is enabled to perform the method according to any one of claims 1 to 9.

21. A computer-readable storage medium, wherein the computer-readable storage medium stores a video bitstream decoded by one or more processors according to the method described in any one of claims 10 to 14.

22. A computer-readable storage medium, the computer-readable storage medium storing a video bitstream obtained by encoding it with one or more processors according to the method described in any one of claims 1 to 9.

Images