Encoding method and device, and decoding method and device

Abstract

Embodiments of the present application relate to the technical field of media. Disclosed are an encoding method and device and a decoding method and device, capable of performing quality adjustment on image content in the spatial dimension by means of JPEG AI. The encoding method comprises: acquiring a target quality matrix; scaling a first residual plot and / or first Gaussian distribution parameter information on the basis of the target quality matrix to obtain a second residual plot and / or second Gaussian distribution parameter information; and generating a bit stream on the basis of the target quality matrix, the second residual plot and / or the second Gaussian distribution parameter information, wherein the target quality matrix is used for representing the image quality of each region in a residual plot of a feature domain, the first residual plot is the residual plot of the feature domain, and the first Gaussian distribution parameter information is Gaussian distribution parameter information of the residual plot of the feature domain.

Description

Coding and decoding method and device

[0001] The present application claims priority to the Chinese patent application No. 202310852519.1, filed on July 11, 2023, and entitled "Coding and decoding method and device", the content of which is incorporated herein by reference in its entirety. TECHNICAL FIELD

[0002] Embodiments of the present application relate to the field of media technology, in particular to a coding and decoding method and device. BACKGROUND

[0003] Joint photographic experts group (JPEG) artificial intelligence (AI) is a learning-based image coding standard that provides a single-stream, compact, compression-domain representation, significantly improving compression efficiency over commonly used image coding standards at the same subjective quality. JPEG AI is widely used in various fields. For example, JPEG AI can be applied to cloud storage, visual monitoring, self-driving cars and devices, image acquisition, storage and management, real-time monitoring of visual data, and media distribution

[0004] In the actual application of video image coding, it is usually necessary to allocate different code rates to the region of interest and the background region for coding, so as to meet the needs of customers with less code rate.

[0005] Therefore, how to adjust the quality of image content in the spatial dimension through JPEG AI is one of the problems that those skilled in the art need to solve.

[0006] SUMMARY

[0007] Embodiments of the present application provide a coding and decoding method and device, which can adjust the quality of image content in the spatial dimension through JPEG AI. To achieve the above purpose, the technical solutions adopted by embodiments of the present application are as follows:

[0008] In a first aspect, the present application provides an encoding method, which comprises: obtaining a target quality matrix. Scaling a first residual map and / or first Gaussian distribution parameter information according to the target quality matrix to obtain a second residual map and / or second Gaussian distribution parameter information. Generating a bitstream according to the target quality matrix, the second residual map and / or the second Gaussian distribution parameter information. Wherein the target quality matrix is used to represent the image quality of each region in the residual map of the feature domain, the first residual map is the residual map of the feature domain, and the first Gaussian distribution parameter information is the Gaussian distribution parameter information of the residual map of the feature domain.

[0009] It can be seen that the method provided in the embodiments of the present application introduces an objective quality matrix representing the image quality of each region in the JPEG AI encoding process. Since different positions in the objective quality matrix can have different image quality values, using the objective quality matrix for encoding can achieve code rate allocation of different regions in the image space, thereby enabling the JPEG AI to adjust the quality in the spatial dimension of the image content.

[0010] In a possible implementation, the target quality scaling matrix can be determined according to the target quality matrix, the target quality scaling matrix being used to represent the scaling amount of each region in the residual map of the feature domain and / or the Gaussian distribution parameter information. The target quality scaling tensor is determined according to the target quality scaling matrix, the target quality scaling tensor being used to represent the scaling tensor of the residual map of the feature domain and / or the Gaussian distribution parameter information in a three-dimensional space. The first residual map is scaled according to the target quality scaling tensor to obtain the second residual map, and / or the first Gaussian distribution parameter information is scaled according to the target quality scaling tensor to obtain the second Gaussian distribution parameter information.

[0011] It can be seen that the method provided in the embodiments of the present application introduces an objective quality matrix representing the image quality of each region in the JPEG AI encoding process. The target quality scaling tensor is determined through the objective quality matrix. Since different positions in the objective quality matrix can have different image quality values, using the target quality scaling tensor for encoding can achieve code rate allocation of different regions in the image space, thereby enabling the JPEG AI to adjust the quality in the spatial dimension of the image content.

[0012] In a possible implementation, the target quality scaling tensor can be determined according to the target quality scaling matrix and a gain parameter. The gain parameter includes a gain vector of channel-level quality adjustment and / or an image-level quality control factor.

[0013] It can be seen that the method provided in the embodiments of the present application introduces an objective quality matrix representing the image quality of each region in the JPEG AI encoding process. The target quality scaling tensor is determined through the objective quality matrix. Since different positions in the objective quality matrix can have different image quality values, using the target quality scaling tensor for encoding can achieve code rate allocation of different regions in the image space, thereby enabling the JPEG AI to adjust the quality in the spatial dimension of the image content.

[0014] In a possible implementation, the target quality matrix and the second residual map can be encoded to generate a code stream. The second residual map is encoded according to the second Gaussian distribution parameter information.

[0015] It can be seen that the method provided in the embodiments of the present application can generate a code stream by encoding the target quality matrix and the second residual image. Since different positions in the target quality matrix can have different image quality values, using a target quality scaling tensor for encoding can achieve code rate allocation for different regions in the image space, thereby enabling the JPEG AI to adjust the quality in the spatial dimension of the image content.

[0016] In a possible implementation, the target quality matrix or the target quality residual matrix of the target quality matrix can be encoded to generate a code stream.

[0017] It can be seen that the method provided in the embodiments of the present application encodes the target quality matrix or the target quality residual matrix of the target quality matrix to generate a code stream. Since different positions in the target quality matrix can have different image quality values, using a target quality scaling tensor for encoding can achieve code rate allocation for different regions in the image space, thereby enabling the JPEG AI to adjust the quality in the spatial dimension of the image content.

[0018] In a possible implementation, a target quality residual matrix of the target quality matrix can be generated according to the target quality matrix, and the target quality residual matrix is used to represent residual values of each image quality in the target quality matrix.

[0019] It can be understood that the data amount of the residual value is smaller than that of the quality value, and therefore, by generating the target quality residual matrix of the target quality matrix from the target quality matrix and then encoding the target quality residual matrix, the complexity of generating the code stream can be reduced.

[0020] In a possible implementation, a Gaussian distribution parameter of the target quality residual matrix can be determined, a probability distribution of the target quality residual matrix can be determined according to the Gaussian distribution parameter, the Gaussian distribution parameter can be written into the code stream, and the target quality residual matrix can be entropy encoded according to the probability distribution.

[0021] It can be seen that the embodiments of the present application can analyze the target quality residual matrix to obtain a Gaussian distribution parameter of the target quality residual matrix, determine a probability distribution of the target quality residual matrix from the Gaussian distribution parameter, and encode the target quality residual matrix into a code stream through the probability distribution of the target quality residual matrix. The target quality matrix can be determined from the target quality residual matrix. Since different positions in the target quality matrix can have different image quality values, using a target quality scaling tensor for encoding can achieve code rate allocation for different regions in the image space, thereby enabling the JPEG AI to adjust the quality in the spatial dimension of the image content.

[0022] In a possible implementation, a Gaussian distribution parameter of the target quality matrix can be determined. A probability distribution of the target quality matrix is determined according to the Gaussian distribution parameter. The Gaussian distribution parameter is written into the bitstream. The target quality matrix is entropy encoded according to the probability distribution.

[0023] It can be seen that the embodiments of the present application can obtain a Gaussian distribution parameter of a target quality matrix by analyzing the target quality matrix, determine a probability distribution of the target quality matrix by using the Gaussian distribution parameter, and encode a target quality residual matrix into a bitstream by using the probability distribution of the target quality matrix. Since different positions in the target quality matrix can have different image quality values, using a target quality scaling tensor for encoding can achieve code rate allocation in different regions of an image space, thereby enabling the JPEG AI to adjust the quality in the spatial dimension of image content.

[0024] In a possible implementation, a target probability distribution can be determined from a plurality of candidate probability distributions according to the target quality residual matrix. An index number of the target probability distribution is written into the bitstream. The target quality residual matrix is entropy encoded according to the target probability distribution.

[0025] It can be seen that the embodiments of the present application can determine a target probability distribution that matches a target quality residual matrix from a plurality of candidate probability distributions by analyzing the target quality residual matrix, and encode the target quality residual matrix into a bitstream by using the target probability distribution. The target quality matrix can be determined by using the target quality residual matrix. Since different positions in the target quality matrix can have different image quality values, using a target quality scaling tensor for encoding can achieve code rate allocation in different regions of an image space, thereby enabling the JPEG AI to adjust the quality in the spatial dimension of image content.

[0026] In a possible implementation, a target probability distribution can be determined from a plurality of candidate probability distributions according to the target quality matrix. An index number of the target probability distribution is written into the bitstream. The target quality matrix is entropy encoded according to the target probability distribution.

[0027] It can be seen that the embodiments of the present application can determine a target probability distribution that matches a target quality matrix from a plurality of candidate probability distributions by analyzing the target quality matrix, and encode the target quality matrix into a bitstream by using the target probability distribution. Since different positions in the target quality matrix can have different image quality values, using a target quality scaling tensor for encoding can achieve code rate allocation in different regions of an image space, thereby enabling the JPEG AI to adjust the quality in the spatial dimension of image content.

[0028] In a possible implementation, a quality map can be acquired, where the quality map is used to record image quality of each region in the residual map of the feature domain. The target quality matrix is determined according to the quality map.

[0029] It can be seen that, in the method provided by the embodiments of the present application, the target quality matrix can be determined through the quality map, and different positions in the target quality matrix can have different image quality values, so that the image space can be allocated with code rates in different regions by using the target quality scaling tensor for encoding, thereby enabling the JPEG AI to adjust the quality in the spatial dimension of the image content.

[0030] In a possible implementation, the image is input into an encoding network to obtain a first feature map. The first feature map is input into a context network to obtain a first prediction map. The first residual map is determined according to the first feature map and the first prediction map. The first feature map is input into a hyper-prior encoding network to obtain first hyper-prior information. The first hyper-prior information is quantized to obtain second hyper-prior information. The second hyper-prior information is input into a hyper-prior variance decoding network to obtain the first Gaussian distribution parameter information.

[0031] It can be seen that, in the method provided by the embodiments of the present application, the first Gaussian distribution parameter information can be obtained by inputting the image, and after the first Gaussian distribution parameter information is obtained, the first Gaussian distribution parameter information can be scaled by using the target quality matrix, different positions in the target quality matrix can have different image quality values, so that the image space can be allocated with code rates in different regions by using the target quality scaling tensor for encoding, thereby enabling the JPEG AI to adjust the quality in the spatial dimension of the image content.

[0032] In a second aspect, the embodiments of the present application further provide a decoding method, which includes: acquiring a code stream. A target quality matrix is determined according to the code stream, where the target quality matrix is used to represent image quality of each region in a residual map of a feature domain. Third Gaussian distribution parameter information is scaled according to the target quality matrix to obtain fourth Gaussian distribution parameter information, where the third Gaussian distribution parameter information is Gaussian distribution parameter information of the residual map of the feature domain. The code stream is decoded according to the fourth Gaussian distribution parameter information to obtain a third residual map, where the third residual map is the residual map of the feature domain. The third residual map is dequantized according to the target quality matrix to obtain a fourth residual map. A reconstructed image is determined according to the fourth residual map.

[0033] In a possible implementation, the target quality scaling matrix can be determined according to the target quality matrix, the target quality scaling matrix being used to represent a scaling amount of each region in the residual map of the feature domain and the Gaussian distribution parameter information. A target quality scaling tensor is determined according to the target quality scaling matrix, the target quality scaling tensor being used to represent a scaling tensor of the residual map of the feature domain and the Gaussian distribution parameter information in a three-dimensional space. The third Gaussian distribution parameter information is scaled according to the target quality scaling tensor to obtain fourth Gaussian distribution parameter information.

[0034] In a possible implementation, the target quality scaling tensor can be determined according to the target quality scaling matrix and a gain parameter, the gain parameter including a gain vector of a channel-level quality adjustment and / or an image-level quality control factor.

[0035] In a possible implementation, the code stream can be decoded to obtain a Gaussian distribution parameter of a target quality residual matrix of the target quality matrix. A probability distribution of the target quality residual matrix is determined according to the Gaussian distribution parameter. The code stream is decoded according to the probability distribution to obtain the target quality residual matrix. The target quality matrix is determined according to the target quality residual matrix.

[0036] In a possible implementation, the code stream can be decoded to obtain a Gaussian distribution parameter of the target quality matrix. A probability distribution of the target quality matrix is determined according to the Gaussian distribution parameter. The code stream is decoded according to the probability distribution to obtain the target quality matrix.

[0037] In a possible implementation, the code stream can be decoded to obtain an index number of a target probability distribution, the target probability distribution being determined from a plurality of candidate probability distributions according to a target quality residual matrix of the target quality matrix. The code stream is decoded according to the target probability distribution to obtain the target quality residual matrix. The target quality matrix is determined according to the target quality residual matrix.

[0038] In a possible implementation, the code stream can be decoded to obtain an index number of a target probability distribution, the target probability distribution being determined from a plurality of candidate probability distributions according to the target quality matrix. The code stream is decoded according to the target probability distribution to obtain the target quality matrix.

[0039] In a possible implementation, the code stream can be decoded to obtain a second feature map. The second feature map is input into a hyper-prior variance decoding network to obtain the third Gaussian distribution parameter information.

[0040] In a third aspect, an embodiment of the present application further provides a decoding method, which comprises: obtaining a code stream. Determining a target quality matrix according to the code stream, the target quality matrix being used to represent image quality of each region in a residual image of a feature domain. Scaling third Gaussian distribution parameter information according to the target quality matrix to obtain fourth Gaussian distribution parameter information, the third Gaussian distribution parameter information being Gaussian distribution parameter information of the residual image of the feature domain. Decoding the code stream according to the fourth Gaussian distribution parameter information to obtain a third residual image, the third residual image being the residual image of the feature domain. Determining a reconstructed image according to the third residual image.

[0041] In a fourth aspect, an embodiment of the present application further provides a decoding method, which comprises: obtaining a code stream. Determining a target quality matrix according to the code stream, the target quality matrix being used to represent image quality of each region in a residual image of a feature domain. Decoding the code stream according to third Gaussian distribution parameter information to obtain a third residual image, the third Gaussian distribution parameter information being Gaussian distribution parameter information of the residual image of the feature domain, and the third residual image being the residual image of the feature domain. Dequantizing the third residual image according to the target quality matrix to obtain a fourth residual image. Determining a reconstructed image according to the fourth residual image.

[0042] In a fifth aspect, an embodiment of the present application further provides an encoding device, which comprises: a transceiving unit and a processing unit. The transceiving unit is configured to obtain a target quality matrix, the target quality matrix being used to represent image quality of each region in a residual image of a feature domain. The processing unit is configured to scale a first residual image and / or first Gaussian distribution parameter information according to the target quality matrix to obtain a second residual image and / or second Gaussian distribution parameter information, the first residual image being the residual image of the feature domain, and the first Gaussian distribution parameter information being Gaussian distribution parameter information of the residual image of the feature domain. The processing unit is further configured to generate a code stream according to the target quality matrix, the second residual image and / or the second Gaussian distribution parameter information.

[0043] In a possible implementation, the processing unit is specifically configured to: determine a target quality scaling matrix according to the target quality matrix, the target quality scaling matrix being used to represent a scaling amount of each region in the residual image of the feature domain and / or the Gaussian distribution parameter information. Determine a target quality scaling tensor according to the target quality scaling matrix, the target quality scaling tensor being used to represent a scaling tensor of the residual image of the feature domain and / or the Gaussian distribution parameter information in a three-dimensional space. Scale the first residual image according to the target quality scaling tensor to obtain the second residual image and / or scale the first Gaussian distribution parameter information according to the target quality scaling tensor to obtain the second Gaussian distribution parameter information.

[0044] In a possible implementation, the processing unit is specifically configured to: determine the target quality scaling tensor according to the target quality scaling matrix and gain parameters, the gain parameters including a gain vector of channel-level quality adjustment and / or an image-level quality control factor.

[0045] In a possible implementation, the processing unit is specifically configured to: encode the target quality matrix and a second residual map to generate a bitstream, the second residual map being encoded according to the second Gaussian distribution parameter information.

[0046] In a possible implementation, the processing unit is specifically configured to: encode the target quality matrix or a target quality residual matrix of the target quality matrix to generate a bitstream.

[0047] In a possible implementation, the processing unit is further configured to: generate a target quality residual matrix of the target quality matrix according to the target quality matrix, the target quality residual matrix being used to represent residual values of image quality in the target quality matrix.

[0048] In a possible implementation, the processing unit is specifically configured to: determine Gaussian distribution parameters of the target quality residual matrix, determine a probability distribution of the target quality residual matrix according to the Gaussian distribution parameters, write the Gaussian distribution parameters into the bitstream, and entropy encode the target quality residual matrix according to the probability distribution.

[0049] In a possible implementation, the processing unit is specifically configured to: determine Gaussian distribution parameters of the target quality matrix, determine a probability distribution of the target quality matrix according to the Gaussian distribution parameters, write the Gaussian distribution parameters into the bitstream, and entropy encode the target quality matrix according to the probability distribution.

[0050] In a possible implementation, the processing unit is specifically configured to: determine a target probability distribution from a plurality of candidate probability distributions according to the target quality residual matrix, write an index number of the target probability distribution into the bitstream, and entropy encode the target quality residual matrix according to the target probability distribution.

[0051] In a possible implementation, the processing unit is specifically configured to: determine a target probability distribution from a plurality of candidate probability distributions according to the target quality matrix, write an index number of the target probability distribution into the bitstream, and entropy encode the target quality matrix according to the target probability distribution.

[0052] In a possible implementation, the obtaining unit is specifically configured to: obtain a quality map, where the quality map is used to record image quality of each region in the residual map of the feature domain; and determine the target quality matrix according to the quality map.

[0053] In a possible implementation, the processing unit is further configured to: input the image into an encoding network to obtain a first feature map; input the first feature map into a context network to obtain a first prediction map; determine the first residual map according to the first feature map and the first prediction map; input the first feature map into a hyper-prior encoding network to obtain first hyper-prior information; quantize the first hyper-prior information to obtain second hyper-prior information; and input the second hyper-prior information into a hyper-prior variance decoding network to obtain the first Gaussian distribution parameter information.

[0054] In a sixth aspect, an embodiment of the present application further provides a decoding apparatus, which comprises a transceiving unit and a processing unit. The transceiving unit is configured to obtain a code stream. The processing unit is configured to determine a target quality matrix according to the code stream, where the target quality matrix is used to represent image quality of each region in a residual map of a feature domain. The processing unit is further configured to scale third Gaussian distribution parameter information according to the target quality matrix to obtain fourth Gaussian distribution parameter information, where the third Gaussian distribution parameter information is Gaussian distribution parameter information of the residual map of the feature domain. The processing unit is further configured to decode the code stream according to the fourth Gaussian distribution parameter information to obtain a third residual map, where the third residual map is the residual map of the feature domain. The processing unit is further configured to dequantize the third residual map according to the target quality matrix to obtain a fourth residual map. The processing unit is further configured to determine a reconstructed image according to the fourth residual map.

[0055] In a possible implementation, the processing unit is specifically configured to: determine a target quality scaling matrix according to the target quality matrix, where the target quality scaling matrix is used to represent a scaling amount of each region in the residual map of the feature domain and the Gaussian distribution parameter information; determine a target quality scaling tensor according to the target quality scaling matrix, where the target quality scaling tensor is used to represent a scaling tensor of the residual map of the feature domain and the Gaussian distribution parameter information in a three-dimensional space; and scale the third Gaussian distribution parameter information according to the target quality scaling tensor to obtain the fourth Gaussian distribution parameter information.

[0056] In a possible implementation, the processing unit is specifically configured to: determine the target quality scaling tensor according to the target quality scaling matrix and a gain parameter, where the gain parameter comprises a gain vector of a channel-level quality adjustment and / or an image-level quality control factor.

[0057] In a possible implementation, the processing unit is specifically configured to: decode the code stream to obtain a Gaussian distribution parameter of a target quality residual matrix of a target quality matrix; determine a probability distribution of the target quality residual matrix according to the Gaussian distribution parameter; and decode the code stream according to the probability distribution to obtain the target quality residual matrix, and determine the target quality matrix according to the target quality residual matrix.

[0058] In a possible implementation, the processing unit is specifically configured to: decode the code stream to obtain a Gaussian distribution parameter of a target quality matrix; determine a probability distribution of the target quality matrix according to the Gaussian distribution parameter; and decode the code stream according to the probability distribution to obtain the target quality matrix.

[0059] In a possible implementation, the processing unit is specifically configured to: decode the code stream to obtain an index number of a target probability distribution, the target probability distribution being determined from a plurality of candidate probability distributions according to a target quality residual matrix of a target quality matrix; decode the code stream according to the target probability distribution to obtain the target quality residual matrix; and determine the target quality matrix according to the target quality residual matrix.

[0060] In a possible implementation, the processing unit is specifically configured to: decode the code stream to obtain an index number of a target probability distribution, the target probability distribution being determined from a plurality of candidate probability distributions according to a target quality matrix; and decode the code stream according to the target probability distribution to obtain the target quality matrix.

[0061] In a possible implementation, the processing unit is further configured to: decode the code stream to obtain a second feature map; and input the second feature map into a hyper-prior variance decoding network to obtain the third Gaussian distribution parameter information.

[0062] In a seventh aspect, an embodiment of the present application further provides a decoding apparatus, which comprises a transceiver unit and a processing unit. The transceiver unit is configured to acquire a code stream. The processing unit is configured to determine a target quality matrix according to the code stream, the target quality matrix being used to represent image quality of each region in a residual map of a feature domain. The processing unit is further configured to scale third Gaussian distribution parameter information according to the target quality matrix to obtain fourth Gaussian distribution parameter information, the third Gaussian distribution parameter information being Gaussian distribution parameter information of the residual map of the feature domain. The processing unit is further configured to decode the code stream according to the fourth Gaussian distribution parameter information to obtain a third residual map, the third residual map being the residual map of the feature domain. The processing unit is further configured to determine a reconstructed image according to the third residual map.

[0063] In an eighth aspect, an embodiment of the present application further provides a decoding apparatus, which comprises a transceiver and a processing unit. The transceiver is configured to acquire a bitstream. The processing unit is configured to determine a target quality matrix according to the bitstream, the target quality matrix being used to represent image quality of each region in a residual image of a feature domain. The processing unit is further configured to decode the bitstream according to third Gaussian distribution parameter information to obtain a third residual image, the third Gaussian distribution parameter information being Gaussian distribution parameter information of the residual image of the feature domain, and the third residual image being the residual image of the feature domain. The processing unit is further configured to dequantize the third residual image according to the target quality matrix to obtain a fourth residual image. The processing unit is further configured to determine a reconstructed image according to the fourth residual image.

[0064] In a ninth aspect, an embodiment of the present application further provides a bitstream, which comprises a target quality matrix, a third residual image and third Gaussian distribution parameter information, the target quality matrix being used to represent image quality of each region in a residual image of a feature domain, the third residual image being the residual image of the feature domain, and the third Gaussian distribution parameter information being Gaussian distribution parameter information of the residual image of the feature domain. The target quality matrix is used to scale the third Gaussian distribution parameter information to obtain fourth Gaussian distribution parameter information. The target quality matrix is further used to dequantize the third residual image to obtain a fourth residual image.

[0065] In a tenth aspect, an embodiment of the present application further provides an encoding apparatus, which comprises at least one processor, and the at least one processor is configured to implement the method in the first aspect or any possible implementation manner thereof when executing program code or instructions.

[0066] Optionally, the apparatus can further comprise at least one memory configured to store the program code or instructions.

[0067] In an eleventh aspect, an embodiment of the present application further provides a decoding apparatus, which comprises at least one processor, and the at least one processor is configured to implement the method in the second aspect or any possible implementation manner thereof when executing program code or instructions.

[0068] Optionally, the apparatus can further comprise at least one memory configured to store the program code or instructions.

[0069] In a twelfth aspect, an embodiment of the present application further provides a chip, which comprises an input interface, an output interface and at least one processor. Optionally, the chip further comprises a memory. The at least one processor is configured to execute code in the memory, and the chip implements the method in the first aspect or any possible implementation manner thereof when the at least one processor executes the code.

[0070] Optionally, the chip can also be an integrated circuit.

[0071] In a thirteenth aspect, the embodiments of the present application further provide a computer readable storage medium for storing a computer program, the computer program comprising instructions for implementing the method in the first aspect or any possible implementation manner thereof.

[0072] In a fourteenth aspect, the embodiments of the present application further provide a computer program product comprising instructions, which, when executed on a computer, cause the computer to carry out the method in the first aspect or any possible implementation manner thereof.

[0073] In a fifteenth aspect, the embodiments of the present application further provide a computer readable storage medium, the computer readable storage medium having stored thereon a video bitstream obtained by executing the method in the first aspect or any possible implementation manner thereof by one or more processors.

[0074] The coding device, computer storage medium, computer program product and chip provided by the embodiments have the beneficial effects of the coding method provided above, and thus the beneficial effects of the coding method provided above are referred to here, and will not be described again. BRIEF DESCRIPTION OF DRAWINGS

[0075] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings needed in the embodiment description will be briefly introduced as follows. Obviously, the drawings in the following description are only some embodiments of the present application, and for those skilled in the art, other drawings can also be obtained without creative effort based on these drawings.

[0076] FIG. 1a is an exemplary block diagram of a coding system provided by the embodiments of the present application;

[0077] FIG. 1b is an exemplary block diagram of a video coding system provided by the embodiments of the present application;

[0078] FIG. 2 is an exemplary block diagram of a video encoder provided by the embodiments of the present application;

[0079] FIG. 3 is an exemplary block diagram of a video decoder provided by the embodiments of the present application;

[0080] FIG. 4 is an exemplary block diagram of a video coding device provided by the embodiments of the present application;

[0081] FIG. 5 is an exemplary block diagram of an apparatus provided by the embodiments of the present application;

[0082] FIG. 6 is a schematic diagram of a neural network-based image compression method provided by the embodiments of the present application;

[0083] FIG. 7 is a schematic diagram of an end-to-end image coding framework according to an embodiment of the present application;

[0084] FIG. 8 is a schematic diagram of a neural network according to an embodiment of the present application;

[0085] FIG. 9 is an exemplary block diagram of a coding network according to an embodiment of the present application;

[0086] FIG. 10 is another exemplary block diagram of a coding network according to an embodiment of the present application;

[0087] FIG. 11 is a schematic diagram of an encoding method according to an embodiment of the present application;

[0088] FIG. 12 is a schematic diagram of a process of generating a target quality scaling tensor according to an embodiment of the present application;

[0089] FIG. 13 is an encoding-end architecture according to an embodiment of the present application;

[0090] FIG. 14 is a schematic diagram of a decoding method according to an embodiment of the present application;

[0091] FIG. 15 is a schematic diagram of another decoding method according to an embodiment of the present application;

[0092] FIG. 16 is a schematic diagram of yet another decoding method according to an embodiment of the present application;

[0093] FIG. 17 is a decoding-end architecture according to an embodiment of the present application;

[0094] FIG. 18 is a schematic diagram of an encoding apparatus according to an embodiment of the present application;

[0095] FIG. 19 is a schematic diagram of a decoding apparatus according to an embodiment of the present application;

[0096] FIG. 20 is a schematic diagram of another decoding apparatus according to an embodiment of the present application;

[0097] FIG. 21 is a schematic diagram of yet another decoding apparatus according to an embodiment of the present application;

[0098] FIG. 22 is a schematic diagram of a chip according to an embodiment of the present application;

[0099] FIG. 23 is a schematic diagram of another neural network according to an embodiment of the present application;

[0100] FIG. 24 is a schematic diagram of a machine video coding system according to an embodiment of the present application;

[0101] FIG. 25 is a schematic diagram of a bitstream according to an embodiment of the present application. DETAILED DESCRIPTION

[0102] The technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all the embodiments of the present application. Based on the embodiments in the embodiments of the present application, all other embodiments obtained by a person of ordinary skill in the art without creative effort belong to the scope of protection of the embodiments of the present application.

[0103] The term "and / or" in the present application is only used to describe the association relationship of the associated objects, which means that there can be three relationships, for example, A and / or B can mean that there are three cases of A alone, A and B together, and B alone.

[0104] The terms "first" and "second" and the like in the description of the embodiments of the present application and the drawings are used to distinguish different objects or different treatments of the same object, rather than to describe the specific order of the objects.

[0105] In addition, the terms "include" and "have" and any variations thereof mentioned in the description of the embodiments of the present application are intended to cover the inclusions without exclusivity. For example, a process, method, system, product or device that includes a series of steps or units is not limited to the listed steps or units, but can optionally include other steps or units not listed or can optionally include other steps or units inherent to the process, method, product or device.

[0106] It should be noted that in the description of the embodiments of the present application, the words "exemplarily" or "for example" are used to represent as an example, illustration or explanation. Any embodiment or design scheme described as "exemplarily" or "for example" in the embodiments of the present application should not be interpreted as more preferred or more advantageous than other embodiments or design schemes. Rather, the words "exemplarily" or "for example" are intended to present the relevant concept in a specific manner.

[0107] In the description of the embodiments of the present application, the meaning of "a plurality of" is two or more, unless otherwise specified.

[0108] First, the terms related to the embodiments of the present application are explained.

[0109] Data coding includes both data encoding and data decoding. Data encoding is performed at a source side (or usually referred to as an encoder side), and usually includes processing (e.g., compressing) original data to reduce the amount of data required to represent the original data (so as to more efficiently store and / or transmit). Data decoding is performed at a destination side (or usually referred to as a decoder side), and usually includes inverse processing relative to the encoder side to reconstruct the original data. The “coding” of data referred to in embodiments of the present application should be understood as either “encoding” or “decoding” of data. The encoding part and the decoding part are also collectively referred to as coding (encoding and decoding, CODEC).

[0110] In the case of lossless data encoding, the original data can be reconstructed, i.e., the reconstructed original data has the same quality as the original data (assuming no transmission loss or other data loss during storage or transmission). In the case of lossy data encoding, further compression is performed by quantization, etc., to reduce the amount of data required to represent the original data, and the decoder side cannot completely reconstruct the original data, i.e., the quality of the reconstructed original data is lower or worse than that of the original data.

[0111] Embodiments of the present application can be applied to video data and other data with compression / decompression requirements, etc. The following describes embodiments of the present application by taking video data encoding (referred to as video encoding for short) as an example. Other types of data (e.g., image data, audio data, integer type data, and other data with compression / decompression requirements) can refer to the following description, and embodiments of the present application will not be described again. It should be noted that, relative to video encoding, the encoding process of audio data and integer type data, etc. does not need to divide the data into blocks, but can directly encode the data.

[0112] Video encoding usually refers to processing a sequence of images forming a video or a video sequence. In the field of video encoding, the terms “picture”, “frame” or “image” can be used as synonyms.

[0113] Several video coding standards belong to the category of "hybrid video coding" (i.e., combining spatial and temporal prediction in the pixel domain with 2D transform coding in the transform domain for applying quantization). Each picture in a video sequence is typically partitioned into a set of non-overlapping blocks, which are usually encoded at the block level. In other words, an encoder usually processes, i.e., encodes, video at the block (video block) level, e.g., by spatial (intra) prediction and temporal (inter) prediction to produce a predicted block; subtracting the predicted block from the current block (block currently being processed / pending processing) results in a residual block; the residual block is transformed in the transform domain and quantized to reduce the amount of data to be transmitted (compressed), while the decoder side applies inverse processing parts with respect to the encoder to the encoded or compressed block to reconstruct the current block for representation. In addition, the encoder needs to repeat the processing steps of the decoder such that the encoder and the decoder generate the same predictions (e.g., intra and inter predictions) and / or reconstruct the pixels for processing, i.e., encoding subsequent blocks.

[0114] In the following embodiments of the coding system 10, the encoder 20 and the decoder 30 are described in accordance with FIGS. 1a-3.

[0115] FIG. 1a is an exemplary block diagram of a coding system 10, e.g., a video coding system 10 (or simply coding system 10) that can utilize techniques of embodiments of the present application. Video encoder 20 (or simply encoder 20) and video decoder 30 (or simply decoder 30) in the video coding system 10 are representative of devices or the like that can be used to perform various techniques described in accordance with various examples described in embodiments of the present application.

[0116] As shown in FIG. 1a, the coding system 10 includes a source device 12 that provides encoded pictures, e.g., encoded picture data 21, to a destination device 14 that decodes the encoded picture data 21.

[0117] The source device 12 includes the encoder 20, and additionally, optionally, an image source 16, a pre-processor (or pre-processing unit) 18, e.g., an image pre-processor, and a communication interface (or communication unit) 22.

[0118] The image source 16 can include or be any type of image capture device for capturing real-world images, and / or any type of image generation device, e.g., a computer graphics processor for generating computer-animated images or any type of device for acquiring and / or providing real-world images, computer-generated images (e.g., screen content, virtual reality (VR) images, and / or any combination thereof (e.g., augmented reality (AR) images). The image source can be any type of memory or storage for storing any of the above images.

[0119] For distinguishing the processing performed by the pre-processor (or pre-processing unit) 18, the image (or image data) 17 can also be referred to as original image (or original image data) 17.

[0120] The pre-processor 18 is configured to receive the original image data 17 and to pre-process the original image data 17 to obtain pre-processed image (or pre-processed image data) 19. For example, the pre-processing performed by the pre-processor 18 can include cropping, color format conversion (e.g. from RGB to YCbCr), color adjustment or de-noising. It can be appreciated that the pre-processing unit 18 can be an optional component.

[0121] The video encoder (or encoder) 20 is configured to receive the pre-processed image data 19 and to provide encoded image data 21 (to be further described below, e.g. with respect to Fig. 2).

[0122] The communication interface 22 in the source device 12 can be configured to receive the encoded image data 21 and to transmit the encoded image data 21 (or any other processed version thereof) to another device, e.g. the destination device 14, or any other device, over the communication channel 13 for storage or direct reconstruction.

[0123] The destination device 14 comprises a decoder 30 and, additionally, optionally, a communication interface (or communication unit) 28, a post-processor (or post-processing unit) 32 and a display device 34.

[0124] The communication interface 28 in the destination device 14 can be configured to receive the encoded image data 21 (or any other processed version thereof) directly from the source device 12 or from any other source device, e.g. a storage device, e.g. an encoded image data storage device, and to provide the encoded image data 21 to the decoder 30.

[0125] The communication interface 22 and the communication interface 28 can be configured to transmit or receive the encoded image data (or encoded data) 21 over a direct communication link, e.g. a direct wired or wireless connection, between the source device 12 and the destination device 14 or over any type of network, e.g. a wired network, a wireless network or any combination thereof, any type of private network and public network or any combination thereof.

[0126] For example, the communication interface 22 can be configured to encapsulate the encoded image data 21 into a suitable format, e.g. a packet, and / or to process the encoded image data using any type of transmission encoding or processing for transmission over the communication link or the communication network.

[0127] The communication interface 28 can correspond to the communication interface 22 and can be configured to receive the transmitted data and to process the transmitted data using any type of corresponding transmission decoding or processing and / or de-encapsulation to obtain the encoded image data 21.

[0128] The communication interface 22 and the communication interface 28 can each be configured as a unidirectional communication interface or a bidirectional communication interface, as indicated by the corresponding communication channels 13 in Fig. 1a from the source device 12 to the destination device 14, and can be used to send and receive messages, etc., to establish a connection, to acknowledge and exchange any other information related to the communication link and / or data transmission, e.g., encoded image data transmission, etc.

[0129] The video decoder (or decoder) 30 is configured to receive the encoded image data 21 and to provide decoded image data (or decoded image data) 31 (to be further described below with respect to Fig. 3, etc.).

[0130] The post-processor 32 is configured to post-process the decoded image data 31 (also referred to as reconstructed image data) to obtain post-processed image data 33 (also referred to as post-processed image data). The post-processing performed by the post-processing unit 32 can include, for example, color format conversion (e.g., from YCbCr to RGB), toning, pruning or resampling, or any other processing for producing the decoded image data 31 for display by the display device 34, etc.

[0131] The display device 34 is configured to receive the post-processed image data 33 for displaying images to a user or viewer, etc. The display device 34 can be or include any type of display for representing reconstructed images, e.g., an integrated or external display screen or display. For example, the display screen can include a liquid crystal display (LCD), an organic light emitting diode (OLED) display, a plasma display, a projector, a micro-LED display, a liquid crystal on silicon (LCoS) display, a digital light processor (DLP), or any other type of display screen.

[0132] The coding system 10 further comprises a training engine 25 configured to train the encoder 20 (in particular the entropy encoding unit 270 of the encoder 20) or the decoder 30 (in particular the entropy decoding unit 304 of the decoder 30) to entropy encode a to-be-encoded image block according to an estimated probability distribution estimated by the estimation engine 24. The training engine 25 will be further described below with respect to the method test example.

[0133] Although FIG. 1a shows source device 12 and destination device 14 as separate devices, device embodiments can also include source device 12 and destination device 14 or functionality of source device 12 and destination device 14 simultaneously, i.e., include source device 12 or corresponding functionality and destination device 14 or corresponding functionality simultaneously. In these embodiments, source device 12 or corresponding functionality and destination device 14 or corresponding functionality can be implemented using the same hardware and / or software or through separate hardware and / or software or any combination thereof.

[0134] From the description it will be apparent to a skilled person that the presence and (precise) division of the different units or functionalities in source device 12 and / or destination device 14 as shown in FIG. 1a can vary depending on the actual device and application.

[0135] Referring to FIG. 1b, FIG. 1b is an exemplary block diagram of a video coding system 40 in accordance with an embodiment of the present application. The encoder 20 (e.g., video encoder 20) or the decoder 30 (e.g., video decoder 30) or both can be implemented through processing circuitry in video coding system 40 as shown in FIG. 1b, such as one or more microprocessors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), discrete logic, hardware, video coding dedicated processors, or any combinations thereof. Referring to FIG. 2 and FIG. 3, FIG. 2 is an exemplary block diagram of a video encoder in accordance with an embodiment of the present application, and FIG. 3 is an exemplary block diagram of a video decoder in accordance with an embodiment of the present application. The encoder 20 can be implemented through processing circuitry 46 to include various modules discussed with respect to the encoder 20 of FIG. 2 and / or any other encoder systems or subsystems described herein. The decoder 30 can be implemented through processing circuitry 46 to include various modules discussed with respect to the decoder 30 of FIG. 3 and / or any other decoder systems or subsystems described herein. The processing circuitry 46 can be used to perform various operations discussed below. If portions of the technology are implemented in software, the devices can store instructions for the software in a suitable, non- transitory computer-readable storage medium and execute the instructions in hardware using one or more processors to perform the technology as described herein. One of the video encoder 20 and the video decoder 30 can be integrated as part of a combined encoder / decoder (CODEC) in a single device, as shown in FIG. 1b.

[0136] The source device 12 and the destination device 14 can include any of various devices, including any type of handheld or stationary device, such as a notebook or laptop computer, a mobile phone, a smart phone, a tablet or tablet computer, a camera, a desktop computer, a set-top box, a television, a display device, a digital media player, a video gaming console, a video streaming device (e.g., a content service server or a content distribution server), a broadcast receiving device, a broadcast transmitting device, a surveillance device, and so on, and can use or not use any type of operating system. The source device 12 and the destination device 14 can also be devices in a cloud computing scenario, such as virtual machines in a cloud computing scenario, and so on. In some cases, the source device 12 and the destination device 14 can be equipped with components for wireless communication. Thus, the source device 12 and the destination device 14 can be wireless communication devices.

[0137] The source device 12 and the destination device 14 can install a virtual scene application (APP), such as a virtual reality (VR) application, an augmented reality (AR) application, or a mixed reality (MR) application, and can run the VR application, the AR application, or the MR application based on a user operation (e.g., a click, a touch, a swipe, a shake, a voice control, and so on). The source device 12 and the destination device 14 can capture an image / video of any object in an environment through a camera and / or a sensor, and display a virtual object on a display device according to the captured image / video, where the virtual object can be a virtual object (i.e., an object in a virtual environment) in a VR scene, an AR scene, or an MR scene.

[0138] It should be noted that, in the embodiments of the present application, the virtual scene application in the source device 12 and the destination device 14 can be an application built in the source device 12 and the destination device 14, or an application provided by a third-party service provider and installed by a user, and the present application is not limited in this regard.

[0139] In addition, the source device 12 and the destination device 14 can install a real-time video transmission application, such as a live broadcast application. The source device 12 and the destination device 14 can capture an image / video through a camera, and display the captured image / video on a display device.

[0140] In some cases, the video coding system 10 of FIG. 1a is merely exemplary, and the techniques provided by embodiments of the present disclosure can be applicable to video coding settings (e.g., video encoding or video decoding) that do not necessarily include any data communication between an encoding device and a decoding device. In other examples, data is retrieved from local storage, transmitted over a network, and the like. A video encoding device can encode data and store the data to a memory, and / or a video decoding device can retrieve data from a memory and decode the data. In some examples, encoding and decoding are performed by devices that do not communicate with each other, but only encode data to a memory and / or retrieve and decode data from a memory.

[0141] Referring to FIG. 1b, FIG. 1b is an exemplary block diagram of a video coding system 40 in accordance with embodiments of the present disclosure. As shown in FIG. 1b, the video coding system 40 can include an imaging device 41, a video encoder 20, a video decoder 30 (and / or a video encoder / decoder implemented by processing circuitry 46), an antenna 42, one or more processors 43, one or more memory stores 44, and / or a display device 45.

[0142] As shown in FIG. 1b, the imaging device 41, the antenna 42, the processing circuitry 46, the video encoder 20, the video decoder 30, the processor(s) 43, the memory store(s) 44, and / or the display device 45 can be in communication with each other. In different examples, the video coding system 40 can include only the video encoder 20 or only the video decoder 30.

[0143] In some examples, the antenna 42 can be used to transmit or receive an encoded bitstream of video data. Additionally, in some examples, the display device 45 can be used to present video data. The processing circuitry 46 can include application-specific integrated circuit (ASIC) logic, a graphics processor, a general purpose processor, etc. The video coding system 40 can also include an optional processor 43, which similarly can include ASIC logic, a graphics processor, a general purpose processor, etc. Additionally, the memory storage 44 can be any type of memory, such as volatile memory (e.g., static random access memory (SRAM), dynamic random access memory (DRAM), etc.) or non-volatile memory (e.g., flash memory, etc.), etc. In non-limiting examples, the memory storage 44 can be implemented by a cache memory. In other examples, the processing circuitry 46 can include memory (e.g., a cache, etc.) for implementing an image buffer, etc.

[0144] In some examples, the video encoder 20 implemented by logic circuitry can include an image buffer (implemented by the processing circuitry 46 or the memory storage 44) and a graphics processing unit (implemented by the processing circuitry 46). The graphics processing unit can be communicatively coupled to the image buffer. The graphics processing unit can include the video encoder 20 implemented by the processing circuitry 46 to implement various modules discussed with reference to the video encoder 20 of FIG. 2 and / or any other encoder system or subsystem described herein. The logic circuitry can be used to perform various operations discussed herein.

[0145] In some examples, the video decoder 30 can be implemented by logic circuitry in a similar manner by the processing circuitry 46 to implement various modules discussed with reference to the video decoder 30 of FIG. 3 and / or any other decoder system or subsystem described herein. In some examples, the video decoder 30 implemented by logic circuitry can include an image buffer (implemented by the processing circuitry 46 or the memory storage 44) and a graphics processing unit (implemented by the processing circuitry 46). The graphics processing unit can be communicatively coupled to the image buffer. The graphics processing unit can include the video decoder 30 implemented by the processing circuitry 46 to implement various modules discussed with reference to the video decoder 30 of FIG. 3 and / or any other decoder system or subsystem described herein.

[0146] In some examples, the antenna 42 can be used to receive an encoded bitstream of video data. As discussed, the encoded bitstream can include data, indicators, index values, mode selection data, etc. discussed herein related to encoding video frames, e.g., data related to encoding partitions (e.g., transform coefficients or quantized transform coefficients, optional indicators (as discussed), and / or data defining encoding partitions). The video coding system 40 can also include a video decoder 30 coupled to the antenna 42 and used to decode the encoded bitstream. The display device 45 is used to present video frames.

[0147] It should be understood that the video decoder 30 can be used to perform the inverse process of the examples described in this application with respect to the reference video encoder 20. With respect to signaling syntax elements, the video decoder 30 can be used to receive and parse such syntax elements and decode the related video data accordingly. In some examples, the video encoder 20 can entropy encode the syntax elements into the encoded video bitstream. In such examples, the video decoder 30 can parse such syntax elements and decode the related video data accordingly.

[0148] For ease of description, the embodiments of the present application are described with reference to the Versatile Video Coding (VVC) reference software or High-Efficiency Video Coding (HEVC) developed by the Joint Collaboration Team on Video Coding (JCT-VC) of ITU-T Video Coding Experts Group (VCEG) and ISO / IEC Motion Picture Experts Group (MPEG). It is understood by those of ordinary skill in the art that the embodiments of the present application are not limited to HEVC or VVC.

[0149] Encoder and encoding method

[0150] As shown in FIG. 2, video encoder 20 includes an input (or input interface) 201, a residual calculation unit 204, a transform processing unit 206, a quantization unit 208, an inverse quantization unit 210, an inverse transform processing unit 212, a reconstruction unit 214, an in-loop filter 220, a decoded picture buffer (DPB) 230, a mode selection unit 260, an entropy encoding unit 270, and an output (or output interface) 272. Mode selection unit 260 can include an inter-prediction unit 244, an intra-prediction unit 254, and a partition unit 262. Inter-prediction unit 244 can include a motion estimation unit and a motion compensation unit (not shown). Video encoder 20 shown in FIG. 2 can also be referred to as a hybrid video encoder or a video encoder based on a hybrid video codec.

[0151] Images and image partitioning (images and blocks)

[0152] Encoder 20 can be configured to receive an image (or image data) 17, e.g., an image in a sequence of images forming a video or a video sequence, via input 201 or the like. The received image or image data can also be a pre-processed image (or pre-processed image data) 19. For simplicity, the following description uses image 17. Image 17 can also be referred to as a current image or an image to be encoded (especially when the current image is distinguished from other images in video encoding, e.g., previously encoded and / or decoded images in the same video sequence, i.e., a video sequence also including the current image).

[0153] A (digital) image is or can be considered as a two-dimensional array or matrix of pixel points with intensity values. A pixel point in the array can also be referred to as a pixel (short for picture element). The number of pixel points in the array or image in horizontal and vertical direction (or axis) determines the size and / or resolution of the image. For representing colors, usually three color components are used, i.e. the image can be represented or comprise three pixel point arrays. In an RBG format or color space, the image comprises corresponding arrays of red, green and blue pixel points. However, in video coding, each pixel is usually represented in a luma / chroma format or color space, e.g. YCbCr, comprising a luma component indicated by Y (sometimes also denoted by L) and two chroma components indicated by Cb and Cr. The luma component Y represents the luminance or gray level intensity (e.g. both in a gray scale image), while the two chroma components Cb and Cr represent the chrominance or color information components. Accordingly, a YCbCr formatted image comprises a luma pixel point array of luma pixel point values (Y) and two chroma pixel point arrays of chroma values (Cb and Cr). An RGB formatted image can be converted or transformed into a YCbCr format and vice versa, a process also referred to as color transform or conversion. If the image is black and white, the image can comprise only a luma pixel point array. Accordingly, the image can be a luma pixel point array in e.g. monochrome format or a luma pixel point array and two corresponding chroma pixel point arrays in 4:2:0, 4:2:2 and 4:4:4 color formats.

[0154] In one embodiment, the embodiment of the video encoder 20 can comprise an image partitioning unit (not shown in Fig. 2) for partitioning the image 17 into a plurality of (usually non-overlapping) image blocks 203. These blocks can also be referred to as root blocks, macroblocks (H.264 / AVC) or coding tree blocks (CTB), or coding tree units (CTU) in the H.265 / HEVC and VVC standards. The partitioning unit can be configured to use the same block size for all images in a video sequence and a corresponding grid of defined block sizes, or to vary the block size between images or subsets or groups of images and to partition each image into corresponding blocks.

[0155] In other embodiments, the video encoder can be configured to directly receive the blocks 203 of the image 17, e.g. one, several or all blocks constituting the image 17. The image blocks 203 can also be referred to as current image blocks or to-be-coded image blocks.

[0156] As with image 17, image block 203 is or can be considered to be a two-dimensional array or matrix of pixels with intensity values (pixel values), but block 203 is smaller than image 17. In other words, block 203 can comprise one array of pixels (e.g., an array of luminance values in case of monochrome image 17 or an array of luminance values or an array of chrominance values in case of a color image) or three arrays of pixels (e.g., one array of luminance values and two arrays of chrominance values in case of a color image 17) or any other number and / or type of arrays depending on the color format employed. The number of pixels in the horizontal and vertical direction (or axis) of block 203 defines the size of block 203. Accordingly, a block can be an M x N (M columns x N rows) array of pixels or an M x N array of transform coefficients or the like.

[0157] In an embodiment, video encoder 20 shown in FIG. 2 is configured to encode image 17 on a block-by-block basis, e.g., to perform encoding and prediction for each block 203.

[0158] In an embodiment, video encoder 20 shown in FIG. 2 can also be configured to partition and / or encode an image using tiles (also referred to as video tiles), wherein the image can be partitioned or encoded using one or more tiles (typically non-overlapping). Each tile can comprise one or more blocks (e.g., coding tree units, CTUs) or one or more groups of blocks (e.g., tiles in the H.265 / HEVC / VVC standards and bricks in the VVC standard).

[0159] In an embodiment, video encoder 20 shown in FIG. 2 can also be configured to partition and / or encode an image using tiles / bricks (also referred to as video tile / brick groups) and / or tiles (also referred to as video tiles), wherein the image can be partitioned or encoded using one or more tiles / bricks (typically non-overlapping), each tile / brick can comprise one or more blocks (e.g., CTUs) or one or more tiles or the like, wherein each tile can be rectangular or the like in shape and can comprise one or more complete or partial blocks (e.g., CTUs).

[0160] Residual calculation

[0161] Residual calculation unit 204 is configured to calculate residual block 205 from image block (or original block) 203 and prediction block 265 (prediction block 265 is described in more detail later) by, for example, subtracting the pixel values of prediction block 265 from the pixel values of image block 203 on a pixel-by-pixel basis (pixel-wise) to obtain residual block 205 in the pixel domain.

[0162] Quantization

[0163] Quantization unit 208 quantizes transform coefficients 207 to provide quantized transform coefficients 209, e.g., by scalar quantization or vector quantization. Quantized transform coefficients 209 can also be referred to as quantized residual coefficients 209.

[0164] The quantization process can reduce the bit depth associated with some or all of transform coefficients 207. For example, n-bit transform coefficients can be rounded down to m-bit transform coefficients during quantization, where n is greater than m. The degree of quantization can be modified by adjusting a quantization parameter (QP). For example, for scalar quantization, different degrees of scaling can be applied to achieve finer or coarser quantization. A smaller quantization step size corresponds to finer quantization, while a larger quantization step size corresponds to coarser quantization. An appropriate quantization step size can be indicated by a quantization parameter (QP). For example, the quantization parameter can be an index to a predefined set of quantization step sizes. For example, a smaller quantization parameter can correspond to fine quantization (smaller quantization step size), a larger quantization parameter can correspond to coarse quantization (larger quantization step size), and vice versa. Quantization can include division by the quantization step size, while corresponding or inverse dequantization performed by inverse quantization unit 210, etc., can include multiplication by the quantization step size. Some embodiments according to, e.g., HEVC, can be used to determine the quantization step size using a quantization parameter. In general, the quantization step size can be computed using a fixed-point approximation of the equation that includes the division, according to the quantization parameter. Other scaling factors can be introduced for quantization and dequantization to recover the norm of the residual block that can have been modified due to scaling used in the fixed-point approximation of the equation for the quantization step size and the quantization parameter. In one exemplary implementation, the scaling for the inverse transform and dequantization can be combined. Alternatively, a custom quantization table can be used and signaled from the encoder to the decoder in the bitstream, etc. Quantization is a lossy operation, where the larger the quantization step size, the greater the loss.

[0165] In one embodiment, video encoder 20 (correspondingly, quantization unit 208) can be configured to output a quantization parameter (QP), e.g., directly or after being encoded or compressed by entropy encoding unit 270, such that video decoder 30 can receive and use the quantization parameter for decoding.

[0166] Inverse quantization

[0167] Inverse quantization unit 210 inverse quantizes quantized coefficients to provide dequantized coefficients 211, e.g., according to or using the same quantization step size as quantization unit 208, to perform an inverse quantization scheme to the quantization scheme performed by quantization unit 208. Dequantized coefficients 211 can also be referred to as dequantized residual coefficients 211, corresponding to transform coefficients 207, but generally not identical to transform coefficients 207 due to the loss caused by quantization.

[0168] reconstruction

[0169] The reconstruction unit 214 (e.g., a summer 214) is configured to add the transform block 213 (i.e., the reconstructed residual block 213) to the prediction block 265 to obtain a reconstructed block 215 in the pixel domain, e.g., by adding the pixel values of the reconstructed residual block 213 and the pixel values of the prediction block 265.

[0170] partitioning

[0171] The partitioning unit 262 can partition (or divide) one image block (or CTU) 203 into smaller portions, e.g., square or rectangular shaped small blocks. For an image having three pixel arrays, a CTU is composed of N x N blocks of luma pixels and two corresponding blocks of chroma pixels. The maximum allowed size of luma blocks in a CTU is specified as 128 x 128 in the developing versatile video coding (VVC) standard, but can be specified as a value other than 128 x 128 in the future, e.g., 256 x 256. The CTUs of an image can be grouped into slices / tile groups, coding blocks or tiles. A coding block covers a rectangular region of an image, and a coding block can be divided into one or more tiles. A tile is composed of multiple CTU rows within a coding block. A coding block that is not partitioned into multiple tiles can be referred to as a tile. However, a tile is a true subset of a coding block, and thus is not referred to as a coding block. VVC supports two tile group modes, a raster-scan tile group mode and a rectangular slice mode. In the raster-scan tile group mode, a slice / tile group contains a sequence of coding blocks in a raster scan of an image. In the rectangular slice mode, a slice contains multiple tiles of an image that together form a rectangular region of the image. The tiles within a rectangular slice are arranged in the order of tile raster scan of the slice. These smaller blocks (also referred to as sub-blocks) can be further partitioned into smaller portions. This is also referred to as tree partitioning or hierarchical tree partitioning, where a root block at root tree level 0 (hierarchy level 0, depth 0) can be recursively partitioned into two or more blocks at a next lower tree level, e.g., nodes at tree level 1 (hierarchy level 1, depth 1). These blocks can be further partitioned into two or more blocks at a next lower level, e.g., tree level 2 (hierarchy level 2, depth 2), and so on until the partitioning ends (because an end criterion is met, e.g., a maximum tree depth is reached or a minimum block size is reached). A block that is not further partitioned is also referred to as a leaf block or a leaf node of the tree. A tree that is partitioned into two portions is referred to as a binary-tree (BT), a tree that is partitioned into three portions is referred to as a ternary-tree (TT), and a tree that is partitioned into four portions is referred to as a quad-tree (QT).

[0172] entropy coding

[0173] Entropy encoding unit 270 is configured to apply an entropy encoding algorithm or scheme (e.g., a variable length coding (VLC) scheme, a context adaptive VLC (CALVC) scheme, an arithmetic coding scheme, a binarization algorithm, a context adaptive binary arithmetic coding (CABAC) scheme, a syntax-based context-adaptive binary arithmetic coding (SBAC) scheme, a probability interval partitioning entropy (PIPE) coding, or other entropy encoding methods or techniques) to quantized residual coefficients 209, inter prediction parameters, intra prediction parameters, loop filter parameters, and / or other syntax elements to obtain encoded picture data 21 that can be output via output terminal 272 in the form of an encoded bitstream 21, such that a video decoder 30, for example, can receive and use the parameters for decoding. Encoded bitstream 21 can be transmitted to video decoder 30 or saved in a memory for later transmission or retrieval by video decoder 30.

[0174] Other structural variants of video encoder 20 can be used to encode a video stream. For example, a non-transform-based encoder 20 can directly quantize residual signals without transform processing unit 206 for certain blocks or frames. In another implementation, encoder 20 can have quantization unit 208 and inverse quantization unit 210 combined into a single unit.

[0175] Decoder and decoding method

[0176] As shown in FIG. 3, video decoder 30 is configured to receive encoded picture data 21 (e.g., an encoded bitstream 21) encoded by, for example, encoder 20, to obtain decoded pictures 331. The encoded picture data or bitstream includes information for decoding the encoded picture data, such as data representing image blocks of encoded video slices (and / or encoded groups of blocks or encoded blocks) and associated syntax elements.

[0177] In the example of FIG. 3, the decoder 30 includes an entropy decoding unit 304, an inverse quantization unit 310, an inverse transform processing unit 312, a reconstruction unit 314 (e.g., a summer 314), an in-loop filter 320, a decoded picture buffer (DPB) 330, a mode application unit 360, an inter prediction unit 344, and an intra prediction unit 354. The inter prediction unit 344 can be or include a motion compensation unit. In some examples, the video decoder 30 can perform a decoding process generally reciprocal to the encoding process described with respect to the video encoder 100 of FIG. 2.

[0178] As described for the encoder 20, the inverse quantization unit 210, the inverse transform processing unit 212, the reconstruction unit 214, the in-loop filter 220, the decoded picture buffer DPB 230, the inter prediction unit 344, and the intra prediction unit 354 also constitute a "built-in decoder" of the video encoder 20. Accordingly, the inverse quantization unit 310 can be functionally identical to the inverse quantization unit 110, the inverse transform processing unit 312 can be functionally identical to the inverse transform processing unit 122, the reconstruction unit 314 can be functionally identical to the reconstruction unit 214, the in-loop filter 320 can be functionally identical to the in-loop filter 220, and the decoded picture buffer 330 can be functionally identical to the decoded picture buffer 230. Thus, the explanations of the respective units and functions of the video encoder 20 apply accordingly to the respective units and functions of the video decoder 30.

[0179] Entropy decoding

[0180] The entropy decoding unit 304 is configured to parse the bitstream 21 (or, generally, the encoded picture data 21) and perform entropy decoding on the encoded picture data 21 to obtain, among other things, quantized coefficients 309 and / or decoded encoding parameters (not shown in FIG. 3), such as any or all of inter prediction parameters (e.g., reference picture indices and motion vectors), intra prediction parameters (e.g., intra prediction modes or indices), transform parameters, quantization parameters, in-loop filter parameters, and / or other syntax elements. The entropy decoding unit 304 can be configured to apply a decoding algorithm or scheme corresponding to the encoding scheme of the entropy encoding unit 270 of the encoder 20. The entropy decoding unit 304 can also be configured to provide inter prediction parameters, intra prediction parameters, and / or other syntax elements to the mode application unit 360, and to provide other parameters to other units of the decoder 30. The video decoder 30 can receive syntax elements at a video slice and / or video block level. Additionally, or as an alternative to slices and corresponding syntax elements, coded group of blocks and / or coded blocks and corresponding syntax elements can be received or used.

[0181] Inverse quantization

[0182] The inverse quantization unit 310 can be configured to receive quantization parameters (QP) (or in general information related to inverse quantization) and quantized coefficients from the encoded picture data 21 (e.g. parsed and / or decoded by the entropy decoding unit 304) and inverse quantize the decoded quantized coefficients 309 based on the quantization parameters to obtain inverse quantized coefficients 311, which can also be referred to as transform coefficients 311. The inverse quantization process can include determining a degree of quantization using quantization parameters calculated by the video encoder 20 for each video block in a video slice, and likewise determining a degree of inverse quantization to be performed.

[0183] reconstruction

[0184] The reconstruction unit 314 (e.g. a summer 314) is configured to add the reconstructed residual block 313 to the prediction block 365 to obtain a reconstructed block 315 in the pixel domain, e.g. by adding the pixel values of the reconstructed residual block 313 to the pixel values of the prediction block 365.

[0185] Other variants of the video decoder 30 can be used to decode the encoded picture data 21. For example, the decoder 30 can produce an output video stream without the loop filter unit 320. For example, a decoder 30 based on non-transform can directly inverse quantize the residual signal without the inverse transform processing unit 312 for certain blocks or frames. In another implementation, the video decoder 30 can have the inverse quantization unit 310 and the inverse transform processing unit 312 combined into a single unit.

[0186] It should be understood that in the encoder 20 and the decoder 30, the processing results of the current step can be further processed before being output to the next step. For example, after the interpolation filtering, the motion vector derivation or the loop filtering, the processing results of the interpolation filtering, the motion vector derivation or the loop filtering can be further operated, e.g. clip or shift operations.

[0187] It should be noted that further operations can be performed on the derived motion vectors of the current block, including but not limited to the control point motion vectors of affine mode, the sub-block motion vectors of affine, planar, ATMVP mode, the temporal motion vectors, etc. For example, the value of the motion vector is limited in a predefined range according to the representation bits of the motion vector. If the representation bits of the motion vector is bitDepth, the range is -2^(bitDepth-1) to 2^(bitDepth-1)-1, where "^" represents the power. For example, if bitDepth is set to 16, the range is -32768~32767; if bitDepth is set to 18, the range is -131072~131071. For example, the value of the derived motion vector (e.g. the MVs of 4 4x4 sub-blocks in one 8x8 block) is limited so that the maximum difference between the integer parts of the 4 4x4 sub-block MVs does not exceed N pixels, e.g. does not exceed 1 pixel. Two methods of limiting the motion vector according to bitDepth are provided herein.

[0188] Although the above embodiments are mainly described for video coding, it should be noted that the embodiments of the coding system 10, the encoder 20 and the decoder 30, as well as other embodiments described herein, can also be used for still image processing or coding, i.e. processing or coding of a single image independent of any previous or consecutive image in video coding. Generally, if the image processing is limited to a single image 17, the inter prediction unit 244 (encoder) and the inter prediction unit 344 (decoder) can not be available. All other functionalities (also referred to as tools or techniques) of the video encoder 20 and the video decoder 30 can likewise be used for still image processing, e.g. residual calculation 204 / 304, transform 206, quantization 208, inverse quantization 210 / 310, (inverse) transform 212 / 312, partitioning 262 / 362, intra prediction 254 / 354 and / or loop filtering 220 / 320, entropy encoding 270 and entropy decoding 304.

[0189] Referring to FIG. 4, FIG. 4 is an exemplary block diagram of a video coding device 400 according to an embodiment of the present disclosure. The video coding device 400 is suitable for implementing the disclosed embodiments described herein. In one embodiment, the video coding device 400 can be a decoder, such as the video decoder 30 in FIG. 1a, or an encoder, such as the video encoder 20 in FIG. 1a.

[0190] The video coding device 400 includes an ingress port 410 (or input port 410) and a receiver unit (Rx) 420 for receiving data, a processor, logic unit, or central processing unit (CPU) 430 for processing data, e.g., the processor 430 here can be a neural network processor 430, a transmitter unit (Tx) 440 and an egress port 450 (or output port 550) for transmitting data, and a memory 460 for storing data. The video coding device 400 can also include optical-to-electrical (OE) components and electrical-to-optical (EO) components coupled to the ingress port 410, receiver unit 420, transmitter unit 440, and egress port 450 for egress or ingress of optical or electrical signals.

[0191] The processor 430 is implemented by hardware and software. The processor 430 can be implemented as one or more processor chips, cores (e.g., a multi-core processor), FPGAs, ASICs, and DSPs. The processor 430 is in communication with the ingress port 410, receiver unit 420, transmitter unit 440, egress port 450, and memory 460. The processor 430 includes a neural network based codec 470. The neural network based codec 470 implements the embodiments disclosed above. For example, the neural network based codec 470 performs, processes, prepares, or provides various encoding operations. Thus, the neural network based codec 470 provides substantial improvement to the functionality of the video coding device 400 and affects the switching of the video coding device 400 to different states. Alternatively, the neural network based codec 470 is implemented in instructions stored in the memory 460 and executed by the processor 430.

[0192] The memory 460 includes one or more disks, tape drives, and solid-state drives, can be used as an over-flow data storage device, to store programs when such programs are selected for execution, and to store instructions and data read from a program as such program is executed. The memory 460 can be volatile and / or non-volatile, and can be read-only memory (ROM), random access memory (RAM), ternary content-addressable memory (TCAM), and / or static random-access memory (SRAM).

[0193] Referring to FIG. 5, FIG. 5 is an exemplary block diagram of an apparatus 500 according to an embodiment of the present disclosure. The apparatus 500 can be used as either or both of the source device 12 and the destination device 14 in FIG. 1a.

[0194] The processor 502 in the apparatus 500 can be a central processing unit. Alternatively, the processor 502 can be any other type of device, either existing or yet to be developed, that is capable of manipulating or processing information. Although a single processor 502 as shown can be used to implement the disclosed implementations, using more than one processor can be faster and more efficient.

[0195] In one implementation, the memory 504 in the apparatus 500 can be a read-only memory (ROM) device or a random access memory (RAM) device. Any other suitable type of storage device can be used as the memory 504. The memory 504 can include code and data 506 that is accessed by the processor 502 via the bus 512. The memory 504 can also include an operating system 508 and application programs 510, including at least one program for allowing the processor 502 to perform the methods described herein. For example, the application programs 510 can include applications 1 through N, including a video coding application that performs the methods described herein.

[0196] The apparatus 500 can also include one or more output devices, such as a display 518. In one example, the display 518 can be a touch-sensitive display that combines a display with a touch-sensitive element that can be used to sense touch input. The display 518 can be coupled to the processor 502 via the bus 512.

[0197] Although the bus 512 in the apparatus 500 is described herein as a single bus, the bus 512 can include multiple buses. Also, auxiliary storage can be directly coupled to other components of the apparatus 500 or accessed via a network, and can include a single integrated unit such as a memory card or multiple units such as multiple memory cards. Thus, the apparatus 500 can have a wide variety of configurations.

[0198] Image coding:

[0199] Nowadays, multimedia data occupies a large portion of the Internet traffic. Compression of image data plays an important role in storage and efficient transmission of multimedia data. Therefore, image coding technology is a very practical technology. It should be noted that the Chinese translation of the English terms "image coding" and "image encoding" is usually "image coding". Image coding is broad and includes the process of encoding an image into a bitstream and the process of decoding (decoding) a bitstream into an image.

[0200] Image coding refers to the process of encoding an image into a bitstream. The study of image coding has a long history. Researchers have proposed a large number of methods and formulated I-frame coding methods for various image coding standards and video coding standards such as JPEG, JPEG2000, JPEG-XL, JPEG-XX, WebP, H.264 / AVC, H.264 / HEVC, H.26 / VVC, AVS3, and AV1. These coding methods are mostly based on transform, prediction, and entropy coding techniques. Although these coding methods are currently widely used, due to the increase in image data volume and the emergence of new media types, coding methods with higher compression efficiency are needed.

[0201] Deep learning-based image coding:

[0202] In recent years, researchers have studied deep learning-based image coding methods. Some researchers have achieved good results. For example, Balle et al. proposed an end-to-end optimized image coding method that outperforms existing best image coding and even outperforms the existing best traditional coding standard H.265 / HEVC.

[0203] Deep learning-based image coding is based on deep neural networks, usually convolutional neural networks. Some research work proposes an image coding method based on a Transformer network. The structure of the deep neural network can be manually designed or obtained through neural architecture search (NAS). The parameters of the deep neural network are obtained by using a loss function and a backpropagation algorithm.

[0204] Figure 6 shows a typical deep learning-based image compression method, also known as neural network-based image compression. Generally, a neural network-based image compression method includes the following parts: a feature extraction module, a feature quantization module, an entropy coding module, an entropy decoding module, a feature dequantization module, and a feature decoding module. On the encoder side, the feature extraction module can use a nonlinear mapping activation function to obtain extracted three-dimensional feature maps through multiple layers of convolution stacking. The feature quantization module quantizes the floating-point feature values through feature value quantization to obtain quantized feature values. The quantized feature values are losslessly entropy coded to obtain the coded bitstream. When receiving the entropy-coded bitstream, the decoder performs lossless entropy decoding to obtain three-dimensional quantized feature values. The feature decoding module decodes the features into a reconstructed image to achieve decoding.

[0205] After the to-be-compressed image passes through the feature extraction module and the feature quantization module, a three-dimensional feature quantization graph is obtained. When processing each feature value in the three-dimensional feature quantization graph, the entropy coding module can estimate and obtain the probability distribution of the feature value by using the feature values in the processed neighborhood as the context, and perform subsequent coding based on the probability distribution to obtain the encoded bit stream.

[0206] [According to Rule 26 Correction 02.02.2026] With the outstanding performance of deep learning in various fields, researchers have proposed an end-to-end image coding solution based on deep learning. Figure 7 shows the coding framework. The specific technical solution is as follows: on the encoder side, the original image is input to the feature extraction module, and the feature map is output. The feature map passes through the side information extraction module, and the side information is output on the encoder and decoder sides, is input to the probability estimation module, and the probability distribution of each feature element is output to obtain the value of the to-be-coded feature element In addition, the feature map is input into the quantization module to obtain the quantized feature map The entropy coding module entropy encodes each feature element in the quantized feature map based on the probability distribution of each feature element to obtain the encoded bit stream.

[0207] [According to Rule 26 Correction 02.02.2026] On the decoder side, the decoder parses the bit stream and outputs the probability distribution of the to-be-coded symbol based on the accompanying information to obtain the value of the to-be-decoded feature element The k-entropy decoding module arithmetically decodes each feature element in the quantized feature map based on the probability distribution of each feature element to obtain the value of the feature element The feature map is input to the image reconstruction module, and the reconstructed image is output.

[0208] [According to Rule 26 Correction 02.02.2026] The neural network

[0209] [According to Rule 26 Correction 02.02.2026] The neural network can include neurons. The neuron can be an operation unit using xs and intercept 1 as input. The output of the operation unit can be:

[0210] Here, s = s = 1, 2,..., n, n is a natural number greater than 1, Ws is a weight of xs, b is a bias of the neuron, and f is an activation function of the neuron (activation function) for introducing a nonlinear characteristic into the neural network to convert an input signal in the neuron into an output signal. The output signal of the activation function can be used as an input of a next convolutional layer. The activation function can be a sigmoid function. The neural network is a network constituted by connecting a plurality of single neurons together. Specifically, an output of one neuron can be an input to another neuron. An input of each neuron can be connected to a local receptive field of a previous layer to extract a feature of the local receptive field. The local receptive field can be a region including several neurons.

[0211] Convolutional neural network:

[0212] A convolutional neural network (CNN) is a kind of deep neural network having a convolutional structure. The convolutional neural network includes a feature extractor including a convolutional layer and a subsampling layer. The feature extractor can be considered as a filter. The convolutional layer is a layer of neurons in the convolutional neural network that performs convolution processing on an input signal. In the convolutional layer of the convolutional neural network, one neuron can be connected to only a part of neurons of an adjacent layer. The convolutional layer usually includes several feature planes, each of which can include some neurons arranged in a rectangle. Neurons on the same feature plane share one weight, where the shared weight is a convolution kernel. The shared weight can be understood as a way of extracting image information regardless of the location. The convolution kernel can be initialized in the form of a matrix of random size. During the training process of the convolutional neural network, appropriate weights can be obtained for the convolution kernel by learning. In addition, the shared weight directly reduces the connections between the layers of the convolutional neural network and reduces the risk of overfitting.

[0213] Figure 8 schematically illustrates the general concept of processing by a neural network such as a CNN. A convolutional neural network is composed of an input layer and an output layer and a plurality of hidden layers. The input layer is the layer that provides the input, e.g. a portion of an image as shown in Figure 8, to be processed. The hidden layers of a CNN are usually composed of a series of convolutional layers that convolve with a multiplication or other dot product. The result of a layer is one or more feature maps, sometimes also called channels. Subsampling can be involved in some or all layers. Thus, as shown in Figure 8, the feature maps can become smaller. The activation function in a CNN is usually a RELU (Rectified Linear Unit) layer, followed by additional convolutions, such as pooling layers, fully connected layers, and normalization layers, called hidden layers because their inputs and outputs are masked by activation functions and final convolutions. Although these layers are colloquially called convolutions, this is just a convention. Mathematically, it is technically a sliding dot product or cross-correlation. This has important implications for the indexing in the matrix, as it affects how the weights are determined at a particular index point.

[0214] When programming a CNN for processing images, as shown in FIG. 8, the input is a tensor with shape (number of images) x (image width) x (image height) x (image depth). Then, after passing through a convolutional layer, the image is abstracted into a feature map with shape (number of images) x (feature map width) x (feature map height) x (feature map channels). A convolutional layer in a neural network should have the following properties. Convolutional kernel defined by width and height (hyperparameters). Number of input channels and output channels (hyperparameters). The depth of the convolutional filter (input channels) should be equal to the number of channels (depth) of the input feature map.

[0215] In the past, traditional multilayer perceptron (MLP) models have been used for image recognition. However, they have a high dimensionality due to the full connectivity between nodes and do not scale well with higher resolution images. A 1000x1000 pixel image with RGB color channels has 3 million weights, which is too high to be efficiently processed at scale with full connectivity. Furthermore, this network architecture does not take into account the spatial structure of the data, treating input pixels that are far apart the same way as those that are close. This ignores the locality of reference in image data both computationally and semantically. Therefore, the full connectivity of neurons is wasteful for purposes such as image recognition, which is dominated by spatially local input patterns.

[0216] Convolutional neural networks are biologically inspired variants of multilayer perceptrons specifically designed to mimic the behavior of the visual cortex. These models alleviate the challenges posed by the MLP architecture by exploiting the strong spatial locality correlations present in natural images. The convolutional layer is the core building block of CNNs. The parameters of this layer consist of a set of learnable filters (the aforementioned kernels) with a small receptive field, but with a large stride that slides nicely over the entire depth of the input volume. During the forward pass, each filter is convolved across the width, height, and depth of the input volume, computing an activation map of that filter by applying the dot product between the filter’s weights and the input at a certain spatial position. The network learns this filter by using backpropagation, and when it detects some specific type of feature at a certain spatial position in the input, the filter becomes active.

[0217] Stacking the activation maps of all filters along the depth dimension forms the complete output volume of the convolutional layer. Therefore, each entry in the output volume can also be interpreted as the output of a neuron that observes a small region in the input and shares parameters with neurons in the same activation map. A feature map or activation map is the output activation of a given filter. Feature map and activation have the same meaning. In some papers, it is called an activation map because it is a map of activations corresponding to different parts of the image, and also a feature map because it is also a map of finding some feature in the image. High activation means that some function is found.

[0218] Another important concept for cell neural networks is pooling, a form of nonlinear down-sampling. There are several nonlinear functions that can implement pooling, with max-pooling being the most common. It divides the input image into a set of non-overlapping rectangles and, for each such sub-region, outputs the maximum value.

[0219] Intuitively, the exact position of a feature is less important than its coarse position relative to other features. This is the idea behind the use of pooling in convolutional neural networks. Pooling layers are used to gradually reduce the spatial size of the representations, reducing the number of parameters, memory, and computation in the network, and thus also controlling overfitting. It is common in CNN architectures to periodically insert a pooling layer between successive convolutional layers. Pooling operations provide another form of translation invariance.

[0220] Pooling layers operate independently on each depth slice of the input and adjust its size spatially. The most common form is a pooling layer with a filter of size 2x2, applying 2 steps of down-sampling along width and height on each depth slice in the input, discarding 75% of the activations. In this case, each max operation exceeds 4 numbers. The depth dimension remains unchanged.

[0221] In addition to max-pool, other functions can be used for the pooling unit, such as average-pool or 2-norm-pool. Average-pool has been used historically, but has fallen out of favor compared to max-pool, which performs better in practice. Due to the large reduction in representation size, there has been a recent trend to use smaller filters or to drop the pooling layer altogether. “Region of interest” pooling (also called ROI-pooling) is a variant of max-pool, where the output size is fixed and the input rectangle is a parameter. Pooling is an important component of convolutional neural networks for object detection, based on the fast R-CNN architecture.

[0222] ReLU, short for rectified linear unit, applies a non-saturating activation function. By setting negative values to zero, it effectively removes negative values from the activation mapping. It increases the non-linear properties of the decision function and the entire network without affecting the receptive field of the convolutional layers. Other functions are also used to increase non-linearity, such as hyperbolic tangent and sigmoid functions. ReLU is generally more popular than other functions because it trains neural networks several times faster without a significant impact on the generalization accuracy.

[0223] After several convolutional and max-pooling layers, high-level reasoning in a neural network is done by fully connected layers. Neurons in a fully connected layer have connections to all activations in the previous layer, as seen in regular (non-convolutional) artificial neural networks. Their activations can thus be computed as an affine transformation, a matrix multiplication followed by a bias offset (vector addition of learned or fixed bias terms).

[0224] A "loss layer" specifies how to penalize deviations between predictions (output) and true labels, typically the last layer of a neural network. Various loss functions can be used that are appropriate for different tasks. Softmax loss is used for predicting a single class out of K mutually exclusive classes. Sigmoid cross-entropy loss is used for predicting K independent probability values in [0, 1]. Euclidean loss is used for regression to real-valued labels.

[0225] In summary, Fig. 8 shows the data flow in a typical convolutional neural network. First, an input image is passed through a convolutional layer and abstracted into a feature map that includes several channels, corresponding to the number of filters in the set of learnable filters of that layer (e.g., one channel per filter). The feature map is then subsampled using, e.g., a pooling layer, which reduces the dimensionality of each channel in the feature map. The data then enters another convolutional layer, which can have a different number of output channels, resulting in a different number of channels in the feature map. As mentioned above, the number of input and output channels is a hyperparameter of the layer. To establish the connectivity of the network, these parameters need to be synchronized between two connected layers, e.g., the number of input channels of the current layer should equal the number of output channels of the previous layer. For the first layer that processes input data (e.g., an image), the number of input channels is typically equal to the number of channels in the data representation, e.g., 3 channels for an RGB or YUV representation of an image or video, or 1 channel for a grayscale image or video representation.

[0226] In the practical application of video image coding, it is usually necessary to code the region of interest and the background region with different code rates to meet the customer's demand with less code rate.

[0227] To this end, an embodiment of the present application provides an encoding method, which is applicable to a codec network.

[0228] Fig. 9 shows a possible implementation of the codec network. As shown in Fig. 9, the codec network includes an encoding network module, a code rate control module, an entropy encoding module, a hyper-encoding module, a hyper-decoding module, an entropy decoding module, an inverse quantization module, and a decoding network module. The code rate control module can control the code rate of the code stream generated by the encoder through a quality matrix.

[0229] The image input into the encoder of the codec network is encoded via the encoding network module, the code rate control module, the entropy encoding module, and the hyper-encoding module to obtain a code stream. After the code stream is input into the decoder of the codec network, the input image is reconstructed via the hyper-decoding module, the entropy decoding module, the inverse quantization module, and the decoding network module to obtain a reconstructed image.

[0230] The above image can be a YUV domain image or an RGB domain image.

[0231] FIG. 10 shows another possible implementation of the codec network. As shown in FIG. 10, the codec network includes a Y component encoding network module, a Y component code rate control module, a Y component hyper-encoding module, a UV component encoding network module, a UV component code rate control module, a UV component hyper-encoding module, an entropy encoding module, an entropy decoding module, a Y component hyper-decoding module, a Y component inverse quantization module, a Y component decoding network module, a UV component hyper-decoding module, a UV component inverse quantization module, and a UV component decoding network module.

[0232] Unlike the codec network shown in FIG. 9, the Y component and the UV component of the YUV image can be respectively encoded and decoded by the codec network shown in FIG. 10.

[0233] Referring to FIG. 11, FIG. 11 is an encoding method provided by an embodiment of the present application. As shown in FIG. 11, the encoding method can include:

[0234] S1101, obtaining a target quality matrix.

[0235] The target quality matrix is used to represent the image quality of each region in the residual map of the feature domain. The target quality matrix includes a plurality of quality values, and each quality value corresponds to the image quality of each region in the residual map of the feature domain.

[0236] In a possible implementation, a quality map can be obtained, and the target quality matrix can be determined according to the quality map. The quality map is used to record the image quality of each region in the residual map of the feature domain.

[0237] It should be noted that the specific manner of obtaining the quality map and the target quality matrix can be any manner thought of by those skilled in the art, and the embodiments of the present application do not limit the same.

[0238] For example, the quality value (image quality value) of each point in the quality map and the target quality matrix can be set by a user.

[0239] For another example, the quality map or the quality matrix can be obtained by analyzing an input image (or a feature map) through a code control algorithm, a region of interest detection algorithm, etc. The size of the input image (or the feature map) and the size of the quality map can be the same.

[0240] In a possible implementation, the input image can be a complete image, or the Y component and the UV component of the image.

[0241] S1102, scaling the first residual map and / or the first Gaussian distribution parameter information according to the target quality matrix to obtain a second residual map and / or a second Gaussian distribution parameter information.

[0242] Wherein, the first residual map is the residual map of the feature domain, and the first Gaussian distribution parameter information is the Gaussian distribution parameter information of the residual map of the feature domain.

[0243] In one possible implementation, a target quality scaling matrix can be determined based on the aforementioned target quality matrix. This target quality scaling matrix characterizes the scaling amount of each region in the residual map of the aforementioned feature domain and / or the aforementioned Gaussian distribution parameter information. A target quality scaling tensor is then determined based on the target quality scaling matrix. This target quality scaling tensor characterizes the scaling tensor of the residual map of the aforementioned feature domain and / or the aforementioned Gaussian distribution parameter information in three-dimensional space. The first residual map is then scaled according to the target quality scaling tensor to obtain the second residual map, and / or the first Gaussian distribution parameter information is scaled according to the target quality scaling tensor to obtain the second Gaussian distribution parameter information.

[0244] For example, for the encoded number domain, the quality scaling value of each point in the image can be determined based on the quality value (i.e., quality level index) Q[i,j] of each point in the target quality matrix. Here, the quality scaling value Q[i,j] of any point in the target quality scaling matrix... m [i,j] satisfies:

[0245] That is, quality scaling value

[0246] Where k is an integer and n is a real number. For example, k can be 4 and n can be 0.

[0247] In one possible implementation, Q can be... m Perform fixed-point processing to Q int .

[0248] Optionally, Q int It can satisfy: Q int =round(Q m *2 denom )+b

[0249] Where b is an integer.

[0250] by For example, when the quality level Q[i,j] ranges from [-8,8], Q is represented by 6 bits in a fixed-point representation. m The specific values ​​are shown in Table 1, where Q m =Q int >>denom.

[0251] Table 1

[0252] For example, for the encoded log field, in addition to calculating the quality scaling value, it is also necessary to calculate the quality scaling value Q of the encoded log field. m_log The quality scaling value Q of the encoded log field m_log It can satisfy:

[0253] Right now

[0254] In practical applications, for encoding the log field, Q is... m_log When performing fixed-point conversion, the relevant parameters of the log field can be used. k 2*a is used for fixed-point transformation, and the quality scaling value Q of the fixed-point transformation is... m_log It can satisfy:

[0255] Q m_log [i,j]=t*Q[i,j]>>(2+precision)

[0256] For example, for coded 3D maps, the quality scaling value for each point in the image can be determined based on the image quality value (i.e., quality level index) Q[c,i,j] of each point in the target quality matrix. Here, the quality scaling value Q... m [c,i,j] satisfies:

[0257] That is, quality scaling value

[0258] For encoding a 3D map, it is also necessary to additionally calculate the quality scaling value Q of the encoded log domain. m_log The quality scaling value Q of the encoded log field m_log It can satisfy:

[0259] Right now

[0260] In practical applications, for encoding 3D maps, Q... m_log When performing fixed-point conversion, the relevant parameters of the log field can be used. k 2*a is used for fixed-point transformation, and the quality scaling value Q of the fixed-point transformation is... m_log It can satisfy: Q m_log [c,i,j]=t*Q[c,i,j]>>(2+precision)

[0261] In one possible implementation, the target quality scaling tensor can be determined based on the target quality scaling matrix and the gain vector, wherein the gain vector includes a channel-level quality adjustment gain vector and / or an image-level quality control factor.

[0262] It can be understood that the feature map CxHxW three-dimensional tensor can be two-dimensional HxW quality scaling matrix Q m extended to a three-dimensional tensor.

[0263] In a possible implementation, the target quality scaling tensor is determined according to the target quality scaling matrix and the gain parameter, and the gain parameter includes a gain vector of channel-level quality adjustment and / or an image-level quality control factor.

[0264] As shown in FIG. 12, a quality scaling graph representing the target quality scaling matrix can be determined according to a quality graph representing the target quality matrix. Then, the scaling tensor (target quality scaling tensor) is determined according to the quality scaling graph representing the target quality scaling matrix and the gain parameter.

[0265] Exemplarily, for the encoding number domain, in the case that there is a gain vector m t of channel-level quality adjustment and an image-level quality control factor βdisplacement, the target quality scaling tensor m[c,i,j] can satisfy: m[c,i,j] = Q m [i,j]·βdisplacement·m t [c]

[0266] or: m[c,i,j] = (Q int [i,j]·βdisplacement>>denom)·m t [c]

[0267] or: m[c,i,j] = (Q int [i,j]·βdisplacement·m t [c])>>denom

[0268] For the encoding log domain, in the case that there is a gain vector m t of channel-level quality adjustment and an image-level quality control factor βdisplacement, the target quality scaling tensor m[c,i,j] can satisfy: m[c,i,j] = Q m [i,j]·βdisplacement·m t [c]

[0269] or: m[c,i,j] = (Q int [i,j]·βdisplacement>>denom)·m t [c]

[0270] or: m[c,i,j] = (Q int[i,j] • βdisplacement • m t [c]) » denom

[0271] For encoding log domain, additional computation of target quality scaling tensor m_log in log domain is needed, which satisfies m_log[c,i,j] = Q m_ log[i,j] + βdisplacement_log + m t_log [c]

[0272] For encoding three-dimensional map, in the presence of channel-level quality adjustment gain vector m t , and image-level quality control factor βdisplacement, target quality scaling tensor m[c,i,j] can satisfy: m[c,i,j] = Q m [c,i,j] • βdisplacement • m t [c]

[0273] Or: m[c,i,j] = (Q int [c,i,j] • βdisplacement » denom) • m t [c]

[0274] Or: m[c,i,j] = (Q int [c,i,j] • βdisplacement • m t [c]) » denom

[0275] For encoding three-dimensional map, additional computation of target quality scaling tensor m_log in log domain is needed, which satisfies m_log[c,i,j] = Q m_ log[c,i,j] + βdisplacement_log + m t_log [c]

[0276] Again exemplarily, for encoding number domain, in the presence of frame-level quality control factor βdisplacement, target quality scaling tensor m[c,i,j] can satisfy: m[c,i,j] = Q m [i,j] • βdisplacement,

[0277] Or: m[c,i,j] = (Q int [c,i,j] » denom) • βdisplacement,

[0278] Or: m[c,i,j] = (Q intm[c, i, j] = (Q

[0279] For encoding log-domain, the target quality scaling tensor m[c, i, j] can satisfy: m[c, i, j] = Q m [i, j] · βdisplacement,

[0280] or:

[0281] m[c, i, j] = (Q int [i, j] » denom) · βdisplacement,

[0282] or:

[0283] m[c, i, j] = (Q int [i, j] · βdisplacement) » denom

[0284] For encoding log-domain, the target quality scaling tensor m_log in log-domain, which can satisfy: m_log[c, i, j] = Q m_ log[i, j] + βdisplacement_log

[0285] For encoding three-dimensional map, the target quality scaling tensor m[c, i, j] can satisfy: m[c, i, j] = Q m [c, i, j] · βdisplacement,

[0286] or: m[c, i, j] = (Q int [c, i, j] » denom) · βdisplacement,

[0287] or: m[c, i, j] = (Q int [c, i, j] · βdisplacement) » denom

[0288] For encoding three-dimensional map, the target quality scaling tensor m_log in log-domain, which can satisfy: m_log[c, i, j] = Q m_ log[c, i, j] + βdisplacement_log

[0289] In another possible implementation, the target quality scaling tensor can be determined according to the target quality scaling matrix.

[0290] Exemplarily, for the coding number domain, without other quality control factors (gain parameters), the target quality scaling tensor m[c,i,j] can satisfy: m[c,i,j] = Q m [i,j]

[0291] Or: m[c,i,j] = Q int [i,j] >> denom

[0292] For the coding log domain, without other quality control factors (gain parameters), the target quality scaling tensor m[c,i,j] can satisfy: m[c,i,j] = Q m [i,j]

[0293] Or: m[c,i,j] = Q int [i,j] >> denom

[0294] For the coding log domain, it is also necessary to additionally calculate the target quality scaling tensor m_log of the log domain, and the target quality scaling tensor m_log of the log domain satisfies: m_log[c,i,j] = Q m_log [i,j]

[0295] For the coding three-dimensional map, without other quality control factors (gain parameters), the target quality scaling tensor m[c,i,j] can satisfy: m[c,i,j] = Q m [c,i,j]

[0296] Or: m[c,i,j] = Q int [c,i,j] >> denom

[0297] For the coding three-dimensional map, it is also necessary to additionally calculate the target quality scaling tensor m_log of the log domain, and the target quality scaling tensor m_log of the log domain satisfies: m_log[c,i,j] = Q m_log [c,i,j]

[0298] In a possible implementation, for the coding number domain, the second residual image r'[c,i,j] can satisfy: r'[c,i,j] = (m[c,i,j] r[c,i,j]) >> (Precision)

[0299] Wherein, r[c,i,j] is the first residual image, and Precision can be any non-negative number. For example, Precision can be 0, 1, 13, etc. non-negative numbers.

[0300] In a possible implementation, for the coding number domain, the second Gaussian distribution parameter information (second variance) σ' can satisfy: σ'[c, i, j] = (m[c, i, j]·σ[c, i, j]) » (Precision2)

[0301] wherein σ is the first Gaussian distribution parameter information (first variance), and Precision2 can be any non-negative number. For example, Precision2 can be 0, 1, 13, and the like.

[0302] In a possible implementation, for the coding log domain, the second residual image r'[c, i, j] can satisfy: r'[c, i, j] = (m[c, i, j]·r[c, i, j]) » (Precision)

[0303] In a possible implementation, for the coding log domain, the second Gaussian distribution parameter information (second variance) σ' can satisfy: σ'[c, i, j] = m log [c, i, j] + σ[c, i, j] - Precision2_log

[0304] S1103. Generating a code stream according to the target quality matrix, the second residual image, and / or the second Gaussian distribution parameter information.

[0305] In a possible implementation, the target quality matrix and the second residual image can be encoded to generate the code stream, and the second residual image is encoded according to the second Gaussian distribution parameter information.

[0306] In a possible implementation, the target quality matrix or a target quality residual matrix of the target quality matrix can be encoded to generate the code stream, and the target quality residual matrix is used to represent a residual value of each image quality in the target quality matrix.

[0307] In a possible implementation, the target quality residual matrix of the target quality matrix can be generated according to the target quality matrix.

[0308] It should be noted that the specific manner of generating the target quality residual matrix of the target quality matrix according to the target quality matrix can be any manner thought by those skilled in the art, and the embodiments of the present application do not limit the same.

[0309] For example, a prediction matrix of the target quality matrix can be determined according to the target quality matrix, and then a target quality residual matrix can be determined according to the prediction matrix of the target quality matrix. Wherein, a prediction value of any point in the prediction matrix of the target quality matrix can be calculated according to the left and upper adjacent points of the quality value corresponding to the prediction value of the point in the target quality matrix. In the case of only existing the upper adjacent point, the prediction value of the point can be calculated only according to the upper adjacent point of the point. In the case of only existing the left adjacent point, the prediction value of the point can be calculated only according to the left adjacent point of the point. In the case of only not existing the left and upper adjacent points, the prediction value of the point can be determined as 0.

[0310] For example, the prediction value Q_pred[i,j] of any point in the prediction matrix of the target quality can satisfy: Q_pred[i,j] = (Q[i,j-1] + Q[i-1,j]) / 2

[0311] For example, the residual value Q[i,j] of any point in the target quality residual matrix can satisfy: Q[i,j] = Q[i,j] - Q_pred[i,j] res res Q[i,j] = Q[i,j] - Q_pred[i,j]

[0312] In practical applications, the value range of the residual value in the target quality residual matrix can be N times of the value range of the quality value in the target quality matrix. N is a positive number. For example, the value range of the residual value in the target quality residual matrix can be 2 times of the value range of the quality value in the target quality matrix. If the value range of the quality value in the target quality matrix is [-8, 8], the value range of the residual value in the target quality residual matrix can be [-16, 16].

[0313] In a possible implementation, a Gaussian distribution parameter of the target quality residual matrix can be determined. A probability distribution of the target quality residual matrix is determined according to the Gaussian distribution parameter. The Gaussian distribution parameter is written into the code stream. The target quality residual matrix is entropy encoded according to the probability distribution.

[0314] For example, the Gaussian distribution parameter of the target quality residual matrix can be determined. A probability distribution table of the target quality residual matrix is determined according to the Gaussian distribution parameter. The Gaussian distribution parameter is written into the code stream. The target quality residual matrix is entropy encoded (such as me-tANS entropy encoding) according to the probability distribution table.

[0315] Optionally, the Gaussian distribution parameter index number can also be written into the code stream.

[0316] As shown in Table 2, if the probability distribution of the Gaussian distribution parameter is 0.2, the Gaussian distribution parameter index number 0 can be written into the code stream.

[0317] Table 2 ​

[0318] In another possible implementation, the target probability distribution can be determined from a plurality of candidate probability distributions according to the target quality residual matrix. An index number of the target probability distribution is written into the bitstream. The target quality residual matrix is entropy encoded according to the target probability distribution.

[0319] Exemplarily, the probability distribution of the residual values in the target quality residual matrix can be matched from a plurality of preset candidate probability distributions to determine a probability distribution with a higher similarity to the probability distribution of the residual values. An index number of the target probability distribution is written into the bitstream, and then the probability distribution obtained through the matching is used to entropy encode the residual matrix based on me-tANS.

[0320] Optionally, the target probability distribution can also be written into the bitstream.

[0321] In yet another possible implementation, a Gaussian distribution parameter of the target quality matrix can be determined. A probability distribution of the target quality matrix is determined according to the Gaussian distribution parameter. The Gaussian distribution parameter is written into the bitstream. The target quality matrix is entropy encoded according to the probability distribution.

[0322] Optionally, the index number of the Gaussian distribution parameter can also be written into the bitstream.

[0323] In yet another possible implementation, a target probability distribution is determined from a plurality of candidate probability distributions according to the target quality matrix. An index number of the target probability distribution is written into the bitstream. The target quality matrix is entropy encoded according to the target probability distribution.

[0324] Optionally, the target probability distribution can also be written into the bitstream.

[0325] In a possible implementation, the encoding method provided by the embodiment of the present application can further include: inputting the image into an encoding network to obtain a first feature map. Inputting the first feature map into a context network to obtain a first prediction map. Determining the first residual map according to the first feature map and the first prediction map. Inputting the first feature map into a hyper-prior encoding network to obtain first hyper-prior information. Quantizing the first hyper-prior information to obtain second hyper-prior information. Inputting the second hyper-prior information into a hyper-prior variance decoding network to obtain the first Gaussian distribution parameter information.

[0326] FIG. 13 is an encoding end architecture provided by an embodiment of the present application, and the above encoding method is described below in combination with the encoding end architecture.

[0327] As shown in FIG. 13, an input encoding network (analysis transform) of an image x input encoding end can obtain a feature map y (i.e., the first feature map described above). The image x can be an entire image, or can also be a Y component or a UV component of a YUV image.

[0328] The obtained feature map y is input into a hyper encoder to obtain hyper prior information z (i.e., the first hyper prior information described above). The quantized z is subjected to me-tANS entropy encoding to be written into a bitstream.

[0329] The obtained hyper prior information z is quantized to obtain quantized hyper prior information z' (i.e., the second hyper prior information described above). As shown in FIG. 13, the quantized hyper prior information z' can also be subjected to entropy encoding (me-tANS) to be written into a bitstream.

[0330] The quantized hyper prior information z' is input into a hyper scale decoder to obtain Gaussian distribution parameter information σ (i.e., the first Gaussian distribution parameter information described above).

[0331] The quantized hyper prior information z' is input into a hyper decoder.

[0332] The Gaussian distribution parameter information σ is input into a sigma scale module to be scaled by a quality matrix Q m (i.e., the target quality matrix described above) to obtain Gaussian distribution parameter information σ' (i.e., the second Gaussian distribution parameter information described above).

[0333] The obtained feature map y is input into a joint context network (MSM) to obtain a predicted map u of the feature map y (i.e., the first predicted map described above).

[0334] A residual r is calculated by the feature map y and the predicted map u to obtain a residual map r (i.e., the first residual map described above).

[0335] The obtained residual map r is input into a gain unit module to be scaled by a quality matrix Q m (i.e., the target quality matrix described above) to obtain a residual map r' (i.e., the second residual map described above).

[0336] Optionally, the encoding end architecture can be a JPEG AI encoding end architecture.

[0337] As can be seen, the gain unit and / or the sigma scale module in the encoding end architecture are improved in the embodiments of the present application, so that the JPEG AI encoding end architecture supports spatial domain rate adjustment.

[0338] Referring to FIG. 14, FIG. 14 is a decoding method provided by an embodiment of the present application. As shown in FIG. 14, the decoding method can include the following steps.

[0339] S1401, obtaining a code stream.

[0340] The target quality matrix is used to represent the image quality of each region in the residual map of the feature domain.

[0341] S1402, determining a target quality matrix according to the code stream.

[0342] The target quality matrix is used to represent the image quality of each region in the residual map of the feature domain.

[0343] In a possible implementation, the code stream can be decoded to obtain a Gaussian distribution parameter of a target quality residual matrix of the target quality matrix. A probability distribution of the target quality residual matrix is determined according to the Gaussian distribution parameter. The code stream is decoded according to the probability distribution to obtain the target quality residual matrix. The target quality matrix is determined according to the target quality residual matrix.

[0344] Alternatively, the code stream can also be decoded to obtain an index number of a Gaussian distribution parameter of a target quality residual matrix of the target quality matrix. For example, if the index number of the Gaussian distribution parameter is 0, a probability distribution of 0.2 can be determined according to Table 2.

[0345] In another possible implementation, the code stream can be decoded to obtain a Gaussian distribution parameter of the target quality matrix. A probability distribution of the target quality matrix is determined according to the Gaussian distribution parameter. The code stream is decoded according to the probability distribution to obtain the target quality matrix.

[0346] In yet another possible implementation, the code stream can be decoded to obtain an index number of a target probability distribution, which is determined from a plurality of candidate probability distributions according to a target quality residual matrix of the target quality matrix. The code stream is decoded according to the target probability distribution to obtain the target quality residual matrix. The target quality matrix is determined according to the target quality residual matrix.

[0347] In yet another possible implementation, the code stream can be decoded to obtain an index number of a target probability distribution, which is determined from a plurality of candidate probability distributions according to the target quality matrix. The code stream is decoded according to the target probability distribution to obtain the target quality matrix.

[0348] In a possible implementation, the target quality prediction matrix of the target quality matrix can be determined according to the target quality residual matrix, and the target quality matrix can be determined according to the target quality residual matrix and the target quality prediction matrix.

[0349] It should be noted that the specific manner of determining the target quality prediction matrix of the target quality matrix can be any manner conceived by those skilled in the art, and the embodiments of the present application do not limit the same.

[0350] For example, the prediction matrix of the target quality matrix can be determined according to the target quality matrix, and then the target quality residual matrix can be determined according to the prediction matrix of the target quality matrix. Wherein, the prediction value of any point of the prediction matrix of the target quality matrix can be calculated according to the left and upper adjacent points of the quality value corresponding to the point in the target quality matrix. In the case of only existing upper adjacent points, the prediction value of the point can be calculated only according to the upper adjacent points of the point. In the case of only existing left adjacent points, the prediction value of the point can be calculated only according to the left adjacent points of the point. In the case of only not existing left and upper adjacent points, the prediction value of the point can be determined as 0.

[0351] Exemplarily, the prediction value Q_pred[i,j] of any point in the target quality prediction matrix can satisfy: Q_pred[i,j] = (Q[i,j-1] + Q[i-1,j]) / 2

[0352] Exemplarily, the quality value Q[i,j] of any point in the target quality matrix can satisfy: Q[i,j] = Q res [i,j] + Q_pred[i,j]

[0353] S1403, scaling the third Gaussian distribution parameter information according to the target quality matrix to obtain fourth Gaussian distribution parameter information.

[0354] Wherein, the third Gaussian distribution parameter information is the Gaussian distribution parameter information of the residual map of the feature field.

[0355] In a possible implementation, the target quality scaling matrix can be determined according to the target quality matrix. The target quality scaling tensor can be determined according to the target quality scaling matrix. The third Gaussian distribution parameter information is scaled according to the target quality scaling tensor to obtain the fourth Gaussian distribution parameter information. Wherein, the target quality scaling matrix is used to represent the scaling amount of each region of the residual map of the feature field and the Gaussian distribution parameter information. The target quality scaling tensor is used to represent the scaling tensor of the residual map of the feature field and the Gaussian distribution parameter information in three-dimensional space.

[0356] It should be noted that the specific implementation of determining the target quality scaling matrix according to the target quality matrix can refer to the description of determining the target quality scaling matrix according to the target quality matrix in S1402 described above, and will not be repeated here.

[0357] It can be understood that the feature map CxHxW three-dimensional tensor can be scaled by the target quality scaling matrix (and the gain vector) to a two-dimensional HxW quality scaling matrix Q m , which is expanded to a three-dimensional tensor.

[0358] In a possible implementation, the target quality scaling tensor is determined according to the target quality scaling matrix and a gain parameter, and the gain parameter includes a gain vector of channel-level quality adjustment and / or an image-level quality control factor.

[0359] Exemplarily, for the coding number domain, in the case that there is a gain vector m t of channel-level quality adjustment and an image-level quality control factor βdisplacement, the target quality scaling tensor m[c,i,j] can satisfy: m[c,i,j] = Q m [i,j]·βdisplacement·m t [c]

[0360] Or: m[c,i,j] = (Q int [i,j]·βdisplacement>>denom)·m t [c]

[0361] Or: m[c,i,j] = (Q int [i,j]·βdisplacement·m t [c])>>denom

[0362] For the coding log domain, in the case that there is a gain vector m t of channel-level quality adjustment and an image-level quality control factor βdisplacement, the target quality scaling tensor m[c,i,j] can satisfy: m[c,i,j] = Q m [i,j]·βdisplacement·m t [c]

[0363] Or: m[c,i,j] = (Q int [i,j]·βdisplacement>>denom)·m t [c]

[0364] Or: m[c,i,j] = (Q int [i,j]·βdisplacement·mt [c]) » denom

[0365] For encoding the log domain, an additional calculation of a target quality scaling tensor m_log in the log domain is required, which satisfies m_log[c,i,j] = Q m_ log[i,j] + βdisplacement_log + m t_log [c]

[0366] For encoding the three-dimensional map, in the presence of a gain vector m t and an image-level quality control factor βdisplacement, the target quality scaling tensor m[c,i,j] can satisfy: m[c,i,j] = Q m [c,i,j] · βdisplacement · m t [c]

[0367] or: m[c,i,j] = (Q int [c,i,j] · βdisplacement » denom) · m t [c]

[0368] or: m[c,i,j] = (Q int [c,i,j] · βdisplacement · m t [c]) » denom

[0369] For encoding the three-dimensional map, an additional calculation of a target quality scaling tensor m_log in the log domain is required, which satisfies m_log[c,i,j] = Q m_ log[c,i,j] + βdisplacement_log + m t_log [c]

[0370] Yet another example, for encoding the number domain, in the presence of a frame-level quality control factor βdisplacement, the target quality scaling tensor m[c,i,j] can satisfy: m[c,i,j] = Q m [i,j] · βdisplacement,

[0371] or: m[c,i,j] = (Q int [c,i,j] » denom) · βdisplacement,

[0372] or: m[c,i,j] = (Q int[c, i, j] · βdisplacement) » denom

[0373] For encoding log domain, the target quality scaling tensor m[c, i, j] can satisfy: m[c, i, j] = Q m [i, j] · βdisplacement,

[0374] Or: m[c, i, j] = (Q int [i, j] » denom) · βdisplacement,

[0375] Or: m[c, i, j] = (Q int [i, j] · βdisplacement) » denom

[0376] For encoding log domain, the target quality scaling tensor m_log in log domain also needs to be calculated additionally, and the target quality scaling tensor m_log in log domain can satisfy: m_log[c, i, j] = Q m_ log[i, j] + βdisplacement_log

[0377] For encoding three-dimensional map, the target quality scaling tensor m[c, i, j] can satisfy: m[c, i, j] = Q m [c, i, j] · βdisplacement,

[0378] Or: m[c, i, j] = (Q int [c, i, j] » denom) · βdisplacement,

[0379] Or: m[c, i, j] = (Q int [c, i, j] · βdisplacement) » denom

[0380] For encoding three-dimensional map, the target quality scaling tensor m_log in log domain also needs to be calculated additionally, and the target quality scaling tensor m_log in log domain can satisfy: m_log[c, i, j] = Q m_ log[c, i, j] + βdisplacement_log

[0381] In another possible implementation, the target quality scaling tensor can be determined according to the target quality scaling matrix.

[0382] Exemplarily, for the coding number domain, without other quality control factors (gain parameters), the target quality scaling tensor m[c,i,j] can satisfy:

[0383] m[c,i,j] = Q m [i,j]

[0384] Or: m[c,i,j] = Q int [i,j] >> denom

[0385] For the coding log domain, without other quality control factors (gain parameters), the target quality scaling tensor m[c,i,j] can satisfy: m[c,i,j] = Q m [i,j]

[0386] Or: m[c,i,j] = Q int [i,j] >> denom

[0387] For the coding log domain, in addition, the target quality scaling tensor m_log of the log domain needs to be additionally calculated, and the target quality scaling tensor m_log of the log domain satisfies: m_log[c,i,j] = Q m_log [i,j]

[0388] For the coding three-dimensional map, without other quality control factors (gain parameters), the target quality scaling tensor m[c,i,j] can satisfy: m[c,i,j] = Q m [c,i,j]

[0389] Or: m[c,i,j] = Q int [c,i,j] >> denom

[0390] For the coding three-dimensional map, in addition, the target quality scaling tensor m_log of the log domain needs to be additionally calculated, and the target quality scaling tensor m_log of the log domain satisfies: m_log[c,i,j] = Q m_log [c,i,j]

[0391] In a possible implementation, for the coding number domain, the fourth Gaussian distribution parameter information (fourth variance) σ' can satisfy: σ'[c,i,j] = (m[c,i,j]·σ[c,i,j]) >> (Precision2)

[0392] Wherein, σ is the third Gaussian distribution parameter information (third variance), and Precision2 can be any non-negative number. For example, Precision2 can be 0, 1, 13, and the like.

[0393] In a possible implementation, for the encoded log domain and the encoded three-dimensional map, the fourth Gaussian distribution parameter information σ' can satisfy: σ' [c, i, j] = mlog[c, i, j] + σ[c, i, j] - Precision2_log

[0394] where σ is the third Gaussian distribution parameter information, and Precision2 can be any non-negative number. For example, Precision2 can be 0, 1, 13, or the like.

[0395] S1404, decoding the code stream according to the fourth Gaussian distribution parameter information to obtain a third residual map.

[0396] where the third residual map is a residual map of the feature domain.

[0397] S1405, dequantizing the third residual map according to a target quality matrix to obtain a fourth residual map.

[0398] In a possible implementation, the target quality scaling matrix can be determined according to the target quality matrix. The target quality de-scaling tensor is determined according to the target quality scaling matrix. The third residual map is dequantized according to the target quality de-scaling tensor to obtain the fourth residual map.

[0399] In a possible implementation, the target quality de-scaling tensor is determined according to the target quality scaling matrix and a gain parameter. The gain parameter includes a gain vector of channel-level quality adjustment and / or an image-level quality control factor.

[0400] For example, when there is a gain vector m t of channel-level quality adjustment and an image-level quality control factor βdisplacement, the target quality de-scaling tensor m -1 [c, i, j] can satisfy:

[0401] m -1 [c, i, j] = Q m -1 [i, j]·1 / βdisplacement·m -1 t [c]

[0402] or: m -1 [c, i, j] = (Q int -1 [i, j]·1 / βdisplacement)<<denom·m -1 t [c]

[0403] or: m-1 [c, i, j] = (Q int -1 [i, j] · 1 / βdisplacement·m -1 t [c]) << denom

[0404] Exemplarily, in the case where there is a frame-level quality control factor βdisplacement, the target quality dezooming tensor m -1 [c, i, j] can satisfy: m -1 [c, i, j] = Q m -1 [i, j] · 1 / βdisplacement

[0405] Or: m -1 [c, i, j] = (Q int -1 [i, j] · 1 / βdisplacement) << denom

[0406] Or: m -1 [c, i, j] = (Q int -1 [i, j] << denom) · 1 / βdisplacement

[0407] In another possible implementation, the target quality dezooming tensor can be determined according to the target quality zooming matrix described above.

[0408] Exemplarily, in the case where there is no other quality control factor (gain parameter), the target quality dezooming tensor m -1 [c, i, j] can satisfy: m -1 [c, i, j] = Q m -1 [i, j]

[0409] Or: m -1 [c, i, j] = Q int -1 [i, j] << denom

[0410] In a possible implementation, the fourth residual map r'[c, i, j] described above can satisfy: r'[c, i, j] = (m -1 [c, i, j] · r[c, i, j]) << (Precision)

[0411] Wherein, r[c, i, j] is the third residual map, and Precision can be any non-negative number. For example, Precision can be 0, 1, 13, and the like non-negative numbers.

[0412] S1406, determining the reconstructed image according to the fourth residual map.

[0413] Exemplarily, the fourth residual map can be input into a decoding network (Synthesis Transform) to obtain the reconstructed image. The reconstructed image can be a complete image or a Y component or a UV component of a YUV image.

[0414] Please refer to FIG. 15, which is another decoding method provided by the embodiments of the present application. As shown in FIG. 15, the decoding method can include:

[0415] S1501, obtaining the bitstream.

[0416] S1502, determining the target quality matrix according to the bitstream.

[0417] The target quality matrix is used to represent the image quality of each region in the residual map of the feature domain.

[0418] S1503, scaling the third Gaussian distribution parameter information according to the target quality matrix to obtain fourth Gaussian distribution parameter information.

[0419] The third Gaussian distribution parameter information is the Gaussian distribution parameter information of the residual map of the feature domain.

[0420] S1504, decoding the bitstream according to the fourth Gaussian distribution parameter information to obtain a third residual map.

[0421] The third residual map is the residual map of the feature domain.

[0422] S1505, determining the reconstructed image according to the third residual map.

[0423] Please refer to FIG. 16, which is another decoding method provided by the embodiments of the present application. As shown in FIG. 16, the decoding method can include:

[0424] S1601, obtaining the bitstream.

[0425] S1602, determining the target quality matrix according to the bitstream.

[0426] The target quality matrix is used to represent the image quality of each region in the residual map of the feature domain.

[0427] S1603, decoding the bitstream according to the third Gaussian distribution parameter information to obtain a third residual map.

[0428] The third Gaussian distribution parameter information is the Gaussian distribution parameter information of the residual map of the feature domain, and the third residual map is the residual map of the feature domain.

[0429] S1604, dequantize the third residual map according to the target quality matrix to obtain a fourth residual map.

[0430] S1605, determine the reconstructed image according to the fourth residual map.

[0431] FIG. 17 is a decoding end architecture provided by an embodiment of the present application. The above encoding method will be described below in combination with the encoding end architecture.

[0432] As shown in FIG. 17, the feature map y can be obtained by decoding the code stream input into the decoding end.

[0433] The obtained feature map y is input into a hyper scale decoder to obtain Gaussian distribution parameter information (i.e., the third Gaussian distribution parameter information described above).

[0434] The obtained Gaussian distribution parameter information is input into a sigma scale module to be scaled by a quality matrix Q m (i.e., the target quality matrix described above) to obtain scaled Gaussian distribution parameter information (i.e., the fourth Gaussian distribution parameter information described above).

[0435] The residual map (i.e., the third residual map described above) can be obtained by decoding the code stream input into the decoding end according to the scaled Gaussian distribution parameter information.

[0436] The obtained residual map is input into an inv gain unit module to be scaled and dequantized to obtain a dequantized residual map (i.e., the fourth residual map described above).

[0437] The fourth residual map described above is input into a decoding network (Synthesis Transform) to obtain a reconstructed image. The reconstructed image can be a complete image, or a Y component or UV component of a YUV image.

[0438] Optionally, the decoding end architecture can be a JPEG AI decoding end architecture.

[0439] As can be seen, the inv gain unit and / or the sigma scale module in the decoding end architecture are improved in the embodiment of the present application, so that the JPEG AI decoding end architecture supports spatial domain rate adjustment.

[0440] The encoding apparatus for performing the above encoding method will be described below in combination with FIG. 18.

[0441] It can be understood that, to achieve the above functions, the encoding device comprises hardware and / or software modules corresponding to the functions. The algorithm steps of each example described in combination with the embodiments disclosed herein can be implemented in hardware or a combination of hardware and computer software. Whether a certain function is implemented in hardware or computer software driven hardware depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application in combination with the embodiments. However, such implementation should not be considered beyond the scope of the embodiments of the present application.

[0442] The embodiments of the present application can divide the functional modules of the encoding device according to the above method examples. For example, each functional module can be divided according to each function, or two or more functions can be integrated into one processing module. The integrated module can be implemented in the form of hardware. It should be noted that the division of modules in the embodiments is illustrative and is only a logical functional division. Actual implementation can have another division method.

[0443] In the case of dividing each functional module according to each function, FIG. 18 shows a possible composition schematic diagram of the encoding device involved in the above embodiments. As shown in FIG. 18, the encoding device 1800 can include a transceiver unit 1801 and a processing unit 1802.

[0444] The transceiver unit 1801 is configured to obtain a target quality matrix, wherein the target quality matrix is used to represent the image quality of each region in the residual map of the feature domain.

[0445] The processing unit 1802 is configured to scale the first residual map and / or the first Gaussian distribution parameter information according to the target quality matrix to obtain the second residual map and / or the second Gaussian distribution parameter information, wherein the first residual map is the residual map of the feature domain, and the first Gaussian distribution parameter information is the Gaussian distribution parameter information of the residual map of the feature domain.

[0446] The processing unit 1802 is further configured to generate a bitstream according to the target quality matrix, the second residual map and / or the second Gaussian distribution parameter information.

[0447] In a possible implementation, the processing unit 1802 is specifically configured to: determine a target quality scaling matrix according to the target quality matrix, the target quality scaling matrix being used to represent a scaling amount of each region in the residual map of the feature domain and / or the Gaussian distribution parameter information; determine a target quality scaling tensor according to the target quality scaling matrix, the target quality scaling tensor being used to represent a scaling tensor of the residual map of the feature domain and / or the Gaussian distribution parameter information in a three-dimensional space; and scale the first residual map according to the target quality scaling tensor to obtain the second residual map and / or scale the first Gaussian distribution parameter information according to the target quality scaling tensor to obtain the second Gaussian distribution parameter information.

[0448] In a possible implementation, the processing unit 1802 is specifically configured to: determine the target quality scaling tensor according to the target quality scaling matrix and a gain parameter, the gain parameter including a gain vector of a channel-level quality adjustment and / or an image-level quality control factor.

[0449] In a possible implementation, the processing unit 1802 is specifically configured to: encode the target quality matrix and a second residual map to generate a bitstream, the second residual map being encoded according to the second Gaussian distribution parameter information.

[0450] In a possible implementation, the processing unit 1802 is specifically configured to: encode the target quality matrix or a target quality residual matrix of the target quality matrix to generate a bitstream.

[0451] In a possible implementation, the processing unit 1802 is further configured to: generate a target quality residual matrix of the target quality matrix according to the target quality matrix, the target quality residual matrix being used to represent a residual value of each image quality in the target quality matrix.

[0452] In a possible implementation, the processing unit 1802 is specifically configured to: determine a Gaussian distribution parameter of the target quality residual matrix; determine a probability distribution of the target quality residual matrix according to the Gaussian distribution parameter; write the Gaussian distribution parameter into the bitstream; and entropy encode the target quality residual matrix according to the probability distribution.

[0453] In a possible implementation, the processing unit 1802 is specifically configured to: determine a Gaussian distribution parameter of the target quality matrix; determine a probability distribution of the target quality matrix according to the Gaussian distribution parameter; write the Gaussian distribution parameter into the bitstream; and entropy encode the target quality matrix according to the probability distribution.

[0454] In a possible implementation, the processing unit 1802 is specifically configured to determine the target probability distribution from a plurality of candidate probability distributions according to the target quality residual matrix. An index number of the target probability distribution is written into the bitstream. The target quality residual matrix is entropy encoded according to the target probability distribution.

[0455] In a possible implementation, the processing unit 1802 is specifically configured to determine the target probability distribution from a plurality of candidate probability distributions according to the target quality matrix. An index number of the target probability distribution is written into the bitstream. The target quality matrix is entropy encoded according to the target probability distribution.

[0456] In a possible implementation, the obtaining unit 1801 is specifically configured to obtain a quality map, where the quality map is used to record image quality of each region in the residual map of the feature domain. The target quality matrix is determined according to the quality map.

[0457] In a possible implementation, the processing unit 1802 is further configured to input the image into an encoding network to obtain a first feature map. The first feature map is input into a context network to obtain a first prediction map. The first residual map is determined according to the first feature map and the first prediction map. The first feature map is input into a hyper-prior encoding network to obtain first hyper-prior information. The first hyper-prior information is quantized to obtain second hyper-prior information. The second hyper-prior information is input into a hyper-prior variance decoding network to obtain the first Gaussian distribution parameter information.

[0458] The decoding apparatus for performing the decoding method will be described below with reference to FIG. 19.

[0459] It can be understood that, to implement the above functions, the decoding apparatus includes hardware and / or software modules corresponding to the functions. The algorithm steps of each example described in combination with the embodiments disclosed herein can be implemented in the form of hardware or a combination of hardware and computer software. Whether a certain function is implemented in hardware or computer software driven hardware depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application in combination with the embodiments. However, such implementation should not be considered beyond the scope of the embodiments of the present application.

[0460] The embodiments of the present application can divide the functional modules of the decoding device according to the above method examples. For example, each functional module can be divided according to each function, or two or more functions can be integrated into one processing module. The integrated module can be realized in the form of hardware. It should be noted that the division of the modules in the embodiments is illustrative, and is only a logical function division. In actual implementation, another division manner can be used.

[0461] In the case of dividing each functional module according to each function, FIG. 19 shows a possible composition schematic diagram of the decoding device involved in the above embodiments. As shown in FIG. 19, the decoding device 1900 can include a transceiver unit 1901 and a processing unit 1902.

[0462] The transceiver unit 1901 is configured to obtain a bitstream.

[0463] The processing unit 1902 is configured to determine a target quality matrix according to the bitstream, and the target quality matrix is used to represent the image quality of each region in a residual map of a feature domain.

[0464] The processing unit 1902 is further configured to scale third Gaussian distribution parameter information according to the target quality matrix to obtain fourth Gaussian distribution parameter information, and the third Gaussian distribution parameter information is Gaussian distribution parameter information of the residual map of the feature domain.

[0465] The processing unit 1902 is further configured to decode the bitstream according to the fourth Gaussian distribution parameter information to obtain a third residual map, and the third residual map is the residual map of the feature domain.

[0466] The processing unit 1902 is further configured to dequantize the third residual map according to the target quality matrix to obtain a fourth residual map.

[0467] The processing unit 1902 is further configured to determine a reconstructed image according to the fourth residual map.

[0468] In a possible implementation manner, the processing unit 1902 is specifically configured to: determine a target quality scaling matrix according to the target quality matrix, and the target quality scaling matrix is used to represent the scaling amount of each region in the residual map of the feature domain and the Gaussian distribution parameter information; determine a target quality scaling tensor according to the target quality scaling matrix, and the target quality scaling tensor is used to represent the scaling tensor of the residual map of the feature domain and the Gaussian distribution parameter information in a three-dimensional space; and scale the third Gaussian distribution parameter information according to the target quality scaling tensor to obtain the fourth Gaussian distribution parameter information.

[0469] In a possible implementation, the processing unit 1902 is specifically configured to: determine the target quality scaling tensor according to the target quality scaling matrix and gain parameters, and the gain parameters include a gain vector of channel-level quality adjustment and / or an image-level quality control factor.

[0470] In a possible implementation, the processing unit 1902 is specifically configured to: decode the bitstream to obtain a Gaussian distribution parameter of a target quality residual matrix of the target quality matrix. Determine a probability distribution of the target quality residual matrix according to the Gaussian distribution parameter. Decode the bitstream according to the probability distribution to obtain the target quality residual matrix. Determine the target quality matrix according to the target quality residual matrix.

[0471] In a possible implementation, the processing unit 1902 is specifically configured to: decode the bitstream to obtain a Gaussian distribution parameter of the target quality matrix. Determine a probability distribution of the target quality matrix according to the Gaussian distribution parameter. Decode the bitstream according to the probability distribution to obtain the target quality matrix.

[0472] In a possible implementation, the processing unit 1902 is specifically configured to: decode the bitstream to obtain an index number of a target probability distribution, and the target probability distribution is determined from a plurality of candidate probability distributions according to a target quality residual matrix of the target quality matrix. Decode the bitstream according to the target probability distribution to obtain the target quality residual matrix. Determine the target quality matrix according to the target quality residual matrix.

[0473] In a possible implementation, the processing unit 1902 is specifically configured to: decode the bitstream to obtain an index number of a target probability distribution, and the target probability distribution is determined from a plurality of candidate probability distributions according to the target quality matrix. Decode the bitstream according to the target probability distribution to obtain the target quality matrix.

[0474] In a possible implementation, the processing unit 1902 is further configured to: decode the bitstream to obtain a second feature map. Input the second feature map into a hyper-prior variance decoding network to obtain the third Gaussian distribution parameter information.

[0475] In the case of dividing each functional module corresponding to each function, FIG. 20 shows another possible composition diagram of the decoding apparatus involved in the above embodiment, as shown in FIG. 20, the decoding apparatus 2000 can include: a transceiver unit 2001 and a processing unit 2002.

[0476] The transceiver unit 2001 is configured to obtain a bitstream.

[0477] The processing unit 2002 is configured to determine a target quality matrix according to the bitstream, and the target quality matrix is used to represent image quality of each region in the residual map of the feature domain.

[0478] The processing unit 2002 is further configured to scale the third Gaussian distribution parameter information according to the target quality matrix to obtain fourth Gaussian distribution parameter information, and the third Gaussian distribution parameter information is Gaussian distribution parameter information of the residual map of the feature domain.

[0479] The processing unit 2002 is further configured to decode the bitstream according to the fourth Gaussian distribution parameter information to obtain a third residual map, and the third residual map is the residual map of the feature domain.

[0480] The processing unit 2002 is further configured to determine a reconstructed image according to the third residual map.

[0481] In the case of dividing each functional module according to each function, FIG. 21 shows another possible composition diagram of the decoding apparatus involved in the above embodiment, as shown in FIG. 21, the decoding apparatus 2100 can include a transceiver unit 2101 and a processing unit 2102.

[0482] The transceiver unit 2101 is configured to obtain a bitstream.

[0483] The processing unit 2102 is configured to determine a target quality matrix according to the bitstream, and the target quality matrix is used to represent image quality of each region in the residual map of the feature domain.

[0484] The processing unit 2102 is further configured to decode the bitstream according to the third Gaussian distribution parameter information to obtain a third residual map, and the third Gaussian distribution parameter information is Gaussian distribution parameter information of the residual map of the feature domain, and the third residual map is the residual map of the feature domain.

[0485] The processing unit 2102 is further configured to dequantize the third residual map according to the target quality matrix to obtain a fourth residual map.

[0486] The processing unit 2102 is further configured to determine a reconstructed image according to the fourth residual map.

[0487] Embodiments of the present application further provide an encoding apparatus, which comprises at least one processor, when the at least one processor executes program code or instructions, the above-mentioned related method steps are realized to realize the encoding method in the above-mentioned embodiments.

[0488] Optionally, the apparatus can further comprise at least one memory for storing the program code or instructions.

[0489] The embodiment of the present application further provides a decoding device, which comprises at least one processor, and when the at least one processor executes program codes or instructions, the decoding method in the above embodiment is realized.

[0490] Optionally, the device further comprises at least one memory for storing the program codes or instructions.

[0491] The embodiment of the present application further provides a code stream, which comprises a target quality matrix, a third residual image and third Gaussian distribution parameters, wherein the target quality matrix is used to represent the image quality of each region in a residual image of a feature domain, the third residual image is the residual image of the feature domain, and the third Gaussian distribution parameter information is the Gaussian distribution parameter information of the residual image of the feature domain; the target quality matrix is used to scale the third Gaussian distribution parameter information to obtain fourth Gaussian distribution parameter information; and the target quality matrix is also used to perform inverse quantization on the third residual image to obtain a fourth residual image.

[0492] The embodiment of the present application further provides a computer storage medium, which stores computer instructions, and when the computer instructions run on an encoding device, the encoding device executes the steps of the related method to realize the encoding and decoding method in the above embodiment.

[0493] The embodiment of the present application further provides a computer program product, and when the computer program product runs on a computer, the computer executes the related steps to realize the encoding and decoding method in the above embodiment.

[0494] The embodiment of the present application further provides an encoding and decoding device, which can be a chip, an integrated circuit, an assembly or a module. Specifically, the device can comprise a processor and a memory for storing instructions, or the device comprises at least one processor for obtaining instructions from an external memory. When the device runs, the processor can execute the instructions to make the chip execute the encoding and decoding method in the above method embodiments.

[0495] FIG. 22 shows a structural schematic diagram of a chip 2300. The chip 2200 comprises one or more processors 2201 and interface circuits 2202. Optionally, the chip 2200 can further comprise a bus 2203.

[0496] The processor 2201 can be an integrated circuit chip with a signal processing capability. In the implementation process, the steps of the encoding and decoding method can be completed by integrated logic circuits or instructions in the form of software in the processor 2201.

[0497] Optionally, the processor 2201 can be a general processor, a digital signal processing (DSP) processor, an application specific integrated circuit (ASIC), a field-programmable gate array (FPGA) or other programmable logic device, a discrete gate or transistor logic component, a discrete hardware component. The methods and steps disclosed in the embodiments of the present application can be implemented or executed. The general processor can be a microprocessor or the processor can be any conventional processor.

[0498] The interface circuit 2202 can be used for sending or receiving data, instructions or information. The processor 2201 can process the data, instructions or other information received by the interface circuit 2202, and can send the processed information out through the interface circuit 2202.

[0499] Optionally, the chip further includes a memory, which can include a read-only memory and a random access memory, and provide operation instructions and data for the processor. Part of the memory can also include a non-volatile random access memory (NVRAM).

[0500] Optionally, the memory stores executable software modules or data structures, and the processor can execute corresponding operations by calling operation instructions stored in the memory (which can be stored in an operating system).

[0501] Optionally, the chip can be used in the encoding device or DOP related to the embodiments of the present application. Optionally, the interface circuit 2202 can be used to output the execution result of the processor 2201. The encoding and decoding method provided by one or more embodiments of the present application can refer to the foregoing embodiments, which will not be described here.

[0502] It should be noted that the functions of the processor 2201 and the interface circuit 2202 respectively can be realized by hardware design, software design or combination of software and hardware, which is not limited here.

[0503] Referring to FIG. 23, FIG. 23 schematically illustrates the general concept of processing by a neural network such as a convolutional neural network (CNN). A convolutional neural network is composed of an input layer and an output layer and multiple hidden layers. The input layer is the layer that provides the input (such as a portion of an input image as shown in FIG. 23) for processing. The hidden layers of a CNN are typically composed of a series of convolutional layers that convolve with a multiplication or other dot product. The result of a layer is one or more feature maps (represented by the empty solid-line rectangles), sometimes also called channels. Resampling (such as subsampling) can be involved in some or all of the layers. Thus, the feature maps can become smaller, as shown in FIG. 23. Note that convolution with a stride can also reduce the size of the input feature maps (resampling). The activation function in a CNN is typically a ReLU (rectified linear unit) layer, followed by additional convolutions such as pooling layers, fully connected layers, and normalization layers, called hidden layers because their inputs and outputs are masked by activation functions and final convolutions. Although these layers are colloquially called convolutions, this is just a convention. Mathematically, it is technically a sliding dot product or cross-correlation. This has important implications for the indexing in the matrices, as it affects how the weights are determined at a particular index point.

[0504] When programming a CNN for processing images, as shown in FIG. 23, the input is a tensor of shape x (number of images) x (image width) x (image height) x (image depth). It should be known that the image depth can be composed of the channels of the image. After passing through a convolutional layer, the image is abstracted into a feature map with shape (number of images) x (feature map width) x (feature map height) x (feature map channels). A convolutional layer in a neural network should have the following properties. A convolution kernel defined by width and height (hyperparameters). The number of input channels and output channels (hyperparameters). The depth of the convolutional filter (input channels) should be equal to the number of channels (depth) of the input feature map.

[0505] Machine video coding (VCM) is another popular direction in computer science today. The main idea behind this approach is to transmit an encoded representation of the image or video information for further processing by computer vision (CV) algorithms such as object segmentation, detection, and recognition. In contrast to traditional image and video coding, which targets human perception, the quality feature is the performance of the computer vision task, e.g., object detection accuracy, rather than the reconstruction quality. As shown in FIG. 24.

[0506] Mobile cloud infrastructure. By splitting the network between the mobile side 2410 and the cloud side 2490 (e.g., cloud servers), the computational workload can be distributed such that the total energy and / or latency of the system is minimized. In general, cooperative intelligence is a paradigm in which the processing of a neural network is distributed among two or more different computing nodes; e.g., devices, but in general, any functionally defined node. Here, the term “node” does not refer to the above-mentioned neural network nodes. Rather, a (computing) node here refers to a (physically or at least logically) independent device / module, which implements a part of the neural network. Such devices can be different servers, different end-user devices, a mix of servers and / or user devices and / or clouds and / or processors, etc. In other words, the computing nodes can be considered to be nodes of the same neural network and communicate with each other to communicate encoded data within / for the neural network. For example, to be able to perform complex computations, one or more layers can be executed on a first device, such as a device on the mobile side 2410, and one or more layers can be executed in another device, such as a cloud server on the cloud side 2490. However, the distribution can also be finer and a single layer can be executed on multiple devices. In this disclosure, the term “multiple” refers to two or more. In some existing solutions, a part of the neural network functionality is executed in a device (user device or edge device, etc.) or multiple such devices, and then the output (feature map) is passed to the cloud. The cloud is a collection of processing or computing systems that are located outside the device that is operating a part of the neural network. The concept of cooperative intelligence is also extended to model training. In this case, data flows bi-directionally: from the cloud to the mobile device in the backward pass of the training, and from the mobile device to the cloud in the forward pass of the training and inference (as illustrated in FIG. 24).

[0507] Some works propose semantic image compression by encoding deep features and then reconstructing the input image from them. Uniform quantization based compression is demonstrated, followed by context-based adaptive arithmetic coding (CABAC) from H.264. In some scenarios, it can be more efficient to send the output of the hidden layers (deep feature maps) from the mobile component 2410 to the cloud 2490, rather than sending compressed natural image data to the cloud and performing object detection using the reconstructed image. Thus, it can be advantageous to compress the data (features) generated by the mobile side 2410, which can include a quantization layer 2420 for this purpose. Correspondingly, the cloud side 2490 can include an inverse quantization layer 2460. Efficient compression of feature maps is advantageous for image and video compression and reconstruction for both human perception and machine vision. Entropy coding methods, such as arithmetic coding, are a popular method for compressing deep features, i.e., feature maps.

[0508] Referring to FIG. 25, FIG. 25 shows a bitstream structure provided by an embodiment of the present application. As shown in FIG. 25, the bitstream includes a start of image (Start of Image), a file header (File Header), entropy encoded data (Entropy Encoded Data) and an end of image (End of Image).

[0509] In a possible implementation, the above-mentioned Gaussian distribution parameter index number, the above-mentioned Gaussian distribution parameter, the target probability distribution and the target probability distribution index number can be stored in the file header.

[0510] In a possible implementation, the above-mentioned target quality residual matrix or the target quality matrix can be stored in the entropy encoded data.

[0511] The apparatus, the computer storage medium, the computer program product or the chip provided by the embodiments of the present application are all used for executing the corresponding method provided above, and thus the beneficial effects achieved by the apparatus, the computer storage medium, the computer program product or the chip can refer to the beneficial effects of the corresponding method provided above, which will not be repeated here.

[0512] It should be understood that, in various embodiments of the embodiments of the present application, the size of the serial number of each process does not mean the order of execution, and the execution order of each process should be determined according to its function and inherent logic, and should not constitute any limitation on the implementation process of the embodiments of the present application.

[0513] Those skilled in the art can appreciate that the units and algorithm steps of the examples described in combination with the embodiments disclosed herein can be realized by electronic hardware or a combination of computer software and electronic hardware. Whether the functions are realized in hardware or software depends on the specific application and design constraints of the technical solution. A person skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of the embodiments of the present application.

[0514] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working process of the above-mentioned system, device and unit can refer to the corresponding process in the foregoing method embodiments, which will not be repeated here.

[0515] In several embodiments provided by the embodiments of the present application, it should be understood that the disclosed system, device and method can be implemented in other manners. For example, the embodiments of the device described above are merely schematic. For example, the division of the units is merely logical function division. There can be another division manner for the actual implementation. For example, a plurality of units or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the displayed or discussed mutual couplings or direct couplings or communication connections can be indirect couplings or communication connections through some interfaces, devices or units, and can be in electrical, mechanical or other forms.

[0516] The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they can be located in one place, or can be distributed on a plurality of network units. Some or all of the units can be selected according to actual needs to achieve the purpose of the embodiments of the present application.

[0517] In addition, each functional unit in each embodiment of the present application can be integrated into a processing unit, or each unit can exist physically, or two or more units can be integrated into one unit.

[0518] If the above functions are realized in the form of software function units and sold or used as independent products, they can be stored in a computer readable storage medium. Based on this understanding, the technical solutions of the embodiments of the present application essentially or the parts that make contributions to the prior art or parts of the technical solutions can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes several instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods described above in the various embodiments of the present application. The foregoing storage medium includes: U disk, mobile hard disk, read-only memory (ROM), random access memory (RAM), magnetic disk or optical disk, and various media that can store program codes.

[0519] The above description is merely specific embodiments of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art can easily think of changes or replacements within the technical range disclosed by the embodiments of the present application, which should be covered within the protection scope of the embodiments of the present application. Therefore, the protection scope of the embodiments of the present application should be subject to the protection scope of the claims.

Claims

An encoding method characterized by comprising: The method comprises: obtaining a target quality matrix, the target quality matrix being used to represent image quality of each region in a residual image of a feature field; scaling a first residual image and / or first Gaussian distribution parameter information according to the target quality matrix to obtain a second residual image and / or second Gaussian distribution parameter information, the first residual image being a residual image of the feature field, and the first Gaussian distribution parameter information being Gaussian distribution parameter information of the residual image of the feature field; generating a bitstream according to the target quality matrix, the second residual image and / or the second Gaussian distribution parameter information. The method of claim 1, wherein The scaling of the first residual image and the first Gaussian distribution parameter information according to the target quality matrix to obtain the second residual image and the second Gaussian distribution parameter information comprises: determining a target quality scaling matrix according to the target quality matrix, the target quality scaling matrix being used to represent a scaling amount of each region in the residual image of the feature field and / or the Gaussian distribution parameter information; determining a target quality scaling tensor according to the target quality scaling matrix, the target quality scaling tensor being used to represent a scaling tensor of the residual image of the feature field and / or the Gaussian distribution parameter information in a three-dimensional space; scaling the first residual image according to the target quality scaling tensor to obtain the second residual image and / or scaling the first Gaussian distribution parameter information according to the target quality scaling tensor to obtain the second Gaussian distribution parameter information. The method according to claim 2, characterized in that The determining of the target quality scaling tensor according to the target quality scaling matrix comprises: determining the target quality scaling tensor according to the target quality scaling matrix and a gain parameter, the gain parameter comprising a gain vector of a channel-level quality adjustment and / or an image-level quality control factor. The encoding method according to any one of claims 1 to 3, characterized in that, The generating of the bitstream according to the target quality matrix, the second residual image and / or the second Gaussian distribution parameter information comprises: encoding the target quality matrix and the second residual image to generate the bitstream, the second residual image being encoded according to the second Gaussian distribution parameter information. The method according to claim 4, characterized in that The encoding of the target quality matrix to generate the bitstream comprises: encoding the target quality matrix or a target quality residual matrix of the target quality matrix to generate the bitstream, the target quality residual matrix being used to represent a residual value of each image quality in the target quality matrix. The method according to claim 5, characterized in that The method further comprises: generating a target quality residual matrix of the target quality matrix according to the target quality matrix. The method according to claim 5 or 6, characterized in that The encoding of the target quality residual matrix of the target quality matrix to generate the bitstream comprises: determining a Gaussian distribution parameter of the target quality residual matrix; determining a probability distribution of the target quality residual matrix according to the Gaussian distribution parameter; writing the Gaussian distribution parameter into the bitstream; and entropy-encoding the target quality residual matrix according to the probability distribution. The method according to claim 5 or 6, characterized in that The encoding of the target quality matrix to generate the bitstream comprises: determining a Gaussian distribution parameter of the target quality matrix; determining a probability distribution of the target quality matrix according to the Gaussian distribution parameter; writing the Gaussian distribution parameter into the bitstream; and entropy-encoding the target quality matrix according to the probability distribution. The method according to claim 5 or 6, characterized in that The encoding of the target quality residual matrix of the target quality matrix generates a code stream, comprising: determining a target probability distribution from a plurality of candidate probability distributions according to the target quality residual matrix; writing an index number of the target probability distribution into the code stream; entropy encoding the target quality residual matrix according to the target probability distribution. The method according to claim 5 or 6, characterized in that The encoding of the target quality matrix generates a code stream, comprising: determining a target probability distribution from a plurality of candidate probability distributions according to the target quality matrix; writing an index number of the target probability distribution into the code stream; entropy encoding the target quality matrix according to the target probability distribution. The method according to any one of claims 1 to 10, characterized in that The target quality matrix comprises: obtaining a quality map, the quality map being used to record image quality of each region in the residual map of the feature domain; determining the target quality matrix according to the quality map. The method according to any one of claims 1 to 11, characterized in that The method further comprises: inputting the image into an encoding network to obtain a first feature map; inputting the first feature map into a context network to obtain a first prediction map; determining the first residual map according to the first feature map and the first prediction map; inputting the first feature map into a hyper-prior encoding network to obtain first hyper-prior information; quantizing the first hyper-prior information to obtain second hyper-prior information; inputting the second hyper-prior information into a hyper-prior variance decoding network to obtain the first Gaussian distribution parameter information. A decoding method characterized by comprising: comprising: obtaining a code stream; determining a target quality matrix according to the code stream, the target quality matrix being used to represent image quality of each region in a residual map of a feature domain; scaling third Gaussian distribution parameter information according to the target quality matrix to obtain fourth Gaussian distribution parameter information, the third Gaussian distribution parameter information being Gaussian distribution parameter information of the residual map of the feature domain; decoding the code stream according to the fourth Gaussian distribution parameter information to obtain a third residual map, the third residual map being the residual map of the feature domain; dequantizing the third residual map according to the target quality matrix to obtain a fourth residual map; determining a reconstructed image according to the fourth residual map. The method of claim 13, wherein The scaling of the third Gaussian distribution parameter information according to the target quality matrix to obtain the fourth Gaussian distribution parameter information comprises: determining a target quality scaling matrix according to the target quality matrix, the target quality scaling matrix being used to represent a scaling amount of each region in the residual map of the feature domain and the Gaussian distribution parameter information; determining a target quality scaling tensor according to the target quality scaling matrix, the target quality scaling tensor being used to represent a scaling tensor of the residual map of the feature domain and the Gaussian distribution parameter information in a three-dimensional space; scaling the third Gaussian distribution parameter information according to the target quality scaling tensor to obtain the fourth Gaussian distribution parameter information. The method of claim 14, wherein The determination of the target quality scaling matrix according to the target quality matrix comprises: determining the target quality scaling tensor according to the target quality scaling matrix and a gain parameter, the gain parameter comprising a gain vector of a channel-level quality adjustment and / or an image-level quality control factor. The method according to any one of claims 13 to 15, characterized in that The determination of the target quality matrix according to the code stream comprises: decoding the code stream to obtain a Gaussian distribution parameter of a target quality residual matrix of the target quality matrix; determining a probability distribution of the target quality residual matrix according to the Gaussian distribution parameter; decoding the code stream according to the probability distribution to obtain the target quality residual matrix; determining the target quality matrix according to the target quality residual matrix. The method according to any one of claims 13 to 15, characterized in that The determining the target quality matrix according to the code stream comprises: decoding the code stream to obtain a Gaussian distribution parameter of a target quality residual matrix of the target quality matrix; determining a probability distribution of the target quality matrix according to the Gaussian distribution parameter; decoding the code stream according to the probability distribution to obtain the target quality matrix. The method according to any one of claims 13 to 15, characterized in that The determining the target quality matrix according to the code stream comprises: decoding the code stream to obtain an index number of a target probability distribution, the target probability distribution being determined from a plurality of candidate probability distributions according to a target quality residual matrix of the target quality matrix; decoding the code stream according to the target probability distribution to obtain the target quality residual matrix; determining the target quality matrix according to the target quality residual matrix. The method according to any one of claims 13 to 15, wherein the determining the target quality matrix according to the code stream comprises: decoding the code stream to obtain an index number of a target probability distribution, the target probability distribution being determined from a plurality of candidate probability distributions according to the target quality matrix; decoding the code stream according to the target probability distribution to obtain the target quality matrix. The method according to any one of claims 13 to 19, characterized in that The method further comprises: decoding the code stream to obtain a second feature map; inputting the second feature map into a hyper-prior variance decoding network to obtain the third Gaussian distribution parameter information. A decoding method characterized by comprising: comprises: obtaining a code stream; determining a target quality matrix according to the code stream, the target quality matrix being used to represent image quality of each region in a residual map of a feature domain; scaling third Gaussian distribution parameter information according to the target quality matrix to obtain fourth Gaussian distribution parameter information, the third Gaussian distribution parameter information being Gaussian distribution parameter information of the residual map of the feature domain; decoding the code stream according to the fourth Gaussian distribution parameter information to obtain a third residual map, the third residual map being the residual map of the feature domain; determining a reconstructed image according to the third residual map. A decoding method characterized by comprising: comprises: obtaining a code stream; determining a target quality matrix according to the code stream, the target quality matrix being used to represent image quality of each region in a residual map of a feature domain; decoding the code stream according to third Gaussian distribution parameter information to obtain a third residual map, the third Gaussian distribution parameter information being Gaussian distribution parameter information of the residual map of the feature domain, the third residual map being the residual map of the feature domain; dequantizing the third residual map according to the target quality matrix to obtain a fourth residual map; determining a reconstructed image according to the fourth residual map. An encoding device characterized by comprising: comprises: a transceiver and a processing unit; the transceiver is configured to obtain a target quality matrix, the target quality matrix being used to represent image quality of each region in a residual map of a feature domain; The processing unit is configured to scale the first residual map and / or the first Gaussian distribution parameter information according to the target quality matrix to obtain a second residual map and / or a second Gaussian distribution parameter information, the first residual map being a residual map of the feature domain, and the first Gaussian distribution parameter information being Gaussian distribution parameter information of the residual map of the feature domain. The processing unit is further configured to generate a bitstream according to the target quality matrix, the second residual map, and / or the second Gaussian distribution parameter information. The apparatus of claim 23, wherein The processing unit is specifically configured to: determine a target quality scaling matrix according to the target quality matrix, the target quality scaling matrix being used to represent a scaling amount of each region in the residual map of the feature domain and / or the Gaussian distribution parameter information; determine a target quality scaling tensor according to the target quality scaling matrix, the target quality scaling tensor being used to represent a scaling tensor of the residual map of the feature domain and / or the Gaussian distribution parameter information in a three-dimensional space; scale the first residual map according to the target quality scaling tensor to obtain the second residual map and / or scale the first Gaussian distribution parameter information according to the target quality scaling tensor to obtain the second Gaussian distribution parameter information. The apparatus of claim 24, wherein The processing unit is specifically configured to: determine the target quality scaling tensor according to the target quality scaling matrix and a gain parameter, the gain parameter including a gain vector of a channel-level quality adjustment and / or an image-level quality control factor. The apparatus of any one of claims 23 to 25, wherein The processing unit is specifically configured to: encode the target quality matrix and a second residual map to generate a bitstream, the second residual map being encoded according to the second Gaussian distribution parameter information. The apparatus of claim 26, wherein The processing unit is specifically configured to: encode the target quality matrix or a target quality residual matrix of the target quality matrix to generate a bitstream. The apparatus of claim 27, wherein The processing unit is further configured to: generate a target quality residual matrix of the target quality matrix according to the target quality matrix, the target quality residual matrix being used to represent a residual value of each image quality in the target quality matrix. The apparatus of claim 27 or 28, wherein The processing unit is specifically configured to: determine a Gaussian distribution parameter of the target quality residual matrix; determine a probability distribution of the target quality residual matrix according to the Gaussian distribution parameter; write the Gaussian distribution parameter into the bitstream; entropy encode the target quality residual matrix according to the probability distribution. The apparatus of claim 27 or 28, wherein The processing unit is specifically configured to: determine a Gaussian distribution parameter of the target quality matrix; determine a probability distribution of the target quality matrix according to the Gaussian distribution parameter; write the Gaussian distribution parameter into the bitstream; entropy encode the target quality matrix according to the probability distribution. The apparatus of claim 27 or 28, wherein The processing unit is specifically configured to: determine a target probability distribution from a plurality of candidate probability distributions according to the target quality residual matrix; write an index number of the target probability distribution into the bitstream; entropy encode the target quality residual matrix according to the target probability distribution. The apparatus of claim 27 or 28, wherein The processing unit is specifically configured to: determine a target probability distribution from a plurality of candidate probability distributions according to the target quality matrix; write an index number of the target probability distribution into the bitstream; Entropy encode the target quality matrix according to the target probability distribution. The apparatus according to any one of claims 23 to 32, characterized in that The obtaining unit is specifically configured to: obtain a quality map, the quality map being used to record image quality of each region in the residual map of the feature domain; determine the target quality matrix according to the quality map. The apparatus according to any one of claims 23 to 33, characterized in that The processing unit is further configured to: input the image into an encoding network to obtain a first feature map; input the first feature map into a context network to obtain a first prediction map; determine the first residual map according to the first feature map and the first prediction map; input the first feature map into a hyper-prior encoding network to obtain first hyper-prior information; quantize the first hyper-prior information to obtain second hyper-prior information; input the second hyper-prior information into a hyper-prior variance decoding network to obtain the first Gaussian distribution parameter information. A decoding device characterized by comprising: comprise: a transceiving unit and a processing unit; the transceiving unit is configured to obtain a bitstream; the processing unit is configured to determine a target quality matrix according to the bitstream, the target quality matrix being used to represent image quality of each region in a residual map of a feature domain; the processing unit is further configured to scale third Gaussian distribution parameter information according to the target quality matrix to obtain fourth Gaussian distribution parameter information, the third Gaussian distribution parameter information being Gaussian distribution parameter information of the residual map of the feature domain; the processing unit is further configured to decode the bitstream according to the fourth Gaussian distribution parameter information to obtain a third residual map, the third residual map being the residual map of the feature domain; the processing unit is further configured to dequantize the third residual map according to the target quality matrix to obtain a fourth residual map; the processing unit is further configured to determine a reconstructed image according to the fourth residual map. The apparatus of claim 35, wherein The processing unit is specifically configured to: determine a target quality scaling matrix according to the target quality matrix, the target quality scaling matrix being used to represent a scaling amount of each region in the residual map of the feature domain and the Gaussian distribution parameter information; determine a target quality scaling tensor according to the target quality scaling matrix, the target quality scaling tensor being used to represent a scaling tensor of the residual map of the feature domain and the Gaussian distribution parameter information in a three-dimensional space; scale the third Gaussian distribution parameter information according to the target quality scaling tensor to obtain the fourth Gaussian distribution parameter information. The apparatus of claim 36, wherein The processing unit is specifically configured to: determine the target quality scaling tensor according to the target quality scaling matrix and a gain parameter, the gain parameter comprising a gain vector of a channel-level quality adjustment and / or an image-level quality control factor. The apparatus of any one of claims 35 to 37, wherein The processing unit is specifically configured to: decode the bitstream to obtain Gaussian distribution parameters of a target quality residual matrix of the target quality matrix; determine a probability distribution of the target quality residual matrix according to the Gaussian distribution parameters; decode the bitstream according to the probability distribution to obtain the target quality residual matrix; determine the target quality matrix according to the target quality residual matrix. The apparatus of any one of claims 35 to 37, wherein The processing unit is specifically configured to: decode the bitstream to obtain Gaussian distribution parameters of a target quality matrix; determine a probability distribution of the target quality matrix according to the Gaussian distribution parameters; decode the code stream according to the target probability distribution to obtain the target quality matrix. The apparatus of any one of claims 35 to 37, wherein The processing unit is specifically configured to: decode the code stream to obtain an index number of a target probability distribution, the target probability distribution being determined from a plurality of candidate probability distributions according to a target quality residual matrix of the target quality matrix; decode the code stream according to the target probability distribution to obtain the target quality residual matrix; determine the target quality matrix according to the target quality residual matrix. The apparatus of any one of claims 35 to 37, wherein The processing unit is specifically configured to: decode the code stream to obtain an index number of a target probability distribution, the target probability distribution being determined from a plurality of candidate probability distributions according to a target quality matrix; decode the code stream according to the target probability distribution to obtain the target quality matrix. The apparatus of any one of claims 35 to 41, wherein The processing unit is further configured to: decode the code stream to obtain a second feature map; input the second feature map into a hyper-prior variance decoding network to obtain the third Gaussian distribution parameter information. A decoding device characterized by comprising: comprise: a transceiver and a processing unit; the transceiver is configured to acquire a code stream; the processing unit is configured to determine a target quality matrix according to the code stream, the target quality matrix being used to represent image quality of each region in a residual map of a feature domain; the processing unit is further configured to scale third Gaussian distribution parameter information according to the target quality matrix to obtain fourth Gaussian distribution parameter information, the third Gaussian distribution parameter information being Gaussian distribution parameter information of the residual map of the feature domain; the processing unit is further configured to decode the code stream according to the fourth Gaussian distribution parameter information to obtain a third residual map, the third residual map being the residual map of the feature domain; the processing unit is further configured to determine a reconstructed image according to the third residual map. A decoding device characterized by comprising: comprise a transceiver and a processing unit; the transceiver is configured to acquire a code stream; the processing unit is configured to determine a target quality matrix according to the code stream, the target quality matrix being used to represent image quality of each region in a residual map of a feature domain; the processing unit is further configured to decode the code stream according to third Gaussian distribution parameter information to obtain a third residual map, the third Gaussian distribution parameter information being Gaussian distribution parameter information of the residual map of the feature domain, the third residual map being the residual map of the feature domain; the processing unit is further configured to dequantize the third residual map according to the target quality matrix to obtain a fourth residual map; the processing unit is further configured to determine a reconstructed image according to the fourth residual map. A bitstream, characterized in that comprise: a target quality matrix, a third residual map and third Gaussian distribution parameter information, the target quality matrix being used to represent image quality of each region in a residual map of a feature domain, the third residual map being the residual map of the feature domain, the third Gaussian distribution parameter information being Gaussian distribution parameter information of the residual map of the feature domain; the target quality matrix is configured to scale the third Gaussian distribution parameter information to obtain fourth Gaussian distribution parameter information; the target quality matrix is further configured to dequantize the third residual map to obtain a fourth residual map. An encoding device includes at least one processor and a memory, wherein the memory stores instructions that, when executed by the at least one processor, cause the encoding device to: The at least one processor executes programs or instructions stored in the memory, so that the encoding device implements the method in any one of claims 1-12. A decoding apparatus comprising at least one processor and a memory, characterized in that The at least one processor executes programs or instructions stored in the memory, so that the decoding device implements the method in any one of claims 13-22. A computer-readable storage medium, characterized by, The computer readable storage medium stores a video bitstream obtained by executing the method in any one of claims 1-12 by one or more processors. A computer-readable storage medium for storing a computer program, characterized in that When the computer program runs on the computer or the processor, the computer or the processor implements the method in any one of claims 1-12 or any one of claims 13-22. When the instructions run on the computer or the processor, the computer or the processor implements the method in any one of claims 1-12 or any one of claims 13-22. A computer program product comprising instructions therein, characterized in that ​

Images