Coding and decoding method and device

Abstract

The embodiment of the invention discloses a coding and decoding method and device, relates to the technical field of media, and can perform spatial dimension quality adjustment on image content through JPEGAI. The method comprises the following steps: acquiring a target quality matrix; and scaling the first residual image and / or the first Gaussian distribution parameter information according to the target quality matrix to obtain a second residual image and / or second Gaussian distribution parameter information. And generating a code stream according to the target quality matrix, the second residual image and / or the second Gaussian distribution parameter information. Wherein the target quality matrix is used for representing image quality of each region in a residual image of a feature domain, the first residual image is the residual image of the feature domain, and the first Gaussian distribution parameter information is Gaussian distribution parameter information of the residual image of the feature domain.

Description

Technical Field

[0001] The embodiments of the present application relate to the field of media technology, and in particular to a coding and decoding method and device. Background Art

[0002] The Joint Photographic Experts Group (JPEG) artificial intelligence (AI) is a learning-based image coding standard that provides a single-stream, compact compressed domain representation, which significantly improves compression efficiency over commonly used image coding standards at the same subjective quality. JPEG AI is widely used in various fields. For example, JPEG AI can be applied to cloud storage, visual surveillance, autonomous vehicles and equipment, image acquisition storage and management, real-time monitoring of visual data, and media distribution.

[0003] In actual applications of video image coding, it is usually necessary to allocate different bit rates to the region of interest and the background region for coding, so as to meet customer needs with a lower bit rate.

[0004] Therefore, how to adjust the quality of image content in spatial dimension through JPEG AI is one of the problems that technicians in this field need to solve urgently. Summary of the invention

[0005] The embodiment of the present application provides a coding method and a device, which can adjust the quality of the image content in the spatial dimension through JPEG AI. To achieve the above purpose, the embodiment of the present application adopts the following technical solutions:

[0006] In a first aspect, an embodiment of the present application provides a coding method, the method comprising: obtaining a target quality matrix. Scaling a first residual map and / or first Gaussian distribution parameter information according to the target quality matrix to obtain a second residual map and / or second Gaussian distribution parameter information. Generate a code stream according to the target quality matrix, the second residual map and / or the second Gaussian distribution parameter information. The target quality matrix is ​​used to characterize the image quality of each area in the residual map of the feature domain, the first residual map is the residual map of the feature domain, and the first Gaussian distribution parameter information is the Gaussian distribution parameter information of the residual map of the feature domain.

[0007] It can be seen that the method provided in the embodiment of the present application additionally introduces a target quality matrix that characterizes the image quality of each area in the JPEG AI encoding process. Since different positions in the target quality matrix can have different image quality values, encoding using the target quality matrix can achieve bit rate allocation for different areas of the image space, thereby allowing JPEG AI to adjust the quality of the image content in the spatial dimension.

[0008] In a possible implementation, a target mass scaling matrix can be determined according to the target mass matrix, and the target mass scaling matrix is ​​used to characterize the scaling amount of each region in the residual map of the feature domain and / or the Gaussian distribution parameter information. A target mass scaling tensor is determined according to the target mass scaling matrix, and the target mass scaling tensor is used to characterize the scaling tensor of the residual map of the feature domain and / or the Gaussian distribution parameter information in three-dimensional space. The first residual map is scaled according to the target mass scaling tensor to obtain the second residual map and / or the first Gaussian distribution parameter information is scaled according to the target mass scaling tensor to obtain the second Gaussian distribution parameter information.

[0009] It can be seen that the method provided in the embodiment of the present application additionally introduces a target quality matrix that characterizes the image quality of each area in the JPEG AI encoding process, and determines the target quality scaling tensor through the target quality matrix. Since different positions in the target quality matrix can have different image quality values, encoding using the target quality scaling tensor can achieve code rate allocation for different areas of the image space, thereby allowing JPEG AI to adjust the quality of the image content in the spatial dimension.

[0010] In a possible implementation, the target quality scaling tensor may be determined according to the target quality scaling matrix and a gain parameter, where the gain parameter includes a gain vector for channel-level quality adjustment and / or an image-level quality control factor.

[0011] It can be seen that the method provided in the embodiment of the present application additionally introduces a target quality matrix that characterizes the image quality of each area in the JPEG AI encoding process, determines the target quality scaling through the target quality matrix, and then determines the above-mentioned target quality scaling tensor based on the above-mentioned target quality scaling matrix and the gain parameter. Since different positions in the target quality matrix can have different image quality values, encoding using the target quality scaling tensor can achieve code rate allocation for different areas of the image space, thereby allowing JPEG AI to adjust the quality of the image content in the spatial dimension.

[0012] In a possible implementation manner, the target quality matrix and the second residual map may be encoded to generate a bitstream, and the second residual map is encoded according to the second Gaussian distribution parameter information.

[0013] It can be seen that the method provided in the embodiment of the present application can generate a bitstream by encoding the target quality matrix and the second residual map. Since different positions in the target quality matrix can have different image quality values, encoding using the target quality scaling tensor can achieve bitrate allocation in different areas of the image space, thereby allowing JPEG AI to adjust the quality of the image content in the spatial dimension.

[0014] In a possible implementation manner, the target quality matrix or the target quality residual matrix of the target quality matrix may be encoded to generate a bitstream.

[0015] It can be seen that the method provided in the embodiment of the present application encodes the above-mentioned target quality matrix or the target quality residual matrix of the above-mentioned target quality matrix to generate a bitstream. Since different positions in the target quality matrix can have different image quality values, encoding using the target quality scaling tensor can achieve code rate allocation in different areas of the image space, thereby allowing JPEG AI to adjust the quality of the image content in the spatial dimension.

[0016] In a possible implementation manner, a target quality residual matrix of the target quality matrix may be generated according to the target quality matrix, and the target quality residual matrix is ​​used to represent the residual value of each image quality in the target quality matrix.

[0017] It is understandable that the residual value has a smaller data volume than the quality value, so the target quality residual matrix of the above target quality matrix is ​​generated through the target quality matrix, and then encoded through the target quality residual matrix, which can reduce the complexity of generating the bitstream.

[0018] In a possible implementation, Gaussian distribution parameters of the target quality residual matrix may be determined; the probability distribution of the target quality residual matrix may be determined based on the Gaussian distribution parameters; the Gaussian distribution parameters may be written into the bitstream; and entropy encoding may be performed on the target quality residual matrix based on the probability distribution.

[0019] It can be seen that the embodiment of the present application can analyze the target quality residual matrix to obtain the Gaussian distribution parameters of the target quality residual matrix, determine the probability distribution of the target quality residual matrix through the Gaussian distribution parameters, and then encode the target quality residual matrix into the bitstream through the probability distribution of the target quality residual matrix. The target quality matrix can be determined through the target quality residual matrix. Since different positions in the target quality matrix can have different image quality values, encoding using the target quality scaling tensor can achieve code rate allocation for different areas of the image space, thereby allowing JPEG AI to adjust the quality of the image content in the spatial dimension.

[0020] In a possible implementation, Gaussian distribution parameters of the target quality matrix may be determined, probability distribution of the target quality matrix may be determined according to the Gaussian distribution parameters, the Gaussian distribution parameters may be written into the bitstream, and entropy coding may be performed on the target quality matrix according to the probability distribution.

[0021] It can be seen that the embodiment of the present application can analyze the target quality matrix to obtain the Gaussian distribution parameters of the target quality matrix, determine the probability distribution of the target quality matrix through the Gaussian distribution parameters, and then encode the target quality residual matrix into the bitstream through the probability distribution of the target quality matrix. Since different positions in the target quality matrix can have different image quality values, encoding using the target quality scaling tensor can achieve bit rate allocation in different areas of the image space, thereby allowing JPEG AI to adjust the quality of the image content in the spatial dimension.

[0022] In a possible implementation, a target probability distribution may be determined from a plurality of candidate probability distributions according to the target quality residual matrix; the index number of the target probability distribution may be written into the bitstream; and the target quality residual matrix may be entropy encoded according to the target probability distribution.

[0023] It can be seen that the embodiment of the present application can analyze the target quality residual matrix to obtain a target probability distribution that matches the target quality residual matrix from multiple candidate probability distributions, thereby encoding the target quality residual matrix into the bitstream through the target probability distribution. The target quality matrix can be determined through the target quality residual matrix. Since different positions in the target quality matrix can have different image quality values, encoding using the target quality scaling tensor can achieve code rate allocation for different areas of the image space, thereby allowing JPEG AI to adjust the quality of the image content in the spatial dimension.

[0024] In a possible implementation, a target probability distribution may be determined from a plurality of candidate probability distributions according to the target quality matrix, an index number of the target probability distribution may be written into the bitstream, and entropy coding may be performed on the target quality matrix according to the target probability distribution.

[0025] It can be seen that the embodiments of the present application can analyze the target quality matrix to obtain a target probability distribution that matches the target quality matrix from multiple candidate probability distributions, and then encode the target quality matrix into the bitstream through the target probability distribution. Since different positions in the target quality matrix can have different image quality values, encoding using the target quality scaling tensor can achieve bit rate allocation for different areas of the image space, thereby allowing JPEG AI to adjust the quality of the image content in the spatial dimension.

[0026] In a possible implementation, a quality map may be obtained, the quality map being used to record the image quality of each region in the residual map of the feature domain, and the target quality matrix may be determined according to the quality map.

[0027] It can be seen that in the method provided in the embodiment of the present application, the above-mentioned target quality matrix can be determined through the quality map. Since different positions in the target quality matrix can have different image quality values, the use of the target quality scaling tensor for encoding can achieve bit rate allocation in different areas of the image space, thereby allowing JPEG AI to adjust the quality of the image content in the spatial dimension.

[0028] In a possible implementation, the image is input into a coding network to obtain a first feature map. The first feature map is input into a context network to obtain a first prediction map. The first residual map is determined according to the first feature map and the first prediction map. The first feature map is input into a super-prior coding network to obtain first super-prior information. The first super-prior information is quantized to obtain second super-prior information. The second super-prior information is input into a super-prior variance decoding network to obtain the first Gaussian distribution parameter information.

[0029] It can be seen that in the method provided in the embodiment of the present application, the first Gaussian distribution parameter information can be obtained by inputting an image. After obtaining the first Gaussian distribution parameter information, the first Gaussian distribution parameter information can be scaled by the target quality matrix. Since different positions in the target quality matrix can have different image quality values, encoding using the target quality scaling tensor can achieve code rate allocation in different areas of the image space, thereby allowing JPEG AI to adjust the quality of the image content in the spatial dimension.

[0030] In a second aspect, an embodiment of the present application further provides a decoding method, the method comprising: obtaining a code stream. Determine a target quality matrix based on the code stream, the target quality matrix being used to characterize the image quality of each region in the residual map of the feature domain. Scaling the third Gaussian distribution parameter information based on the target quality matrix to obtain fourth Gaussian distribution parameter information, the third Gaussian distribution parameter information being the Gaussian distribution parameter information of the residual map of the feature domain. Decoding the code stream based on the fourth Gaussian distribution parameter information to obtain a third residual map, the third residual map being the residual map of the feature domain. De-quantizing the third residual map based on the target quality matrix to obtain a fourth residual map. Determining a reconstructed image based on the fourth residual map.

[0031] In a possible implementation, a target mass scaling matrix may be determined according to the target mass matrix, and the target mass scaling matrix is ​​used to characterize the residual map of the feature domain and the scaling amount of each region in the Gaussian distribution parameter information. A target mass scaling tensor is determined according to the target mass scaling matrix, and the target mass scaling tensor is used to characterize the residual map of the feature domain and the scaling tensor of the Gaussian distribution parameter information in three-dimensional space. The third Gaussian distribution parameter information is scaled according to the target mass scaling tensor to obtain fourth Gaussian distribution parameter information.

[0032] In a possible implementation, the target quality scaling tensor may be determined according to the target quality scaling matrix and a gain parameter, where the gain parameter includes a gain vector for channel-level quality adjustment and / or an image-level quality control factor.

[0033] In a possible implementation, the bitstream may be decoded to obtain Gaussian distribution parameters of a target quality residual matrix of the target quality matrix. The probability distribution of the target quality residual matrix is ​​determined according to the Gaussian distribution parameters. The bitstream is decoded according to the probability distribution to obtain the target quality residual matrix. The target quality matrix is ​​determined according to the target quality residual matrix.

[0034] In a possible implementation, the bitstream may be decoded to obtain Gaussian distribution parameters of the target quality matrix, the probability distribution of the target quality matrix may be determined according to the Gaussian distribution parameters, and the bitstream may be decoded according to the probability distribution to obtain the target quality matrix.

[0035] In a possible implementation, the bitstream may be decoded to obtain an index number of a target probability distribution, where the target probability distribution is determined from a plurality of candidate probability distributions according to a target quality residual matrix of the target quality matrix. The bitstream is decoded according to the target probability distribution to obtain the target quality residual matrix. The target quality matrix is ​​determined according to the target quality residual matrix.

[0036] In a possible implementation, the bitstream may be decoded to obtain an index number of a target probability distribution, where the target probability distribution is determined from a plurality of candidate probability distributions according to the target quality matrix. The bitstream is decoded according to the target probability distribution to obtain the target quality matrix.

[0037] In a possible implementation, the code stream may be decoded to obtain a second feature map, and the second feature map is input into a super prior variance decoding network to obtain the third Gaussian distribution parameter information.

[0038] In a third aspect, an embodiment of the present application further provides a decoding method, the method comprising: obtaining a code stream. Determine a target quality matrix based on the code stream, the target quality matrix being used to characterize the image quality of each region in the residual map of the feature domain. Scaling the third Gaussian distribution parameter information based on the target quality matrix to obtain fourth Gaussian distribution parameter information, the third Gaussian distribution parameter information being the Gaussian distribution parameter information of the residual map of the feature domain. Decoding the code stream based on the fourth Gaussian distribution parameter information to obtain a third residual map, the third residual map being the residual map of the feature domain. Determine a reconstructed image based on the third residual map.

[0039] In a fourth aspect, an embodiment of the present application further provides a decoding method, the method comprising: obtaining a code stream. Determining a target quality matrix based on the code stream, the target quality matrix is ​​used to characterize the image quality of each region in the residual map of the feature domain. Decoding the code stream based on third Gaussian distribution parameter information to obtain a third residual map, the third Gaussian distribution parameter information is Gaussian distribution parameter information of the residual map of the feature domain, and the third residual map is the residual map of the feature domain. Dequantizing the third residual map based on the target quality matrix to obtain a fourth residual map. Determining a reconstructed image based on the fourth residual map.

[0040] In a fifth aspect, an embodiment of the present application further provides an encoding device, which includes: a transceiver unit and a processing unit. The transceiver unit is used to obtain a target quality matrix, and the target quality matrix is ​​used to characterize the image quality of each area in the residual map of the feature domain. The processing unit is used to scale the first residual map and / or the first Gaussian distribution parameter information according to the target quality matrix to obtain a second residual map and / or the second Gaussian distribution parameter information, the first residual map is the residual map of the feature domain, and the first Gaussian distribution parameter information is the Gaussian distribution parameter information of the residual map of the feature domain. The processing unit is also used to generate a code stream according to the target quality matrix, the second residual map and / or the second Gaussian distribution parameter information.

[0041] In a possible implementation, the processing unit is specifically used to: determine a target mass scaling matrix according to the target mass matrix, the target mass scaling matrix is ​​used to characterize the scaling amount of each region in the residual map of the feature domain and / or the Gaussian distribution parameter information. Determine a target mass scaling tensor according to the target mass scaling matrix, the target mass scaling tensor is used to characterize the scaling tensor of the residual map of the feature domain and / or the Gaussian distribution parameter information in three-dimensional space. Scale the first residual map according to the target mass scaling tensor to obtain the second residual map and / or scale the first Gaussian distribution parameter information according to the target mass scaling tensor to obtain the second Gaussian distribution parameter information.

[0042] In a possible implementation, the processing unit is specifically used to determine the target quality scaling tensor according to the target quality scaling matrix and a gain parameter, where the gain parameter includes a gain vector for channel-level quality adjustment and / or an image-level quality control factor.

[0043] In a possible implementation manner, the processing unit is specifically configured to: encode the target quality matrix and the second residual map to generate a bitstream, wherein the second residual map is encoded according to the second Gaussian distribution parameter information.

[0044] In a possible implementation manner, the processing unit is specifically configured to: encode the target quality matrix or a target quality residual matrix of the target quality matrix to generate a bitstream.

[0045] In a possible implementation manner, the processing unit is further used to: generate a target quality residual matrix of the target quality matrix according to the target quality matrix, wherein the target quality residual matrix is ​​used to characterize a residual value of each image quality in the target quality matrix.

[0046] In a possible implementation, the processing unit is specifically used to: determine Gaussian distribution parameters of the target quality residual matrix, determine probability distribution of the target quality residual matrix according to the Gaussian distribution parameters, write the Gaussian distribution parameters into the bitstream, and perform entropy coding on the target quality residual matrix according to the probability distribution.

[0047] In a possible implementation, the processing unit is specifically used to: determine Gaussian distribution parameters of the target quality matrix, determine probability distribution of the target quality matrix according to the Gaussian distribution parameters, write the Gaussian distribution parameters into the bitstream, and perform entropy coding on the target quality matrix according to the probability distribution.

[0048] In a possible implementation, the processing unit is specifically configured to: determine a target probability distribution from a plurality of candidate probability distributions according to the target quality residual matrix, write an index number of the target probability distribution into the bitstream, and perform entropy coding on the target quality residual matrix according to the target probability distribution.

[0049] In a possible implementation, the processing unit is specifically configured to: determine a target probability distribution from a plurality of candidate probability distributions according to the target quality matrix, write an index number of the target probability distribution into the bitstream, and perform entropy coding on the target quality matrix according to the target probability distribution.

[0050] In a possible implementation, the acquisition unit is specifically used to: acquire a quality map, the quality map being used to record the image quality of each region in the residual map of the feature domain, and determine the target quality matrix according to the quality map.

[0051] In a possible implementation, the processing unit is further used to: input the image into a coding network to obtain a first feature map. Input the first feature map into a context network to obtain a first prediction map. Determine the first residual map based on the first feature map and the first prediction map. Input the first feature map into a super-prior coding network to obtain first super-prior information. Quantize the first super-prior information to obtain second super-prior information. Input the second super-prior information into a super-prior variance decoding network to obtain the first Gaussian distribution parameter information.

[0052] In a sixth aspect, an embodiment of the present application also provides a decoding device, which includes: a transceiver unit and a processing unit. The transceiver unit is used to obtain a code stream. The processing unit is used to determine a target quality matrix based on the code stream, and the target quality matrix is ​​used to characterize the image quality of each area in the residual map of the feature domain. The processing unit is also used to scale the third Gaussian distribution parameter information according to the target quality matrix to obtain fourth Gaussian distribution parameter information, and the third Gaussian distribution parameter information is the Gaussian distribution parameter information of the residual map of the feature domain. The processing unit is also used to decode the code stream according to the fourth Gaussian distribution parameter information to obtain a third residual map, and the third residual map is the residual map of the feature domain. The processing unit is also used to dequantize the third residual map according to the target quality matrix to obtain a fourth residual map. The processing unit is also used to determine a reconstructed image based on the fourth residual map.

[0053] In a possible implementation, the processing unit is specifically used to: determine a target mass scaling matrix according to the target mass matrix, the target mass scaling matrix is ​​used to characterize the residual map of the feature domain and the scaling amount of each region in the Gaussian distribution parameter information. Determine a target mass scaling tensor according to the target mass scaling matrix, the target mass scaling tensor is used to characterize the residual map of the feature domain and the scaling tensor of the Gaussian distribution parameter information in three-dimensional space. Scale the third Gaussian distribution parameter information according to the target mass scaling tensor to obtain fourth Gaussian distribution parameter information.

[0054] In a possible implementation, the processing unit is specifically used to determine the target quality scaling tensor according to the target quality scaling matrix and a gain parameter, where the gain parameter includes a gain vector for channel-level quality adjustment and / or an image-level quality control factor.

[0055] In a possible implementation, the processing unit is specifically used to: decode the bitstream to obtain Gaussian distribution parameters of a target quality residual matrix of the target quality matrix. Determine the probability distribution of the target quality residual matrix according to the Gaussian distribution parameters. Decode the bitstream according to the probability distribution to obtain the target quality residual matrix. Determine the target quality matrix according to the target quality residual matrix.

[0056] In a possible implementation, the processing unit is specifically configured to: decode the bitstream to obtain Gaussian distribution parameters of the target quality matrix, determine the probability distribution of the target quality matrix according to the Gaussian distribution parameters, and decode the bitstream according to the probability distribution to obtain the target quality matrix.

[0057] In a possible implementation, the processing unit is specifically configured to: decode the bitstream to obtain an index number of a target probability distribution, where the target probability distribution is determined from a plurality of candidate probability distributions according to a target quality residual matrix of the target quality matrix; decode the bitstream according to the target probability distribution to obtain the target quality residual matrix; and determine the target quality matrix according to the target quality residual matrix.

[0058] In a possible implementation, the processing unit is specifically configured to: decode the bitstream to obtain an index number of a target probability distribution, where the target probability distribution is determined from a plurality of candidate probability distributions according to the target quality matrix; and decode the bitstream according to the target probability distribution to obtain the target quality matrix.

[0059] In a possible implementation, the processing unit is further used to: decode the bitstream to obtain a second feature map, and input the second feature map into a super priori variance decoding network to obtain the third Gaussian distribution parameter information.

[0060] In the seventh aspect, an embodiment of the present application also provides a decoding device, which includes: a transceiver unit and a processing unit. The transceiver unit is used to obtain a code stream. The processing unit is used to determine a target quality matrix based on the code stream, and the target quality matrix is ​​used to characterize the image quality of each area in the residual map of the feature domain. The processing unit is also used to scale the third Gaussian distribution parameter information according to the target quality matrix to obtain fourth Gaussian distribution parameter information, and the third Gaussian distribution parameter information is the Gaussian distribution parameter information of the residual map of the feature domain. The processing unit is also used to decode the code stream according to the fourth Gaussian distribution parameter information to obtain a third residual map, and the third residual map is the residual map of the feature domain. The processing unit is also used to determine a reconstructed image based on the third residual map.

[0061] In an eighth aspect, an embodiment of the present application further provides a decoding device, which includes: a transceiver unit and a processing unit. The transceiver unit is used to obtain a code stream. The processing unit is used to determine a target quality matrix based on the code stream, and the target quality matrix is ​​used to characterize the image quality of each area in the residual map of the feature domain. The processing unit is also used to decode the code stream according to the third Gaussian distribution parameter information to obtain a third residual map, and the third Gaussian distribution parameter information is the Gaussian distribution parameter information of the residual map of the feature domain, and the third residual map is the residual map of the feature domain. The processing unit is also used to dequantize the third residual map according to the target quality matrix to obtain a fourth residual map. The processing unit is also used to determine a reconstructed image based on the fourth residual map.

[0062] In the ninth aspect, an embodiment of the present application further provides a code stream, which includes: a target quality matrix, a third residual map and a third Gaussian distribution parameter, wherein the target quality matrix is ​​used to characterize the image quality of each area in the residual map of the feature domain, the third residual map is the residual map of the feature domain, and the third Gaussian distribution parameter information is the Gaussian distribution parameter information of the residual map of the feature domain. The target quality matrix is ​​used to scale the third Gaussian distribution parameter information to obtain fourth Gaussian distribution parameter information. The target quality matrix is ​​also used to dequantize the third residual map to obtain a fourth residual map.

[0063] In the tenth aspect, an embodiment of the present application also provides a coding device, which includes: at least one processor, when the at least one processor executes program code or instructions, it implements the method described in the above first aspect or any possible implementation method thereof.

[0064] Optionally, the device may further include at least one memory, and the at least one memory is used to store the program code or instruction.

[0065] In the eleventh aspect, an embodiment of the present application further provides a decoding device, which includes: at least one processor, when the at least one processor executes program code or instructions, it implements the method described in the above second aspect or any possible implementation method thereof.

[0066] Optionally, the device may further include at least one memory, and the at least one memory is used to store the program code or instruction.

[0067] In a twelfth aspect, an embodiment of the present application further provides a chip, comprising: an input interface, an output interface, and at least one processor. Optionally, the chip also includes a memory. The at least one processor is used to execute the code in the memory, and when the at least one processor executes the code, the chip implements the method described in the first aspect or any possible implementation thereof.

[0068] Optionally, the above chip may also be an integrated circuit.

[0069] In a thirteenth aspect, an embodiment of the present application further provides a computer-readable storage medium for storing a computer program, wherein the computer program includes methods for implementing the method described in the above-mentioned first aspect or any possible implementation thereof.

[0070] In a fourteenth aspect, an embodiment of the present application further provides a computer program product comprising instructions, which, when executed on a computer, enables the computer to implement the method described in the above-mentioned first aspect or any possible implementation thereof.

[0071] In a fifteenth aspect, an embodiment of the present application further provides a computer-readable storage medium, on which is stored a video code stream obtained by one or more processors executing the method described in the first aspect above or any possible implementation thereof.

[0072] The encoding and decoding device, computer storage medium, computer program product and chip provided in this embodiment are all used to execute the encoding and decoding method provided above. Therefore, the beneficial effects that can be achieved can refer to the beneficial effects in the encoding and decoding method provided above, and will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0073] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0074] Figure 1a An exemplary block diagram of a decoding system provided in an embodiment of the present application;

[0075] Figure 1b An exemplary block diagram of a video decoding system provided in an embodiment of the present application;

[0076] Figure 2 An exemplary block diagram of a video encoder provided in an embodiment of the present application;

[0077] Figure 3 An exemplary block diagram of a video decoder provided in an embodiment of the present application;

[0078] Figure 4 An exemplary block diagram of a video decoding device provided in an embodiment of the present application;

[0079] Figure 5An exemplary block diagram of a device provided in an embodiment of the present application;

[0080] Figure 6 A schematic diagram of a neural network-based image compression method provided in an embodiment of the present application;

[0081] Figure 7 A schematic diagram of an end-to-end image coding framework provided in an embodiment of the present application;

[0082] Figure 8 A schematic diagram of the structure of a neural network provided in an embodiment of the present application;

[0083] Fig. 9 An exemplary block diagram of a codec network provided in an embodiment of the present application;

[0084] Fig.10 Another exemplary block diagram of a codec network provided in an embodiment of the present application;

[0085] Fig.11 A schematic diagram of an encoding method provided in an embodiment of the present application;

[0086] Fig.12 A schematic diagram of a process for generating a target mass scaling tensor provided in an embodiment of the present application;

[0087] Fig.13 An encoding end architecture provided in an embodiment of the present application;

[0088] Fig.14 A schematic diagram of a decoding method provided in an embodiment of the present application;

[0089] Fig.15 A schematic diagram of another decoding method provided in an embodiment of the present application;

[0090] Fig.16 A schematic diagram of another decoding method provided in an embodiment of the present application;

[0091] Fig.17 A decoding end architecture provided in an embodiment of the present application;

[0092] Fig.18 A schematic diagram of an encoding device provided in an embodiment of the present application;

[0093] Fig.19 A schematic diagram of a decoding device provided in an embodiment of the present application;

[0094] Fig. 20 A schematic diagram of another decoding device provided in an embodiment of the present application;

[0095] Fig.21A schematic diagram of another decoding device provided in an embodiment of the present application;

[0096] Fig. 22 A schematic diagram of the structure of a chip provided in an embodiment of the present application;

[0097] Fig.23 A schematic diagram of the structure of another neural network provided in an embodiment of the present application;

[0098] Fig.24 A schematic diagram of the structure of a machine video encoding system provided in an embodiment of the present application;

[0099] Fig.25 A schematic diagram of the structure of a code stream provided in an embodiment of the present application. DETAILED DESCRIPTION

[0100] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the embodiments of the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the embodiments of the present application.

[0101] The term "and / or" in this article is merely a description of the association relationship of associated objects, indicating that three relationships may exist. For example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone.

[0102] The terms "first" and "second" and the like in the description and drawings of the embodiments of the present application are used to distinguish different objects, or to distinguish different processing of the same object, rather than to describe a specific order of objects.

[0103] In addition, the terms "including" and "having" and any variations thereof mentioned in the description of the embodiments of the present application are intended to cover non-exclusive inclusions. For example, a process, method, system, product or device including a series of steps or units is not limited to the listed steps or units, but may optionally include other steps or units that are not listed, or may optionally include other steps or units inherent to these processes, methods, products or devices.

[0104] It should be noted that, in the description of the embodiments of the present application, words such as "exemplarily" or "for example" are used to indicate examples, illustrations or descriptions. Any embodiment or design described as "exemplarily" or "for example" in the embodiments of the present application should not be interpreted as being more preferred or more advantageous than other embodiments or designs. Specifically, the use of words such as "exemplarily" or "for example" is intended to present related concepts in a specific way.

[0105] In the description of the embodiments of the present application, unless otherwise specified, “plurality” means two or more.

[0106] First, the terms involved in the embodiments of the present application are explained.

[0107] Data encoding and decoding includes two parts: data encoding and data decoding. Data encoding is performed on the source side (or commonly referred to as the encoder side), and generally includes processing (e.g., compressing) the original data to reduce the amount of data required to represent the original data (thereby more efficiently storing and / or transmitting). Data decoding is performed on the destination side (or commonly referred to as the decoder side), and generally includes inverse processing relative to the encoder side to reconstruct the original data. The "encoding and decoding" of the data involved in the embodiments of the present application should be understood as the "encoding" or "decoding" of the data. The encoding part and the decoding part are also collectively referred to as encoding and decoding (encoding and decoding, CODEC).

[0108] In the case of lossless data encoding, the original data can be reconstructed, that is, the reconstructed original data has the same quality as the original data (assuming no transmission loss or other data loss during storage or transmission). In the case of lossy data encoding, further compression is performed by quantization, etc. to reduce the amount of data required to represent the original data, but the decoder side cannot completely reconstruct the original data, that is, the quality of the reconstructed original data is lower or worse than the quality of the original data.

[0109] The embodiments of the present application can be applied to video data and other data with compression / decompression requirements. The following takes the encoding of video data (referred to as video encoding) as an example to illustrate the embodiments of the present application. Other types of data (such as image data, audio data, integer data and other data with compression / decompression requirements) can refer to the following description, and the embodiments of the present application will not be repeated. It should be noted that, compared with video encoding, the encoding process of audio data and integer data does not need to divide the data into blocks, but the data can be directly encoded.

[0110] Video coding generally refers to processing a sequence of images to form a video or video sequence. In the field of video coding, the terms "picture", "frame" or "image" can be used as synonyms.

[0111] Several video coding standards belong to the category of "lossy hybrid video codecs" (i.e., combining spatial and temporal prediction in the pixel domain with 2D transform coding in the transform domain for applying quantization). Each image in a video sequence is usually divided into a set of non-overlapping blocks, which are usually encoded at the block level. In other words, the encoder usually processes, i.e., encodes the video at the block (video block) level, for example, by generating a prediction block through spatial (intra-frame) prediction and temporal (inter-frame) prediction; subtracting the prediction block from the current block (currently processed / to be processed block) to obtain a residual block; transforming the residual block in the transform domain and quantizing the residual block to reduce the amount of data to be transmitted (compressed), while the decoder side applies the inverse processing part relative to the encoder to the encoded or compressed block to reconstruct the current block for representation. In addition, the encoder needs to repeat the processing steps of the decoder so that the encoder and the decoder generate the same predictions (e.g., intra-frame predictions and inter-frame predictions) and / or reconstructed pixels for processing, i.e., encoding subsequent blocks.

[0112] In the following embodiment of the decoding system 10, the encoder 20 and the decoder 30 are based on Figures 1a to 3 Give a description.

[0113] Figure 1a An exemplary block diagram of a decoding system 10 provided in an embodiment of the present application, for example, a video decoding system 10 (or simply a decoding system 10) that can utilize the techniques of the embodiments of the present application. The video encoder 20 (or simply an encoder 20) and the video decoder 30 (or simply a decoder 30) in the video decoding system 10 represent devices that can be used to perform various techniques according to various examples described in the embodiments of the present application.

[0114] like Figure 1a As shown, the decoding system 10 includes a source device 12, which is used to provide encoded image data 21 such as encoded images to a destination device 14 for decoding the encoded image data 21.

[0115] The source device 12 includes an encoder 20 , and may additionally or optionally include an image source 16 , a preprocessor (or a preprocessing unit) 18 such as an image preprocessor, and a communication interface (or a communication unit) 22 .

[0116] The image source 16 may include or may be any type of image capture device for capturing real-world images, etc., and / or any type of image generation device, such as a computer graphics processor for generating computer-animated images or any type of device for acquiring and / or providing real-world images, computer-generated images (e.g., screen content, virtual reality (VR) images, and / or any combination thereof (e.g., augmented reality (AR) images). The image source may be any type of memory or storage that stores any of the above images.

[0117] In order to distinguish the processing performed by the pre-processor (or pre-processing unit) 18 , the image (or image data) 17 may also be referred to as a raw image (or raw image data) 17 .

[0118] The preprocessor 18 is used to receive the original image data 17 and preprocess the original image data 17 to obtain a preprocessed image (or preprocessed image data) 19. For example, the preprocessing performed by the preprocessor 18 may include trimming, color format conversion (e.g., from RGB to YCbCr), color adjustment, or denoising. It is understood that the preprocessing unit 18 may be an optional component.

[0119] The video encoder (or encoder) 20 is used to receive the pre-processed image data 19 and provide the encoded image data 21 (hereinafter referred to as Figure 2 etc. for further description).

[0120] The communication interface 22 in the source device 12 can be used to receive the encoded image data 21 and send the encoded image data 21 (or any other processed version) to another device such as the destination device 14 or any other device through the communication channel 13 for storage or direct reconstruction.

[0121] The destination device 14 includes a decoder 30 and, in addition or alternatively, may include a communication interface (or communication unit) 28 , a post-processor (or post-processing unit) 32 , and a display device 34 .

[0122] The communication interface 28 in the destination device 14 is used to receive the encoded image data 21 (or any other processed version) directly from the source device 12 or from any other source device such as a storage device, for example, the storage device is a encoded image data storage device, and provide the encoded image data 21 to the decoder 30.

[0123] The communication interface 22 and the communication interface 28 can be used to send or receive encoded image data (or encoded data) 21 through a direct communication link between the source device 12 and the destination device 14, such as a direct wired or wireless connection, or through any type of network, such as a wired network, a wireless network or any combination thereof, any type of private network and public network or any type of combination thereof.

[0124] For example, the communication interface 22 may be used to encapsulate the encoded image data 21 into a suitable format such as a message, and / or process the encoded image data using any type of transmission coding or processing for transmission over a communication link or network.

[0125] The communication interface 28 corresponds to the communication interface 22 , for example, and can be used to receive transmission data and process the transmission data using any type of corresponding transmission decoding or processing and / or decapsulation to obtain the encoded image data 21 .

[0126] Both the communication interface 22 and the communication interface 28 can be configured as follows Figure 1a A unidirectional communication interface, or a bidirectional communication interface, indicated by an arrow pointing from the source device 12 to the corresponding communication channel 13 of the destination device 14, and can be used to send and receive messages, etc. to establish a connection, confirm and exchange any other information related to the communication link and / or data transmission such as encoded image data transmission, etc.

[0127] The video decoder (or decoder) 30 is used to receive the encoded image data 21 and provide decoded image data (or decoded image data) 31 (hereinafter referred to as Figure 3 etc. for further description).

[0128] The post-processor 32 is used to post-process the decoded image data 31 (also called reconstructed image data) such as the decoded image to obtain the post-processed image data 33 such as the post-processed image. The post-processing performed by the post-processing unit 32 may include, for example, color format conversion (e.g., from YCbCr to RGB), color adjustment, cropping or resampling, or any other processing for generating the decoded image data 31 for display by the display device 34 or the like.

[0129] The display device 34 is used to receive the post-processed image data 33 to display the image to a user or viewer, etc. The display device 34 can be or include any type of display for displaying the reconstructed image, such as an integrated or external display screen or display. For example, the display screen can include a liquid crystal display (LCD), an organic light emitting diode (OLED) display, a plasma display, a projector, a micro LED display, a liquid crystal on silicon (LCoS), a digital light processor (DLP), or any other type of display screen.

[0130] The decoding system 10 also includes a training engine 25, which is used to train the encoder 20 (especially the entropy coding unit 270 in the encoder 20) or the decoder 30 (especially the entropy decoding unit 304 in the decoder 30) to perform entropy coding on the image block to be encoded according to the estimated probability distribution. For a detailed description of the training engine 25, please refer to the following method test example.

[0131] although Figure 1a The source device 12 and the destination device 14 are shown as independent devices, but the device embodiment may also include the source device 12 and the destination device 14 or the functions of the source device 12 and the destination device 14 at the same time, that is, the source device 12 or the corresponding function and the destination device 14 or the corresponding function at the same time. In these embodiments, the source device 12 or the corresponding function and the destination device 14 or the corresponding function can be implemented using the same hardware and / or software or by separate hardware and / or software or any combination thereof.

[0132] According to the description, Figure 1a The presence and (exact) division of different units or functions in the source device 12 and / or the destination device 14 shown may vary depending on the actual device and application, which will be obvious to the skilled person.

[0133] Please refer to Figure 1b , Figure 1b An exemplary block diagram of a video decoding system 40 provided in an embodiment of the present application, wherein the encoder 20 (eg, the video encoder 20) or the decoder 30 (eg, the video decoder 30) or both may be configured as follows: Figure 1bThe processing circuits in the video decoding system 40 shown are implemented, for example, by one or more microprocessors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), discrete logic, hardware, video encoding dedicated processors, or any combination thereof. Figure 2 and Figure 3 , Figure 2 An exemplary block diagram of a video encoder provided in an embodiment of the present application is shown in FIG. Figure 3 An exemplary block diagram of a video decoder provided in an embodiment of the present application. The encoder 20 may be implemented by a processing circuit 46 to include a reference Figure 2 The various modules discussed in encoder 20 and / or any other encoder system or subsystem described herein. Decoder 30 may be implemented by processing circuit 46 to include reference Figure 3 The processing circuit 46 may be used to perform the various operations discussed below. Figure 4 As shown, if part of the technology is implemented in software, the device can store the software instructions in a suitable non-transitory computer-readable storage medium, and use one or more processors to execute the instructions in hardware, thereby performing the technology of the embodiment of the present application. One of the video encoder 20 and the video decoder 30 can be integrated into a single device as part of a combined codec (encoder / decoder, CODEC), such as Figure 1b shown.

[0134] The source device 12 and the destination device 14 may include any of a variety of devices, including any type of handheld or fixed device, such as a notebook or laptop computer, a mobile phone, a smart phone, a tablet or a tablet computer, a camera, a desktop computer, a set-top box, a television, a display device, a digital media player, a video game console, a video streaming device (e.g., a content service server or a content distribution server), a broadcast receiving device, a broadcast transmitting device, and a monitoring device, etc., and may not use or use any type of operating system. The source device 12 and the destination device 14 may also be devices in a cloud computing scenario, such as a virtual machine in a cloud computing scenario. In some cases, the source device 12 and the destination device 14 may be equipped with components for wireless communication. Therefore, the source device 12 and the destination device 14 may be wireless communication devices.

[0135] The source device 12 and the destination device 14 may install virtual scene applications (applications, APPs) such as virtual reality (VR) applications, augmented reality (AR) applications, or mixed reality (MR) applications, and may run VR applications, AR applications, or MR applications based on user operations (such as clicks, touches, slides, shakes, voice control, etc.). The source device 12 and the destination device 14 may collect images / videos of any object in the environment through cameras and / or sensors, and then display virtual objects on the display device based on the collected images / videos. The virtual objects may be virtual objects in VR scenes, AR scenes, or MR scenes (i.e., objects in the virtual environment).

[0136] It should be noted that in the embodiment of the present application, the virtual scene application in the source device 12 and the destination device 14 can be a built-in application of the source device 12 and the destination device 14 themselves, or it can be an application provided by a third-party service provider and installed by the user. There is no specific limitation on this.

[0137] In addition, the source device 12 and the destination device 14 may be installed with a real-time video transmission application, such as a live broadcast application. The source device 12 and the destination device 14 may collect images / videos through cameras and then display the collected images / videos on a display device.

[0138] In some cases, Figure 1a The video decoding system 10 shown is merely exemplary, and the techniques provided in the embodiments of the present application may be applicable to video encoding settings (e.g., video encoding or video decoding), which do not necessarily include any data communication between the encoding device and the decoding device. In other examples, the data is retrieved from a local memory, sent over a network, and so on. The video encoding device can encode the data and store the data in a memory, and / or the video decoding device can retrieve the data from the memory and decode the data. In some examples, encoding and decoding are performed by devices that do not communicate with each other but only encode data to a memory and / or retrieve and decode data from a memory.

[0139] Please refer to Figure 1b , Figure 1b An exemplary block diagram of a video decoding system 40 provided in an embodiment of the present application is shown in FIG. Figure 1b As shown, the video decoding system 40 may include an imaging device 41, a video encoder 20, a video decoder 30 (and / or a video encoder / decoder implemented by a processing circuit 46), an antenna 42, one or more processors 43, one or more memory storages 44 and / or a display device 45.

[0140] like Figure 1b As shown, imaging device 41, antenna 42, processing circuit 46, video encoder 20, video decoder 30, processor 43, memory storage 44 and / or display device 45 can communicate with each other. In different examples, video decoding system 40 can include only video encoder 20 or only video decoder 30.

[0141] In some instances, antenna 42 may be used to transmit or receive a coded bit stream of video data. In addition, in some instances, display device 45 may be used to present video data. Processing circuit 46 may include application-specific integrated circuit (ASIC) logic, graphics processor, general purpose processor, etc. Video decoding system 40 may also include an optional processor 43, which may similarly include application-specific integrated circuit (ASIC) logic, graphics processor, general purpose processor, etc. In addition, memory storage 44 may be any type of memory, such as volatile memory (e.g., static random access memory (SRAM), dynamic random access memory (DRAM), etc.) or non-volatile memory (e.g., flash memory, etc.). In a non-limiting example, memory storage 44 may be implemented by cache memory. In other instances, processing circuit 46 may include memory (e.g., cache, etc.) for implementing image buffers, etc.

[0142] In some examples, video encoder 20 implemented by logic circuits may include an image buffer (e.g., implemented by processing circuit 46 or memory storage 44) and a graphics processing unit (e.g., implemented by processing circuit 46). The graphics processing unit may be communicatively coupled to the image buffer. The graphics processing unit may include video encoder 20 implemented by processing circuit 46 to implement reference Figure 2 and / or the various modules discussed in connection with any other encoder system or subsystem described herein. Logic circuits may be used to perform the various operations discussed herein.

[0143] In some examples, video decoder 30 may be implemented in a similar manner by processing circuitry 46 to implement reference Figure 3The various modules discussed in connection with video decoder 30 and / or any other decoder system or subsystem described herein may be included in the video decoder 30 implemented by logic circuitry. In some examples, video decoder 30 implemented by logic circuitry may include an image buffer (implemented by processing circuitry 46 or memory storage 44) and a graphics processing unit (implemented by processing circuitry 46, for example). The graphics processing unit may be communicatively coupled to the image buffer. The graphics processing unit may include video decoder 30 implemented by processing circuitry 46 to implement the video decoder 30 implemented by processing circuitry 46. Figure 3 and / or the various modules discussed above for any other decoder system or subsystem described herein.

[0144] In some examples, antenna 42 may be used to receive an encoded bitstream of video data. As discussed, the encoded bitstream may include data related to the encoded video frames discussed herein, indicators, index values, mode selection data, etc., such as data related to the encoded partitions (e.g., transform coefficients or quantized transform coefficients, (as discussed) optional indicators, and / or data defining the encoded partitions). Video decoding system 40 may also include video decoder 30 coupled to antenna 42 and used to decode the encoded bitstream. Display device 45 is used to present the video frames.

[0145] It should be understood that for the examples described with reference to the video encoder 20 in the embodiments of the present application, the video decoder 30 can be used to perform the reverse process. With respect to the signaling syntax elements, the video decoder 30 can be used to receive and parse such syntax elements and decode the related video data accordingly. In some examples, the video encoder 20 can entropy encode the syntax elements into an encoded video bitstream. In such examples, the video decoder 30 can parse such syntax elements and decode the related video data accordingly.

[0146] For ease of description, the embodiments of the present application are described with reference to the universal video coding (VVC) reference software or the high-efficiency video coding (HEVC) developed by the video coding joint working group (joint collaboration team on video coding, JCT-VC) of the ITU-T video coding experts group (VCEG) and the ISO / IEC motion picture experts group (MPEG). Those of ordinary skill in the art understand that the embodiments of the present application are not limited to HEVC or VVC.

[0147] Encoders and encoding methods

[0148] like Figure 2As shown, the video encoder 20 includes an input terminal (or input interface) 201, a residual calculation unit 204, a transform processing unit 206, a quantization unit 208, an inverse quantization unit 210, an inverse transform processing unit 212, a reconstruction unit 214, a loop filter 220, a decoded picture buffer (DPB) 230, a mode selection unit 260, an entropy coding unit 270 and an output terminal (or output interface) 272. The mode selection unit 260 may include an inter-frame prediction unit 244, an intra-frame prediction unit 254 and a segmentation unit 262. The inter-frame prediction unit 244 may include a motion estimation unit and a motion compensation unit (not shown). Figure 2 The illustrated video encoder 20 may also be referred to as a hybrid video encoder or a hybrid video codec-based video encoder.

[0149] Images and Image Segmentation (Images and Blocks)

[0150] The encoder 20 may be configured to receive, via an input 201 or the like, an image (or image data) 17, e.g., an image in a sequence of images forming a video or a video sequence. The received image or image data may also be a pre-processed image (or pre-processed image data) 19. For simplicity, the following description uses image 17. Image 17 may also be referred to as a current image or an image to be encoded (particularly when the current image is to be distinguished from other images in video encoding, such as previously encoded images and / or decoded images in the same video sequence, i.e., a video sequence also including the current image).

[0151] A (digital) image is or can be considered as a two-dimensional array or matrix of pixels with intensity values. The pixels in the array can also be called pixels (pixel or pel) (short for picture elements). The number of pixels in the array or image in the horizontal and vertical directions (or axes) determines the size and / or resolution of the image. In order to represent color, three color components are usually used, that is, the image can be represented as or include three pixel arrays. In the RBG format or color space, the image includes corresponding red, green and blue pixel arrays. However, in video coding, each pixel is usually represented in a brightness / chrominance format or color space, such as YCbCr, including a brightness component indicated by Y (sometimes also represented by L) and two chrominance components represented by Cb and Cr. The brightness (luma) component Y represents the brightness or grayscale level intensity (for example, the two are the same in grayscale images), while the two chrominance (abbreviated as chroma) components Cb and Cr represent the chrominance or color information components. Accordingly, an image in YCbCr format includes a luminance pixel array of luminance pixel values ​​(Y) and two chrominance pixel arrays of chrominance values ​​(Cb and Cr). An image in RGB format can be converted or transformed into YCbCr format and vice versa, a process also referred to as color conversion or transformation. If the image is black and white, the image may include only a luminance pixel array. Accordingly, an image may be, for example, a luminance pixel array in a monochrome format or a luminance pixel array and two corresponding chrominance pixel arrays in 4:2:0, 4:2:2 and 4:4:4 color formats.

[0152] In one embodiment, an embodiment of the video encoder 20 may include an image segmentation unit ( Figure 2 ), for dividing the image 17 into a plurality of (usually non-overlapping) image blocks 203. These blocks may also be referred to as root blocks, macroblocks (H.264 / AVC) or coding tree blocks (CTB), or coding tree units (CTU) in the H.265 / HEVC and VVC standards. The partitioning unit may be used to use the same block size for all images in a video sequence and a corresponding grid of defined block sizes, or to change the block size between images or subsets of images or groups of images, and to partition each image into corresponding blocks.

[0153] In other embodiments, the video encoder may be configured to directly receive a block 203 of the image 17, for example, one, several or all blocks constituting the image 17. The image block 203 may also be referred to as a current image block or an image block to be encoded.

[0154] Like image 17, image block 203 is also or can be considered as a two-dimensional array or matrix composed of pixels with intensity values ​​(pixel values), but image block 203 is smaller than image 17. In other words, block 203 may include one pixel array (e.g., a brightness array in the case of monochrome image 17 or a brightness array or chrominance array in the case of a color image) or three pixel arrays (e.g., one brightness array and two chrominance arrays in the case of a color image 17) or any other number and / or type of arrays according to the color format adopted. The number of pixels in the horizontal and vertical directions (or axes) of block 203 defines the size of block 203. Accordingly, the block can be an M×N (M columns×N rows) pixel array, or an M×N transform coefficient array, etc.

[0155] In one embodiment, Figure 2 The video encoder 20 shown is used to encode the image 17 block by block, for example, encoding and prediction are performed on each block 203.

[0156] In one embodiment, Figure 2 The illustrated video encoder 20 may also be used to partition and / or encode an image using slices (also referred to as video slices), wherein an image may be partitioned or encoded using one or more slices (usually non-overlapping). Each slice may include one or more blocks (e.g., coding tree units CTUs) or one or more groups of blocks (e.g., coding tiles in the H.265 / HEVC / VVC standard and bricks in the VVC standard).

[0157] In one embodiment, Figure 2 The video encoder 20 shown can also be used to partition and / or encode an image using slices / coding block groups (also called video coding block groups) and / or coding blocks (also called video coding blocks), wherein an image can be partitioned or encoded using one or more slices / coding block groups (usually non-overlapping), each slice / coding block group may include one or more blocks (e.g., CTU) or one or more coding blocks, etc., wherein each coding block may be in a rectangular shape, etc., and may include one or more complete or partial blocks (e.g., CTU).

[0158] Residual calculation

[0159] The residual calculation unit 204 is used to calculate the residual block 205 (the prediction block 265 is described in detail later) based on the image block (or original block) 203 and the prediction block 265 in the following manner: for example, the pixel value of the prediction block 265 is subtracted from the pixel value of the image block 203 pixel by pixel (pixel by pixel) to obtain the residual block 205 in the pixel domain.

[0160] Quantification

[0161] The quantization unit 208 is used to quantize the transform coefficient 207 by, for example, scalar quantization or vector quantization to obtain a quantized transform coefficient 209 . The quantized transform coefficient 209 may also be referred to as a quantized residual coefficient 209 .

[0162] The quantization process may reduce the bit depth associated with some or all of the transform coefficients 207. For example, an n-bit transform coefficient may be rounded down to an m-bit transform coefficient during quantization, where n is greater than m. The degree of quantization may be modified by adjusting a quantization parameter (QP). For example, for scalar quantization, different degrees of scaling may be applied to achieve finer or coarser quantization. A smaller quantization step size corresponds to a finer quantization, while a larger quantization step size corresponds to a coarser quantization. A suitable quantization step size may be indicated by a quantization parameter (QP). For example, a quantization parameter may be an index of a predefined set of suitable quantization step sizes. For example, a smaller quantization parameter may correspond to fine quantization (a smaller quantization step size), a larger quantization parameter may correspond to coarse quantization (a larger quantization step size), and vice versa. Quantization may include dividing by a quantization step size, while a corresponding or inverse dequantization performed by an inverse quantization unit 210 or the like may include multiplying by a quantization step size. Embodiments according to some standards, such as HEVC, may be used to determine a quantization step size using a quantization parameter. In general, the quantization step size may be calculated using a fixed-point approximation of an equation containing division according to the quantization parameter. Other scaling factors may be introduced for quantization and dequantization to recover the norm of the residual block that may be modified due to the scaling used in the fixed-point approximation of the equations for the quantization step size and the quantization parameter. In one exemplary implementation, the scaling of the inverse transform and dequantization may be combined. Alternatively, a custom quantization table may be used and indicated from the encoder to the decoder in the bitstream, etc. Quantization is a lossy operation, where the larger the quantization step size, the greater the loss.

[0163] In one embodiment, the video encoder 20 (correspondingly, the quantization unit 208) may be configured to output a quantization parameter (QP), for example, directly output or output after being encoded or compressed by the entropy coding unit 270, for example, so that the video decoder 30 may receive and use the quantization parameter for decoding.

[0164] Dequantization

[0165] The inverse quantization unit 210 is used to perform inverse quantization of the quantization unit 208 on the quantized coefficients to obtain dequantized coefficients 211, for example, performing an inverse quantization scheme of the quantization scheme performed by the quantization unit 208 according to or using the same quantization step size as the quantization unit 208. The dequantized coefficients 211 may also be referred to as dequantized residual coefficients 211, corresponding to the transform coefficients 207, but due to the loss caused by quantization, the dequantized coefficients 211 are usually not completely the same as the transform coefficients.

[0166] reconstruction

[0167] The reconstruction unit 214 (e.g., the summer 214) is used to add the transform block 213 (i.e., the reconstructed residual block 213) to the prediction block 265 to obtain the reconstructed block 215 in the pixel domain, for example, by adding the pixel point values ​​of the reconstructed residual block 213 and the pixel point values ​​of the prediction block 265.

[0168] segmentation

[0169] The segmentation unit 262 may segment (or divide) an image block (or CTU) 203 into smaller parts, such as small blocks in a square or rectangular shape. For an image with three pixel arrays, a CTU consists of N×N luminance pixel blocks and two corresponding chrominance pixel blocks. The maximum allowed size of a luminance block in a CTU is specified as 128×128 in the developing universal video coding (VVC) standard, but may be specified as a value different from 128×128 in the future, such as 256×256. The CTUs of an image may be concentrated / grouped into slices / coding block groups, coding blocks, or bricks. A coding block covers a rectangular area of ​​an image, and a coding block may be divided into one or more bricks. A brick consists of multiple CTU rows within a coding block. A coding block that is not segmented into multiple bricks may be called a brick. However, a brick is a true subset of a coding block and is therefore not called a coding block. VVC supports two coding block group modes, namely raster scan slice / coding block group mode and rectangular slice mode. In raster scan CBG mode, a slice / CBG contains a sequence of CBs in a raster scan of CBs of an image. In rectangular slice mode, a slice contains multiple bricks of an image, which together form a rectangular region of the image. The bricks within a rectangular slice are arranged in the order of the bricks in the slice's raster scan. These smaller blocks (also called sub-blocks) can be further split into smaller parts. This is also called tree partitioning or hierarchical tree partitioning, where a root block at root tree level 0 (hierarchy level 0, depth 0), etc., can be recursively split into two or more blocks at the next lower tree level, such as nodes at tree level 1 (hierarchy level 1, depth 1). These blocks can be split into two or more blocks at the next lower level, such as tree level 2 (hierarchy level 2, depth 2), etc., until the partitioning ends (because the end criteria are met, such as reaching the maximum tree depth or minimum block size). Blocks that are not further split are also called leaf blocks or leaf nodes of the tree. A tree divided into two parts is called a binary tree (BT), a tree divided into three parts is called a ternary tree (TT), and a tree divided into four parts is called a quadtree (QT).

[0170] Entropy Coding

[0171] The entropy coding unit 270 is used to apply an entropy coding algorithm or scheme (e.g., a variable length coding (VLC) scheme, a context adaptive VLC scheme (CALVC), an arithmetic coding scheme, a binarization algorithm, a context adaptive binary arithmetic coding (CABAC), a syntax-based context-adaptive binary arithmetic coding (SBAC), a probability interval partitioning entropy (PIPE) coding, or other entropy coding methods or techniques) to the quantized residual coefficients 209, inter-frame prediction parameters, intra-frame prediction parameters, loop filter parameters, and / or other syntax elements to obtain coded image data 21 that can be output in the form of a coded bit stream 21 through an output terminal 272, so that the video decoder 30, etc. can receive and use the parameters for decoding. The coded bit stream 21 can be transmitted to the video decoder 30, or it can be stored in a memory for later transmission or retrieval by the video decoder 30.

[0172] Other structural variations of the video encoder 20 may be used to encode the video stream. For example, a non-transform based encoder 20 may directly quantize the residual signal without a transform processing unit 206 for certain blocks or frames. In another implementation, the encoder 20 may have a quantization unit 208 and an inverse quantization unit 210 combined into a single unit.

[0173] Decoder and decoding method

[0174] like Figure 3 As shown, the video decoder 30 is used to receive the coded image data 21 (e.g., the coded bitstream 21) encoded by the encoder 20, and obtain a decoded image 331. The coded image data or bitstream includes information for decoding the coded image data, such as data representing image blocks of a coded video slice (and / or a coding block group or coding block) and related syntax elements.

[0175] exist Figure 3In the example of FIG. 3 , decoder 30 includes an entropy decoding unit 304, an inverse quantization unit 310, an inverse transform processing unit 312, a reconstruction unit 314 (e.g., a summer 314), a loop filter 320, a decoded picture buffer (DBP) 330, a mode application unit 360, an inter-frame prediction unit 344, and an intra-frame prediction unit 354. Inter-frame prediction unit 344 may be or include a motion compensation unit. In some examples, video decoder 30 may perform substantially the same as reference Figure 2 The decoding process of the video encoder 100 is the reverse of the encoding process described above.

[0176] As described for encoder 20, inverse quantization unit 210, inverse transform processing unit 212, reconstruction unit 214, loop filter 220, decoded picture buffer DPB 230, inter prediction unit 344, and intra prediction unit 354 also constitute a “built-in decoder” of video encoder 20. Accordingly, inverse quantization unit 310 may be functionally identical to inverse quantization unit 110, inverse transform processing unit 312 may be functionally identical to inverse transform processing unit 122, reconstruction unit 314 may be functionally identical to reconstruction unit 214, loop filter 320 may be functionally identical to loop filter 220, and decoded picture buffer 330 may be functionally identical to decoded picture buffer 230. Therefore, the explanation of the corresponding units and functions of video encoder 20 applies accordingly to the corresponding units and functions of video decoder 30.

[0177] Entropy decoding

[0178] The entropy decoding unit 304 is used to parse the bit stream 21 (or generally the encoded image data 21) and perform entropy decoding on the encoded image data 21 to obtain the quantization coefficients 309 and / or the decoded encoding parameters ( Figure 3 The entropy decoding unit 304 may be used to provide the inter-frame prediction parameters, intra-frame prediction parameters and / or other syntax elements to the mode application unit 360, and to provide other parameters to other units of the decoder 30. The video decoder 30 may receive syntax elements at the video slice and / or video block level. In addition, or as an alternative to slices and corresponding syntax elements, coding block groups and / or coding blocks and corresponding syntax elements may be received or used.

[0179] Dequantization

[0180] The inverse quantization unit 310 may be configured to receive a quantization parameter (QP) (or generally information related to inverse quantization) and a quantization coefficient from the encoded image data 21 (e.g., parsed and / or decoded by the entropy decoding unit 304), and inverse quantize the decoded quantization coefficient 309 based on the quantization parameter to obtain an inverse quantization coefficient 311, which may also be referred to as a transform coefficient 311. The inverse quantization process may include using the quantization parameter calculated by the video encoder 20 for each video block in the video slice to determine a degree of quantization, and also determine a degree of inverse quantization that needs to be performed.

[0181] reconstruction

[0182] The reconstruction unit 314 (eg, the summer 314 ) is used to add the reconstructed residual block 313 to the prediction block 365 to obtain the reconstructed block 315 in the pixel domain, for example, by adding the pixel point values ​​of the reconstructed residual block 313 and the pixel point values ​​of the prediction block 365 .

[0183] Other variations of the video decoder 30 may be used to decode the encoded image data 21. For example, the decoder 30 may generate an output video stream without the loop filter unit 320. For example, a non-transform based decoder 30 may directly dequantize the residual signal without the inverse transform processing unit 312 for certain blocks or frames. In another implementation, the video decoder 30 may have the dequantization unit 310 and the inverse transform processing unit 312 combined into a single unit.

[0184] It should be understood that the processing result of the current step can be further processed in the encoder 20 and the decoder 30 and then output to the next step. For example, after interpolation filtering, motion vector derivation or loop filtering, the processing result of interpolation filtering, motion vector derivation or loop filtering can be further operated, such as clipping or shifting operation.

[0185] It should be noted that the derived motion vector of the current block (including but not limited to the control point motion vector of the affine mode, the sub-block motion vector of the affine, plane, ATMVP mode, the temporal motion vector, etc.) can be further operated. For example, the value of the motion vector is limited to a predefined range according to the representation bit of the motion vector. If the representation bit of the motion vector is bitDepth, the range is -2^(bitDepth-1) to 2^(bitDepth-1)-1, where "^" represents the power. For example, if bitDepth is set to 16, the range is -32768 to 32767; if bitDepth is set to 18, the range is -131072 to 131071. For example, the value of the derived motion vector (e.g., the MV of 4 4×4 sub-blocks in an 8×8 block) is limited so that the maximum difference between the integer parts of the 4 4×4 sub-block MVs does not exceed N pixels, for example, not more than 1 pixel. Two methods of limiting motion vectors according to bitDepth are provided here.

[0186] Although the above embodiments mainly describe video coding and decoding, it should be noted that the embodiments of the decoding system 10, the encoder 20 and the decoder 30 and other embodiments described herein can also be used for still image processing or coding and decoding, that is, the processing or coding and decoding of a single image in video coding and decoding that is independent of any previous or consecutive images. In general, if the image processing is limited to a single image 17, the inter-frame prediction unit 244 (encoder) and the inter-frame prediction unit 344 (decoder) may not be available. All other functions (also called tools or techniques) of the video encoder 20 and the video decoder 30 can also be used for still image processing, such as residual calculation 204 / 304, transformation 206, quantization 208, inverse quantization 210 / 310, (inverse) transformation 212 / 312, segmentation 262 / 362, intra-frame prediction 254 / 354 and / or loop filtering 220 / 320, entropy encoding 270 and entropy decoding 304.

[0187] Please refer to Figure 4 , Figure 4 An exemplary block diagram of a video decoding device 400 provided in an embodiment of the present application. The video decoding device 400 is suitable for implementing the disclosed embodiments described herein. In one embodiment, the video decoding device 400 may be a decoder, such as Figure 1a The video decoder 30 in the embodiment may also be an encoder, for example Figure 1a The video encoder 20 in.

[0188] The video decoding device 400 includes: an input port 410 (or input port 410) and a receiving unit (Rx) 420 for receiving data; a processor, a logic unit or a central processing unit (CPU) 430 for processing data; for example, the processor 430 here can be a neural network processor 430; a transmitter unit (Tx) 440 and an output port 450 (or output port 550) for transmitting data; and a memory 460 for storing data. The video decoding device 400 may also include an optical-to-electrical (OE) component and an electrical-to-optical (EO) component coupled to the input port 410, the receiving unit 420, the transmitting unit 440 and the output port 450 for the output or input of an optical signal or an electrical signal.

[0189] The processor 430 is implemented by hardware and software. The processor 430 can be implemented as one or more processor chips, cores (e.g., multi-core processors), FPGAs, ASICs, and DSPs. The processor 430 communicates with the input port 410, the receiving unit 420, the sending unit 440, the output port 450, and the memory 460. The processor 430 includes a neural network-based codec 470. The neural network-based codec 470 implements the embodiments disclosed above. For example, the neural network-based codec 470 performs, processes, prepares, or provides various encoding operations. Therefore, a substantial improvement is provided to the function of the video decoding device 400 by the neural network-based codec 470, and the switching of the video decoding device 400 to different states is affected. Alternatively, the neural network-based codec 470 is implemented with instructions stored in the memory 460 and executed by the processor 430.

[0190] The memory 460 includes one or more disks, tape drives, and solid-state hard disks, and can be used as an overflow data storage device for storing such programs when such programs are selected for execution, and for storing instructions and data read during program execution. The memory 460 can be volatile and / or non-volatile, and can be a read-only memory (ROM), a random access memory (RAM), a ternary content-addressable memory (TCAM), and / or a static random-access memory (SRAM).

[0191] Please refer to Figure 5 , Figure 5An exemplary block diagram of a device 500 provided in an embodiment of the present application, the device 500 can be used as Figure 1a Either or both of the source device 12 and the destination device 14 in .

[0192] The processor 502 in the device 500 may be a central processing unit. Alternatively, the processor 502 may be any other type of device or devices that are currently available or will be developed in the future and are capable of manipulating or processing information. Although a single processor such as the processor 502 shown in the figure may be used to implement the disclosed implementation, using more than one processor is faster and more efficient.

[0193] In one implementation, the memory 504 in the apparatus 500 may be a read-only memory (ROM) device or a random access memory (RAM) device. Any other suitable type of storage device may be used as the memory 504. The memory 504 may include code and data 506 accessed by the processor 502 via a bus 512. The memory 504 may also include an operating system 508 and an application 510, which includes at least one program that allows the processor 502 to perform the methods described herein. For example, the application 510 may include applications 1 to N, and also include a video decoding application that performs the methods described herein.

[0194] The apparatus 500 may also include one or more output devices, such as a display 518. In one example, the display 518 may be a touch-sensitive display that combines a display with a touch-sensitive element that may be used to sense touch input. The display 518 may be coupled to the processor 502 via the bus 512.

[0195] Although bus 512 in device 500 is described herein as a single bus, bus 512 may include multiple buses. In addition, auxiliary storage may be directly coupled to other components of device 500 or accessed through a network, and may include a single integrated unit such as a memory card or multiple units such as multiple memory cards. Therefore, device 500 may have a variety of configurations.

[0196] Image encoding:

[0197] Nowadays, multimedia data occupies a large part of Internet traffic. The compression of image data plays an important role in the storage and efficient transmission of multimedia data. Therefore, image coding technology is a technology with great practical value. It should be noted that the Chinese translation of the English terms "image coding" and "image encoding" is usually "image coding". Image coding is broad and includes the process of encoding an image into a bit stream and the process of decoding (decoding) a bit stream into an image.

[0198] Image coding simply refers to the process of encoding an image into a bit stream. Image coding research has a long history. Researchers have proposed a large number of methods and developed I-frame coding methods for a variety of image coding standards and video coding standards such as JPEG, JPEG2000, JPEG-XL, JPEG-XX, WebP, H. 264 / AVC, H. 264 / HEVC, H. 26 / VVC, AVS3 and AV1. Most of these coding methods are based on transform, prediction and entropy coding techniques. Although these coding methods are currently widely used, due to the increase in the amount of image data and the emergence of new media types, coding methods with higher compression efficiency are needed.

[0199] Image encoding based on deep learning:

[0200] In recent years, researchers have studied image coding methods based on deep learning. Some researchers have achieved good results. For example, Balle et al. proposed an end-to-end optimized image coding method that outperforms the existing best image coding and even outperforms the existing best traditional coding standard H. 265 / HEVC.

[0201] Image coding based on deep learning is based on deep neural networks, usually convolutional neural networks. Some research works have proposed an image coding method based on Transformer networks. The structure of the deep neural network can be designed manually or obtained through neural architecture search (NAS). The parameters of the deep neural network are obtained by using a loss function and a back-propagation algorithm.

[0202] Figure 6 A typical deep learning-based image compression method, also known as a neural network-based image compression method, is shown. Generally, a neural network-based image compression method includes the following parts: a feature extraction module, a feature quantization module, an entropy coding module, an entropy decoding module, a feature dequantization module, and a feature decoding module. On the encoder side, the feature extraction module can use a nonlinear mapping activation function to obtain an extracted three-dimensional feature map through multi-layer convolution superposition. The feature quantization module quantizes the eigenvalues ​​of the floating-point number through eigenvalue quantization to obtain the quantized eigenvalues. The quantized eigenvalues ​​are losslessly entropy encoded to obtain an encoded bit stream. When the entropy-coded bit stream is received, the decoder performs lossless entropy decoding to obtain the three-dimensional quantized eigenvalues. The feature decoding module decodes the features into a reconstructed image to achieve decoding.

[0203] After the image to be compressed passes through the feature extraction module and the feature quantization module, a three-dimensional feature quantization map is obtained. When processing each eigenvalue in the three-dimensional feature quantization map, the entropy coding module can estimate and obtain the probability distribution of the eigenvalue by using the eigenvalue in the processed neighborhood as context, and perform subsequent coding based on the probability distribution to obtain a coded bit stream.

[0204] With the outstanding performance of deep learning in various fields, researchers proposed an end-to-end image coding solution based on deep learning. Figure 7 The coding framework is shown. The specific technical scheme is as follows: On the encoder side, the original image is input to the feature extraction module, and the feature map is output. The feature map passes through the side information extraction module, and the side information is output. On the encoder and decoder side, is input to the probability estimation module, and each feature element The probability distribution of is output to obtain the value of the feature element to be encoded In addition, the feature map is input into the quantization module to obtain a quantized feature map The entropy coding module performs entropy coding on each feature element. In the quantized feature map Based on the probability distribution of each feature element to obtain the encoded bit stream.

[0205] On the decoder side, the decoder parses the bitstream and outputs the probability distribution of the symbols to be encoded Based on incidental information To obtain the value of the characteristic element to be decoded The k-entropy decoding module performs arithmetic decoding on each feature element. In the quantized feature map Based on the probability distribution of each feature element To get the value of a feature element Feature Map is input to the image reconstruction module, and the reconstructed image is output.

[0206] Neural Networks

[0207] A neural network may include neurons. A neuron may be an operation unit that uses xs and an intercept 1 as input. The output of the operation unit may be:

[0208]

[0209] Here, s=s=1,2,...,n, n is a natural number greater than 1, Ws is the weight of xs, b is the bias of the neuron, and f is the activation function of the neuron (activation function), which is used to introduce nonlinear characteristics into the neural network to convert the input signal in the neuron into the output signal. The output signal of the activation function can be used as the input of the next convolutional layer. The activation function can be a sigmoid function. A neural network is a network formed by connecting multiple single neurons together. Specifically, the output of one neuron can be the input to another neuron. The input of each neuron can be connected to the local receptive field of the previous layer to extract the characteristics of the local receptive field. The local receptive field can be an area including several neurons.

[0210] Convolutional Neural Networks:

[0211] A convolutional neural network (CNN) is a deep neural network with a convolutional structure. A convolutional neural network includes a feature extractor, which includes a convolution layer and a subsampling layer. The feature extractor can be considered as a filter. A convolution layer is a neuron layer in a convolutional neural network that performs convolution processing on an input signal. In a convolutional layer of a convolutional neural network, a neuron may be connected to only a portion of neurons in an adjacent layer. A convolutional layer typically includes several feature planes, each of which may include a number of neurons arranged in a rectangular shape. Neurons on the same feature plane share a weight, where the shared weight is a convolution kernel. Shared weights can be understood as a way of extracting image information that is independent of position. The convolution kernel can be initialized in the form of a matrix of random size. During the training process of a convolutional neural network, appropriate weights can be obtained for the convolution kernel through learning. In addition, shared weights directly reduce the connections between the layers of a convolutional neural network and reduce the risk of overfitting.

[0212] Figure 8 Schematic diagram showing the general concept of processing by a neural network such as a CNN. A convolutional neural network consists of an input layer and an output layer and multiple hidden layers. The input layer is where the input (e.g. Figure 8 The hidden layers of a CNN are usually composed of a series of convolutional layers that perform convolutions with multiplications or other dot products. The result of the layer is one or more feature maps, sometimes called channels. Subsampling may be involved in some or all layers. So, for example Figure 8As shown, the feature maps can become smaller. The activation function in a CNN is usually a RELU (Rectified Linear Unit) layer, followed by additional convolutions such as pooling layers, fully connected layers, and normalization layers, called hidden layers because their inputs and outputs are masked by the activation function and the final convolution. Although these layers are colloquially called convolutions, this is just convention. Mathematically speaking, it is technically a sliding dot product or cross correlation. This has important implications for the indexing in the matrix because it affects how the weight is determined at a specific index point.

[0213] When programming a CNN for processing images, such as Figure 8 As shown, the input is a tensor with shape (number of images) x (image width) x (map height) x (image depth). Then, after passing through the convolution layer, the image is abstracted into a feature map with shape (number of images) x (feature map width) x (feature map height) x (feature map channels). The convolution layer in a neural network should have the following properties. The convolution kernel defined by width and height (hyperparameters). The number of input channels and output channels (hyperparameters). The depth of the convolution filter (input channels) should be equal to the number of channels of the input feature map (depth).

[0214] In the past, traditional multilayer perceptron (MLP) models have been used for image recognition. However, due to the full connectivity between nodes, they are of high dimensionality and do not scale well with higher resolution images. A 1000×1000 pixel image with RGB color channels has 3 million weights, which is too high to be processed efficiently at scale with full connectivity. Furthermore, this network architecture does not account for the spatial structure of the data, treating input pixels that are far apart in the same way as pixels that are close together. This ignores the locality of reference in image data both computationally and semantically. Therefore, full connectivity of neurons is wasteful for purposes such as image recognition, which is dominated by spatially local input patterns.

[0215] Convolutional Neural Networks are biologically inspired variants of Multilayer Perceptrons, specifically designed to mimic the behavior of the visual cortex. These models mitigate the challenges posed by MLP architectures by exploiting the strong spatially local correlations present in natural images. The convolutional layer is the core building block of a CNN. The parameters of this layer consist of a set of learnable filters (the kernels mentioned above) that have small receptive fields but extend to the full depth of the input volume. During the forward pass, each filter is convolved across the width and height of the input volume, computing the dot product between the entries of the filter and the input, and producing a 2D activation map for that filter. Thus, the network learns filters that activate when it detects a certain type of feature at a certain spatial location in the input.

[0216] Stacking the activation maps of all filters along the depth dimension forms the complete output volume of the convolutional layer. Therefore, each entry in the output volume can also be interpreted as the output of a neuron that observes a small region in the input and shares parameters with neurons in the same activation map. A feature map or activation map is the output activation of a given filter. Feature map and activation have the same meaning. In some papers, it is called an activation map because it is a map corresponding to the activations of different parts of the image, and a feature map because this is also a map of some kind of feature found in the image. A high activation means that a certain feature was found.

[0217] Another important concept in CNNs is pooling, which is a form of nonlinear downsampling. There are several nonlinear functions that can implement pooling, of which max pooling is the most common. It divides the input image into a set of non-overlapping rectangles and for each such sub-region, outputs the maximum value.

[0218] Intuitively, the exact location of a feature relative to other features is less important than its coarse location. This is the idea behind using pooling in convolutional neural networks. Pooling layers are used to gradually reduce the spatial size of the representation, reducing the number of parameters, memory usage, and computation in the network, thus also controlling overfitting. In CNN architectures, it is common to periodically insert pooling layers between consecutive convolutional layers. The pooling operation provides another form of translation invariance.

[0219] Pooling layers operate independently on each depth slice of the input, and resize it spatially. The most common form is a pooling layer with filters of size 2×2, applying a downsampling stride of 2 along width and height on each depth slice in the input, discarding 75% of the activations. In this case, each maximum operation is over 4 numbers. The depth dimension remains unchanged.

[0220] In addition to max pooling, pooling units can use other functions such as average pooling or l2-norm pooling. Average pooling was often used historically, but has recently fallen out of favor compared to max pooling, which performs better in practice. Due to the large reduction in representation size, there has been a recent trend to use smaller filters or to drop pooling layers entirely. "Region of interest" pooling (also known as ROI pooling) is a variant of max pooling where the output size is fixed and the input rectangle is a parameter. Pooling is an important component of convolutional neural networks for object detection based on the Fast R-CNN architecture.

[0221] The above ReLU is short for Rectified Linear Unit, which applies a non-saturated activation function. It effectively removes negative values ​​from the activation map by setting them to zero. It adds nonlinearity to the decision function and the entire network without affecting the receptive field of the convolutional layer. Other functions are also used to add nonlinearity, such as saturated hyperbolic tangent and sigmoid function. ReLU is often preferred over other functions because it trains neural networks several times faster without significantly affecting generalization accuracy.

[0222] After several convolutional and max pooling layers, high-level reasoning in a neural network is done with fully connected layers. Neurons in a fully connected layer have connections to all activations in the previous layer, as seen in regular (non-convolutional) artificial neural networks. Therefore, their activations can be calculated as affine transformations, matrix multiplications followed by bias shifts (vector addition of learned or fixed bias terms).

[0223] The "loss layer" specifies how the training penalizes the deviation between the prediction (output) and the true label and is usually the last layer of the neural network. A variety of loss functions suitable for different tasks can be used. Softmax loss is used to predict a single class among K mutually exclusive classes. Sigmoid cross entropy loss is used to predict K independent probability values ​​in [0, 1]. Euclidean loss is used to regress to real-valued labels.

[0224] In conclusion, Figure 8 shows the data flow in a typical convolutional neural network. First, the input image passes through a convolutional layer and is abstracted into a feature map consisting of several channels, corresponding to the number of filters in the set of learnable filters of the layer (e.g., one channel per filter). The feature map is then subsampled using, for example, a pooling layer, which reduces the dimensionality of each channel in the feature map. The data next enters another convolutional layer, which may have a different number of output channels, resulting in a different number of channels in the feature map. As mentioned above, the number of input channels and output channels are hyperparameters of the layer. In order to establish the connectivity of the network, these parameters need to be synchronized between two connected layers, for example, the number of input channels of the current layer should be equal to the number of output channels of the previous layer. For the first layer that processes the input data (e.g., an image), the number of input channels is usually equal to the number of channels of the data representation, such as 3 channels for RGB or YUV representation of images or videos, or 1 channel for grayscale image or video representation.

[0225] In actual applications of video image coding, it is usually necessary to allocate different bit rates to the region of interest and the background region for coding, so as to meet customer needs with a lower bit rate.

[0226] To this end, an embodiment of the present application provides a coding method, which is suitable for a coding and decoding network.

[0227] Fig. 9 A possible implementation of the codec network is shown. Fig. 9 As shown, the codec network includes a coding network module, a bit rate control module, an entropy coding module, a super coding module, a super decoding module, an entropy decoding module, an inverse quantization module, and a decoding network module. Among them, the bit rate control module can control the bit rate of the code stream generated by the encoder through the quality matrix.

[0228] The image input to the codec network encoder is encoded by the coding network module, the bit rate control module, the entropy coding module, and the super coding module to obtain a bit stream. After the bit stream is input to the codec network decoder, the input image is reconstructed by the super decoding module, the entropy decoding module, the inverse quantization module, and the decoding network module to obtain a reconstructed image.

[0229] The above image can be a YUV domain image or an RGB domain image.

[0230] Fig.10 Another possible implementation of the codec network is shown. Fig.10 As shown, the encoding and decoding network includes a Y component encoding network module, a Y component rate control module, a Y component super encoding module, a UV component encoding network module, a UV component rate control module, a UV component super encoding module, an entropy encoding module, an entropy decoding module, a Y component super decoding module, a Y component dequantization module, a Y component decoding network module, a UV component super decoding module, a UV component dequantization module, and a UV component decoding network module.

[0231] and Fig. 9 The difference between the codec network shown is that Fig.10 The encoding and decoding network shown can encode and decode the Y component and UV component of the YUV image respectively.

[0232] Please refer to Fig.11 , Fig.11 A coding method is provided in the embodiment of the present application. Fig.11 As shown, the encoding method may include:

[0233] S1101. Obtain target mass matrix.

[0234] The target quality matrix is ​​used to characterize the image quality of each region in the residual image of the feature domain. The target quality matrix includes multiple quality values, each of which corresponds to the image quality of each region in the residual image of the feature domain.

[0235] In a possible implementation, a quality map may be obtained, and the target quality matrix may be determined according to the quality map, wherein the quality map is used to record the image quality of each region in the residual map of the feature domain.

[0236] It should be noted that the specific method for obtaining the quality map and the target quality matrix may be any method that can be conceived by a person skilled in the art, and the embodiments of the present application are not limited thereto.

[0237] For example, the quality value (image quality value) of each point in the above-mentioned quality map and target quality matrix may be set by the user.

[0238] For another example, the quality map or quality matrix can be obtained by analyzing the input image (or feature map) through a coding algorithm, a region of interest detection algorithm, etc. The size of the input image (or feature map) and the size of the quality map can be the same.

[0239] In a possible implementation, the input image may be a complete image, or may be a Y component and a UV component of an image.

[0240] S1102: Scale the first residual map and / or the first Gaussian distribution parameter information according to the target quality matrix to obtain a second residual map and / or second Gaussian distribution parameter information.

[0241] Among them, the above-mentioned first residual map is the residual map of the above-mentioned feature domain, and the above-mentioned first Gaussian distribution parameter information is the Gaussian distribution parameter information of the residual map of the above-mentioned feature domain.

[0242] In a possible implementation, a target mass scaling matrix can be determined according to the target mass matrix, and the target mass scaling matrix is ​​used to characterize the scaling amount of each region in the residual map of the feature domain and / or the Gaussian distribution parameter information. A target mass scaling tensor is determined according to the target mass scaling matrix, and the target mass scaling tensor is used to characterize the scaling tensor of the residual map of the feature domain and / or the Gaussian distribution parameter information in three-dimensional space. The first residual map is scaled according to the target mass scaling tensor to obtain the second residual map and / or the first Gaussian distribution parameter information is scaled according to the target mass scaling tensor to obtain the second Gaussian distribution parameter information.

[0243] For example, for the coded number domain, the quality scaling value of each point in the image can be determined according to the quality value (i.e., quality level index) Q[i, j] of each point in the target quality scaling matrix. m [i, j] can satisfy:

[0244] Quality scaling value

[0245] Wherein, k is an integer and n is a real number. For example, K can be 4 and n can be 0.

[0246] In one possible implementation, Q m Fixed point processing is Q int .

[0247] Optionally, Q int Can satisfy:

[0248] Q int =round(Q m *2 denom )+b

[0249] Wherein, b is an integer.

[0250] by For example, when the value range of quality level Q[i, j] is [-8, 8], a 6-bit fixed point is used to represent Q m , the specific values ​​are shown in Table 1, where Q m =Q int >>denom.

[0251] Table 1

[0252] Quality value Q -8 -7 -6 -5 -4 -3 -2 -1 0 1 2 3 4 5 6 7 8 <![CDATA[Q int ]]> 4 5 6 7 8 10 12 14 16 20 23 27 32 39 46 54 64 denom 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4

[0253] For example, for the encoded log domain, in addition to calculating the quality scaling value, it is also necessary to additionally calculate the quality scaling value Q of the encoded log domain m_log , encodes the quality scaling value Q in the log domain m_log Can satisfy:

[0254] Right now

[0255] In practical applications, for the encoded log domain, Q m_log When performing fixed-point conversion, you can use the relevant parameters log in the log domain k 2*a is fixed-point, and the fixed-point quality scaling value Q m_log Can satisfy:

[0256] Q m_log [i,j]=t*Q[i,j]>>(2+precision)

[0257] As another example, for a coded three-dimensional map, the quality scaling value of each image point can be determined according to the (image) quality value (i.e., quality level index) Q[c, i, j] of each image point in the target quality matrix. m [c, i, j] can satisfy:

[0258] Quality scaling value

[0259] For encoding three-dimensional maps, it is also necessary to calculate the quality scaling value Q in the encoding log domain m_log , encodes the quality scaling value Q in the log domain m_log Can satisfy:

[0260] Right now

[0261] In practical applications, for encoding three-dimensional maps, Q m_log When performing fixed-point conversion, you can use the relevant parameters log in the log domain k 2*a is fixed-point, and the fixed-point quality scaling value Q m_log Can satisfy:

[0262] Q m_log [c,i,j]=t*Q[c,i,j]>>(2+precision)

[0263] In a possible implementation, the target quality scaling tensor may be determined according to the target quality scaling matrix and a gain vector, where the gain vector includes a gain vector for channel-level quality adjustment and / or an image-level quality control factor.

[0264] It can be understood that the feature map C×H×W three-dimensional tensor can be converted into a two-dimensional H×W quality scaling matrix Q through the target quality scaling matrix (and gain vector) m Expanded to a three-dimensional tensor.

[0265] In a possible implementation, the target quality scaling tensor is determined according to the target quality scaling matrix and a gain parameter, where the gain parameter includes a gain vector for channel-level quality adjustment and / or an image-level quality control factor.

[0266] like Fig.12 As shown, a mass scaling map representing a target mass scaling matrix can be determined according to a mass map representing a target mass matrix. Then, a scaling tensor (target mass scaling tensor) is determined according to the mass scaling map representing the target mass scaling matrix and a gain parameter.

[0267] For example, for the coded number domain, in the presence of a channel-level quality adjustment gain vector m t , and image-level quality control factor β displacement In the case of , the target mass scaling tensor m[c,i,j] can satisfy:

[0268] m[c,i,j]=Q m [i,j]·β displacement ·m t [c]

[0269] or:

[0270] m[c,i,j]=(Q int [i,j]·β displacement >>denom)·m t [c]

[0271] or:

[0272] m[c,i,j]=(Q int [i,j]·β displacement ·m t [c])>>denom

[0273] For the encoding log domain, there is a gain vector m for channel-level quality adjustment t , and image-level quality control factor β displacement In the case of , the target mass scaling tensor m[c,i,j] can satisfy:

[0274] m[c,i,j]=Q m [i,j]·β displacement ·m t [c]

[0275] or:

[0276] m[c,i,j]=(Q int [i,j]·β displacement >>denom)·m t [c]

[0277] or:

[0278] m[c,i,j]=(Q int [i,j]·β displacement ·m t [c])>>denom

[0279] For encoding log domain, it is also necessary to additionally calculate the target quality scaling tensor m_log in the logarithmic domain. The target quality scaling tensor m_log in the logarithmic domain satisfies:

[0280] m_log[c,i,j]=Q m _log[i,j]+β displacement_log +m t_log [c]

[0281] For the encoded 3D map, there is a gain vector m for channel-level quality adjustment t , and image-level quality control factor β displacement In the case of , the target mass scaling tensor m[c,i,j] can satisfy:

[0282] m[c,i,j]=Q m [c,i,j]·β displacement ·m t [c]

[0283] or:

[0284] m[c,i,j]=(Q int [c,i,j]·β displacement >>denom)·m t [c]

[0285] or:

[0286] m[c,i,j]=(Q int [c,i,j]·β displacement ·m t [c])>>denom

[0287] For encoding three-dimensional maps, it is also necessary to additionally calculate the target quality scaling tensor m_log in the logarithmic domain. The target quality scaling tensor m_log in the logarithmic domain satisfies:

[0288] m_log[c,i,j]=Q m _log[c,i,j]+β displacement_log +m t_log [c]

[0289] As another example, for the coding domain, in the presence of a frame-level quality control factor β displacement In the case of , the target mass scaling tensor m[c,i,j] can satisfy:

[0290] m[c,i,j]=Q m [i,j]·β displacement ,

[0291] or:

[0292] m[c,i,j]=(Q int [c,i,j]>>denom)·β displacement ,

[0293] or:

[0294] m[c,i,j]=(Q int [c,i,j]·β displacement )>>denom

[0295] For encoding in the log domain, there is a frame-level quality control factor β displacement In the case of , the target mass scaling tensor m[c,i,j] can satisfy:

[0296] m[c,i,j]=Q m [i,j]·β displacement ,

[0297] or:

[0298] m[c,i,j]=(Q int [i,j]>>denom)·β displacement ,

[0299] or:

[0300] m[c,i,j]=(Q int [i,j]·β displacement )>>denom

[0301] For encoding log domain, it is also necessary to additionally calculate the target quality scaling tensor m_log in the logarithmic domain. The target quality scaling tensor m_log in the logarithmic domain satisfies:

[0302] m_log[c,i,j]=Q m _log[i,j]+β displacement_log

[0303] For encoding 3D maps, there is a frame-level quality control factor β displacement In the case of , the target mass scaling tensor m[c,i,j] can satisfy:

[0304] m[c,i,j]=Q m [c,i,j]·β displacement ,

[0305] or:

[0306] m[c,i,j]=(Q int [c,i,j]>>denom)·β displacement ,

[0307] or:

[0308] m[c,i,j]=(Q int [c,i,j]·β displacement )>>denom

[0309] For encoding three-dimensional maps, it is also necessary to additionally calculate the target quality scaling tensor m_log in the logarithmic domain. The target quality scaling tensor m_log in the logarithmic domain satisfies:

[0310] m_log[c,i,j]=Q m _log[c,i,j]+β displacement_log

[0311] In another possible implementation manner, the target mass scaling tensor may be determined according to the target mass scaling matrix.

[0312] Exemplarily, for the coded number domain, in the absence of other quality control factors (gain parameters), the target quality scaling tensor m[c,i,j] may satisfy:

[0313] m[c,i,j]=Q m [i,j]

[0314] or:

[0315] m[c,i,j]=Q int [i,j]>>denom

[0316] For encoding in the log domain, without other quality control factors (gain parameters), the target quality scaling tensor m[c,i,j] can satisfy:

[0317] m[c,i,j]=Q m [i,j]

[0318] or:

[0319] m[c,i,j]=Q int [i,j]>>denom

[0320] For encoding log domain, it is also necessary to additionally calculate the target quality scaling tensor m_log in the logarithmic domain. The target quality scaling tensor m_log in the logarithmic domain satisfies:

[0321] m_log[c,i,j]=Q m_log [i,j]

[0322] For the encoded 3D map, in the absence of other quality control factors (gain parameters), the target quality scaling tensor m[c,i,j] can satisfy:

[0323] m[c,i,j]=Q m [c,i,j]

[0324] or:

[0325] m[c,i,j]=Q int [c,i,j]>>denom

[0326] For encoding three-dimensional maps, it is also necessary to additionally calculate the target quality scaling tensor m_log in the logarithmic domain. The target quality scaling tensor m_log in the logarithmic domain satisfies:

[0327] m_log[c,i,j]=Q m_log [c,i,j]

[0328] In a possible implementation, for the coded number field, the second residual graph r'[c,i,j] may satisfy:

[0329] r'[c,i,j]=(m[c,i,j]·r[c,i,j])>>(Precision)

[0330] Where r[c,i,j] is the first residual graph, and Precision can be any non-negative number. For example, Precision can be a non-negative number such as 0, 1, or 13.

[0331] In a possible implementation, for the coded number field, the second Gaussian distribution parameter information (second variance) σ' may satisfy:

[0332] σ'[c,i,j]=(m[c,i,j]·σ[c,i,j])>>(Precision2)

[0333] Wherein, σ is the first Gaussian distribution parameter information (first variance), and Precision2 can be any non-negative number. For example, Precision2 can be a non-negative number such as 0, 1, or 13.

[0334] In a possible implementation, for the encoded log domain, the second residual graph r'[c,i,j] may satisfy:

[0335] r'[c,i,j]=(m[c,i,j]·r[c,i,j])>>(Precision)

[0336] In a possible implementation, for the encoded log domain, the second Gaussian distribution parameter information (second variance) σ' may satisfy:

[0337] σ'[c,i,j]=m log [c,i,j]+σ[c,i,j]-Precision2_log

[0338] S1103: Generate a bitstream according to the target quality matrix, the second residual map and / or the second Gaussian distribution parameter information.

[0339] In a possible implementation manner, the target quality matrix and the second residual map may be encoded to generate a bitstream, and the second residual map is encoded according to the second Gaussian distribution parameter information.

[0340] In a possible implementation manner, the target quality matrix or the target quality residual matrix of the target quality matrix may be encoded to generate a bitstream, and the target quality residual matrix is ​​used to represent the residual value of each image quality in the target quality matrix.

[0341] In a possible implementation, a target quality residual matrix of the target quality matrix can be generated according to the target quality matrix:

[0342] It should be noted that the specific method of generating the target quality residual matrix of the target quality matrix according to the target quality matrix may adopt any method conceived by those skilled in the art, and the embodiments of the present application are not limited thereto.

[0343] For example, the prediction matrix of the target mass matrix can be determined according to the target mass matrix, and then the target mass residual matrix can be determined according to the prediction matrix of the target mass matrix. Among them, the prediction value of any point in the prediction matrix of the target mass matrix can be calculated according to the left and upper adjacent points of the mass quantity corresponding to the prediction quantity of the point in the target mass matrix. In the case where only the upper adjacent point exists, the prediction value of the point can be calculated only according to the upper adjacent point of the point. In the case where only the left adjacent point exists, the prediction value of the point can be calculated only according to the left adjacent point of the point. In the case where only the left adjacent point and the upper adjacent point do not exist, the prediction value of the point can be determined to be 0.

[0344] Exemplarily, the predicted value Q_pred[i,j] of any point in the target quality prediction matrix may satisfy:

[0345] Q_pred[i,j]=(Q[i,j-1]+Q[i-1,j]) / 2

[0346] For example, the residual value Q of any point in the target quality residual matrix res [i,j] can satisfy:

[0347] Q res [i,j]=Q[i,j]-Q_pred[i,j]

[0348] In practical applications, the range of residual values ​​in the target quality residual matrix can be N times the range of mass values ​​in the target quality matrix. N is a positive number. For example, if the range of mass values ​​in the target quality matrix is ​​[-8, 8], the range of residual values ​​in the target quality residual matrix can be [-16, 16].

[0349] In a possible implementation, Gaussian distribution parameters of the target quality residual matrix may be determined, probability distribution of the target quality residual matrix may be determined according to the Gaussian distribution parameters, the Gaussian distribution parameters may be written into the bitstream, and entropy coding may be performed on the target quality residual matrix according to the probability distribution.

[0350] For example, the Gaussian distribution parameters of the target quality residual matrix can be determined, a probability distribution table of the target quality residual matrix can be determined according to the Gaussian distribution parameters, the Gaussian distribution parameters can be written into the bitstream, and entropy coding (such as me-tANS entropy coding) can be performed on the target quality residual matrix according to the probability distribution table.

[0351] Optionally, the Gaussian distribution parameter index number may also be written into the bitstream.

[0352] As shown in Table 2, if the Gaussian distribution parameter is a probability distribution of 0.2, the Gaussian distribution parameter index number 0 can be written into the bitstream.

[0353] Table 2

[0354] Index Number 0 1 2 3 4 5 6 7 Gaussian distribution parameters 0.2 0.25 0.3 0.4 0.5 0.8 1.6 4

[0355] In another possible implementation, a target probability distribution may be determined from a plurality of candidate probability distributions according to the target quality residual matrix, an index number of the target probability distribution may be written into the bitstream, and entropy coding may be performed on the target quality residual matrix according to the target probability distribution.

[0356] Exemplarily, a probability distribution with a high similarity to the probability distribution of the residual value can be determined from a plurality of preset candidate probability distributions according to the probability distribution of the residual value in the target quality residual matrix. The index number of the target probability distribution is written into the bitstream, and then the residual matrix is ​​entropy encoded using me-tANS based on the probability distribution obtained by the match.

[0357] Optionally, the above target probability distribution may also be written into the bitstream.

[0358] In another possible implementation, Gaussian distribution parameters of the target quality matrix may be determined, probability distribution of the target quality matrix may be determined according to the Gaussian distribution parameters, the Gaussian distribution parameters may be written into the bitstream, and entropy coding may be performed on the target quality matrix according to the probability distribution.

[0359] Optionally, the Gaussian distribution parameter index number may also be written into the bitstream.

[0360] In another possible implementation, a target probability distribution is determined from a plurality of candidate probability distributions according to the target quality matrix, an index number of the target probability distribution is written into the bitstream, and entropy coding is performed on the target quality matrix according to the target probability distribution.

[0361] Optionally, the above target probability distribution may also be written into the bitstream.

[0362] In a possible implementation, the encoding method provided by the embodiment of the present application may also include: inputting the above-mentioned image into the encoding network to obtain a first feature map. Inputting the above-mentioned first feature map into the context network to obtain a first prediction map. Determining the above-mentioned first residual map based on the above-mentioned first feature map and the above-mentioned first prediction map. Inputting the above-mentioned first feature map into the super-prior encoding network to obtain the first super-prior information. Quantizing the above-mentioned first super-prior information to obtain the second super-prior information. Inputting the above-mentioned second super-prior information into the super-prior variance decoding network to obtain the above-mentioned first Gaussian distribution parameter information.

[0363] Fig.13 An encoding end architecture is provided in an embodiment of the present application, and the above encoding method is described below in conjunction with the encoding end architecture.

[0364] like Fig.13 As shown, the input encoding network (analysis transform) of the image x input encoding end can obtain the feature map y (that is, the first feature map mentioned above). The image x can be the entire image, or the Y component or UV component of the YUV image.

[0365] The obtained feature map y is input into a hyper encoder to obtain hyper information z (i.e., the first hyper information mentioned above). The quantized z is entropy encoded by me-tANS and written into the bitstream.

[0366] The obtained super prior information z is quantized to obtain quantized super prior information z' (i.e., the second super prior information mentioned above). Fig.13 Not shown is that the quantized super-prior information z' can also be entropy-coded (me-tANS) and written into the bitstream.

[0367] Inputting the quantized hyper-prior information z' into a hyper-scale decoder network can obtain Gaussian distribution parameter information σ (ie, the first Gaussian distribution parameter information mentioned above).

[0368] The quantized hyper-prior information z' is input into the hyper-prior decoding network (hyper decoder).

[0369] The Gaussian distribution parameter information σ is input into the variance scaling (sigma scale) module through the above mass matrix Q m (i.e. the target mass matrix mentioned above) can be scaled to obtain Gaussian distribution parameter information σ' (i.e. the second Gaussian distribution parameter information mentioned above)

[0370] The obtained feature map y is input into the joint context network (MSM) to obtain the prediction map u of the feature map y (i.e. the first prediction map mentioned above).

[0371] By calculating the residual by the feature map y and the prediction map u, the residual map r (i.e. the first residual map mentioned above) can be obtained.

[0372] The obtained residual graph r is input into the gain unit module through the mass matrix Q m (i.e., the above-mentioned target mass matrix) can be scaled to obtain the residual graph r' (i.e., the above-mentioned second residual graph).

[0373] Optionally, the above encoding end architecture may be a JPEG AI encoding end architecture.

[0374] It can be seen that the embodiment of the present application improves the gain unit and / or variance scaling (sigma scale) module in the encoding end architecture so that the JPEG AI encoding end architecture supports spatial domain bit rate adjustment.

[0375] Please refer to Fig.14 , Fig.14 A decoding method is provided in an embodiment of the present application. Fig.14 As shown, the decoding method may include:

[0376] S1401: Obtain code stream.

[0377] The target quality matrix is ​​used to characterize the image quality of each region in the residual image of the feature domain.

[0378] S1402: Determine a target quality matrix according to the bitstream.

[0379] The target quality matrix is ​​used to characterize the image quality of each region in the residual image of the feature domain.

[0380] In a possible implementation, the bitstream may be decoded to obtain Gaussian distribution parameters of a target quality residual matrix of the target quality matrix. The probability distribution of the target quality residual matrix is ​​determined according to the Gaussian distribution parameters. The bitstream is decoded according to the probability distribution to obtain the target quality residual matrix. The target quality matrix is ​​determined according to the target quality residual matrix.

[0381] Optionally, the code stream may be decoded to obtain the index number of the Gaussian distribution parameter of the target quality residual matrix of the target quality matrix. For example, if the Gaussian distribution parameter index number is 0, a probability distribution with a probability distribution of 0.2 may be determined according to Table 2 above.

[0382] In another possible implementation, the bitstream may be decoded to obtain Gaussian distribution parameters of the target quality matrix, the probability distribution of the target quality matrix may be determined according to the Gaussian distribution parameters, and the bitstream may be decoded according to the probability distribution to obtain the target quality matrix.

[0383] In another possible implementation, the bitstream may be decoded to obtain an index number of a target probability distribution, where the target probability distribution is determined from a plurality of candidate probability distributions according to a target quality residual matrix of the target quality matrix. The bitstream is decoded according to the target probability distribution to obtain the target quality residual matrix. The target quality matrix is ​​determined according to the target quality residual matrix.

[0384] In another possible implementation, the bitstream may be decoded to obtain an index number of a target probability distribution, where the target probability distribution is determined from a plurality of candidate probability distributions according to the target quality matrix. The bitstream is decoded according to the target probability distribution to obtain the target quality matrix.

[0385] In a possible implementation manner, a target quality prediction matrix of a target quality matrix may be determined according to the target quality residual matrix, and the target quality matrix may be determined according to the target quality residual matrix and the target quality prediction matrix.

[0386] It should be noted that the specific method for determining the target quality prediction matrix of the target quality matrix may adopt any method conceivable by a person skilled in the art, and the embodiments of the present application are not limited thereto.

[0387] For example, the prediction matrix of the target mass matrix can be determined according to the target mass matrix, and then the target mass residual matrix can be determined according to the prediction matrix of the target mass matrix. Among them, the prediction value of any point in the prediction matrix of the target mass matrix can be calculated according to the left and upper adjacent points of the mass quantity corresponding to the prediction quantity of the point in the target mass matrix. In the case where only the upper adjacent point exists, the prediction value of the point can be calculated only according to the upper adjacent point of the point. In the case where only the left adjacent point exists, the prediction value of the point can be calculated only according to the left adjacent point of the point. In the case where only the left adjacent point and the upper adjacent point do not exist, the prediction value of the point can be determined to be 0.

[0388] Exemplarily, the predicted value Q_pred[i,j] of any point in the target quality prediction matrix may satisfy:

[0389] Q_pred[i,j]=(Q[i,j-1]+Q[i-1,j]) / 2

[0390] For example, the quality value Q[i,j] of any point in the target quality matrix may satisfy:

[0391] Q[i,j]=Q res [i,j]+Q_pred[i,j]

[0392] S1403 . Scale the third Gaussian distribution parameter information according to the target mass matrix to obtain fourth Gaussian distribution parameter information.

[0393] Among them, the above-mentioned third Gaussian distribution parameter information is the Gaussian distribution parameter information of the residual map of the above-mentioned feature domain.

[0394] In a possible implementation, a target mass scaling matrix may be determined according to the target mass matrix. A target mass scaling tensor may be determined according to the target mass scaling matrix. The third Gaussian distribution parameter information is scaled according to the target mass scaling tensor to obtain fourth Gaussian distribution parameter information. The target mass scaling matrix is ​​used to characterize the residual map of the feature domain and the scaling amount of each region in the Gaussian distribution parameter information. The target mass scaling tensor is used to characterize the residual map of the feature domain and the scaling tensor of the Gaussian distribution parameter information in three-dimensional space.

[0395] It should be noted that the specific implementation method of determining the target mass scaling matrix according to the target mass matrix can refer to the description of determining the target mass scaling matrix according to the target mass matrix in S1402, which will not be repeated here.

[0396] It can be understood that the feature map C×H×W three-dimensional tensor can be converted into a two-dimensional H×W quality scaling matrix Q through the target quality scaling matrix (and gain vector) m Expanded to a three-dimensional tensor.

[0397] In a possible implementation, the target quality scaling tensor is determined according to the target quality scaling matrix and a gain parameter, where the gain parameter includes a gain vector for channel-level quality adjustment and / or an image-level quality control factor.

[0398] For example, for the coded number domain, in the presence of a channel-level quality adjustment gain vector m t , and image-level quality control factor β displacement In the case of , the target mass scaling tensor m[c,i,j] can satisfy:

[0399] m[c,i,j]=Q m [i,j]·β displacement·m t [c]

[0400] or:

[0401] m[c,i,j]=(Q int [i,j]·β displacement >>denom)·m t [c]

[0402] or:

[0403] m[c,i,j]=(Q int [i,j]·β displacement ·m t [c])>>denom

[0404] For the encoding log domain, there is a gain vector m for channel-level quality adjustment t , and image-level quality control factor β displacement In the case of , the target mass scaling tensor m[c,i,j] can satisfy:

[0405] m[c,i,j]=Q m [i,j]·β displacement ·m t [c]

[0406] or:

[0407] m[c,i,j]=(Q int [i,j]·β displacement >>denom)·m t [c]

[0408] or:

[0409] m[c,i,j]=(Q int [i,j]·β displacement ·m t [c])>>denom

[0410] For encoding log domain, it is also necessary to additionally calculate the target quality scaling tensor m_log in the logarithmic domain. The target quality scaling tensor m_log in the logarithmic domain satisfies:

[0411] m_log[c,i,j]=Q m _log[i,j]+β displacement_log +m t_log [c]

[0412] For the encoded 3D map, there is a gain vector m for channel-level quality adjustment t , and image-level quality control factor β displacementIn the case of , the target mass scaling tensor m[c,i,j] can satisfy:

[0413] m[c,i,j]=Q m [c,i,j]·β displacement ·m t [c]

[0414] or:

[0415] m[c,i,j]=(Q int [c,i,j]·β displacement >>denom)·m t [c]

[0416] or:

[0417] m[c,i,j]=(Q int [c,i,j]·β displacement ·m t [c])>>denom

[0418] For encoding three-dimensional maps, it is also necessary to additionally calculate the target quality scaling tensor m_log in the logarithmic domain. The target quality scaling tensor m_log in the logarithmic domain satisfies:

[0419] m_log[c,i,j]=Q m _log[c,i,j]+β displacement_log +m t_log [c]

[0420] As another example, for the coding domain, in the presence of a frame-level quality control factor β displacement In the case of , the target mass scaling tensor m[c,i,j] can satisfy:

[0421] m[c,i,j]=Q m [i,j]·β displacement ,

[0422] or:

[0423] m[c,i,j]=(Q int [c,i,j]>>denom)·β displacement ,

[0424] or:

[0425] m[c,i,j]=(Q int [c,i,j]·β displacement )>>denom

[0426] For encoding in the log domain, there is a frame-level quality control factor β displacementIn the case of , the target mass scaling tensor m[c,i,j] can satisfy:

[0427] m[c,i,j]=Q m [i,j]·β displacement ,

[0428] or:

[0429] m[c,i,j]=(Q int [i,j]>>denom)·β displacement ,

[0430] or:

[0431] m[c,i,j]=(Q int [i,j]·β displacement )>>denom

[0432] For encoding log domain, it is also necessary to additionally calculate the target quality scaling tensor m_log in the logarithmic domain. The target quality scaling tensor m_log in the logarithmic domain satisfies:

[0433] m_log[c,i,j]=Q m _log[i,j]+β displacement_log

[0434] For encoding 3D maps, there is a frame-level quality control factor β displacement In the case of , the target mass scaling tensor m[c,i,j] can satisfy:

[0435] m[c,i,j]=Q m [c,i,j]·β displacement ,

[0436] or:

[0437] m[c,i,j]=(Q int [c,i,j]>>denom)·β displacement ,

[0438] or:

[0439] m[c,i,j]=(Q int [c,i,j]·β displacement )>>denom

[0440] For encoding three-dimensional maps, it is also necessary to additionally calculate the target quality scaling tensor m_log in the logarithmic domain. The target quality scaling tensor m_log in the logarithmic domain satisfies:

[0441] m_log[c,i,j]=Q m _log[c,i,j]+βdisplacement_log

[0442] In another possible implementation manner, the target mass scaling tensor may be determined according to the target mass scaling matrix.

[0443] Exemplarily, for the coded number domain, in the absence of other quality control factors (gain parameters), the target quality scaling tensor m[c,i,j] may satisfy:

[0444] m[c,i,j]=Q m [i,j]

[0445] or:

[0446] m[c,i,j]=Q int [i,j]>>denom

[0447] For encoding in the log domain, without other quality control factors (gain parameters), the target quality scaling tensor m[c,i,j] can satisfy:

[0448] m[c,i,j]=Q m [i,j]

[0449] or:

[0450] m[c,i,j]=Q int [i,j]>>denom

[0451] For encoding log domain, it is also necessary to additionally calculate the target quality scaling tensor m_log in the logarithmic domain. The target quality scaling tensor m_log in the logarithmic domain satisfies:

[0452] m_log[c,i,j]=Q m_log [i,j]

[0453] For the encoded 3D map, in the absence of other quality control factors (gain parameters), the target quality scaling tensor m[c,i,j] can satisfy:

[0454] m[c,i,j]=Q m [c,i,j]

[0455] or:

[0456] m[c,i,j]=Q int [c,i,j]>>denom

[0457] For encoding three-dimensional maps, it is also necessary to additionally calculate the target quality scaling tensor m_log in the logarithmic domain. The target quality scaling tensor m_log in the logarithmic domain satisfies:

[0458] m_log[c,i,j]=Q m _ log [c,i,j]

[0459] In a possible implementation, for the coded number field, the fourth Gaussian distribution parameter information (fourth variance) σ' may satisfy:

[0460] σ'[c,i,j]=(m[c,i,j]·σ[c,i,j])>>(Precision2)

[0461] Wherein, σ is the parameter information of the third Gaussian distribution (third party difference), and Precision2 can be any non-negative number. For example, Precision2 can be a non-negative number such as 0, 1, or 13.

[0462] In a possible implementation, for the encoded log domain and the encoded three-dimensional map, the fourth Gaussian distribution parameter information σ' may satisfy:

[0463] σ'[c,i,j]=m log[c,i,j] +σ[c,i,j]-Precision2_log

[0464] Wherein, σ is the parameter information of the third Gaussian distribution, and Precision2 can be any non-negative number. For example, Precision2 can be a non-negative number such as 0, 1, or 13.

[0465] S1404: Decode the bitstream according to the fourth Gaussian distribution parameter information to obtain a third residual image.

[0466] Among them, the third residual map is the residual map of the feature domain.

[0467] S1405 . Dequantize the third residual image according to the target quality matrix to obtain a fourth residual image.

[0468] In a possible implementation, a target mass scaling matrix may be determined according to the target mass matrix, a target mass inverse scaling tensor may be determined according to the target mass scaling matrix, and the third residual map may be inversely quantized according to the target mass inverse scaling tensor to obtain a fourth residual map.

[0469] In a possible implementation, the target quality inverse scaling tensor is determined according to the target quality scaling matrix and a gain parameter, where the gain parameter includes a gain vector for channel-level quality adjustment and / or an image-level quality control factor.

[0470] For example, in the presence of a channel-level quality-adjusted gain vector m t , and image-level quality control factor β displacementIn the case of -1 [c,i,j] can satisfy:

[0471] m -1 [c,i,j]=Q m -1 [i,j]·1 / β displacement ·m -1 t [c]

[0472] or:

[0473] m -1 [c,i,j]=(Q int -1 [i,j]·1 / β displacement )<<denom·m -1 t [c]

[0474] or:

[0475] m -1 [c,i,j]=(Q int -1 [i,j]·1 / β displacement ·m -1 t [c])<<denom

[0476] As another example, in the presence of a frame-level quality control factor β displacement In the case of -1 [c,i,j] can satisfy:

[0477] m -1 [c,i,j]=Q m -1 [i,j]·1 / β displacement

[0478] or:

[0479] m -1 [c,i,j]=(Q int -1 [i,j]·1 / β displacement )<<denom

[0480] or:

[0481] m -1 [c,i,j]=(Q int -1 [i,j]<<denom)·1 / β displacement

[0482] In another possible implementation manner, the target mass inverse scaling tensor may be determined according to the target mass scaling matrix.

[0483] Exemplarily, in the absence of other quality control factors (gain parameters), the target quality inverse scales the tensor m -1 [c,i,j] can satisfy:

[0484] m -1 [c,i,j]=Q m -1 [i,j]

[0485] or:

[0486] m -1 [c,i,j]=Q int -1 [i,j]<<denom

[0487] In a possible implementation, the fourth residual graph r'[c,i,j] may satisfy:

[0488] r'[c,i,j]=(m -1 [c,i,j]·r[c,i,j])<<(Precision)

[0489] Where r[c,i,j] is the third residual graph, and Precision can be any non-negative number. For example, Precision can be a non-negative number such as 0, 1, or 13.

[0490] S1406: Determine a reconstructed image according to the fourth residual image.

[0491] Exemplarily, the fourth residual image may be input into a decoding network (Synthesis Transform) to obtain a reconstructed image, wherein the reconstructed image may be a complete image or a Y component or UV component of a YUV image.

[0492] Please refer to Fig.15 , Fig.15 Another decoding method provided by the embodiment of the present application. Fig.15 As shown, the decoding method may include:

[0493] S1501: Obtain code stream.

[0494] S1502: Determine a target quality matrix according to the bitstream.

[0495] The target quality matrix is ​​used to characterize the image quality of each region in the residual image of the feature domain.

[0496] S1503 . Scale the third Gaussian distribution parameter information according to the target mass matrix to obtain fourth Gaussian distribution parameter information.

[0497] Among them, the above-mentioned third Gaussian distribution parameter information is the Gaussian distribution parameter information of the residual map of the above-mentioned feature domain.

[0498] S1504: Decode the bitstream according to the fourth Gaussian distribution parameter information to obtain a third residual image.

[0499] Among them, the third residual map is the residual map of the feature domain.

[0500] S1505: Determine a reconstructed image according to the third residual image.

[0501] Please refer to Fig.16 , Fig.16 Another decoding method provided in the embodiment of the present application. Fig.16 As shown, the decoding method may include:

[0502] S1601. Obtain code stream.

[0503] S1602: Determine a target quality matrix according to the bitstream.

[0504] The target quality matrix is ​​used to characterize the image quality of each region in the residual image of the feature domain.

[0505] S1603: Decode the bitstream according to the third Gaussian distribution parameter information to obtain a third residual image.

[0506] Among them, the above-mentioned third Gaussian distribution parameter information is the Gaussian distribution parameter information of the residual map of the above-mentioned feature domain, and the above-mentioned third residual map is the residual map of the above-mentioned feature domain.

[0507] S1604 . Dequantize the third residual image according to the target quality matrix to obtain a fourth residual image.

[0508] S1605: Determine a reconstructed image according to the fourth residual image.

[0509] Fig.17 A decoding end architecture is provided in an embodiment of the present application, and the above encoding method is described below in conjunction with the encoding end architecture.

[0510] like Fig.17 As shown, the feature map y can be decoded from the bit stream at the input decoding end.

[0511] The obtained feature map y is input into a hyper scale decoder to obtain Gaussian distribution parameter information (i.e., the third Gaussian distribution parameter information mentioned above).

[0512] The obtained Gaussian distribution parameter information is input into the variance scaling (sigma scale) module through the mass matrix Q m (ie, the target mass matrix) can be scaled to obtain the scaled Gaussian distribution parameter information (ie, the fourth Gaussian distribution parameter information).

[0513] According to the scaled Gaussian distribution parameter information, a residual image (ie, the third residual image) can be obtained by decoding the bit stream input to the decoding end.

[0514] The obtained residual image is input into an inv gain unit module to perform scaling and inverse quantization on the residual image, thereby obtaining an inverse quantized residual image (ie, the fourth residual image mentioned above).

[0515] The fourth residual image is input into the decoding network (Synthesis Transform) to obtain a reconstructed image, wherein the reconstructed image can be a complete image or a Y component or UV component of a YUV image.

[0516] Optionally, the above decoding end architecture may be a JPEGAI decoding end architecture.

[0517] It can be seen that the embodiments of the present application improve the inv gain unit and / or the sigma scale module in the decoding end architecture so that the JPEG AI decoding end architecture supports spatial domain bit rate adjustment.

[0518] The following will be combined Fig.18 A coding device for executing the above coding method is introduced.

[0519] It is understandable that, in order to realize the above functions, the encoding device includes hardware and / or software modules corresponding to the execution of each function. In combination with the algorithm steps of each example described in the embodiments disclosed herein, the embodiments of the present application can be implemented in the form of hardware or a combination of hardware and computer software. Whether a function is executed in the form of hardware or computer software driving hardware depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application in combination with the embodiments, but such implementation should not be considered to exceed the scope of the embodiments of the present application.

[0520] The embodiment of the present application can divide the functional modules of the encoding device according to the above method example. For example, each functional module can be divided corresponding to each function, or two or more functions can be integrated into one processing module. The above integrated module can be implemented in the form of hardware. It should be noted that the division of modules in this embodiment is schematic and is only a logical function division. There may be other division methods in actual implementation.

[0521] In the case of dividing each functional module into corresponding functional modules, Fig.18 A possible schematic diagram of the composition of the encoding device involved in the above embodiment is shown. Fig.18 As shown, the encoding device 1800 may include: a transceiver unit 1801 and a processing unit 1802.

[0522] The transceiver unit 1801 is used to obtain a target quality matrix, and the target quality matrix is ​​used to characterize the image quality of each region in the residual image of the feature domain.

[0523] The processing unit 1802 is used to scale the first residual map and / or the first Gaussian distribution parameter information according to the target quality matrix to obtain a second residual map and / or second Gaussian distribution parameter information, wherein the first residual map is the residual map of the feature domain, and the first Gaussian distribution parameter information is the Gaussian distribution parameter information of the residual map of the feature domain.

[0524] The processing unit 1802 is further configured to generate a bitstream according to the target quality matrix, the second residual map and / or the second Gaussian distribution parameter information.

[0525] In a possible implementation, the processing unit 1802 is specifically used to: determine a target quality scaling matrix according to the target quality matrix, the target quality scaling matrix is ​​used to characterize the scaling amount of each region in the residual map of the feature domain and / or the Gaussian distribution parameter information. Determine a target quality scaling tensor according to the target quality scaling matrix, the target quality scaling tensor is used to characterize the scaling tensor of the residual map of the feature domain and / or the Gaussian distribution parameter information in three-dimensional space. Scaling the first residual map according to the target quality scaling tensor to obtain the second residual map and / or scaling the first Gaussian distribution parameter information according to the target quality scaling tensor to obtain the second Gaussian distribution parameter information.

[0526] In a possible implementation, the processing unit 1802 is specifically configured to determine the target quality scaling tensor according to the target quality scaling matrix and a gain parameter, wherein the gain parameter includes a gain vector for channel-level quality adjustment and / or an image-level quality control factor.

[0527] In a possible implementation manner, the processing unit 1802 is specifically configured to: encode the target quality matrix and the second residual map to generate a bitstream, wherein the second residual map is encoded according to the second Gaussian distribution parameter information.

[0528] In a possible implementation manner, the processing unit 1802 is specifically configured to: encode the target quality matrix or the target quality residual matrix of the target quality matrix to generate a bitstream.

[0529] In a possible implementation, the processing unit 1802 is further configured to: generate a target quality residual matrix of the target quality matrix according to the target quality matrix, wherein the target quality residual matrix is ​​used to represent a residual value of each image quality in the target quality matrix.

[0530] In a possible implementation, the processing unit 1802 is specifically configured to: determine Gaussian distribution parameters of the target quality residual matrix, determine probability distribution of the target quality residual matrix according to the Gaussian distribution parameters, write the Gaussian distribution parameters into the bitstream, and perform entropy coding on the target quality residual matrix according to the probability distribution.

[0531] In a possible implementation, the processing unit 1802 is specifically configured to: determine Gaussian distribution parameters of the target quality matrix, determine probability distribution of the target quality matrix according to the Gaussian distribution parameters, write the Gaussian distribution parameters into the bitstream, and perform entropy coding on the target quality matrix according to the probability distribution.

[0532] In a possible implementation, the processing unit 1802 is specifically configured to: determine a target probability distribution from multiple candidate probability distributions according to the target quality residual matrix, write the index number of the target probability distribution into the bitstream, and perform entropy coding on the target quality residual matrix according to the target probability distribution.

[0533] In a possible implementation, the processing unit 1802 is specifically configured to: determine a target probability distribution from a plurality of candidate probability distributions according to the target quality matrix, write the index number of the target probability distribution into the bitstream, and perform entropy coding on the target quality matrix according to the target probability distribution.

[0534] In a possible implementation, the acquisition unit 1801 is specifically used to: acquire a quality map, the quality map being used to record the image quality of each region in the residual map of the feature domain, and determine the target quality matrix according to the quality map.

[0535] In a possible implementation, the processing unit 1802 is further used to: input the image into a coding network to obtain a first feature map. Input the first feature map into a context network to obtain a first prediction map. Determine the first residual map according to the first feature map and the first prediction map. Input the first feature map into a super-prior coding network to obtain first super-prior information. Quantize the first super-prior information to obtain second super-prior information. Input the second super-prior information into a super-prior variance decoding network to obtain the first Gaussian distribution parameter information.

[0536] The following will be combined Fig.19 A decoding device for executing the above decoding method is introduced.

[0537] It is understandable that, in order to implement the above functions, the decoding device includes hardware and / or software modules corresponding to the execution of each function. In combination with the algorithm steps of each example described in the embodiments disclosed herein, the embodiments of the present application can be implemented in the form of hardware or a combination of hardware and computer software. Whether a function is executed in the form of hardware or computer software driving hardware depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application in combination with the embodiments, but such implementation should not be considered to exceed the scope of the embodiments of the present application.

[0538] The embodiment of the present application can divide the functional modules of the decoding device according to the above method example. For example, each functional module can be divided according to each function, or two or more functions can be integrated into one processing module. The above integrated module can be implemented in the form of hardware. It should be noted that the division of modules in this embodiment is schematic and is only a logical function division. There may be other division methods in actual implementation.

[0539] In the case of dividing each functional module into corresponding functional modules, Fig.19 A possible schematic diagram of the composition of the decoding device involved in the above embodiment is shown. Fig.19 As shown, the decoding device 1900 may include: a transceiver unit 1901 and a processing unit 1902 .

[0540] The above-mentioned transceiver unit 1901 is used to obtain a code stream.

[0541] The processing unit 1902 is used to determine a target quality matrix according to the bitstream, and the target quality matrix is ​​used to characterize the image quality of each region in the residual image of the feature domain.

[0542] The processing unit 1902 is further configured to scale the third Gaussian distribution parameter information according to the target mass matrix to obtain fourth Gaussian distribution parameter information, wherein the third Gaussian distribution parameter information is the Gaussian distribution parameter information of the residual map of the feature domain.

[0543] The processing unit 1902 is further configured to decode the bitstream according to the fourth Gaussian distribution parameter information to obtain a third residual image, where the third residual image is a residual image of the feature domain.

[0544] The processing unit 1902 is further configured to perform inverse quantization on the third residual image according to the target quality matrix to obtain a fourth residual image.

[0545] The processing unit 1902 is further configured to determine a reconstructed image according to the fourth residual image.

[0546] In a possible implementation, the processing unit 1902 is specifically used to: determine a target mass scaling matrix according to the target mass matrix, the target mass scaling matrix is ​​used to characterize the residual map of the feature domain and the scaling amount of each region in the Gaussian distribution parameter information. Determine a target mass scaling tensor according to the target mass scaling matrix, the target mass scaling tensor is used to characterize the residual map of the feature domain and the scaling tensor of the Gaussian distribution parameter information in three-dimensional space. Scale the third Gaussian distribution parameter information according to the target mass scaling tensor to obtain fourth Gaussian distribution parameter information.

[0547] In a possible implementation, the processing unit 1902 is specifically used to determine the target quality scaling tensor according to the target quality scaling matrix and gain parameters, where the gain parameters include a gain vector for channel-level quality adjustment and / or an image-level quality control factor.

[0548] In a possible implementation, the processing unit 1902 is specifically configured to: decode the bitstream to obtain Gaussian distribution parameters of a target quality residual matrix of the target quality matrix. Determine the probability distribution of the target quality residual matrix according to the Gaussian distribution parameters. Decode the bitstream according to the probability distribution to obtain the target quality residual matrix. Determine the target quality matrix according to the target quality residual matrix.

[0549] In a possible implementation, the processing unit 1902 is specifically configured to: decode the bitstream to obtain Gaussian distribution parameters of the target quality matrix, determine the probability distribution of the target quality matrix according to the Gaussian distribution parameters, and decode the bitstream according to the probability distribution to obtain the target quality matrix.

[0550] In a possible implementation, the processing unit 1902 is specifically configured to: decode the bitstream to obtain an index number of a target probability distribution, where the target probability distribution is determined from a plurality of candidate probability distributions according to a target quality residual matrix of the target quality matrix. Decode the bitstream according to the target probability distribution to obtain the target quality residual matrix. Determine the target quality matrix according to the target quality residual matrix.

[0551] In a possible implementation, the processing unit 1902 is specifically configured to: decode the bitstream to obtain an index number of a target probability distribution, where the target probability distribution is determined from a plurality of candidate probability distributions according to the target quality matrix. Decode the bitstream according to the target probability distribution to obtain the target quality matrix.

[0552] In a possible implementation, the processing unit 1902 is further configured to: decode the bitstream to obtain a second feature map, and input the second feature map into a super priori variance decoding network to obtain the third Gaussian distribution parameter information.

[0553] In the case of dividing each functional module into corresponding functional modules, Fig. 20 Another possible schematic diagram of the composition of the decoding device involved in the above embodiment is shown. Fig. 20 As shown, the decoding device 2000 may include: a transceiver unit 2001 and a processing unit 2002.

[0554] The above-mentioned transceiver unit 2001 is used to obtain a code stream.

[0555] The processing unit 2002 is used to determine a target quality matrix according to the bitstream, and the target quality matrix is ​​used to characterize the image quality of each region in the residual image of the feature domain.

[0556] The processing unit 2002 is further configured to scale the third Gaussian distribution parameter information according to the target mass matrix to obtain fourth Gaussian distribution parameter information, wherein the third Gaussian distribution parameter information is the Gaussian distribution parameter information of the residual map of the feature domain.

[0557] The processing unit 2002 is further configured to decode the bitstream according to the fourth Gaussian distribution parameter information to obtain a third residual image, where the third residual image is a residual image of the feature domain.

[0558] The processing unit 2002 is further configured to determine a reconstructed image according to the third residual image.

[0559] In the case of dividing each functional module into corresponding functional modules, Fig.21 Another possible schematic diagram of the composition of the decoding device involved in the above embodiment is shown. Fig.21 As shown, the decoding device 2100 may include: a transceiver unit 2101 and a processing unit 2102 .

[0560] The above-mentioned transceiver unit 2101 is used to obtain a code stream.

[0561] The processing unit 2102 is used to determine a target quality matrix according to the bitstream, and the target quality matrix is ​​used to characterize the image quality of each region in the residual image of the feature domain.

[0562] The processing unit 2102 is further used to decode the bitstream according to third Gaussian distribution parameter information to obtain a third residual image, wherein the third Gaussian distribution parameter information is Gaussian distribution parameter information of the residual image of the feature domain, and the third residual image is the residual image of the feature domain.

[0563] The processing unit 2102 is further configured to perform inverse quantization on the third residual image according to the target quality matrix to obtain a fourth residual image.

[0564] The processing unit 2102 is further configured to determine a reconstructed image according to the fourth residual image.

[0565] An embodiment of the present application also provides an encoding device, which includes: at least one processor, when the at least one processor executes program code or instructions, it implements the above-mentioned related method steps to implement the encoding method in the above-mentioned embodiment.

[0566] Optionally, the device may further include at least one memory, and the at least one memory is used to store the program code or instruction.

[0567] An embodiment of the present application also provides a decoding device, which includes: at least one processor, when the at least one processor executes program code or instructions, it implements the above-mentioned related method steps to implement the decoding method in the above embodiment.

[0568] Optionally, the device may further include at least one memory, and the at least one memory is used to store the program code or instruction.

[0569] The embodiment of the present application also provides a code stream, which includes: a target quality matrix, a third residual map and a third Gaussian distribution parameter, wherein the target quality matrix is ​​used to characterize the image quality of each region in the residual map of the feature domain, the third residual map is the residual map of the feature domain, and the third Gaussian distribution parameter information is the Gaussian distribution parameter information of the residual map of the feature domain. The target quality matrix is ​​used to scale the third Gaussian distribution parameter information to obtain fourth Gaussian distribution parameter information. The target quality matrix is ​​also used to dequantize the third residual map to obtain a fourth residual map.

[0570] An embodiment of the present application also provides a computer storage medium, in which computer instructions are stored. When the computer instructions are executed on an encoding device, the encoding device executes the above-mentioned related method steps to implement the encoding and decoding method in the above-mentioned embodiment.

[0571] The embodiment of the present application also provides a computer program product. When the computer program product is run on a computer, the computer is enabled to execute the above-mentioned related steps to implement the encoding and decoding method in the above-mentioned embodiment.

[0572] The embodiment of the present application also provides a coding and decoding device, which can be a chip, an integrated circuit, a component or a module. Specifically, the device may include a connected processor and a memory for storing instructions, or the device includes at least one processor for obtaining instructions from an external memory. When the device is running, the processor can execute instructions so that the chip executes the coding and decoding methods in the above-mentioned method embodiments.

[0573] Fig. 22 2 shows a schematic diagram of the structure of a chip 2300. The chip 2200 includes one or more processors 2201 and an interface circuit 2202. Optionally, the chip 2200 may also include a bus 2203.

[0574] The processor 2201 may be an integrated circuit chip with signal processing capability. In the implementation process, each step of the above encoding and decoding method may be completed by an integrated logic circuit of hardware in the processor 2201 or by instructions in the form of software.

[0575] Optionally, the processor 2201 may be a general purpose processor, a digital signal processing (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. The methods and steps disclosed in the embodiments of the present application may be implemented or executed. The general purpose processor may be a microprocessor or the processor may be any conventional processor, etc.

[0576] The interface circuit 2202 can be used to send or receive data, instructions or information. The processor 2201 can use the data, instructions or other information received by the interface circuit 2202 to process, and can send the processing completion information through the interface circuit 2202.

[0577] Optionally, the chip also includes a memory, which may include a read-only memory and a random access memory, and provides operation instructions and data to the processor. A portion of the memory may also include a non-volatile random access memory (NVRAM).

[0578] Optionally, the memory stores executable software modules or data structures, and the processor can perform corresponding operations by calling operation instructions stored in the memory (the operation instructions can be stored in the operating system).

[0579] Optionally, the chip can be used in the encoding device or DOP involved in the embodiment of the present application. Optionally, the interface circuit 2202 can be used to output the execution result of the processor 2201. The encoding and decoding method provided by one or more embodiments of the embodiment of the present application can refer to the aforementioned embodiments, which will not be repeated here.

[0580] It should be noted that the corresponding functions of the processor 2201 and the interface circuit 2202 can be implemented through hardware design, software design, or a combination of hardware and software, and there is no limitation here.

[0581] Please refer to Fig.23 , Fig.23 The general concept of processing by a neural network such as a convolutional neural network (CNN) is schematically illustrated. A convolutional neural network consists of an input layer and an output layer and multiple hidden layers. The input layer is where the input (e.g. Fig.23 The hidden layers of a CNN are usually composed of a series of convolutional layers that perform convolutions with multiplications or other dot products. The result of the layer is one or more feature maps (represented by empty solid rectangles), sometimes also called channels. Resampling (such as subsampling) may be involved in some or all layers. As a result, the feature maps may become smaller, such as Fig.23 As shown. Note that convolution with stride can also reduce the size of the input feature map (resampling). The activation function in a CNN is typically a ReLU (Rectified Linear Unit) layer, followed by additional convolutions such as pooling layers, fully connected layers, and normalization layers, called hidden layers because their inputs and outputs are masked by the activation function and the final convolution. Although these layers are colloquially called convolutions, this is just convention. Mathematically speaking, it is technically a sliding dot product or cross correlation. This has important implications for the indexing in the matrix because it affects how the weights are determined at a specific index point.

[0582] When programming a CNN for processing images, such as Fig.23As shown, the input is a tensor of shape (number of images) x (image width) x (image height) x (image depth). It should be noted that the image depth can be constructed by the channels of the image. After passing through the convolution layer, the image is abstracted into a feature map with shape (number of images) x (feature map width) x (feature map height) x (feature map channels). The convolution layer in a neural network should have the following properties. The convolution kernel defined by width and height (hyperparameters). The number of input channels and output channels (hyperparameters). The depth of the convolution filter (input channels) should be equal to the number of channels of the input feature map (depth).

[0583] Video Coding Machines (VCM) is another popular direction in computer science today. The main idea behind this approach is to transmit coded representations of image or video information for further processing by computer vision (CV) algorithms, such as object segmentation, detection, and recognition. In contrast to traditional image and video coding, which targets human perception, the quality features are the performance of computer vision tasks, such as object detection accuracy, rather than reconstruction quality. Fig.24 shown.

[0584] Machine video coding, also known as collaborative intelligence, is a relatively new paradigm for efficiently deploying deep neural networks in mobile cloud infrastructure. By dividing the network between the mobile side 2410 and the cloud side 2490 (e.g., cloud servers), the computing workload can be distributed so that the total energy and / or latency of the system is minimized. In general, collaborative intelligence is a paradigm in which the processing of a neural network is distributed between two or more different computing nodes; for example, devices, but in general, any functionally defined nodes. Here, the term "node" does not refer to the neural network nodes mentioned above. Instead, the (computing) nodes here refer to (physically or at least logically) independent devices / modules that implement parts of the neural network. Such devices can be different servers, different end-user devices, a mixture of servers and / or user devices and / or clouds and / or processors, etc. In other words, the computing nodes can be considered as nodes belonging to the same neural network and communicate with each other to transmit encoded data within / for the neural network. For example, in order to be able to perform complex calculations, one or more layers can be executed on a first device (such as a device on the mobile side 2410) and one or more layers can be executed in another device (such as a cloud server on the cloud side 2490). However, the distribution can also be finer, and a single layer can be executed on multiple devices. In this disclosure, the term "multiple" means two or more. In some existing solutions, part of the neural network functionality is executed in a device (user device or edge device, etc.) or multiple such devices, and the output (feature map) is then passed to the cloud. The cloud is a collection of processing or computing systems located outside the device that is operating a part of the neural network. The concept of collaborative intelligence has also been extended to model training. In this case, data flows in both directions: from the cloud to the mobile device during the backpropagation of training, and from the mobile device to the cloud during the forward pass and inference of training (such as Fig.24 shown).

[0585] Some works have proposed semantic image compression by encoding deep features and then reconstructing the input image from them. Compression based on uniform quantization followed by context-based adaptive arithmetic coding (CABAC) from H.264 was demonstrated. In some scenarios, it may be more efficient to send the output of the hidden layer (deep feature map) from the mobile part 2410 to the cloud 2490, rather than sending compressed natural image data to the cloud and performing object detection using the reconstructed image. Therefore, it may be advantageous to compress the data (features) generated by the mobile side 2410, which may include a quantization layer 2420 for this purpose. Accordingly, the cloud side 2490 may include an inverse quantization layer 2460. Efficient compression of feature maps facilitates image and video compression and reconstruction for human perception and machine vision. Entropy coding methods, such as arithmetic coding, are a popular method for compressing deep features (i.e., feature maps).

[0586] Please refer to Fig.25 , Fig.25 A code stream structure provided by an embodiment of the present application is shown, such as Fig.25 The code stream shown includes: image start point (Start of Image), file header (File Header), entropy encoded data (Entropy Encoded Data) and image end point (End of Image).

[0587] In a possible implementation, the Gaussian distribution parameter index number, the Gaussian distribution parameter, the target probability distribution, and the target probability distribution index number may be stored in a file header.

[0588] In a possible implementation manner, the target quality residual matrix or the target quality matrix may be stored in entropy coded data.

[0589] Among them, the device, computer storage medium, computer program product or chip provided in this embodiment is used to execute the corresponding method provided above. Therefore, the beneficial effects that can be achieved can refer to the beneficial effects in the corresponding method provided above, and will not be repeated here.

[0590] It should be understood that in various embodiments of the present application, the size of the serial numbers of the above-mentioned processes does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present application.

[0591] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the embodiments of the present application.

[0592] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0593] In the several embodiments provided in the embodiments of the present application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the above units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0594] The units described above as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0595] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0596] If the above functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the embodiment of the present application is essentially or the part that contributes to the prior art or the part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the above methods of each embodiment of the embodiment of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (Read Only Memory, ROM), random access memory (Random Access Memory, RAM), disk or optical disk and other media that can store program codes.

[0597] The above is only a specific implementation of the embodiment of the present application, but the protection scope of the embodiment of the present application is not limited thereto. Any technician familiar with the technical field can easily think of changes or replacements within the technical scope disclosed in the embodiment of the present application, which should be included in the protection scope of the embodiment of the present application. Therefore, the protection scope of the embodiment of the present application should be based on the protection scope of the claims.

Claims

1. A coding method, characterized in that: include: Obtaining a target quality matrix, where the target quality matrix is ​​used to characterize the image quality of each region in the residual image of the feature domain; Scaling the first residual map and / or the first Gaussian distribution parameter information according to the target mass matrix to obtain a second residual map and / or the second Gaussian distribution parameter information, wherein the first residual map is the residual map of the feature domain, and the first Gaussian distribution parameter information is the Gaussian distribution parameter information of the residual map of the feature domain; A bitstream is generated according to the target quality matrix, the second residual map and / or the second Gaussian distribution parameter information.

2. The method according to claim 1, characterized in that Scaling the first residual map and the first Gaussian distribution parameter information according to the target mass matrix to obtain the second residual map and the second Gaussian distribution parameter information includes: Determine a target quality scaling matrix according to the target quality matrix, wherein the target quality scaling matrix is ​​used to characterize a scaling amount of each region in the residual map of the feature domain and / or the Gaussian distribution parameter information; Determine a target mass scaling tensor according to a target mass scaling matrix, wherein the target mass scaling tensor is used to characterize a residual map of the feature domain and / or a scaling tensor of the Gaussian distribution parameter information in a three-dimensional space; The first residual map is scaled according to the target quality scaling tensor to obtain the second residual map and / or the first Gaussian distribution parameter information is scaled according to the target quality scaling tensor to obtain the second Gaussian distribution parameter information.

3. The method according to claim 2, characterized in that Determining the target mass scaling tensor according to the target mass scaling matrix includes: The target quality scaling tensor is determined according to the target quality scaling matrix and a gain parameter, wherein the gain parameter includes a gain vector for channel-level quality adjustment and / or an image-level quality control factor.

4. The encoding method according to any one of claims 1 to 3, characterized in that: The generating a bitstream according to the second residual map, the second Gaussian distribution parameter information and the target quality matrix includes: The target quality matrix and the second residual map are encoded to generate a bitstream, wherein the second residual map is encoded according to the second Gaussian distribution parameter information.

5. The method according to claim 4, characterized in that The step of encoding the target quality matrix to generate a bitstream includes: The target quality matrix or a target quality residual matrix of the target quality matrix is ​​encoded to generate a bitstream, wherein the target quality residual matrix is ​​used to represent a residual value of each image quality in the target quality matrix.

6. The method according to claim 5, characterized in that The method further comprises: A target mass residual matrix of the target mass matrix is ​​generated according to the target mass matrix.

7. The method according to claim 5 or 6, characterized in that: The step of encoding a target quality residual matrix of the target quality matrix to generate a bitstream includes: Determining Gaussian distribution parameters of the target quality residual matrix; Determining a probability distribution of the target quality residual matrix according to the Gaussian distribution parameters; Writing the Gaussian distribution parameters into the bit stream; The target quality residual matrix is ​​entropy encoded according to the probability distribution.

8. The method according to claim 5 or 6, characterized in that: The step of encoding the target quality matrix to generate a bitstream includes: Determining Gaussian distribution parameters of the target mass matrix; Determining a probability distribution of the target mass matrix according to the Gaussian distribution parameters; Writing the Gaussian distribution parameters into the bit stream; The target quality matrix is ​​entropy encoded according to the probability distribution.

9. The method according to claim 5 or 6, characterized in that: The step of encoding a target quality residual matrix of the target quality matrix to generate a bitstream includes: determining a target probability distribution from a plurality of candidate probability distributions according to the target quality residual matrix; Writing the index number of the target probability distribution into the bitstream; The target quality residual matrix is ​​entropy encoded according to the target probability distribution.

10. The method according to claim 5 or 6, characterized in that: The step of encoding the target quality matrix to generate a bitstream includes: determining a target probability distribution from a plurality of candidate probability distributions according to the target mass matrix; Writing the index number of the target probability distribution into the bitstream; The target quality matrix is ​​entropy encoded according to the target probability distribution.

11. The method according to any one of claims 1 to 10, characterized in that The obtaining of the target mass matrix comprises: Acquire a quality map, where the quality map is used to record the image quality of each region in the residual map of the feature domain; The target quality matrix is ​​determined according to the quality map.

12. The method according to any one of claims 1 to 11, characterized in that The method further comprises: Inputting the image into an encoding network to obtain a first feature map; Inputting the first feature map into a context network to obtain a first prediction map; Determine the first residual map according to the first feature map and the first prediction map; Inputting the first feature map into a super-prior coding network to obtain first super-prior information; quantizing the first super a priori information to obtain second super a priori information; The second super-prior information is input into a super-prior variance decoding network to obtain the first Gaussian distribution parameter information.

13. A decoding method, characterized in that: include: Get the code stream; Determine a target quality matrix according to the bitstream, wherein the target quality matrix is ​​used to characterize the image quality of each region in the residual image of the feature domain; Scaling the third Gaussian distribution parameter information according to the target mass matrix to obtain fourth Gaussian distribution parameter information, wherein the third Gaussian distribution parameter information is Gaussian distribution parameter information of the residual graph of the feature domain; Decoding the bitstream according to the fourth Gaussian distribution parameter information to obtain a third residual map, where the third residual map is a residual map of the feature domain; Dequantizing the third residual map according to the target quality matrix to obtain a fourth residual map; A reconstructed image is determined according to the fourth residual image.

14. The method according to claim 13, characterized in that Scaling the third Gaussian distribution parameter information according to the target mass matrix to obtain fourth Gaussian distribution parameter information includes: Determine a target quality scaling matrix according to the target quality matrix, wherein the target quality scaling matrix is ​​used to characterize the scaling amount of each region in the residual map of the feature domain and the Gaussian distribution parameter information; Determine a target mass scaling tensor according to a target mass scaling matrix, wherein the target mass scaling tensor is used to characterize a scaling tensor of a residual map of the feature domain and the Gaussian distribution parameter information in a three-dimensional space; The third Gaussian distribution parameter information is scaled according to the target mass scaling tensor to obtain fourth Gaussian distribution parameter information.

15. The method according to claim 14, characterized in that The determining a target mass scaling matrix according to the target mass matrix comprises: The target quality scaling tensor is determined according to the target quality scaling matrix and a gain parameter, wherein the gain parameter includes a gain vector for channel-level quality adjustment and / or an image-level quality control factor.

16. The method according to any one of claims 13 to 15, characterized in that The determining a target quality matrix according to the bitstream includes: Decoding the bitstream to obtain Gaussian distribution parameters of a target quality residual matrix of a target quality matrix; Determining a probability distribution of the target quality residual matrix according to the Gaussian distribution parameters; Decoding the bitstream according to the probability distribution to obtain the target quality residual matrix; The target mass matrix is ​​determined according to the target mass residual matrix.

17. The method according to any one of claims 13 to 15, characterized in that The determining a target quality matrix according to the bitstream includes: Decoding the bitstream to obtain Gaussian distribution parameters of a target quality matrix; Determining a probability distribution of the target mass matrix according to the Gaussian distribution parameters; The code stream is decoded according to the probability distribution to obtain the target quality matrix.

18. The method according to any one of claims 13 to 15, characterized in that The determining a target quality matrix according to the bitstream includes: Decoding the bitstream to obtain an index number of a target probability distribution, where the target probability distribution is determined from a plurality of candidate probability distributions according to a target quality residual matrix of the target quality matrix; Decoding the bitstream according to the target probability distribution to obtain the target quality residual matrix; The target mass matrix is ​​determined according to the target mass residual matrix.

19. The method according to any one of claims 13 to 15, wherein determining a target quality matrix according to the bitstream comprises: Decoding the bitstream to obtain an index number of a target probability distribution, wherein the target probability distribution is determined from a plurality of candidate probability distributions according to the target quality matrix; The code stream is decoded according to the target probability distribution to obtain the target quality matrix.

20. The method according to any one of claims 13 to 19, characterized in that The method further comprises: Decoding the code stream to obtain a second feature map; The second feature map is input into a super prior variance decoding network to obtain the third Gaussian distribution parameter information.

21. A decoding method, characterized in that: include: Get the code stream; Determine a target quality matrix according to the bitstream, wherein the target quality matrix is ​​used to characterize the image quality of each region in the residual image of the feature domain; Scaling the third Gaussian distribution parameter information according to the target mass matrix to obtain fourth Gaussian distribution parameter information, wherein the third Gaussian distribution parameter information is Gaussian distribution parameter information of the residual graph of the feature domain; Decoding the bitstream according to the fourth Gaussian distribution parameter information to obtain a third residual map, where the third residual map is a residual map of the feature domain; A reconstructed image is determined according to the third residual image.

22. A decoding method, characterized in that: include: Get the code stream; Determine a target quality matrix according to the bitstream, wherein the target quality matrix is ​​used to characterize the image quality of each region in the residual image of the feature domain; Decoding the bitstream according to third Gaussian distribution parameter information to obtain a third residual image, wherein the third Gaussian distribution parameter information is Gaussian distribution parameter information of the residual image of the feature domain, and the third residual image is the residual image of the feature domain; Dequantizing the third residual map according to the target quality matrix to obtain a fourth residual map; A reconstructed image is determined according to the fourth residual image.

23. A coding device, characterized in that: include: Transceiver unit and processing unit; The transceiver unit is used to obtain a target quality matrix, where the target quality matrix is ​​used to characterize the image quality of each region in the residual image of the feature domain; The processing unit is used to scale the first residual map and / or the first Gaussian distribution parameter information according to the target mass matrix to obtain a second residual map and / or the second Gaussian distribution parameter information, wherein the first residual map is the residual map of the feature domain, and the first Gaussian distribution parameter information is the Gaussian distribution parameter information of the residual map of the feature domain; The processing unit is further configured to generate a bitstream according to the target quality matrix, the second residual map and / or the second Gaussian distribution parameter information.

24. The device according to claim 23, characterized in that The processing unit is specifically used for: Determine a target quality scaling matrix according to the target quality matrix, wherein the target quality scaling matrix is ​​used to characterize a scaling amount of each region in the residual map of the feature domain and / or the Gaussian distribution parameter information; Determine a target mass scaling tensor according to a target mass scaling matrix, wherein the target mass scaling tensor is used to characterize a residual map of the feature domain and / or a scaling tensor of the Gaussian distribution parameter information in a three-dimensional space; The first residual map is scaled according to the target quality scaling tensor to obtain the second residual map and / or the first Gaussian distribution parameter information is scaled according to the target quality scaling tensor to obtain the second Gaussian distribution parameter information.

25. The device according to claim 24, characterized in that The processing unit is specifically used for: The target quality scaling tensor is determined according to the target quality scaling matrix and a gain parameter, wherein the gain parameter includes a gain vector for channel-level quality adjustment and / or an image-level quality control factor.

26. The device according to any one of claims 23 to 25, characterized in that The processing unit is specifically used for: The target quality matrix and the second residual map are encoded to generate a bitstream, wherein the second residual map is encoded according to the second Gaussian distribution parameter information.

27. The device according to claim 26, characterized in that The processing unit is specifically used for: The target quality matrix or a target quality residual matrix of the target quality matrix is ​​encoded to generate a bitstream.

28. The device according to claim 27, characterized in that The processing unit is also used for: A target quality residual matrix of the target quality matrix is ​​generated according to the target quality matrix, and the target quality residual matrix is ​​used to represent the residual value of each image quality in the target quality matrix.

29. The device according to claim 27 or 28, characterized in that The processing unit is specifically used for: Determining Gaussian distribution parameters of the target quality residual matrix; Determining a probability distribution of the target quality residual matrix according to the Gaussian distribution parameters; Writing the Gaussian distribution parameters into the bit stream; The target quality residual matrix is ​​entropy encoded according to the probability distribution.

30. The device according to claim 27 or 28, characterized in that The processing unit is specifically used for: Determining Gaussian distribution parameters of the target mass matrix; Determining a probability distribution of the target mass matrix according to the Gaussian distribution parameters; Writing the Gaussian distribution parameters into the bit stream; The target quality matrix is ​​entropy encoded according to the probability distribution.

31. The device according to claim 27 or 28, characterized in that The processing unit is specifically used for: determining a target probability distribution from a plurality of candidate probability distributions according to the target quality residual matrix; Writing the index number of the target probability distribution into the bitstream; The target quality residual matrix is ​​entropy encoded according to the target probability distribution.

32. The device according to claim 27 or 28, characterized in that The processing unit is specifically used for: determining a target probability distribution from a plurality of candidate probability distributions according to the target mass matrix; Writing the index number of the target probability distribution into the bitstream; The target quality matrix is ​​entropy encoded according to the target probability distribution.

33. The device according to any one of claims 23 to 32, characterized in that The acquisition unit is specifically used for: Acquire a quality map, where the quality map is used to record the image quality of each region in the residual map of the feature domain; The target quality matrix is ​​determined according to the quality map.

34. The device according to any one of claims 23 to 33, characterized in that The processing unit is also used for: Inputting the image into an encoding network to obtain a first feature map; Inputting the first feature map into a context network to obtain a first prediction map; Determine the first residual map according to the first feature map and the first prediction map; Inputting the first feature map into a super-prior coding network to obtain first super-prior information; quantizing the first super a priori information to obtain second super a priori information; The second super-prior information is input into a super-prior variance decoding network to obtain the first Gaussian distribution parameter information.

35. A decoding device, characterized in that: include: Transceiver unit and processing unit; The transceiver unit is used to obtain a code stream; The processing unit is used to determine a target quality matrix according to the bitstream, wherein the target quality matrix is ​​used to characterize the image quality of each region in the residual image of the feature domain; The processing unit is further used to scale the third Gaussian distribution parameter information according to the target mass matrix to obtain fourth Gaussian distribution parameter information, wherein the third Gaussian distribution parameter information is the Gaussian distribution parameter information of the residual map of the feature domain; The processing unit is further configured to decode the bitstream according to the fourth Gaussian distribution parameter information to obtain a third residual map, wherein the third residual map is a residual map of the feature domain; The processing unit is further configured to dequantize the third residual map according to the target quality matrix to obtain a fourth residual map; The processing unit is further used to determine a reconstructed image according to the fourth residual image.

36. The device according to claim 35, characterized in that The processing unit is specifically used for: Determine a target quality scaling matrix according to the target quality matrix, wherein the target quality scaling matrix is ​​used to characterize the scaling amount of each region in the residual map of the feature domain and the Gaussian distribution parameter information; Determine a target mass scaling tensor according to a target mass scaling matrix, wherein the target mass scaling tensor is used to characterize a scaling tensor of a residual map of the feature domain and the Gaussian distribution parameter information in a three-dimensional space; The third Gaussian distribution parameter information is scaled according to the target mass scaling tensor to obtain fourth Gaussian distribution parameter information.

37. The device according to claim 36, characterized in that The processing unit is specifically used for: The target quality scaling tensor is determined according to the target quality scaling matrix and a gain parameter, wherein the gain parameter includes a gain vector for channel-level quality adjustment and / or an image-level quality control factor.

38. The device according to any one of claims 35 to 37, characterized in that The processing unit is specifically used for: Decoding the bitstream to obtain Gaussian distribution parameters of a target quality residual matrix of a target quality matrix; Determining a probability distribution of the target quality residual matrix according to the Gaussian distribution parameters; Decoding the bitstream according to the probability distribution to obtain the target quality residual matrix; The target mass matrix is ​​determined according to the target mass residual matrix.

39. The device according to any one of claims 35 to 37, characterized in that The processing unit is specifically used for: Decoding the bitstream to obtain Gaussian distribution parameters of a target quality matrix; Determining a probability distribution of the target mass matrix according to the Gaussian distribution parameters; The code stream is decoded according to the probability distribution to obtain the target quality matrix.

40. The device according to any one of claims 35 to 37, characterized in that The processing unit is specifically used for: Decoding the bitstream to obtain an index number of a target probability distribution, where the target probability distribution is determined from a plurality of candidate probability distributions according to a target quality residual matrix of the target quality matrix; Decoding the bitstream according to the target probability distribution to obtain the target quality residual matrix; The target mass matrix is ​​determined according to the target mass residual matrix.

41. The device according to any one of claims 35 to 37, characterized in that The processing unit is specifically used for: Decoding the bitstream to obtain an index number of a target probability distribution, wherein the target probability distribution is determined from a plurality of candidate probability distributions according to the target quality matrix; The code stream is decoded according to the target probability distribution to obtain the target quality matrix.

42. The device according to any one of claims 35 to 41, characterized in that The processing unit is also used for: Decoding the code stream to obtain a second feature map; The second feature map is input into a super prior variance decoding network to obtain the third Gaussian distribution parameter information.

43. A decoding device, characterized in that: include: Transceiver unit and processing unit; The transceiver unit is used to obtain a code stream; The processing unit is used to determine a target quality matrix according to the bitstream, wherein the target quality matrix is ​​used to characterize the image quality of each region in the residual image of the feature domain; The processing unit is further used to scale the third Gaussian distribution parameter information according to the target mass matrix to obtain fourth Gaussian distribution parameter information, wherein the third Gaussian distribution parameter information is the Gaussian distribution parameter information of the residual map of the feature domain; The processing unit is further configured to decode the bitstream according to the fourth Gaussian distribution parameter information to obtain a third residual map, wherein the third residual map is a residual map of the feature domain; The processing unit is further configured to determine a reconstructed image according to the third residual image.

44. A decoding device, characterized in that: including a transceiver unit and a processing unit; The transceiver unit is used to obtain a code stream; The processing unit is used to determine a target quality matrix according to the bitstream, wherein the target quality matrix is ​​used to characterize the image quality of each region in the residual image of the feature domain; The processing unit is further configured to decode the bitstream according to third Gaussian distribution parameter information to obtain a third residual image, wherein the third Gaussian distribution parameter information is Gaussian distribution parameter information of the residual image of the feature domain, and the third residual image is the residual image of the feature domain; The processing unit is further configured to dequantize the third residual map according to the target quality matrix to obtain a fourth residual map; The processing unit is further used to determine a reconstructed image according to the fourth residual image.

45. A code stream, characterized in that include: a target quality matrix, a third residual map, and a third Gaussian distribution parameter, wherein the target quality matrix is ​​used to characterize the image quality of each region in the residual map of the feature domain, the third residual map is the residual map of the feature domain, and the third Gaussian distribution parameter information is the Gaussian distribution parameter information of the residual map of the feature domain; The target mass matrix is ​​used to scale the third Gaussian distribution parameter information to obtain fourth Gaussian distribution parameter information; The target quality matrix is ​​further used to dequantize the third residual image to obtain a fourth residual image.

46. ​​A coding device comprising at least one processor and a memory, characterized in that: The at least one processor executes a program or instruction stored in the memory so that the encoding device implements the method according to any one of claims 1 to 12.

47. A decoding device, comprising at least one processor and a memory, characterized in that: The at least one processor executes a program or instruction stored in the memory so that the decoding device implements the method according to any one of claims 13 to 22.

48. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a video code stream obtained by the method according to any one of claims 1 to 12 executed by one or more processors.

49. A computer-readable storage medium for storing a computer program, characterized in that: When the computer program is executed on a computer or a processor, the computer or the processor is enabled to implement the method of any one of claims 1 to 12 or any one of claims 13 to 22.

50. A computer program product, comprising instructions, characterized in that: When the instructions are executed on a computer or a processor, the computer or the processor is caused to implement the method of any one of claims 1 to 12 or any one of claims 13 to 22.

Images