Coding and decoding method and device

Abstract

The invention discloses a coding and decoding method and device. The method comprises the following steps: obtaining a feature map of an image to be coded; obtaining a probability distribution parameter graph of the feature graph; obtaining a first matrix according to the probability distribution parameter graph; a first interval corresponding to the first matrix is determined from multiple intervals, the multiple intervals are not overlapped, and each interval corresponds to at least one scaling factor; according to a first scaling factor corresponding to the first interval, scaling a feature value corresponding to the first matrix in the feature map to obtain a scaled feature map; according to a first scaling factor corresponding to the first interval, scaling a probability distribution parameter corresponding to the first matrix in the probability distribution parameter graph to obtain a scaled probability distribution parameter graph; and entropy coding is carried out on the scaled feature map by using the scaled probability distribution parameter map, and a code stream is written. According to the method, the calculation complexity of the adaptive quantization technology and the potential scaling technology can be reduced while the coding performance is maintained.

Description

[0001] This application claims priority to the Chinese patent application filed with the State Intellectual Property Office on July 11, 2023, with application number 202310854045.4 and invention name “Data processing method and computing device”, all contents of which are incorporated by reference in this application. Technical Field

[0002] The present application relates to the field of video coding and decoding, and more specifically, to a coding and decoding method and device. Background Art

[0003] Data encoding and decoding includes two parts: data encoding and data decoding. Data encoding is performed on the source side (or commonly referred to as the encoder side), and generally includes processing (e.g., compressing) the original data to reduce the amount of data required to represent the original data (thereby more efficiently storing and / or transmitting). Data decoding is performed on the destination side (or commonly referred to as the decoder side), and generally includes inverse processing relative to the encoder side to reconstruct the original data. The "encoding and decoding" of the data involved in the embodiments of the present application should be understood as the "encoding" or "decoding" of the data. The encoding part and the decoding part are also collectively referred to as encoding and decoding (encoding and decoding, CODEC).

[0004] Adaptive quantization technology and latent scaling technology are used in data encoding and decoding, such as the adaptive sigma scale module, residual and variance scale (RVS) module, inverse residual and variance scale (Inv RVS) module, and latent scaling before synthesis (LSBS) module. These modules are mask-based encoding and decoding tools in JPEG AI. These tools can improve the performance of encoding and decoding, but the implementation complexity is relatively high. Taking the adaptive sigma scale module as an example, the variance tensor (sigma) of C×H×W is averaged in NxN blocks to obtain sigma0 of C×H / N×W / N; sigma0 is compared with the threshold T0, and mask0 with a value of 0 / 1 is output; when mask0[c,i,j] is 1, all values ​​in the corresponding NxN area on sigma are multiplied by the scaling value scale0, and this process is repeated m times.

[0005] Therefore, how to reduce the computational complexity of adaptive quantization technology and potential scaling technology while maintaining coding performance has become a technical problem that needs to be solved urgently. Summary of the invention

[0006] The present application provides a coding and decoding method and device, which can reduce the computational complexity of adaptive quantization technology and potential scaling technology while maintaining coding performance.

[0007] In a first aspect, a coding method is provided, the method comprising: obtaining a probability distribution parameter map of a feature map; obtaining a first matrix according to the probability distribution parameter map; determining a first interval corresponding to the first matrix from a plurality of intervals, the plurality of intervals not overlapping each other, and each interval corresponding to at least one scaling factor; scaling the eigenvalues ​​corresponding to the first matrix in the feature map according to the first scaling factor corresponding to the first interval to obtain a scaled feature map; scaling the probability distribution parameters corresponding to the first matrix in the probability distribution parameter map according to the first scaling factor corresponding to the first interval to obtain a scaled probability distribution parameter map; and performing entropy coding on the scaled feature map using the scaled probability distribution parameter map and writing the scaled feature map into a bitstream.

[0008] In the above technical solution, by setting the threshold intervals to non-overlapping intervals, there is no need to repeat multiple times when scaling the feature map or probability distribution parameter map, thereby reducing the complexity of the algorithm without losing coding performance.

[0009] In a possible implementation, the multiple intervals are divided by multiple thresholds, the number of the multiple thresholds is m, the number of the multiple intervals is m+1, and m is an integer greater than 1.

[0010] In a possible implementation manner, the first value of the first matrix is ​​compared with at least one threshold value among the multiple threshold values, and the first interval corresponding to the first value is determined from the multiple intervals.

[0011] In a possible implementation manner, the method further includes: writing the number m of the multiple thresholds and the value of each threshold into the code stream.

[0012] In a possible implementation manner, the method further includes: writing the first scaling factor into a bitstream.

[0013] In a possible implementation, a first scaling factor corresponding to the first interval is multiplied by an eigenvalue in the feature map corresponding to the first matrix to obtain the scaled feature map.

[0014] In a possible implementation manner, a first scaling factor corresponding to the first interval is multiplied by a probability distribution parameter in the probability distribution parameter map corresponding to the first matrix to obtain the scaled probability distribution parameter map.

[0015] In a possible implementation, the probability distribution parameters in the probability distribution parameter map are Gaussian distribution parameters, and the first value of the first matrix is ​​the mean or maximum value of the Gaussian distribution parameters included in the probability distribution parameter map.

[0016] In a possible implementation, the method is applied to an adaptive variance scaling module or a residual quantization scaling module.

[0017] In a possible implementation, the feature map is a residual feature map, and the eigenvalues ​​in the feature map corresponding to the first matrix are residual eigenvalues.

[0018] In a second aspect, a decoding method is provided, the method comprising: decoding from a bitstream to obtain a probability distribution parameter map; obtaining a first matrix based on the probability distribution parameter map; determining a first interval corresponding to the first matrix from a plurality of intervals, the plurality of intervals not overlapping each other, each of the intervals corresponding to at least one scaling factor; scaling the probability distribution parameters corresponding to the first matrix in the probability distribution parameter map according to the first scaling factor corresponding to the first interval to obtain a scaled probability distribution parameter map; and decoding from the bitstream using the scaled probability distribution parameter map to obtain a feature map.

[0019] In a possible implementation, the method further includes: decoding from the code stream to obtain a number m of multiple thresholds and a value of each threshold, the multiple intervals are divided by the multiple thresholds, the number of the multiple intervals is m+1, and m is an integer greater than 1.

[0020] In a possible implementation manner, the first value of the first matrix is ​​compared with at least one threshold value among the multiple threshold values, and the first interval corresponding to the first value is determined from the multiple intervals.

[0021] In a possible implementation manner, the method further includes: decoding from the code stream to obtain the first scaling factor.

[0022] In a possible implementation manner, a first scaling factor corresponding to the first interval is multiplied by a probability distribution parameter in the probability distribution parameter map corresponding to the first matrix to obtain the scaled probability distribution parameter map.

[0023] In a possible implementation, the probability distribution parameters in the probability distribution parameter map are Gaussian distribution parameters, and the first value of the first matrix is ​​the mean or maximum value of the Gaussian distribution parameters included in the probability distribution parameter map.

[0024] In a possible implementation, the method further includes: dividing the feature map by a first scaling factor corresponding to the first interval to obtain a feature map before scaling.

[0025] In a possible implementation, the method is applied to an adaptive variance scaling module or a residual inverse quantization scaling module.

[0026] In a possible implementation, the feature map is a residual feature map, and the residual feature map includes residual feature values.

[0027] In a possible implementation, the feature map includes eigenvalues, the method is applied to a potential scaling module, the first interval corresponds to the first scaling factor and the second scaling factor, and the method also includes: obtaining a prediction map and a residual map of the feature map; scaling the predicted eigenvalues ​​corresponding to the first matrix in the prediction map according to the first scaling factor corresponding to the first interval to obtain a scaled predicted feature map; scaling the residual eigenvalues ​​corresponding to the first matrix in the residual map according to the second scaling factor corresponding to the first interval to obtain a scaled residual feature map; updating the feature map according to the scaled prediction map and the scaled residual map to obtain an updated feature map.

[0028] In a third aspect, a method for data processing is provided, the method comprising: obtaining a first matrix; determining a first interval corresponding to the first matrix from multiple intervals, the multiple intervals do not overlap, and each of the intervals corresponds to at least one scaling factor; scaling the elements included in the first matrix according to the first scaling factor corresponding to the first interval to obtain a second matrix.

[0029] In combination with the third aspect, in an implementation of the third aspect, the method is applied to an adaptive variance scaling module, the first matrix is ​​a first variance map, and the variances included in the first matrix are multiplied by the first scaling factors to obtain a second variance map.

[0030] In combination with the third aspect, in the implementation of the third aspect, the method is applied to a residual quantization scaling module, the first matrix is ​​a first residual map, and the residuals included in the first matrix are multiplied by the first scaling factor respectively to obtain a second residual map.

[0031] In combination with the third aspect, in the implementation of the third aspect, the method is applied to a residual inverse quantization scaling module, the first matrix is ​​a first residual image, and the residuals included in the first matrix are divided by the first scaling factor to obtain a second residual image.

[0032] In combination with the third aspect, in the implementation of the third aspect, the method is applied to a potential scaling module before synthesis, the first matrix is ​​a first feature map, and the residuals included in the first residual map are multiplied by the second scaling factors corresponding to the first interval to obtain a fourth feature map; based on the first feature map, the third feature map and the fourth feature map, a second feature map is obtained.

[0033] In a fourth aspect, a coding method is provided, the method comprising: obtaining a feature map of an image to be encoded; obtaining probability distribution parameters of the feature map; obtaining a first matrix based on the probability distribution parameters; determining a first interval corresponding to the first matrix from a plurality of intervals, the plurality of intervals not overlapping each other, and each of the intervals corresponding to at least one scaling factor; scaling the feature map and probability distribution parameters corresponding to the first matrix based on a first scaling factor corresponding to the first interval to obtain a scaled feature map and a scaled probability distribution parameter; and entropy encoding the scaled feature map using the scaled probability distribution parameters and writing the scaled feature map into a bitstream.

[0034] In combination with the fourth aspect, in an implementation method of the fourth aspect, the probability distribution parameters of the feature map are obtained through super a priori information, and the super a priori information is written into the bit stream.

[0035] In combination with the fourth aspect, in an implementation of the fourth aspect, the multiple intervals are divided by multiple thresholds, the number of the multiple thresholds is m, the number of the multiple intervals is m+1, and m is an integer greater than 1.

[0036] In combination with the fourth aspect, in the implementation method of the fourth aspect, determining the first interval corresponding to the first matrix from multiple intervals includes: comparing the first value of the first matrix with at least one threshold among the multiple thresholds, and determining the first interval corresponding to the first data from the multiple intervals.

[0037] In combination with the fourth aspect, in an implementation of the fourth aspect, the number m of the multiple thresholds and the value of each threshold are written into the bitstream.

[0038] In combination with the fourth aspect, in an implementation manner of the fourth aspect, the first scaling factor is written into a bitstream.

[0039] In combination with the fourth aspect, in the implementation method of the fourth aspect, the feature map corresponding to the first matrix is ​​scaled according to the first scaling factor corresponding to the first interval, and the scaled feature map is obtained, including: multiplying the first scaling factor corresponding to the first interval and the feature map corresponding to the first matrix to obtain the scaled feature map.

[0040] In combination with the fourth aspect, in an implementation manner of the fourth aspect, scaling the probability distribution parameter corresponding to the first matrix according to the first scaling factor corresponding to the first interval, to obtain the scaled probability distribution parameter includes:

[0041] A first scaling factor corresponding to the first interval is multiplied by the probability distribution parameter to obtain the scaled probability distribution parameter.

[0042] In combination with the fourth aspect, in an implementation method of the fourth aspect, the probability distribution parameter is a Gaussian distribution parameter, and the first value of the first matrix is ​​the mean or maximum value of the Gaussian distribution parameter.

[0043] In combination with the fourth aspect, in an implementation manner of the fourth aspect, the method is applied to an adaptive variance scaling module or a residual quantization scaling module.

[0044] In combination with the fourth aspect, in an implementation method of the fourth aspect, the feature map corresponding to the first matrix is ​​a residual feature map.

[0045] In a fifth aspect, a decoding method is provided, the method comprising: decoding from a bit stream to obtain probability distribution parameters; obtaining a first matrix based on the probability distribution parameters; determining a first interval corresponding to the first matrix from a plurality of intervals, the plurality of intervals not overlapping each other, and each of the intervals corresponding to at least one scaling factor; scaling the probability distribution parameters corresponding to the first matrix based on the first scaling factor corresponding to the first interval to obtain scaled probability distribution parameters; and decoding from the bit stream using the scaled probability distribution parameters to obtain a scaled feature map.

[0046] In combination with the fifth aspect, in an implementation of the fifth aspect, decoding from a bitstream to obtain probability distribution parameters includes: decoding from a bitstream to obtain super a priori information; and obtaining the probability distribution parameters based on the super a priori information.

[0047] In combination with the fifth aspect, in the implementation method of the fifth aspect, it also includes: dividing the first scaling factor corresponding to the first interval and the feature map to obtain a scaled feature map.

[0048] If the feature map after scaling on the encoding side refers to a zoom-in operation on the feature map, the feature map after scaling on the decoding side refers to a zoom-out operation on the feature map. Conversely, if the feature map after scaling on the encoding side refers to a zoom-out operation on the feature map, the feature map after scaling on the decoding side refers to a zoom-in operation on the feature map.

[0049] In combination with the fifth aspect, in an implementation method of the fifth aspect, a number m of multiple thresholds and a value of each threshold are obtained by decoding the code stream, the multiple intervals are divided by the multiple thresholds, and the number of the multiple intervals is m+1.

[0050] In combination with the fifth aspect, in the implementation method of the fifth aspect, determining the first interval corresponding to the first matrix from multiple intervals includes: comparing the first value of the first matrix with at least one threshold among the multiple thresholds, and determining the first interval corresponding to the first data from the multiple intervals.

[0051] In combination with the fifth aspect, in an implementation manner of the fifth aspect, the first scaling factor is obtained by decoding a bit stream.

[0052] In combination with the fifth aspect, in the implementation method of the fifth aspect, the probability distribution parameters corresponding to the first matrix are scaled according to the first scaling factor corresponding to the first interval, and the scaled probability distribution parameters are obtained, including: multiplying the first scaling factor corresponding to the first interval and the probability distribution parameters to obtain the scaled probability distribution parameters.

[0053] In combination with the fifth aspect, in an implementation method of the fifth aspect, the probability distribution parameter is a Gaussian distribution parameter, and the first value of the first matrix is ​​the mean or maximum value of the Gaussian distribution parameter.

[0054] In combination with the fifth aspect, in an implementation of the fifth aspect, the method is applied to an adaptive variance scaling module or a residual inverse quantization scaling module or a potential scaling module.

[0055] In combination with the fifth aspect, in an implementation method of the fifth aspect, the feature map corresponding to the first matrix is ​​a residual feature map.

[0056] In a sixth aspect, a coding method is provided, the method comprising: obtaining a feature map of an image to be encoded; obtaining probability distribution parameters of the feature map; obtaining a first matrix based on the probability distribution parameters; determining a first interval corresponding to the first matrix from a plurality of intervals, the plurality of intervals not overlapping each other, and each of the intervals corresponding to at least one scaling factor; scaling the feature map corresponding to the first matrix based on the first scaling factor corresponding to the first interval to obtain a scaled feature map; performing entropy coding on the scaled feature map using the probability distribution parameters, and writing the scaled feature map into a bitstream.

[0057] In the seventh aspect, a coding method is provided, the method comprising: obtaining a feature map of an image to be encoded; obtaining probability distribution parameters of the feature map; obtaining a first matrix based on the probability distribution parameters; determining a first interval corresponding to the first matrix from a plurality of intervals, the plurality of intervals not overlapping each other, and each of the intervals corresponding to at least one scaling factor; scaling the probability distribution parameters corresponding to the first matrix based on the first scaling factor corresponding to the first interval to obtain scaled probability distribution parameters; and entropy encoding the feature map using the scaled probability distribution parameters and writing them into a bitstream.

[0058] In the eighth aspect, a coding device is provided, including a transceiver unit and a processing unit; the processing unit is used to obtain a feature map of an image to be encoded and a probability distribution parameter map of the feature map; and is also used to obtain a first matrix according to the probability distribution parameter map; determine a first interval corresponding to the first matrix from multiple intervals, the multiple intervals do not overlap with each other, and each of the intervals corresponds to at least one scaling factor; according to the first scaling factor corresponding to the first interval, scale the eigenvalues ​​corresponding to the first matrix in the feature map to obtain a scaled feature map; according to the first scaling factor corresponding to the first interval, scale the probability distribution parameters corresponding to the first matrix in the probability distribution parameter map to obtain a scaled probability distribution parameter map; use the scaled probability distribution parameter map to entropy encode the scaled feature map and write it into a bitstream; the transceiver unit is used to send the bitstream.

[0059] In a possible implementation, the multiple intervals are divided by multiple thresholds, the number of the multiple thresholds is m, the number of the multiple intervals is m+1, and m is an integer greater than 1.

[0060] In a possible implementation manner, the processing unit is specifically configured to: compare the first value of the first matrix with at least one threshold value among the multiple threshold values, and determine the first interval corresponding to the first value from the multiple intervals.

[0061] In a possible implementation, the processing unit is further configured to write the number m of the multiple thresholds and the value of each threshold into the code stream.

[0062] In a possible implementation manner, the processing unit is further configured to write the first scaling factor into the bitstream.

[0063] In a possible implementation, the processing unit is specifically used to: multiply a first scaling factor corresponding to the first interval by an eigenvalue in the feature map corresponding to the first matrix to obtain the scaled feature map.

[0064] In a possible implementation, the processing unit is specifically configured to: multiply a first scaling factor corresponding to the first interval by a probability distribution parameter in the probability distribution parameter map corresponding to the first matrix to obtain the scaled probability distribution parameter map.

[0065] In a possible implementation, the probability distribution parameters in the probability distribution parameter map are Gaussian distribution parameters, and the first value of the first matrix is ​​the mean or maximum value of the Gaussian distribution parameters included in the probability distribution parameter map.

[0066] In a possible implementation manner, the device is applied to an adaptive variance scaling module or a residual quantization scaling module.

[0067] In a possible implementation, the feature map is a residual feature map, and the eigenvalues ​​in the feature map corresponding to the first matrix are residual eigenvalues.

[0068] In the ninth aspect, a decoding device is provided, including a transceiver unit and a processing unit; the transceiver unit is used to obtain a code stream; the processing unit is used to decode the code stream to obtain a probability distribution parameter map; and is also used to obtain a first matrix based on the probability distribution parameter map; determine a first interval corresponding to the first matrix from a plurality of intervals, the plurality of intervals do not overlap with each other, and each of the intervals corresponds to at least one scaling factor; according to the first scaling factor corresponding to the first interval, scale the probability distribution parameters corresponding to the first matrix in the probability distribution parameter map to obtain a scaled probability distribution parameter map; and use the scaled probability distribution parameter map to decode the code stream to obtain a feature map.

[0069] In a possible implementation, the processing unit is further used to decode from the code stream to obtain a number m of multiple thresholds and a value of each threshold, the multiple intervals are divided by the multiple thresholds, the number of the multiple intervals is m+1, and m is an integer greater than 1.

[0070] In a possible implementation manner, the processing unit is specifically configured to: compare the first value of the first matrix with at least one threshold value among the multiple threshold values, and determine the first interval corresponding to the first value from the multiple intervals.

[0071] In a possible implementation manner, the processing unit is further configured to decode the bit stream to obtain the first scaling factor.

[0072] In a possible implementation, the processing unit is specifically configured to: multiply a first scaling factor corresponding to the first interval by a probability distribution parameter in the probability distribution parameter map corresponding to the first matrix to obtain the scaled probability distribution parameter map.

[0073] In a possible implementation, the probability distribution parameters in the probability distribution parameter map are Gaussian distribution parameters, and the first value of the first matrix is ​​the mean or maximum value of the Gaussian distribution parameters included in the probability distribution parameter map.

[0074] In a possible implementation, the processing unit is further configured to divide the feature map by a first scaling factor corresponding to the first interval to obtain a feature map before scaling.

[0075] In a possible implementation manner, the device is applied to an adaptive variance scaling module or a residual inverse quantization scaling module.

[0076] In a possible implementation, the feature map is a residual feature map, and the residual feature map includes residual feature values.

[0077] In a possible implementation, the feature map includes eigenvalues, the device is applied to a potential scaling module, the first interval corresponds to the first scaling factor and the second scaling factor, and the processing unit is further used to obtain a prediction map and a residual map of the feature map; the processing unit is further used to scale the predicted eigenvalues ​​corresponding to the first matrix in the prediction map according to the first scaling factor corresponding to the first interval to obtain a scaled predicted feature map; the processing unit is further used to scale the residual eigenvalues ​​corresponding to the first matrix in the residual map according to the second scaling factor corresponding to the first interval to obtain a scaled residual feature map; the processing unit is further used to update the feature map according to the scaled prediction map and the scaled residual map to obtain an updated feature map.

[0078] In the tenth aspect, an embodiment of the present application also provides a coding device, which includes: at least one processor, when the at least one processor executes program code or instructions, it implements the methods in the above-mentioned first, third, fourth, sixth, seventh aspects or any possible implementation methods thereof.

[0079] Optionally, the device may further include at least one memory, and the at least one memory is used to store the program code or instruction.

[0080] In the eleventh aspect, an embodiment of the present application also provides a decoding device, which includes: at least one processor, when the at least one processor executes program code or instructions, it implements the methods in the above-mentioned second and fifth aspects or any possible implementation methods thereof.

[0081] Optionally, the device may further include at least one memory, and the at least one memory is used to store the program code or instruction.

[0082] In a twelfth aspect, an embodiment of the present application further provides a chip, comprising: an input interface, an output interface, and at least one processor. Optionally, the chip also includes a memory. The at least one processor is used to execute the code in the memory, and when the at least one processor executes the code, the chip implements the method in the above-mentioned first to seventh aspects or any possible implementation thereof.

[0083] Optionally, the above chip may also be an integrated circuit.

[0084] In a thirteenth aspect, an embodiment of the present application further provides a computer-readable storage medium for storing a computer program, wherein the computer program includes methods for implementing the above-mentioned first to seventh aspects or any possible implementation thereof.

[0085] In a fourteenth aspect, an embodiment of the present application further provides a computer program product comprising instructions, which, when executed on a computer, enables the computer to implement the methods in the above-mentioned first to seventh aspects or any possible implementation thereof.

[0086] In a fifteenth aspect, an embodiment of the present application further provides a computer-readable storage medium, on which is stored a video code stream obtained by one or more processors executing the methods in the above-mentioned first to seventh aspects or any possible implementation thereof.

[0087] The encoding and decoding device, computer storage medium, computer program product and chip provided in this embodiment are all used to execute the encoding and decoding method provided above. Therefore, the beneficial effects that can be achieved can refer to the beneficial effects in the encoding and decoding method provided above, and will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0088] Figure 1 An exemplary block diagram of a decoding system provided in an embodiment of the present application.

[0089] Figure 2 An exemplary block diagram of a video decoding system provided in an embodiment of the present application.

[0090] Figure 3 An exemplary block diagram of a video encoder provided in an embodiment of the present application.

[0091] Figure 4 A schematic block diagram of a hybrid architecture video encoder.

[0092] Figure 5 An exemplary block diagram of a video decoder provided in an embodiment of the present application.

[0093] Figure 6 An exemplary schematic diagram of a candidate image block provided in an embodiment of the present application.

[0094] Figure 7 A schematic block diagram of a hybrid architecture video decoder.

[0095] Figure 8 It is the JPEG AI encoder structure.

[0096] Fig. 9 It is the JPEG AI decoder structure.

[0097] Fig.10 An exemplary block diagram of a video decoding device provided in an embodiment of the present application.

[0098] Fig.11 An exemplary block diagram of an apparatus provided in an embodiment of the present application.

[0099] Fig.12 A schematic block diagram of a hybrid architecture video encoder applied to an embodiment of the present application.

[0100] Fig.13 A schematic block diagram of a hybrid architecture video decoder applied to an embodiment of the present application.

[0101] Fig.14 It is the specific process of the decoding process.

[0102] Fig.15 The figure is a schematic diagram of the calculation flow of the existing adaptive variance scaling module during scaling.

[0103] Fig.16 Schematic diagram of the calculation flow of the existing Inv RVS module during scaling.

[0104] Fig.17 The figure is a schematic diagram of the calculation flow of the existing LSBS module during scaling.

[0105] Fig.18 It is the specific process of the encoding process.

[0106] Fig.19 It is the system architecture diagram of this application.

[0107] Fig. 20 A schematic flowchart of an encoding method provided in an embodiment of the present application.

[0108] Fig.21 A schematic diagram of m+1 intervals divided based on m thresholds provided in an embodiment of the present application.

[0109] Fig. 22 This is the flow chart of the adaptive variance scaling module (Adpt.Sigma Scale).

[0110] Fig.23 A schematic flowchart of a decoding method provided in an embodiment of the present application.

[0111] Fig.24 It is the Latent Scaling Before Synthesis (LSBS) flow chart.

[0112] Fig.25 A schematic diagram of an encoding device 1600 provided in an embodiment of the present application.

[0113] Fig.26 A schematic diagram of a decoding device 1700 provided in an embodiment of the present application.

[0114] Fig. 27A schematic diagram of the structure of a chip 2300 provided in an embodiment of the present application. DETAILED DESCRIPTION

[0115] The technical solution in this application will be described below in conjunction with the accompanying drawings.

[0116] The present application will present various aspects, embodiments or features around a system including multiple devices, components, modules, etc. It should be understood and appreciated that each system may include additional devices, components, modules, etc., and / or may not include all devices, components, modules, etc. discussed in conjunction with the figures. In addition, combinations of these schemes may also be used.

[0117] In addition, in the embodiments of the present application, words such as "exemplary" and "for example" are used to indicate examples, illustrations or explanations. Any embodiment or design described as "exemplary" in the present application should not be interpreted as being more preferred or more advantageous than other embodiments or designs. Specifically, the use of the word "exemplary" is intended to present concepts in a concrete way.

[0118] In the embodiments of the present application, "corresponding (corresponding, relevant)" and "corresponding (corresponding)" can sometimes be used interchangeably. It should be pointed out that when the distinction between them is not emphasized, the meanings they intend to express are consistent.

[0119] The business scenarios described in the embodiments of the present application are intended to more clearly illustrate the technical solutions of the embodiments of the present application, and do not constitute a limitation on the technical solutions provided in the embodiments of the present application. A person of ordinary skill in the art can appreciate that, with the evolution of network architecture and the emergence of new business scenarios, the technical solutions provided in the embodiments of the present application are also applicable to similar technical problems.

[0120] References to "one embodiment" or "some embodiments" etc. described in this specification mean that a particular feature, structure or characteristic described in conjunction with the embodiment is included in one or more embodiments of the present application. Thus, the phrases "in one embodiment", "in some embodiments", "in some other embodiments", "in some other embodiments", etc. that appear at different places in this specification do not necessarily refer to the same embodiment, but mean "one or more but not all embodiments", unless otherwise specifically emphasized in other ways. The terms "including", "comprising", "having" and their variations all mean "including but not limited to", unless otherwise specifically emphasized in other ways.

[0121] In the present application, "at least one" means one or more, and "plurality" means two or more. "And / or" describes the association relationship of associated objects, indicating that three relationships may exist. For example, A and / or B can mean: including the existence of A alone, the existence of A and B at the same time, and the existence of B alone, where A and B can be singular or plural. The character " / " generally indicates that the previous and next associated objects are in an "or" relationship. "At least one of the following items" or similar expressions refers to any combination of these items, including any combination of single items or plural items. For example, at least one of a, b, or c can mean: a, b, c, ab, ac, bc, or abc, where a, b, c can be single or multiple.

[0122] Data encoding and decoding includes two parts: data encoding and data decoding. Data encoding is performed on the source side (or commonly referred to as the encoder side), and generally includes processing (e.g., compressing) the original data to reduce the amount of data required to represent the original data (thereby more efficiently storing and / or transmitting). Data decoding is performed on the destination side (or commonly referred to as the decoder side), and generally includes inverse processing relative to the encoder side to reconstruct the original data. The "encoding and decoding" of the data involved in the embodiments of the present application should be understood as the "encoding" or "decoding" of the data. The encoding part and the decoding part are also collectively referred to as encoding and decoding (encoding and decoding, CODEC).

[0123] Video compression coding technology is widely used in multimedia services, broadcasting, video communication and storage. H.264 / AVC, jointly developed by ITU-T and ISO / IEC, is the most widely used video coding standard today. The Joint Collaborative Team on Video Coding (JCT-VC) released the High Efficiency Video Coding (HEVC) standard in 2013. HEVC uses many new technologies in both intra-frame and inter-frame coding, and its compression efficiency is about 50% higher than that of H.264 / AVC. The standard has also been widely used in the market in recent years. In 2018, JVET (Joint Video Expert Team) started to formulate the VVC (Versatile Video Coding) standard, which further improved the compression efficiency for applications such as 4K / 8K, HDR high dynamic video, and 360 panoramic video. It can be foreseen that the VVC standard will be deployed in various products and services related to video images in the near future. These standards all adopt a hybrid architecture encoding and decoding solution based on block division and transform quantization, and make continuous technical iterations in specific modules such as block division, prediction, transformation, entropy coding, and loop filtering to continuously improve compression efficiency.

[0124] In the case of lossless data encoding, the original data can be reconstructed, that is, the reconstructed original data has the same quality as the original data (assuming no transmission loss or other data loss during storage or transmission). In the case of lossy data encoding, further compression is performed by quantization, etc. to reduce the amount of data required to represent the original data, but the decoder side cannot completely reconstruct the original data, that is, the quality of the reconstructed original data is lower or worse than the quality of the original data.

[0125] The embodiments of the present application can be applied to video data and other data with compression / decompression requirements. The following takes the encoding of video data (referred to as video encoding) as an example to illustrate the embodiments of the present application. Other types of data (such as image data, audio data, integer data and other data with compression / decompression requirements) can refer to the following description, and the embodiments of the present application will not be repeated. It should be noted that, compared with video encoding, the encoding process of audio data and integer data does not need to divide the data into blocks, but the data can be directly encoded.

[0126] Video coding generally refers to processing a sequence of images to form a video or video sequence. In the field of video coding, the terms "picture", "frame" or "image" can be used as synonyms.

[0127] Several video coding standards belong to the category of "lossy hybrid video codecs" (i.e., combining spatial and temporal prediction in the pixel domain with 2D transform coding in the transform domain for applying quantization). Each image in a video sequence is usually divided into a set of non-overlapping blocks, which are usually encoded at the block level. In other words, the encoder usually processes, i.e., encodes the video at the block (video block) level, for example, by generating a prediction block through spatial (intra-frame) prediction and temporal (inter-frame) prediction; subtracting the prediction block from the current block (currently processed / to be processed block) to obtain a residual block; transforming the residual block in the transform domain and quantizing the residual block to reduce the amount of data to be transmitted (compressed), while the decoder side applies the inverse processing part relative to the encoder to the encoded or compressed block to reconstruct the current block for representation. In addition, the encoder needs to repeat the processing steps of the decoder so that the encoder and the decoder generate the same predictions (e.g., intra-frame predictions and inter-frame predictions) and / or reconstructed pixels for processing, i.e., encoding subsequent blocks.

[0128] In the following embodiment of the decoding system 10, the encoder 20 and the decoder 30 are based on Figures 1 to 4 Give a description.

[0129] Figure 1An exemplary block diagram of a decoding system 10 provided in an embodiment of the present application, for example, a video decoding system 10 (or simply referred to as a decoding system 10). The video encoder 20 (or simply referred to as an encoder 20) and the video decoder 30 (or simply referred to as a decoder 30) in the video decoding system 10 represent devices that can be used to perform various techniques according to various examples described in the embodiments of the present application.

[0130] like Figure 1 As shown, the decoding system 10 includes a source device 12, which is used to provide encoded image data 21 such as encoded images to a destination device 14 for decoding the encoded image data 21.

[0131] The source device 12 includes an encoder 20 , and may additionally or optionally include an image source 16 , a preprocessor (or a preprocessing unit) 18 such as an image preprocessor, and a communication interface (or a communication unit) 22 .

[0132] The image source 16 may include or may be any type of image capture device for capturing real-world images, etc., and / or any type of image generation device, such as a computer graphics processor for generating computer-animated images or any type of device for acquiring and / or providing real-world images, computer-generated images (e.g., screen content, virtual reality (VR) images, and / or any combination thereof (e.g., augmented reality (AR) images). The image source may be any type of memory or storage that stores any of the above images.

[0133] In order to distinguish the processing performed by the pre-processor (or pre-processing unit) 18 , the image (or image data) 17 may also be referred to as a raw image (or raw image data) 17 .

[0134] The preprocessor 18 is used to receive the original image data 17 and preprocess the original image data 17 to obtain a preprocessed image (or preprocessed image data) 19. For example, the preprocessing performed by the preprocessor 18 may include trimming, color format conversion (e.g., from RGB to YCbCr), color adjustment, or denoising. It is understood that the preprocessing unit 18 may be an optional component.

[0135] The video encoder (or encoder) 20 is used to receive the pre-processed image data 19 and provide the encoded image data 21 (hereinafter referred to as Figure 2 etc. for further description).

[0136] The communication interface 22 in the source device 12 can be used to receive the encoded image data 21 and send the encoded image data 21 (or any other processed version) to another device such as the destination device 14 or any other device through the communication channel 13 for storage or direct reconstruction.

[0137] The destination device 14 includes a decoder 30 and, in addition or alternatively, may include a communication interface (or communication unit) 28 , a post-processor (or post-processing unit) 32 , and a display device 34 .

[0138] The communication interface 28 in the destination device 14 is used to receive the encoded image data 21 (or any other processed version) directly from the source device 12 or from any other source device such as a storage device, for example, the storage device is a encoded image data storage device, and provide the encoded image data 21 to the decoder 30.

[0139] The communication interface 22 and the communication interface 28 can be used to send or receive encoded image data (or encoded data) 21 through a direct communication link between the source device 12 and the destination device 14, such as a direct wired or wireless connection, or through any type of network, such as a wired network, a wireless network or any combination thereof, any type of private network and public network or any type of combination thereof.

[0140] For example, the communication interface 22 may be used to encapsulate the encoded image data 21 into a suitable format such as a message, and / or process the encoded image data using any type of transmission coding or processing for transmission over a communication link or network.

[0141] The communication interface 28 corresponds to the communication interface 22 , for example, and can be used to receive transmission data and process the transmission data using any type of corresponding transmission decoding or processing and / or decapsulation to obtain the encoded image data 21 .

[0142] Both the communication interface 22 and the communication interface 28 can be configured as follows Figure 1 A unidirectional communication interface, or a bidirectional communication interface, indicated by an arrow pointing from the source device 12 to the corresponding communication channel 13 of the destination device 14, and can be used to send and receive messages, etc. to establish a connection, confirm and exchange any other information related to the communication link and / or data transmission such as encoded image data transmission, etc.

[0143] The video decoder (or decoder) 30 is used to receive the encoded image data 21 and provide decoded image data (or decoded image data) 31 (hereinafter referred to as Figure 3 etc. for further description).

[0144] The post-processor 32 is used to post-process the decoded image data 31 (also called reconstructed image data) such as the decoded image to obtain the post-processed image data 33 such as the post-processed image. The post-processing performed by the post-processing unit 32 may include, for example, color format conversion (e.g., from YCbCr to RGB), color adjustment, cropping or resampling, or any other processing for generating the decoded image data 31 for display by the display device 34 or the like.

[0145] The display device 34 is used to receive the post-processed image data 33 to display the image to a user or viewer, etc. The display device 34 can be or include any type of display for displaying the reconstructed image, such as an integrated or external display screen or display. For example, the display screen can include a liquid crystal display (LCD), an organic light emitting diode (OLED) display, a plasma display, a projector, a micro LED display, a liquid crystal on silicon (LCoS), a digital light processor (DLP), or any other type of display screen.

[0146] The decoding system 10 also includes a training engine 25, which is used to train the encoder 20 (especially the entropy coding unit 270 in the encoder 20) or the decoder 30 (especially the entropy decoding unit 304 in the decoder 30) to perform entropy coding on the image block to be encoded according to the estimated probability distribution. For a detailed description of the training engine 25, please refer to the following method test example.

[0147] although Figure 1 The source device 12 and the destination device 14 are shown as independent devices, but the device embodiment may also include the source device 12 and the destination device 14 or the functions of the source device 12 and the destination device 14 at the same time, that is, the source device 12 or the corresponding function and the destination device 14 or the corresponding function at the same time. In these embodiments, the source device 12 or the corresponding function and the destination device 14 or the corresponding function can be implemented using the same hardware and / or software or by separate hardware and / or software or any combination thereof.

[0148] According to the description, Figure 1 The presence and (exact) division of different units or functions in the source device 12 and / or the destination device 14 shown may vary depending on the actual device and application, which will be obvious to the skilled person.

[0149] Please refer to Figure 2 , Figure 2An exemplary block diagram of a video decoding system 40 provided in an embodiment of the present application, wherein the encoder 20 (eg, the video encoder 20) or the decoder 30 (eg, the video decoder 30) or both may be configured as follows: Figure 2 The processing circuitry in the video decoding system 40 shown may be implemented, for example, by one or more microprocessors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), discrete logic, hardware, video encoding-specific processors, or any combination thereof. Figure 3 and Figure 4 , Figure 3 An exemplary block diagram of a video encoder provided in an embodiment of the present application is shown in FIG. Figure 4 An exemplary block diagram of a video decoder provided in an embodiment of the present application. The encoder 20 may be implemented by a processing circuit 46 to include a reference Figure 3 The various modules discussed in the encoder 20 of the present invention and / or any other encoder system or subsystem described herein. The decoder 30 may be implemented by the processing circuit 46 to include reference Figure 4 The processing circuit 46 may be used to perform the various operations discussed below. Figure 6 As shown, if part of the technology is implemented in software, the device can store the software instructions in a suitable non-transitory computer-readable storage medium, and use one or more processors to execute the instructions in hardware, thereby performing the technology of the embodiment of the present application. Figure 2 As shown, one of video encoder 20 and video decoder 30 may be integrated into a single device as part of a combined encoder / decoder (CODEC).

[0150] The source device 12 and the destination device 14 may include any of a variety of devices, including any type of handheld or fixed device, such as a notebook or laptop computer, a mobile phone, a smart phone, a tablet or a tablet computer, a camera, a desktop computer, a set-top box, a television, a display device, a digital media player, a video game console, a video streaming device (e.g., a content service server or a content distribution server), a broadcast receiving device, a broadcast transmitting device, and a monitoring device, etc., and may not use or use any type of operating system. The source device 12 and the destination device 14 may also be devices in a cloud computing scenario, such as a virtual machine in a cloud computing scenario. In some cases, the source device 12 and the destination device 14 may be equipped with components for wireless communication. Therefore, the source device 12 and the destination device 14 may be wireless communication devices.

[0151] The source device 12 and the destination device 14 may install virtual scene applications (applications, APPs) such as virtual reality (VR) applications, augmented reality (AR) applications, or mixed reality (MR) applications, and may run VR applications, AR applications, or MR applications based on user operations (such as clicks, touches, slides, shakes, voice control, etc.). The source device 12 and the destination device 14 may collect images / videos of any object in the environment through cameras and / or sensors, and then display virtual objects on the display device based on the collected images / videos. The virtual objects may be virtual objects in VR scenes, AR scenes, or MR scenes (i.e., objects in the virtual environment).

[0152] It should be noted that in the embodiment of the present application, the virtual scene application in the source device 12 and the destination device 14 can be a built-in application of the source device 12 and the destination device 14 themselves, or it can be an application provided by a third-party service provider and installed by the user. There is no specific limitation on this.

[0153] In addition, the source device 12 and the destination device 14 may be installed with a real-time video transmission application, such as a live broadcast application. The source device 12 and the destination device 14 may collect images / videos through cameras and then display the collected images / videos on a display device.

[0154] In some cases, Figure 2The video decoding system 10 shown is merely exemplary, and the techniques provided in the embodiments of the present application may be applicable to video encoding settings (e.g., video encoding or video decoding), which do not necessarily include any data communication between the encoding device and the decoding device. In other examples, the data is retrieved from a local memory, sent over a network, and so on. The video encoding device can encode the data and store the data in a memory, and / or the video decoding device can retrieve the data from the memory and decode the data. In some examples, encoding and decoding are performed by devices that do not communicate with each other but only encode data to a memory and / or retrieve and decode data from a memory.

[0155] Please refer to Figure 2 , Figure 2 An exemplary block diagram of a video decoding system 40 provided in an embodiment of the present application is shown in FIG. Figure 2 As shown, the video decoding system 40 may include an imaging device 47, a video encoder 20, a video decoder 30 (and / or a video encoder / decoder implemented by a processing circuit 46), an antenna 48, one or more processors 43, one or more memory storages 44 and / or a display device 45.

[0156] like Figure 2 As shown, imaging device 47, antenna 48, processing circuit 46, video encoder 20, video decoder 30, processor 43, memory storage 44 and / or display device 45 can communicate with each other. In different examples, video decoding system 40 can include only video encoder 20 or only video decoder 30.

[0157] In some instances, antenna 48 may be used to transmit or receive a coded bit stream of video data. In addition, in some instances, display device 45 may be used to present video data. Processing circuit 46 may include application-specific integrated circuit (ASIC) logic, graphics processor, general purpose processor, etc. Video decoding system 40 may also include an optional processor 43, which may similarly include application-specific integrated circuit (ASIC) logic, graphics processor, general purpose processor, etc. In addition, memory storage 44 may be any type of memory, such as volatile memory (e.g., static random access memory (SRAM), dynamic random access memory (DRAM), etc.) or non-volatile memory (e.g., flash memory, etc.). In a non-limiting example, memory storage 44 may be implemented by cache memory. In other instances, processing circuit 46 may include memory (e.g., cache, etc.) for implementing image buffers, etc.

[0158] In some examples, video encoder 20 implemented by logic circuits may include an image buffer (e.g., implemented by processing circuit 46 or memory storage 44) and a graphics processing unit (e.g., implemented by processing circuit 46). The graphics processing unit may be communicatively coupled to the image buffer. The graphics processing unit may include video encoder 20 implemented by processing circuit 46 to implement reference Figure 3 and / or the various modules discussed in connection with any other encoder system or subsystem described herein. Logic circuits may be used to perform the various operations discussed herein.

[0159] In some examples, video decoder 30 may be implemented in a similar manner by processing circuitry 46 to implement reference Figure 4 The various modules discussed in connection with video decoder 30 and / or any other decoder system or subsystem described herein may be included in the video decoder 30 implemented by logic circuitry. In some examples, video decoder 30 implemented by logic circuitry may include an image buffer (implemented by processing circuitry 46 or memory storage 44) and a graphics processing unit (implemented by processing circuitry 46, for example). The graphics processing unit may be communicatively coupled to the image buffer. The graphics processing unit may include video decoder 30 implemented by processing circuitry 46 to implement the video decoder 30 implemented by processing circuitry 46. Figure 4 and / or the various modules discussed above for any other decoder system or subsystem described herein.

[0160] In some examples, antenna 48 may be used to receive an encoded bitstream of video data. As discussed, the encoded bitstream may include data related to the encoded video frames, indicators, index values, mode selection data, etc. discussed herein, such as data related to the encoded partitions (e.g., transform coefficients or quantized transform coefficients, (as discussed) optional indicators, and / or data defining the encoded partitions). Video decoding system 40 may also include video decoder 30 coupled to antenna 48 and used to decode the encoded bitstream. Display device 45 is used to present the video frames.

[0161] It should be understood that for the examples described with reference to the video encoder 20 in the embodiments of the present application, the video decoder 30 can be used to perform the reverse process. With respect to the signaling syntax elements, the video decoder 30 can be used to receive and parse such syntax elements and decode the related video data accordingly. In some examples, the video encoder 20 can entropy encode the syntax elements into an encoded video bitstream. In such examples, the video decoder 30 can parse such syntax elements and decode the related video data accordingly.

[0162] For ease of description, the embodiments of the present application are described with reference to the universal video coding (VVC) reference software or the high-efficiency video coding (HEVC) developed by the video coding joint working group (joint collaboration team on video coding, JCT-VC) of the ITU-T video coding experts group (VCEG) and the ISO / IEC motion picture experts group (MPEG). Those of ordinary skill in the art understand that the embodiments of the present application are not limited to HEVC or VVC.

[0163] Encoders and encoding methods

[0164] like Figure 3 As shown, the video encoder 20 includes an input terminal (or input interface) 201, a residual calculation unit 2040, a transform processing unit 2060, a quantization unit 2080, an inverse quantization unit 2100, an inverse transform processing unit 212, a reconstruction unit 214, a loop filter 220, a decoded picture buffer (decoded picture buffer, DPB) 230, a mode selection unit 260, an entropy coding unit 270 and an output terminal (or output interface) 272. The mode selection unit 260 may include an inter-frame prediction unit 244, an intra-frame prediction unit 254 and a segmentation unit 262. The inter-frame prediction unit 244 may include a motion estimation unit and a motion compensation unit (not shown). Figure 3 The illustrated video encoder 20 may also be referred to as a hybrid video encoder or a hybrid video codec-based video encoder.

[0165] See also Figure 3 The inter-frame prediction unit is a trained target model (also called a neural network) that processes an input image or an image region or an image block to generate a prediction value of the input image block. For example, the neural network for inter-frame prediction is used to receive an input image or an image region or an image block, and generate a prediction value of the input image or an image region or an image block.

[0166] The residual calculation unit 2040, the transform processing unit 2060, the quantization unit 2080 and the mode selection unit 260 constitute the forward signal path of the encoder 20, and the inverse quantization unit 2100, the inverse transform processing unit 212, the reconstruction unit 214, the buffer 216, the loop filter 220, the decoded picture buffer (DPB) 230, the inter-frame prediction unit 244 and the intra-frame prediction unit 254 constitute the backward signal path of the encoder, wherein the backward signal path of the encoder 20 corresponds to the signal path of the decoder (see Figure 3 The inverse quantization unit 210, the inverse transform processing unit 212, the reconstruction unit 214, the loop filter 220, the decoded image buffer 230, the inter-frame prediction unit 244 and the intra-frame prediction unit 254 also constitute a “built-in decoder” of the video encoder 20.

[0167] Images and image segmentation (images and blocks)

[0168] The encoder 20 may be configured to receive, via an input 201 or the like, an image (or image data) 17, e.g., an image in a sequence of images forming a video or a video sequence. The received image or image data may also be a pre-processed image (or pre-processed image data) 19. For simplicity, the following description uses image 17. Image 17 may also be referred to as a current image or an image to be encoded (particularly when the current image is to be distinguished from other images in video encoding, such as previously encoded images and / or decoded images in the same video sequence, i.e., a video sequence also including the current image).

[0169] A (digital) image is or can be considered as a two-dimensional array or matrix of pixels with intensity values. The pixels in the array can also be called pixels (pixel or pel) (short for picture elements). The number of pixels in the array or image in the horizontal and vertical directions (or axes) determines the size and / or resolution of the image. In order to represent color, three color components are usually used, that is, the image can be represented as or include three pixel arrays. In the RBG format or color space, the image includes corresponding red, green and blue pixel arrays. However, in video coding, each pixel is usually represented in a brightness / chrominance format or color space, such as YCbCr, including a brightness component indicated by Y (sometimes also represented by L) and two chrominance components represented by Cb and Cr. The brightness (luma) component Y represents the brightness or grayscale level intensity (for example, the two are the same in grayscale images), while the two chrominance (abbreviated as chroma) components Cb and Cr represent the chrominance or color information components. Accordingly, an image in YCbCr format includes a luminance pixel array of luminance pixel values ​​(Y) and two chrominance pixel arrays of chrominance values ​​(Cb and Cr). An image in RGB format can be converted or transformed into YCbCr format and vice versa, a process also referred to as color conversion or transformation. If the image is black and white, the image may include only a luminance pixel array. Accordingly, an image may be, for example, a luminance pixel array in a monochrome format or a luminance pixel array and two corresponding chrominance pixel arrays in 4:2:0, 4:2:2 and 4:4:4 color formats.

[0170] In one embodiment, an embodiment of the video encoder 20 may include an image segmentation unit ( Figure 2 ), for partitioning the image 17 into a plurality of (usually non-overlapping) image blocks 2030. These blocks may also be referred to as root blocks, macroblocks (H.264 / AVC) or coding tree blocks (CTB), or coding tree units (CTU) in the H.265 / HEVC and VVC standards. The partitioning unit may be used to use the same block size for all images in a video sequence and a corresponding grid of defined block sizes, or to change the block size between images or subsets of images or groups of images, and to partition each image into corresponding blocks.

[0171] In other embodiments, the video encoder may be configured to directly receive a block 2030 of the image 17, for example, one, several or all blocks constituting the image 17. The image block 2030 may also be referred to as a current image block or an image block to be encoded.

[0172] Like image 17, image block 2030 is also or can be considered as a two-dimensional array or matrix composed of pixels with intensity values ​​(pixel values), but image block 2030 is smaller than image 17. In other words, block 2030 may include one pixel array (e.g., a brightness array in the case of monochrome image 17 or a brightness array or chrominance array in the case of a color image) or three pixel arrays (e.g., one brightness array and two chrominance arrays in the case of a color image 17) or any other number and / or type of arrays according to the color format adopted. The number of pixels in the horizontal and vertical directions (or axes) of block 2030 defines the size of block 2030. Accordingly, the block can be an M×N (M columns×N rows) pixel array, or an M×N transform coefficient array, etc.

[0173] In one embodiment, Figure 3 The video encoder 20 shown is used to encode the image 17 block by block, for example, encoding and prediction are performed on each block 2030.

[0174] In one embodiment, Figure 3 The illustrated video encoder 20 may also be used to partition and / or encode an image using slices (also referred to as video slices), wherein an image may be partitioned or encoded using one or more slices (usually non-overlapping). Each slice may include one or more blocks (e.g., coding tree units CTUs) or one or more groups of blocks (e.g., coding tiles in the H.265 / HEVC / VVC standard and bricks in the VVC standard).

[0175] In one embodiment, Figure 3 The video encoder 20 shown can also be used to partition and / or encode an image using slices / coding block groups (also called video coding block groups) and / or coding blocks (also called video coding blocks), wherein an image can be partitioned or encoded using one or more slices / coding block groups (usually non-overlapping), each slice / coding block group may include one or more blocks (e.g., CTU) or one or more coding blocks, etc., wherein each coding block may be in a rectangular shape, etc., and may include one or more complete or partial blocks (e.g., CTU).

[0176] Residual calculation

[0177] The residual calculation unit 2040 is used to calculate the residual block 2050 (the prediction block 265 is described in detail later) based on the image block (or original block) 2030 and the prediction block 265 in the following manner: for example, the pixel value of the prediction block 265 is subtracted from the pixel value of the image block 2030 pixel by pixel (pixel by pixel) to obtain the residual block 2050 in the pixel domain.

[0178] Transform

[0179] The transform processing unit 2060 is used to perform discrete cosine transform (DCT) or discrete sine transform (DST) on the pixel values ​​of the residual block 2050 to obtain transform coefficients 2070 in the transform domain. The transform coefficients 2070 may also be called transform residual coefficients, representing the residual block 2050 in the transform domain.

[0180] The transform processing unit 2060 may be used to apply an integerized approximation of the DCT / DST, such as the transform specified for H.265 / HEVC. This integerized approximation is typically scaled by a factor compared to an orthogonal DCT transform. In order to maintain the norm of the residual block processed by the forward transform and the inverse transform, other scaling factors are used as part of the transform process. The scaling factor is typically selected based on certain constraints, such as whether the scaling factor is a power of 2 for the shift operation, the bit depth of the transform coefficients, a trade-off between accuracy and implementation cost, etc. For example, a specific scaling factor is specified for the inverse transform on the encoder 20 side by the inverse transform processing unit 212 (and for the corresponding inverse transform on the decoder 30 side by, for example, the inverse transform processing unit 312), and accordingly, a corresponding scaling factor may be specified for the forward transform on the encoder 20 side by the transform processing unit 2060.

[0181] In one embodiment, the video encoder 20 (correspondingly, the transform processing unit 2060) may be used to output transform parameters such as one or more transform types, for example, directly output or output after encoding or compression by the entropy coding unit 270, for example, so that the video decoder 30 may receive and use the transform parameters for decoding.

[0182] Quantification

[0183] The quantization unit 2080 is configured to quantize the transform coefficients 2070 by, for example, scalar quantization or vector quantization to obtain quantized transform coefficients 2090. The quantized transform coefficients 2090 may also be referred to as quantized residual coefficients 2090.

[0184] The quantization process may reduce the bit depth associated with some or all of the transform coefficients 2070. For example, an n-bit transform coefficient may be rounded down to an m-bit transform coefficient during quantization, where n is greater than m. The degree of quantization may be modified by adjusting a quantization parameter (QP). For example, for scalar quantization, different degrees of scaling may be applied to achieve finer or coarser quantization. A smaller quantization step size corresponds to a finer quantization, while a larger quantization step size corresponds to a coarser quantization. A suitable quantization step size may be indicated by a quantization parameter (QP). For example, a quantization parameter may be an index to a predefined set of suitable quantization step sizes. For example, a smaller quantization parameter may correspond to fine quantization (a smaller quantization step size), a larger quantization parameter may correspond to coarse quantization (a larger quantization step size), and vice versa. Quantization may include dividing by a quantization step size, while a corresponding or inverse dequantization performed by the inverse quantization unit 2100 or the like may include multiplying by a quantization step size. Embodiments according to some standards such as HEVC may be used to determine a quantization step size using a quantization parameter. In general, the quantization step size may be calculated using a fixed-point approximation of an equation containing division according to the quantization parameter. Other scaling factors may be introduced for quantization and dequantization to recover the norm of the residual block that may be modified due to the scaling used in the fixed-point approximation of the equations for the quantization step size and the quantization parameter. In one exemplary implementation, the scaling of the inverse transform and dequantization may be combined. Alternatively, a custom quantization table may be used and indicated from the encoder to the decoder in the bitstream, etc. Quantization is a lossy operation, where the larger the quantization step size, the greater the loss.

[0185] In one embodiment, the video encoder 20 (correspondingly, the quantization unit 2080) may be configured to output a quantization parameter (QP), for example, directly output or output after being encoded or compressed by the entropy coding unit 270, for example, so that the video decoder 30 may receive and use the quantization parameter for decoding.

[0186] Dequantization

[0187] The inverse quantization unit 2100 is used to perform the inverse quantization of the quantization unit 2080 on the quantized coefficients to obtain dequantized coefficients 2110, for example, performing the inverse quantization scheme of the quantization scheme performed by the quantization unit 2080 according to or using the same quantization step size as the quantization unit 2080. The dequantized coefficients 2110 may also be called dequantized residual coefficients 2110, corresponding to the transform coefficients 2070, but due to the loss caused by quantization, the dequantized coefficients 2110 are usually not completely the same as the transform coefficients.

[0188] Inverse Transform

[0189] The inverse transform processing unit 212 is used to perform an inverse transform of the transform performed by the transform processing unit 2060, such as an inverse discrete cosine transform (DCT) or an inverse discrete sine transform (DST), to obtain a reconstructed residual block 213 (or a corresponding dequantized coefficient 213) in the pixel domain. The reconstructed residual block 213 may also be referred to as a transform block 213.

[0190] reconstruction

[0191] The reconstruction unit 214 (e.g., the summer 214) is used to add the transform block 213 (i.e., the reconstructed residual block 213) to the prediction block 265 to obtain the reconstructed block 215 in the pixel domain, for example, by adding the pixel point values ​​of the reconstructed residual block 213 and the pixel point values ​​of the prediction block 265.

[0192] Filtering

[0193] The loop filter unit 220 (or simply "loop filter" 220) is used to filter the reconstruction block 215 to obtain the filter block 221, or is generally used to filter the reconstructed pixel points to obtain the filtered pixel point value. For example, the loop filter unit is used to smoothly perform pixel conversion or improve video quality. The loop filter unit 220 may include one or more loop filters, such as a deblocking filter, a sample-adaptive offset (SAO) filter, or one or more other filters, such as an adaptive loop filter (ALF), a noise suppression filter (NSF), or any combination. For example, the loop filter unit 220 may include a deblocking filter, an SAO filter, and an ALF filter. The order of the filtering process may be a deblocking filter, an SAO filter, and an ALF filter. For another example, a process called luma mapping with chromascaling (LMCS) (i.e., an adaptive in-loop shaper) is added. This process is performed before deblocking. For another example, the deblocking filtering process may also be applied to internal sub-block edges, such as affine sub-block edges, ATMVP sub-block edges, sub-block transform (SBT) edges, and intra sub-partition (ISP) edges. Figure 2 2 is shown as a loop filter, but in other configurations, the loop filter unit 220 can be implemented as a post-loop filter. The filter block 221 can also be called a filter reconstruction block 221.

[0194] In one embodiment, the video encoder 20 (correspondingly, the loop filter unit 220) may be used to output loop filter parameters (e.g., SAO filter parameters, ALF filter parameters, or LMCS parameters), for example, directly or after being entropy encoded by the entropy encoding unit 270, such that the decoder 30 may receive and use the same or different loop filter parameters for decoding.

[0195] Decoded Image Buffer

[0196] The decoded picture buffer (DPB) 230 may be a reference picture memory that stores reference picture data for use by the video encoder 20 when encoding video data. The DPB 230 may be formed by any of a variety of memory devices, such as a dynamic random access memory (DRAM), including synchronous DRAM (SDRAM), magnetoresistive RAM (MRAM), resistive RAM (RRAM), or other types of storage devices. The decoded picture buffer 230 may be used to store one or more filter blocks 221. The decoded picture buffer 230 may also be used to store other previous filter blocks of the same current image or a different image such as a previous reconstructed image, such as a previously reconstructed and filtered block 221, and may provide a complete previously reconstructed, i.e., decoded image (and corresponding reference blocks and pixels) and / or a partially reconstructed current image (and corresponding reference blocks and pixels), such as for inter-frame prediction. The decoded image buffer 230 may also be used to store one or more unfiltered reconstructed blocks 215, or generally to store unfiltered reconstructed pixels, for example, reconstructed blocks 215 that have not been filtered by the loop filter unit 220, or reconstructed blocks or reconstructed pixels that have not undergone any other processing.

[0197] Mode selection (segmentation and prediction)

[0198] The mode selection unit 260 includes a segmentation unit 262, an inter-frame prediction unit 244, and an intra-frame prediction unit 254 for obtaining a decoded image from the decoded image buffer 230 or other buffers (eg, column buffers, Figure 2 The decoder 200 (not shown) receives or obtains original image data such as an original block 2030 (current block 2030 of the current image 17) and reconstructed image data, for example, filtered and / or unfiltered reconstructed pixels or reconstructed blocks of the same (current) image and / or one or more previously decoded images. The reconstructed image data is used as reference image data required for prediction such as inter-frame prediction or intra-frame prediction to obtain a prediction block 265 or a prediction value 265.

[0199] The mode selection unit 260 may be used to determine or select a partition for the current block (including no partition) and prediction mode (eg, intra-frame or inter-frame prediction mode), generate a corresponding prediction block 265 , calculate the residual block 2050 , and reconstruct the reconstruction block 215 .

[0200] In one embodiment, the mode selection unit 260 may be used to select a segmentation and prediction mode (e.g., from prediction modes supported or available by the mode selection unit 260), the prediction mode providing the best match or minimum residual (minimum residual means better compression in transmission or storage), or providing minimum signaling overhead (minimum signaling overhead means better compression in transmission or storage), or considering or balancing both of the above. The mode selection unit 260 may be used to determine the segmentation and prediction mode based on rate distortion optimization (RDO), i.e., selecting the prediction mode that provides minimum rate distortion optimization. The terms "best", "lowest", "optimal", etc. herein do not necessarily mean "best", "lowest", "optimal" in general, but may also refer to situations where termination or selection criteria are met, for example, values ​​exceeding or falling below a threshold or other restrictions may result in a "suboptimal selection" but reduce complexity and processing time.

[0201] In other words, the partitioning unit 262 may be used to partition an image in a video sequence into a sequence of coding tree units (CTUs), which may be further partitioned into smaller block portions or sub-blocks (again forming blocks), for example, by iteratively using quad-tree partitioning (QT), binary-tree partitioning (BT) or triple-tree partitioning (TT) or any combination thereof, and for performing prediction on each of the block portions or sub-blocks, for example, wherein the mode selection comprises selecting a tree structure for the partitioned blocks and selecting a prediction mode to be applied to each of the block portions or sub-blocks.

[0202] The segmentation (eg, performed by segmentation unit 262) and prediction processes (eg, performed by inter-prediction unit 244 and intra-prediction unit 254) performed by video encoder 20 are described in detail below.

[0203] segmentation

[0204] The segmentation unit 262 may segment (or divide) an image block (or CTU) into smaller parts, such as small blocks of square or rectangular shape. For an image with three pixel arrays, a CTU consists of N×N luminance pixel blocks and two corresponding chrominance pixel blocks. The maximum allowed size of a luminance block in a CTU is specified as 128×128 in the versatile video coding (VVC) standard under development, but may be specified as a value different from 128×128 in the future, such as 256×256. The CTUs of an image may be concentrated / grouped into slices / coding block groups, coding blocks, or bricks. A coding block covers a rectangular area of ​​an image, and a coding block may be divided into one or more bricks. A brick consists of multiple CTU rows within a coding block. A coding block that is not segmented into multiple bricks may be called a brick. However, a brick is a true subset of a coding block and is therefore not called a coding block. VVC supports two coding block group modes, namely raster scan slice / coding block group mode and rectangular slice mode. In raster scan CBG mode, a slice / CBG contains a sequence of CBs in a raster scan of CBs of an image. In rectangular slice mode, a slice contains multiple bricks of an image, which together form a rectangular region of the image. The bricks within a rectangular slice are arranged in the order of the bricks in the slice's raster scan. These smaller blocks (also called sub-blocks) can be further split into smaller parts. This is also called tree partitioning or hierarchical tree partitioning, where a root block at root tree level 0 (hierarchy level 0, depth 0), etc., can be recursively split into two or more blocks at the next lower tree level, such as nodes at tree level 1 (hierarchy level 1, depth 1). These blocks can be split into two or more blocks at the next lower level, such as tree level 2 (hierarchy level 2, depth 2), etc., until the partitioning ends (because the end criteria are met, such as reaching the maximum tree depth or minimum block size). Blocks that are not further split are also called leaf blocks or leaf nodes of the tree. A tree divided into two parts is called a binary tree (BT), a tree divided into three parts is called a ternary tree (TT), and a tree divided into four parts is called a quadtree (QT).

[0205] For example, a coding tree unit (CTU) may be or include a CTB of luminance pixels, two corresponding CTBs of chrominance pixels of an image with three pixel arrays, or a CTB of pixels of a monochrome image, or a CTB of pixels of an image encoded using three independent color planes and syntax structures (for encoding pixels). Accordingly, a coding tree block (CTB) may be an N×N pixel block, where N may be set to a certain value so that the component is divided into CTBs, which is segmentation. A coding unit (coding unit, CU) may be or include a coding block of luminance pixels, two corresponding coding blocks of chrominance pixels of an image with three pixel arrays, or a coding block of pixels of a monochrome image, or a coding block of pixels of an image encoded using three independent color planes and syntax structures (for encoding pixels). Accordingly, a coding block (CB) may be an M×N pixel block, where M and N may be set to a certain value so that the CTB is divided into coding blocks, which is segmentation.

[0206] For example, in an embodiment, a coding tree unit (CTU) may be divided into multiple CUs according to HEVC by using a quadtree structure represented as a coding tree. A decision is made at the leaf-CU level whether to use inter-frame (temporal) prediction or intra-frame (spatial) prediction to encode an image area. Each leaf-CU may be further divided into one, two, or four PUs according to the PU partition type. The same prediction process is used within a PU, and relevant information is transmitted to the decoder in units of PUs. After the prediction process is applied according to the PU partition type to obtain the residual block, the leaf-CU may be partitioned into transform units (TUs) according to other quadtree structures similar to the coding tree for the CU.

[0207] For example, in an embodiment, according to the latest video coding standard currently under development (referred to as Versatile Video Coding (VVC), a combined quadtree of nested multi-type trees (e.g., binary trees and ternary trees) is used to partition a segment structure for partitioning a coding tree unit. In the coding tree structure within the coding tree unit, the CU may be a square or a rectangle. For example, a coding tree unit (CTU) is first partitioned by a quadtree structure. The quadtree leaf nodes are further partitioned by a multi-type tree structure. The multi-type tree structure has four partition types: vertical binary tree partition (SPLIT_BT_VER), horizontal binary tree partition (SPLIT_BT_HOR), vertical ternary tree partition The leaf nodes of the multi-type tree are called coding units (CUs), unless the CU is too large for the maximum transform length, such segmentation is used for prediction and transform processing without any other segmentation. In most cases, this means that the block sizes of CUs, PUs, and TUs in the coding block structure of the quadtree nested multi-type tree are the same. This exception occurs when the maximum supported transform length is less than the width or height of the color components of the CU. VVC has established a unique signaling mechanism for the split division information in the coding structure with quadtree nested multi-type trees. In the signaling mechanism, the coding The tree unit (CTU) as the root of the quadtree is first split by the quadtree structure. Then each quadtree leaf node (when large enough) is further split into a multi-type tree structure. In the multi-type tree structure, the first flag (mtt_split_cu_flag) indicates whether the node is further split. When the node is further split, the second flag (mtt_split_cu_vertical_flag) is used to indicate the division direction, and then the third flag (mtt_split_cu_binary_flag) is used to indicate whether the division is a binary tree division or a ternary tree division. According to mtt_split_c The values ​​of u_vertical_flag and mtt_split_cu_binary_flag allow the decoder to derive the multi-type tree splitting mode (MttSplitMode) of the CU based on predefined rules or tables. It should be noted that for certain designs, such as the 64×64 luma block and 32×32 chroma pipeline design in the VVC hardware decoder, TT splitting is not allowed when the width or height of the luma coding block is greater than 64. TT splitting is also not allowed when the width or height of the chroma coding block is greater than 32. The pipeline design divides the image into multiple virtual pipeline data units (VPDUs), each VPDU is defined as a non-overlapping unit in the image. In the hardware decoder, consecutive VPDUs are processed simultaneously in multiple pipeline stages. In most pipeline stages, the VPDU size is roughly proportional to the buffer size, so it is necessary to keep the VPDU small.In most hardware decoders, the VPDU size can be set to the maximum transform block (TB) size. However, in VVC, the ternary tree (TT) and binary tree (BT) partitioning may increase the VPDU size.

[0208] In addition, it should be noted that when a part of the tree node block exceeds the bottom or the right boundary of the image, the tree node block is forcibly divided until all the pixels of each coding CU are located within the image boundary.

[0209] For example, the intra sub-partitions (ISP) tool may vertically or horizontally divide the luma intra prediction block into two or four sub-partitions according to the block size.

[0210] In one example, mode select unit 260 of video encoder 20 may be used to perform any combination of the segmentation techniques described above.

[0211] As described above, the video encoder 20 is operable to determine or select the best or optimal prediction mode from a (predetermined) set of prediction modes. The set of prediction modes may include, for example, an intra prediction mode and / or an inter prediction mode.

[0212] Intra-frame prediction

[0213] The intra prediction mode set may include 35 different intra prediction modes, for example, non-directional modes like DC (or mean) mode and plane mode, or directional modes as defined in HEVC, or may include 67 different intra prediction modes, for example, non-directional modes like DC (or mean) mode and plane mode, or directional modes as defined in VVC. For example, several traditional angle intra prediction modes are adaptively replaced with wide-angle intra prediction modes for non-square blocks defined in VVC. For another example, in order to avoid the division operation of DC prediction, only the longer side is used to calculate the average value of non-square blocks. In addition, the intra prediction result of the plane mode can also be modified using the position-dependent intra prediction combination (PDPC) method.

[0214] The intra prediction unit 254 is configured to generate an intra prediction block 265 by reconstructing pixels in adjacent blocks of the same current image according to an intra prediction mode in the intra prediction mode set.

[0215] The intra-frame prediction unit 254 (or generally the mode selection unit 260) is also used to output intra-frame prediction parameters (or generally information indicating the selected intra-frame prediction mode of the block) in the form of syntax elements 266 to the entropy coding unit 270 for inclusion in the encoded image data 21, so that the video decoder 30 can perform operations, such as receiving and using the prediction parameters for decoding.

[0216] The intra prediction modes in HEVC include DC prediction mode, plane prediction mode and 33 angle prediction modes, with a total of 35 candidate prediction modes. The current block can use the pixels of the reconstructed image blocks on the left and above as references for intra prediction. The image blocks in the surrounding area of ​​the current block used for intra prediction of the current block become reference blocks, and the pixels in the reference blocks are called reference pixels. Among the 35 candidate prediction modes, the DC prediction mode is applicable to the area with flat texture in the current block, and all pixels in this area use the average value of the reference pixels in the reference block as prediction; the plane prediction mode is applicable to image blocks with smoothly changing textures. The current block that meets this condition uses the reference pixels in the reference block for bilinear interpolation as the prediction of all pixels in the current block; the angle prediction mode uses the characteristics that the texture of the current block is highly correlated with the texture of the adjacent reconstructed image blocks, and copies the values ​​of the reference pixels in the corresponding reference block along a certain angle as the prediction of all pixels in the current block.

[0217] The HEVC encoder selects an optimal intra-frame prediction mode for the current block from 35 candidate prediction modes and writes the optimal intra-frame prediction mode into the video bitstream. To improve the coding efficiency of intra-frame prediction, the encoder / decoder derives three most likely modes from the optimal intra-frame prediction modes of the reconstructed image blocks in the surrounding area using intra-frame prediction. If the optimal intra-frame prediction mode selected for the current block is one of the three most likely modes, a first index is encoded to indicate that the selected optimal intra-frame prediction mode is one of the three most likely modes; if the selected optimal intra-frame prediction mode is not one of the three most likely modes, a second index is encoded to indicate that the selected optimal intra-frame prediction mode is one of the other 32 modes (other modes among the 35 candidate prediction modes except the aforementioned three most likely modes). The HEVC standard uses a 5-bit fixed-length code as the aforementioned second index.

[0218] The method by which the HEVC encoder derives the three most likely modes includes: selecting the optimal intra-frame prediction modes of the left adjacent image block and the upper adjacent image block of the current block and putting them into a set. If the two optimal intra-frame prediction modes are the same, only one is retained in the set. If the two optimal intra-frame prediction modes are the same and both are angle prediction modes, two angle prediction modes adjacent to the angle direction are selected to be added to the set; otherwise, the plane prediction mode, the DC mode, and the vertical prediction mode are selected in turn to be added to the set until the number of modes in the set reaches 3.

[0219] After the HEVC decoder performs entropy decoding on the bitstream, it obtains the mode information of the current block, which includes an indication flag indicating whether the optimal intra-frame prediction mode of the current block is among the three most likely modes, and the index of the optimal intra-frame prediction mode of the current block among the three most likely modes or the index of the optimal intra-frame prediction mode of the current block among the other 32 modes.

[0220] Inter prediction

[0221] In a possible implementation, the set of inter-prediction modes depends on the available reference picture (i.e., at least part of the previously decoded picture stored in the DBP 230 as mentioned above) and other inter-prediction parameters, for example, on whether to use the entire reference picture or only a part of the reference picture, such as a search window area around the area of ​​the current block, to search for the best matching reference block, and / or on whether to perform pixel interpolation such as half-pixel, quarter-pixel and / or 1 / 16 interpolation, for example.

[0222] In addition to the above prediction modes, skip mode and / or direct mode may also be employed.

[0223] For example, extended merge prediction, the merge candidate list of this mode consists of the following five candidate types in order: spatial MVP from spatially adjacent CUs, temporal MVP from collocated CUs, history-based MVP from FIFO tables, pairwise average MVP, and zero MV. Decoder side motion vector refinement (DMVR) based on bilateral matching can be used to increase the accuracy of the MV of the merge mode. Merge mode with MVD (MMVD) is derived from the merge mode with motion vector difference. The MMVD flag is sent immediately after the skip flag and merge flag are sent to specify whether the CU uses the MMVD mode. The CU-level adaptive motion vector resolution (AMVR) scheme can be used. AMVR supports encoding the MVD of the CU with different precisions. The MVD of the current CU is adaptively selected according to the prediction mode of the current CU. When the CU is encoded in merge mode, the combined inter / intra prediction (CIIP) mode can be applied to the current CU. The CIIP prediction is obtained by weighted averaging the inter and intra prediction signals. For affine motion compensated prediction, the affine motion field of the block is described by the motion information of 2 control points (4 parameters) or 3 control points (6 parameters) motion vectors. Subblock-based temporal motion vector prediction (SbTMVP) is similar to the temporal motion vector prediction (TMVP) in HEVC, but predicts the motion vector of the sub-CU within the current CU. Bidirectional optical flow (BDOF), formerly known as BIO, is a simplified version that reduces calculations, especially in terms of the number of multiplications and the size of the multipliers. In triangle partitioning mode, the CU is evenly divided into two triangular parts using diagonal partitioning and anti-diagonal partitioning. In addition, the bidirectional prediction mode is extended on the basis of simple averaging to support weighted averaging of two prediction signals.

[0224] The inter-frame prediction unit 244 may include a motion estimation (ME) unit and a motion compensation (MC) unit (both in Figure 2). The motion estimation unit may be configured to receive or acquire an image block 2030 (the current image block 2030 of the current image 17) and a decoded image 231, or at least one or more previously reconstructed blocks, for example, reconstructed blocks of one or more other / different previously decoded images 231, for motion estimation. For example, a video sequence may include a current image and a previously decoded image 231, or in other words, the current image and the previously decoded image 231 may be part of or form a sequence of images forming the video sequence.

[0225] For example, the encoder 20 may be configured to select a reference block from a plurality of reference blocks of the same or different images in a plurality of other images, and provide the reference image (or reference image index) and / or the offset (spatial offset) between the position (x, y coordinates) of the reference block and the position of the current block as an inter-frame prediction parameter to the motion estimation unit. The offset is also referred to as a motion vector (MV).

[0226] The motion compensation unit is used to obtain, for example, receive, inter-frame prediction parameters, and perform inter-frame prediction based on or using the inter-frame prediction parameters to obtain an inter-frame prediction block 246. The motion compensation performed by the motion compensation unit may include extracting or generating a prediction block based on a motion / block vector determined by motion estimation, and may also include performing interpolation with sub-pixel accuracy. Interpolation filtering can generate pixel points of other pixels from pixel points of known pixels, thereby potentially increasing the number of candidate prediction blocks that can be used to encode the image block. Once a motion vector corresponding to a PU of a current image block is received, the motion compensation unit can locate the prediction block pointed to by the motion vector in one of the reference image lists.

[0227] The motion compensation unit may also generate syntax elements associated with blocks and video slices for use by the video decoder 30 when decoding image blocks of the video slice. In addition, or as an alternative to slices and corresponding syntax elements, coding block groups and / or coding blocks and corresponding syntax elements may be generated or used.

[0228] In the process of obtaining a candidate motion vector list in an advanced motion vector prediction (AMVP) mode, motion vectors (MVs) that can be added to the candidate motion vector list as alternatives include MVs of image blocks that are spatially adjacent and temporally adjacent to the current block, wherein the MVs of the image blocks that are spatially adjacent may include the MV of a left candidate image block located on the left side of the current block and the MV of an upper candidate image block located above the current block. For example, please refer to Figure 6 , Figure 6 An exemplary schematic diagram of a candidate image block provided in an embodiment of the present application is as follows: Figure 6As shown, the set of left candidate image blocks includes {A0, A1}, the set of upper candidate image blocks includes {B0, B1, B2}, and the set of temporally adjacent candidate image blocks includes {C, T}. All three sets can be added to the candidate motion vector list as alternatives. However, according to the existing coding standard, the maximum length of the candidate motion vector list of AMVP is 2, so it is necessary to determine the MV of adding up to two image blocks to the candidate motion vector list from the three sets according to the prescribed order. The order can be to give priority to the set of left candidate image blocks {A0, A1} of the current block (consider A0 first, and then consider A1 if A0 is not available), then consider the set of upper candidate image blocks {B0, B1, B2} of the current block (consider B0 first, and then consider B1 if B0 is not available, and then consider B2 if B1 is not available), and finally consider the set of temporally adjacent candidate image blocks {C, T} of the current block (consider T first, and then consider C if T is not available).

[0229] After obtaining the above candidate motion vector list, the optimal MV is determined from the candidate motion vector list by the rate distortion cost (RDcost), and the candidate motion vector with the smallest RD cost is used as the motion vector predictor (MVP) of the current block. The rate distortion cost is calculated by the following formula:

[0230] J=SAD+λR

[0231] Wherein, J represents RD cost, SAD is the sum of absolute differences (SAD) between the pixel value of the predicted block obtained after motion estimation using the candidate motion vector and the pixel value of the current block, R represents the bit rate, and λ represents the Lagrange multiplier.

[0232] The encoder passes the index of the determined MVP in the candidate motion vector list to the decoder. Furthermore, a motion search can be performed in the neighborhood centered on the MVP to obtain the actual motion vector of the current block. The encoder calculates the motion vector difference (MVD) between the MVP and the actual motion vector, and also passes the MVD to the decoder. The decoder parses the index, finds the corresponding MVP in the candidate motion vector list based on the index, parses the MVD, and adds the MVD to the MVP to obtain the actual motion vector of the current block.

[0233] In the process of obtaining the candidate motion information list in the merging mode, the motion information that can be added to the candidate motion information list as an alternative includes the motion information of the image blocks that are spatially adjacent or temporally adjacent to the current block, wherein the image blocks that are spatially adjacent and the image blocks that are temporally adjacent can refer to Figure 6 , the candidate motion information corresponding to the spatial domain in the candidate motion information list comes from the five spatially adjacent blocks (A0, A1, B0, B1 and B2). If the spatially adjacent block is not available or is intra-predicted, its motion information will not be added to the candidate motion information list. The candidate motion information of the current block in the time domain is obtained by scaling the MV of the block at the corresponding position in the reference frame according to the picture order count (POC) of the reference frame and the current frame. First, determine whether the block at position T in the reference frame is available. If not, select the block at position C. After obtaining the above candidate motion information list, the optimal motion information is determined from the candidate motion information list through the RD cost as the motion information of the current block. The encoder passes the index value (denoted as mergeindex) of the position of the optimal motion information in the candidate motion information list to the decoder.

[0234] Entropy Coding

[0235] The entropy coding unit 270 is used to apply an entropy coding algorithm or scheme (e.g., a variable length coding (VLC) scheme, a context adaptive VLC scheme (CALVC), an arithmetic coding scheme, a binarization algorithm, a context adaptive binary arithmetic coding (CABAC), a syntax-based context-adaptive binary arithmetic coding (SBAC), a probability interval partitioning entropy (PIPE) coding, or other entropy coding methods or techniques) to the quantized residual coefficients 2090, inter-frame prediction parameters, intra-frame prediction parameters, loop filter parameters, and / or other syntax elements to obtain coded image data 21 that can be output in the form of a coded bit stream 21 through an output terminal 272, so that the video decoder 30, etc. can receive and use the parameters for decoding. The coded bit stream 21 can be transmitted to the video decoder 30, or it can be stored in a memory for later transmission or retrieval by the video decoder 30.

[0236] Other structural variations of the video encoder 20 may be used to encode the video stream. For example, a non-transform based encoder 20 may directly quantize the residual signal without a transform processing unit 2060 for certain blocks or frames. In another implementation, the encoder 20 may have a quantization unit 2080 and an inverse quantization unit 2100 combined into a single unit.

[0237] Figure 4 : is a schematic block diagram of a hybrid architecture video encoder. A to-be-encoded image is first divided into non-overlapping image blocks and each image block is processed in a given order (e.g., a row scanning order). The image block is a set of pixels. For example, if the block size is NxN, it means that the block is a two-dimensional pixel array, and its horizontal and vertical sizes are both N. The encoder performs a prediction operation on the image block to obtain the predicted values ​​of the pixels therein. The set of predicted values ​​of the pixels in the image block is called the prediction of the image block. The difference between the original value of the image block pixel and the predicted value of the image block pixel is further calculated. The set of predicted difference values ​​of the pixels in the image block is called the residual of the image block. The residual is transformed and quantized to obtain the quantized transform coefficients. Finally, the entropy coding operation is performed on the quantized transform coefficients and the bit string output by the entropy coding is written into the bit stream. At the same time, the transformed and quantized transform coefficients are subjected to inverse quantization and inverse transformation operations to obtain the reconstructed residual, and then the reconstructed residual is added to the prediction of the current coding block to obtain the reconstruction of the block. The reconstruction of the current block will be used to predict the subsequent coding block, which can be the image block to be processed in the current image or the image block in the subsequent image to be encoded. The reconstructed image block is subjected to the loop filtering module to remove coding distortions such as block effects and ringing effects, and the output filtered reconstructed image can be used as a reference for intra-frame prediction of the subsequent coded image. Prediction in video coding can generally be divided into two categories: intra-frame prediction and inter-frame prediction. Intra-frame prediction uses the reconstruction of the coded blocks on the left and above to predict the current image block. Each image block can select the optimal prediction mode from a variety of candidate prediction modes to generate a prediction for the block. For example, the HEVC standard specifies 35 candidate intra-frame prediction modes. The prediction mode information is also written into the bitstream via entropy coding. Inter-frame prediction uses the reconstruction of the coded image as a reference image, searches for image blocks similar to the coding blocks in the current coded image from the reference image, and performs necessary interpolation filtering to obtain the prediction of the coding block.

[0238] Decoder and decoding method

[0239] like Figure 5As shown, the video decoder 30 is used to receive the coded image data 21 (e.g., the coded bitstream 21) encoded by the encoder 20, and obtain a decoded image 331. The coded image data or bitstream includes information for decoding the coded image data, such as data representing image blocks of a coded video slice (and / or a coding block group or coding block) and related syntax elements.

[0240] exist Figure 5 In the example of FIG. 3 , decoder 30 includes an entropy decoding unit 304, an inverse quantization unit 310, an inverse transform processing unit 312, a reconstruction unit 314 (e.g., a summer 314), a loop filter 320, a decoded picture buffer (DBP) 330, a mode application unit 360, an inter-frame prediction unit 344, and an intra-frame prediction unit 354. Inter-frame prediction unit 344 may be or include a motion compensation unit. In some examples, video decoder 30 may perform substantially the same as reference Figure 2 The video encoder describes the decoding process which is the opposite of the encoding process.

[0241] As described for encoder 20, inverse quantization unit 2100, inverse transform processing unit 212, reconstruction unit 214, loop filter 220, decoded picture buffer DPB 230, inter prediction unit 344, and intra prediction unit 354 also constitute a "built-in decoder" of video encoder 20. Accordingly, inverse quantization unit 310 may be functionally identical to inverse quantization unit, inverse transform processing unit 312 may be functionally identical to inverse transform processing unit 122, reconstruction unit 314 may be functionally identical to reconstruction unit 214, loop filter 320 may be functionally identical to loop filter 220, and decoded picture buffer 330 may be functionally identical to decoded picture buffer 230. Therefore, the explanation of the corresponding units and functions of video encoder 20 applies accordingly to the corresponding units and functions of video decoder 30.

[0242] Entropy decoding

[0243] The entropy decoding unit 304 is used to parse the bit stream 21 (or generally the encoded image data 21) and perform entropy decoding on the encoded image data 21 to obtain the quantization coefficients 309 and / or the decoded encoding parameters ( Figure 3The entropy decoding unit 304 may be used to provide the inter-frame prediction parameters, intra-frame prediction parameters and / or other syntax elements to the mode application unit 360, and to provide other parameters to other units of the decoder 30. The video decoder 30 may receive syntax elements at the video slice and / or video block level. In addition, or as an alternative to slices and corresponding syntax elements, coding block groups and / or coding blocks and corresponding syntax elements may be received or used.

[0244] Dequantization

[0245] The inverse quantization unit 310 may be configured to receive a quantization parameter (QP) (or generally information related to inverse quantization) and a quantization coefficient from the encoded image data 21 (e.g., parsed and / or decoded by the entropy decoding unit 304), and inverse quantize the decoded quantization coefficient 309 based on the quantization parameter to obtain an inverse quantization coefficient 311, which may also be referred to as a transform coefficient 311. The inverse quantization process may include using the quantization parameter calculated by the video encoder 20 for each video block in the video slice to determine a degree of quantization, and also determine a degree of inverse quantization that needs to be performed.

[0246] Inverse Transform

[0247] The inverse transform processing unit 312 may be configured to receive the dequantized coefficients 311, also referred to as transform coefficients 311, and apply a transform to the dequantized coefficients 311 to obtain a reconstructed residual block 213 in the pixel domain. The reconstructed residual block 213 may also be referred to as a transform block 313. The transform may be an inverse transform, such as an inverse DCT, an inverse DST, an inverse integer transform, or a conceptually similar inverse transform process. The inverse transform processing unit 312 may also be configured to receive transform parameters or corresponding information from the encoded image data 21 (e.g., parsed and / or decoded by the entropy decoding unit 304) to determine the transform applied to the dequantized coefficients 311.

[0248] reconstruction

[0249] The reconstruction unit 314 (eg, the summer 314 ) is used to add the reconstructed residual block 313 to the prediction block 365 to obtain the reconstructed block 315 in the pixel domain, for example, by adding the pixel point values ​​of the reconstructed residual block 313 and the pixel point values ​​of the prediction block 365 .

[0250] Filtering

[0251] The loop filter unit 320 (in or after the encoding loop) is used to filter the reconstruction block 315 to obtain a filter block 321, so as to smoothly perform pixel conversion or improve video quality, etc. The loop filter unit 320 may include one or more loop filters, such as a deblocking filter, a sample-adaptive offset (SAO) filter, or one or more other filters, such as an adaptive loop filter (ALF), a noise suppression filter (NSF), or any combination. For example, the loop filter unit 220 may include a deblocking filter, an SAO filter, and an ALF filter. The order of the filtering process may be a deblocking filter, an SAO filter, and an ALF filter. For another example, a process called luma mapping with chromascaling (LMCS) (i.e., an adaptive in-loop shaper) is added. This process is performed before deblocking. For another example, the deblocking filtering process may also be applied to internal sub-block edges, such as affine sub-block edges, ATMVP sub-block edges, sub-block transform (SBT) edges, and intra sub-partition (ISP) edges. Figure 3 3. Although shown as a loop filter in FIG. 3, in other configurations, the loop filter unit 320 may be implemented as a post-loop filter.

[0252] Decoded Image Buffer

[0253] The decoded video blocks 321 in one picture are then stored in a decoded picture buffer 330 which stores the decoded pictures 331 as reference pictures for subsequent motion compensation of other pictures and / or respective output displays.

[0254] The decoder 30 is used to output the decoded image 311 through the output terminal 312, etc., to display it to the user or for the user to view it.

[0255] predict

[0256] The inter-frame prediction unit 344 may be functionally the same as the inter-frame prediction unit 244 (particularly the motion compensation unit), and the intra-frame prediction unit 354 may be functionally the same as the inter-frame prediction unit 254, and may determine the division or segmentation and perform prediction based on the segmentation and / or prediction parameters or corresponding information received from the coded image data 21 (e.g., parsed and / or decoded by the entropy decoding unit 304). The mode application unit 360 may be used to perform prediction (intra-frame or inter-frame prediction) of each block according to the reconstructed image, block or corresponding pixel point (filtered or unfiltered), and obtain a prediction block 365.

[0257] When the video slice is encoded as an intra coded (I) slice, the intra prediction unit 354 in the mode application unit 360 is used to generate a prediction block 365 for the image block of the current video slice based on the indicated intra prediction mode and data from the previously decoded block of the current image. When the video image is encoded as an inter coded (i.e., B or P) slice, the inter prediction unit 344 (e.g., motion compensation unit) in the mode application unit 360 is used to generate a prediction block 365 for the video block of the current video slice based on the motion vector and other syntax elements received from the entropy decoding unit 304. For inter prediction, these prediction blocks can be generated from one of the reference images in one of the reference image lists. The video decoder 30 can construct reference frame list 0 and list 1 using a default construction technique based on the reference images stored in the DPB 330. The same or similar processes may be applied to embodiments of coding block groups (e.g., video coding block groups) and / or coding blocks (e.g., video coding blocks) in addition to or as an alternative to slices (e.g., video slices), for example, a video may be encoded using I, P or B coding block groups and / or coding blocks.

[0258] The mode application unit 360 is used to determine prediction information for a video block of a current video slice by parsing motion vectors and other syntax elements, and use the prediction information to generate a prediction block for the current video block being decoded. For example, the mode application unit 360 uses some of the received syntax elements to determine the prediction mode (e.g., intra-frame prediction or inter-frame prediction) for encoding the video block of the video slice, the inter-frame prediction slice type (e.g., B slice, P slice, or GPB slice), the construction information of one or more reference picture lists for the slice, the motion vector for each inter-frame coded video block of the slice, the inter-frame prediction state for each inter-frame coded video block of the slice, and other information to decode the video block in the current video slice. In addition to slices (e.g., video slices) or as an alternative to slices, the same or similar process can be applied to embodiments of coding block groups (e.g., video coding block groups) and / or coding blocks (e.g., video coding blocks), for example, the video can be encoded using I, P, or B coding block groups and / or coding blocks.

[0259] In one embodiment, Figure 5The video decoder 30 may also be configured to partition and / or decode an image using slices (also referred to as video slices), wherein an image may be partitioned or decoded using one or more slices (usually non-overlapping). Each slice may include one or more blocks (e.g., CTUs) or one or more groups of blocks (e.g., coding blocks in the H.265 / HEVC / VVC standard and bricks in the VVC standard).

[0260] In one embodiment, Figure 5 The video decoder 30 shown can also be used to segment and / or decode an image using slices / coding block groups (also called video coding block groups) and / or coding blocks (also called video coding blocks), wherein an image can be segmented or decoded using one or more slices / coding block groups (usually non-overlapping), each slice / coding block group may include one or more blocks (e.g., CTU) or one or more coding blocks, etc., wherein each coding block may be in a rectangular shape, etc., and may include one or more complete or partial blocks (e.g., CTU).

[0261] Other variations of the video decoder 30 may be used to decode the encoded image data 21. For example, the decoder 30 may generate an output video stream without the loop filter unit 320. For example, a non-transform based decoder 30 may directly dequantize the residual signal without the inverse transform processing unit 312 for certain blocks or frames. In another implementation, the video decoder 30 may have the dequantization unit 310 and the inverse transform processing unit 312 combined into a single unit.

[0262] It should be understood that the processing result of the current step can be further processed in the encoder 20 and the decoder 30 and then output to the next step. For example, after interpolation filtering, motion vector derivation or loop filtering, the processing result of interpolation filtering, motion vector derivation or loop filtering can be further operated, such as clipping or shifting operation.

[0263] It should be noted that the derived motion vector of the current block (including but not limited to the control point motion vector of the affine mode, the sub-block motion vector of the affine, plane, ATMVP mode, the temporal motion vector, etc.) can be further operated. For example, the value of the motion vector is limited to a predefined range according to the representation bit of the motion vector. If the representation bit of the motion vector is bitDepth, the range is -2^(bitDepth-1) to 2^(bitDepth-1)-1, where "^" represents the power. For example, if bitDepth is set to 16, the range is -32768 to 32767; if bitDepth is set to 18, the range is -131072 to 131071. For example, the value of the derived motion vector (e.g., the MV of 4 4×4 sub-blocks in an 8×8 block) is limited so that the maximum difference between the integer parts of the 4 4×4 sub-block MVs does not exceed N pixels, for example, not more than 1 pixel. Two methods of limiting motion vectors according to bitDepth are provided here.

[0264] Although the above embodiments mainly describe video coding and decoding, it should be noted that the embodiments of the decoding system 10, the encoder 20 and the decoder 30 and other embodiments described herein can also be used for still image processing or coding and decoding, that is, the processing or coding and decoding of a single image in video coding and decoding that is independent of any previous or consecutive images. In general, if the image processing is limited to a single image 17, the inter-frame prediction unit 244 (encoder) and the inter-frame prediction unit 344 (decoder) may not be available. All other functions (also called tools or techniques) of the video encoder 20 and the video decoder 30 can also be used for still image processing, such as residual calculation 2040 / 304, transformation 2060, quantization 2080, inverse quantization 2100 / 310, (inverse) transformation 212 / 312, segmentation 262 / 362, intra-frame prediction 254 / 354 and / or loop filtering 220 / 320, entropy encoding 270 and entropy decoding 304.

[0265] Figure 7 This is a schematic block diagram of a hybrid architecture video decoder. After the decoder obtains the bitstream of an image block encoded and output by the encoder, it obtains the prediction mode information and transform quantization coefficients through entropy decoding. The decoder performs corresponding prediction operations based on the prediction mode with reference to the reference pixels of the surrounding decoded image area to obtain the prediction of the current image block. The decoder performs inverse quantization and inverse transformation operations on the transform quantization coefficients obtained by entropy decoding to obtain the reconstructed residual. Then, the prediction and reconstruction residuals of the current block are added to obtain the reconstruction of the current block. Note that both the encoder and the decoder process each image block in a pre-agreed order, so the encoder does not need to transmit additional information about the processing order of the encoding blocks to the decoder.

[0266] It should be noted that in the hybrid architecture video codec solution, the operations of each module in the codec and decoder have a corresponding relationship. For example, in the encoder and decoder, the predictor operation is exactly the same, while the entropy encoder and entropy decoder, the transformer and the inverse transformer, the quantizer and the inverse quantizer, etc. are all paired inverse operations. Therefore, after the prediction, transformation, quantization, entropy coding and other operations of the encoding end are specified, the prediction, inverse transformation, inverse quantization, entropy decoding and other operations of the decoding end are also determined. And vice versa.

[0267] Please note that Figure 5 and Figure 7 A schematic diagram of a general hybrid architecture video codec is described. The same functional module can be implemented differently in different coding and decoding schemes. Taking the division of the input image into image blocks as an example, the image division can be performed in an equal-sized image manner, or the image blocks of unequal sizes can be obtained by adaptive division based on the content. The image blocks obtained by the division can be square or non-square. For another example, the transformer can select different transform kernels such as DCT (Discrete Cosine Transform), DST (Discrete Sine Transform), KLT (Karhunen Loève Transform) to perform a transform operation on the residual of the input image block, and the decoder can select the same transform kernel to perform the corresponding inverse transform operation. In addition, more functional modules or operations can be added. For example, a loop filtering operation can be performed on the obtained reconstructed image to improve the quality of the reconstructed image and improve the prediction efficiency of the subsequent image to be processed.

[0268] The JPEG AI standard organization adopts an end-to-end image coding scheme based on deep learning. The encoder structure is shown in the figure. The RGB image is converted into the YUV color space. The Y component and the UV component are downsampled according to their respective downsampling factors and processed through a multi-layer network to obtain the feature map y Y and UV , feature map y Y and UV After being processed by their respective super-coding networks, the super-prior information z is obtained Y and z UV , the super prior information is quantized and losslessly encoded (for example, me-tANS encoding) into the bitstream. Subsequently, the encoder needs to perform some steps that are exactly the same as the decoding process. The quantized super prior information z Y and z UV Input their respective variance super decoding networks to obtain the probability distribution related information σ Y and σ UV ; Based on σ Y and σ UV For the feature residual map r Y and r UV (For example,Y =y Y -m Y and r UV =y UV -m UV ) is losslessly encoded and written into the bitstream. In one case, the quantized super prior information z Y and z UV Input each super decoding network respectively to obtain the feature map prediction information m Y and m UV ,like Figure 8 As shown; in another case, the quantized super prior information z Y and z UV Input their respective super decoding network output information and feature map y Y and UV The encoded and decoded context information is used to obtain m Y and m UV .

[0269] The decoder architecture of JPEG AI is as follows Fig. 9 As shown, first decode the super prior information z from the bitstream Y and z UV .z Y and z UV Input their respective variance super decoding networks to obtain the probability distribution related information σ Y and σ UV ; Based on σ Y and σ UV The feature residual map r is obtained from the bitstream entropy decoding Y and r UV In one case, the quantized super prior information z Y and z UV Input each super decoding network respectively to obtain the feature map prediction information m Y and m UV , that is, the predicted feature map, such as Fig. 9 As shown; in another case, the quantized super prior information z Y and z UV Input their respective super decoding network output information and feature map y Y and UV The encoded and decoded context information is used to obtain m Y and m UV . Based on r and m, we get y, for example: y Y =r Y +m Y and UV =r UV +m UV Finally, y Y andUV The YUV domain images are input into the decoder network respectively, and after ICCI filtering, they are transferred to the RGB domain for display.

[0270] Please refer to Fig.10 , Fig.10 An exemplary block diagram of a video decoding device 500 provided in an embodiment of the present application. The video decoding device 500 is suitable for implementing the disclosed embodiments described herein. In one embodiment, the video decoding device 500 may be a decoder, such as Figure 1 The video decoder 30 in the embodiment may also be an encoder, for example Figure 2 The video encoder 20 in.

[0271] The video decoding device 500 includes: an input port 510 (or input port 510) and a receiving unit (Rx) 520 for receiving data; a processor, a logic unit or a central processing unit (CPU) 530 for processing data; for example, the processor 530 here can be a neural network processor 530; a transmitter unit (Tx) 540 and an output port 550 (or output port 550) for transmitting data; and a memory 560 for storing data. The video decoding device 500 may also include an optical-to-electrical (OE) component and an electrical-to-optical (EO) component coupled to the input port 510, the receiving unit 520, the transmitting unit 540 and the output port 550 for the output or output of optical or electrical signals.

[0272] The processor 530 is implemented by hardware and software. The processor 530 can be implemented as one or more processor chips, cores (e.g., multi-core processors), FPGAs, ASICs, and DSPs. The processor 530 communicates with the input port 510, the receiving unit 520, the sending unit 540, the output port 550, and the memory 560. The processor 530 includes a neural network-based codec 570. The neural network-based codec 570 implements the embodiments disclosed above. For example, the neural network-based codec 570 performs, processes, prepares, or provides various encoding operations. Therefore, a substantial improvement is provided for the function of the video decoding device 500 by the neural network-based codec 570, and the switching of the video decoding device 500 to different states is affected. Alternatively, the neural network-based codec 570 is implemented with instructions stored in the memory 560 and executed by the processor 530.

[0273] The memory 560 includes one or more disks, tape drives, and solid-state hard disks, and can be used as an overflow data storage device for storing such programs when such programs are selected for execution, and for storing instructions and data read during program execution. The memory 560 can be volatile and / or non-volatile, and can be a read-only memory (ROM), a random access memory (RAM), a ternary content-addressable memory (TCAM), and / or a static random-access memory (SRAM).

[0274] Please refer to Fig.11 , Fig.11 An exemplary block diagram of a device 600 provided in an embodiment of the present application, the device 600 can be used as Figure 1 Either or both of the source device 12 and the destination device 14 in .

[0275] The processor 602 in the device 600 may be a central processing unit. Alternatively, the processor 602 may be any other type of device or devices that are currently available or will be developed in the future and are capable of manipulating or processing information. Although a single processor such as the processor 602 shown in the figure may be used to implement the disclosed implementation, using more than one processor is faster and more efficient.

[0276] In one implementation, the memory 604 in the apparatus 600 may be a read-only memory (ROM) device or a random access memory (RAM) device. Any other suitable type of storage device may be used as the memory 604. The memory 604 may include code and data 606 accessed by the processor 602 via a bus 612. The memory 604 may also include an operating system 608 and an application 610, which includes at least one program that allows the processor 602 to perform the methods described herein. For example, the application 610 may include applications 1 to N, and also include a video decoding application that performs the methods described herein.

[0277] The apparatus 600 may also include one or more output devices, such as a display 618. In one example, the display 618 may be a touch-sensitive display that combines a display with a touch-sensitive element that may be used to sense touch input. The display 618 may be coupled to the processor 602 via the bus 612.

[0278] Although the bus 612 in the device 600 is described herein as a single bus, the bus 612 may include multiple buses. In addition, the auxiliary storage may be directly coupled to other components of the device 600 or accessed through a network, and may include a single integrated unit such as a memory card or multiple units such as multiple memory cards. Therefore, the device 600 may have a variety of configurations.

[0279] Fig.12 is a schematic block diagram of an encoder architecture applied to an embodiment of the present application. Fig.12 The encoder architecture in the figure describes the data encoding process in detail.

[0280] like Fig.12 As shown, the image x is input into the input encoding network (analysis transform) of the encoding end to obtain the feature map y. The image x can be the entire image, or the Y component or UV component of a YUV image, which is not specifically limited in the embodiment of the present application. The obtained feature map y is input into the hyper encoder to obtain the hyper information z, and the hyper information z is quantized to obtain the quantized hyper information z, and the quantized hyper information z is entropy encoded (me-tANS) and written into the bitstream.

[0281] In a possible implementation, the quantized hyper-prior information z is input into a hyper-prior variance decoding network (hyper scale decoder) to obtain a first probability distribution parameter. The first probability distribution parameter information is input into a variance scaling (sigma scale) module to obtain a second probability distribution parameter, and the second probability distribution parameter is input into an adaptive variance scaling (adaptive sigma scale) module to obtain a third probability distribution parameter.

[0282] In another possible implementation, the prediction image μ of the feature map y is obtained according to the quantized hyper prior information z. In one example, the quantized hyper prior information z is input into the hyper decoder to obtain the prediction image μ of the feature map y. In another example, the quantized hyper prior information z is input into the hyper decoder, and the prediction image μ of the feature map y is obtained according to the joint context network (MSM). The residual image r is obtained by calculating the residual between the feature map y and the prediction image u, and the obtained residual image r is input into the gain unit module for scaling to obtain the residual image r`, and the residual image r` is input into the residual and variance scale (RVS) module to scale the residual image r` to obtain the residual image r``. After obtaining the residual image r``, the residual image r`` is encoded by the third probability distribution parameter obtained above and written into the bitstream.

[0283] This embodiment provides a typical implementation method. The encoding end process of a component (Y component or UV component) in the technical solution of this embodiment is that the image x is input into the encoding network (analysis transform) to obtain the feature map y. The feature map y belongs to the hyper encoder to obtain the feature map z, and the quantized z is subjected to me-tANS entropy coding and written into the bitstream. The quantized z is input into the hyper scale decoder to obtain the Gaussian distribution parameter information σ, which is input into the subsequent module after being scaled by the variance scaling module and the adaptive variance scaling module. The quantized z is input into the hyper decoder, and optionally, the predicted map μ of the feature map y is obtained by the joint context network (MSM). The residual of the feature map y and the predicted map μ is calculated to obtain the residual map r, which is input into the subsequent module after being scaled by the gain unit module and the residual scaling (RVS) module.

[0284] Fig.13 is a schematic block diagram of a decoder architecture applied to an embodiment of the present application. Fig.13 The decoder architecture in the figure describes the data decoding process in detail.

[0285] like Fig.13As shown, first, the quantized hyper-prior information z is decoded from the bitstream. In one possible implementation, the quantized hyper-prior information z is input into the hyper-prior variance decoding network (hyper scale decoder) to obtain the first probability distribution parameter. The first probability distribution parameter information is input into the variance scaling (sigma scale) module to obtain the second probability distribution parameter, and the second probability distribution parameter is input into the adaptive variance scaling (adptive sigma scale) module to obtain the third probability distribution parameter. Based on the third probability distribution parameter, the residual image r`` is obtained by entropy decoding from the bitstream, and the residual image r`` is input into the residual inverse quantization scaling (inverse residual and variance scale, Inv RVS) module to dequantize the residual image r`` to obtain the residual image r`, and the residual image r` is input into the inverse gain unit (Inv gain unit) module to dequantize to obtain the residual image r. In another possible implementation, the prediction image μ of the feature image y is obtained according to the quantized hyper-prior information z. In one example, the quantized hyper-prior information z is input into the hyper-prior decoding network (hyper decoder) to obtain the prediction map μ of the feature map y. In another example, the quantized hyper-prior information z is input into the hyper-prior decoding network (hyper decoder), and the prediction map μ of the feature map y is obtained according to the joint context network (MSM). The feature map y` is obtained by calculating the sum of the prediction map μ and the residual map r, and the feature map y` is corrected according to the latent scaling before synthesis (LSBS) module to obtain the feature map y. The feature map y is input into the decoding network at the decoding end to obtain the image x.

[0286] The specific operation of JPEG AI for a component decoding process is as follows Fig.14As shown, the Adaptive Variance Scaling Module (Adpt.Sigma Scale), the Inverse Residual and Variance Scale (InvRVS) module and the Latent Scaling Before Synthesis-LSBS module are mask-based encoding and decoding tools in JPEG AI. The Adaptive Variance Scaling Module (Adpt.Sigma Scale) and the Inverse Residual and Variance Scale (Inv RVS) module are two processing links of the adaptive quantization technology at the decoding end under the JPEG AI framework, and the Latent Scaling Before Synthesis-LSBS module is the processing link of the latent scaling technology at the decoding end. These two adaptive quantization technologies and latent scaling technologies have a 3% and 2% improvement in coding performance, respectively.

[0287] The above-mentioned adaptive variance scaling (adptive sigma scale), RVS, Inv RVS, and LSBS are mask-based codec tools in JPEG AI. These tools can improve the performance of codecs, but the implementation complexity is relatively high. Taking the adaptive variance scaling (adptive sigma scale) module as an example, the variance tensor (sigma) of C×H×W is averaged in NxN blocks to obtain sigma0 of C×H / N×W / N; sigma0 is compared with the threshold T0, and mask0 with a value of 0 / 1 is output; when mask0[c,i,j] is 1, all values ​​in the corresponding NxN area on sigma are multiplied by the scaling value scale0, and this process is repeated m times. Fig.15 In the example of repeating 3 times, we explain this. Taking Inv RVS as an example, when mask0[c,i,j] is 1, all values ​​in the corresponding NxN region on the C×H×W residual feature map (res) are divided by the scaling value scale0. This process is repeated m times. Fig.16 In the example of repeating 3 times, the following is explained. Taking the LSBS module as an example, the variance tensor (sigma) of C×H×W is averaged in NxN blocks to obtain sigma0 of C×H / N×W / N; sigma0 is compared with the threshold T3, and mask0 with a value of 0 / 1 is output; when mask0[c,i,j] is 1, the value of the image feature point in the corresponding NxN area on the C×H×W image feature map (y) is obtained by linearly weighting the corresponding positions of the image feature map, residual feature map, and predicted feature map. This process is repeated m times. Fig.17 The example of repeating twice is used to illustrate this.

[0288] It should be understood that the potential scaling technology is only used at the decoding end, and the adaptive quantization technology is used at both the encoding end and the decoding end. Fig.18 The specific process of operating a component (Y component or UV component) in JPEGAI is given. The processing link of adaptive quantization technology at the encoding end involves Fig.18 The Adpt.Sigma Scale module and RVS module are shown. The operation of the Adpt.Sigma Scale module is consistent with that of the decoder. The only difference between the RVS module and the Inv RVS module is that when mask0[c,i,j] is 1, all values ​​in the corresponding NxN region on the C×H×W residual feature map (res) are multiplied by the scaling value scale0.

[0289] From the above data scaling process, we can see that from N k ×N k The unit averaging to value scaling process can be repeated m times. This means that, first, the averaging operation with different block sizes needs to be repeated m times, and second, the same K×K unit needs to be scaled m times at worst. The larger the value of m, the more operations need to be repeated, and the longer the entire operation pipeline is.

[0290] That is, the mask-based codec tools in JPEG AI, Adaptive Variance Scaling Module (Adpt.SigmaScale) and Latent Scaling Before Synthesis (LSBS) module, have obvious coding performance improvements, but the implementation complexity is relatively high. As can be seen from the flowchart, the N k ×N k The unit averaging to numerical scaling process can be repeated m times, which means, first, the averaging operation with different block sizes needs to be repeated m times, and second, the same K×K unit needs to be scaled m times at worst. The larger the value of m, the more operations need to be repeated, and the longer the entire operation pipeline is.

[0291] The technical problem to be solved by this embodiment is: to simplify two linear scaling technologies of mask images generated based on variance maps, namely adaptive quantization and potential scaling, and to reduce the complexity of the algorithm without losing the coding performance.

[0292] In view of this, the embodiment of the present application provides a method of encoding and decoding, which can reduce the complexity of the algorithm without losing the encoding performance. It can also be understood as performing interval judgment based on the variance map, and deriving the scaling values ​​at different spatial positions to generate a scaling map. Linear weighting is performed based on the scaling map to obtain the final feature map value.

[0293] The system architecture of this embodiment is as follows Fig.10 As shown: After the video is captured by the video capture device, it goes through a series of pre-processing, and then the processed video is compressed and encoded to obtain the encoded code stream. The code stream is sent to the receiving module through the transmission network, and after being decoded by the decoder, it can be rendered and displayed. In addition, the code stream after video encoding can also be directly stored.

[0294] The method provided in the embodiments of the present application can be used for the adaptive variance scaling module (Adpt.Sigma Scale), the inverse residual and variance scale (InvRVS) module and the latent scaling before synthesis (Latent Scaling Before Synthesis-LSBS) module of a video image decoder; and can be used for the Adpt.Sigma Scale module and the RVS module of a video image encoder.

[0295] Fig. 20 is a schematic flow chart of a coding method provided in an embodiment of the present application. Fig. 20 As shown, the method may include steps 1310-1360, and steps 1310-1360 are described in detail below.

[0296] Step 1310: Obtain a feature map of the image to be encoded.

[0297] As an example, Fig.12 As shown, the input encoding network (analysis transform) of the image x input encoding end can obtain the feature map y. It should be understood that the image x can be the entire image, or the Y component or UV component of the YUV image, and the embodiment of the present application does not specifically limit this.

[0298] The embodiment of the present application does not specifically limit the feature map, which may be a feature map directly obtained based on the image to be encoded, or may be a residual feature map.

[0299] It should be understood that the residual feature map can also be called a residual map, which is determined based on a prediction map in the feature map (referred to as a prediction map) that is different from the feature map. Specifically, in one possible implementation, the feature map includes multiple feature values, the prediction map includes predicted values ​​of multiple feature values ​​(referred to as predicted values), and the residual map includes multiple residual values ​​(also referred to as residual feature values), which are determined based on the difference between the feature values ​​and the predicted values ​​at the same position in the feature map and the prediction map.

[0300] Step 1320: Obtain a probability distribution parameter map of the feature map.

[0301] In the embodiment of the present application, the probability distribution parameter map includes multiple probability distribution parameters. A probability distribution parameter in the probability distribution parameter map can be a probability distribution parameter of a feature value at a corresponding position in the feature map, or can also be a probability distribution parameter of a residual value at a corresponding position in the residual feature map.

[0302] It should be understood that the above-mentioned probability distribution parameters may be parameters of Gaussian distribution, or may be parameters of Laplace distribution, or may also be parameters of mixed Gaussian distribution, and the embodiments of the present application do not make specific limitations on this.

[0303] It should also be understood that the above-mentioned probability distribution parameter may be a variance σ, or a mean, or a standard deviation, and the embodiments of the present application do not specifically limit this.

[0304] In the embodiment of the present application, after obtaining the feature map of the image to be encoded, the feature map can be input into the hyper encoder to obtain the hyper information z, and the probability distribution parameters can be obtained based on the quantized hyper information z and the hyper variance decoding network (hyper scale decoder), the variance scaling (sigma scale) module, and the adaptive variance scaling (adaptive sigma scale) module. For details, please refer to Fig.12 The description in will not be repeated here.

[0305] Step 1330: Obtain a first matrix according to the probability distribution parameter graph.

[0306] The first matrix may be a part of the probability distribution parameter map. For example, the first matrix is ​​an N*N unit in the probability distribution parameter map. The first matrix includes N*N probability distribution parameters, where N is an integer greater than 1.

[0307] Step 1340: Determine a first interval corresponding to the first matrix from a plurality of intervals, the plurality of intervals do not overlap each other, and each interval corresponds to at least one scaling factor.

[0308] In the embodiment of the present application, the multiple intervals are divided by multiple thresholds. As an example, the number of the multiple thresholds is m, and the number of the multiple intervals is m+1, where m is an integer greater than 1.

[0309] In the embodiment of the present application, the multiple intervals divided based on the multiple thresholds do not overlap with each other. Fig.21 As shown, T1-Tm are m thresholds, interval 0-interval m are m+1 intervals, these intervals do not overlap each other, and each interval corresponds to at least one scaling factor.

[0310] In a possible implementation, a first value corresponding to the first matrix may be determined, and the first value may be compared with the above-mentioned multiple thresholds respectively, and a first interval corresponding to the first value may be determined from multiple intervals divided by the multiple thresholds. It should be understood that the first interval is one interval among the multiple intervals. For example, Fig.21 As shown, assuming that the first value is greater than the threshold value T1 and greater than T2, then the interval 1 where T1 and T2 are located can be determined as the first interval corresponding to the first value.

[0311] The embodiment of the present application does not specifically limit the first value corresponding to the first matrix. In one example, the first value may be the maximum value of multiple probability distribution parameters included in the first matrix. In another example, the first value may also be the mean value of multiple probability distribution parameters included in the first matrix.

[0312] Optionally, in the embodiment of the present application, the values ​​of the above-mentioned multiple thresholds and the number of thresholds may also be written into the bitstream, so that the decoding end may obtain the values ​​of the multiple thresholds and the number of thresholds from the bitstream.

[0313] Optionally, in an embodiment of the present application, the scaling factor corresponding to the first interval (e.g., the first scaling factor) may also be written into the bitstream so that the decoding end may obtain the scaling factor corresponding to the first interval (e.g., the first scaling factor) from the bitstream.

[0314] As an example, in an embodiment of the present application, the decoding end can determine the values ​​of multiple thresholds carried in the bitstream, the number of thresholds, related coding information, and scaling factors based on the syntax table.

[0315] For example, the following table 1 shows a syntax table of the adaptive scaling technique, and table 2 shows a syntax table of the potential scaling technique. It should be understood that the adaptive scaling technique is applied to an adaptive sigma scale module or a residual and variance scale (RVS) module, and the potential scaling technique is applied to an LSBS module.

[0316] Table 1 Syntax table of adaptive scaling technology

[0317]

[0318]

[0319] Referring to Table 1, num_rvs_params indicates the number of thresholds m, threshold_rvs indicates the threshold T, scale_rvs indicates the scaling factor s0, presice_flag_thr_rvs is 1 indicating that the threshold is encoded with 16-bit precision, otherwise the threshold is encoded with 8-bit precision; presice_flag_scale_rvs is 1 indicating that the scaling factor is encoded with 16-bit precision, otherwise the scaling factor is encoded with 8-bit precision.

[0320] Table 2 Syntax of potential scaling techniques

[0321]

[0322] See Table 2, num_lsbs_params indicates the number of thresholds m, threshold_lsbs indicates the threshold T, scale0_lsbs indicates the scaling factor s0, scale1_lsbs indicates the scaling factor s1, presice_flag_thr_lsbs is 1 to indicate that the threshold is encoded with 16-bit precision, otherwise the threshold is encoded with 8-bit precision; presice_flag_scale_lsbs is 1 to indicate that the scaling factor is encoded with 16-bit precision, otherwise the scaling factor is encoded with 8-bit precision. application_id_lsbs is 0 to indicate that it is only used for the Y component, 1 to indicate that it is only used for the UV component, and 2 to indicate that it is used for both the Y component and the UV component.

[0323] Step 1350: Scale the feature map and the probability distribution parameter map corresponding to the first matrix according to the first scaling factor corresponding to the first interval to obtain the scaled feature map and the scaled probability distribution parameter map.

[0324] In the embodiment of the present application, it is assumed that the scaling factor corresponding to the first interval is the first scaling factor. Different implementations of scaling using the first scaling factor are described in detail below.

[0325] In a possible implementation, the feature map corresponding to the first matrix may be scaled to obtain a scaled feature map. Specifically, in one example, the first scaling factor corresponding to the first interval and the feature map corresponding to the first matrix may be multiplied to obtain the scaled feature map.

[0326] For example, assuming that the feature map corresponding to the first matrix includes multiple eigenvalues, the multiple eigenvalues ​​can be multiplied by the first scaling factor to obtain multiple scaled eigenvalues, and the multiple scaled eigenvalues ​​constitute the scaled feature map. Assuming that the feature map corresponding to the first matrix includes multiple residual eigenvalues, the multiple residual eigenvalues ​​can be multiplied by the first scaling factor to obtain multiple scaled residual eigenvalues, and the multiple scaled residual eigenvalues ​​constitute the scaled feature map. The above process can be implemented in the residual quantization scaling module.

[0327] In another possible implementation, the probability distribution parameter map corresponding to the first matrix can also be scaled to obtain a scaled probability distribution parameter map. Specifically, in one example, the first scaling factor corresponding to the first interval and the probability distribution parameter map corresponding to the first matrix can be multiplied to obtain the scaled probability distribution parameter map. The above process can be implemented in an adaptive variance scaling module.

[0328] For example, assuming that the probability distribution parameter map corresponding to the first matrix includes multiple probability distribution parameters, the multiple probability distribution parameters can be multiplied by the first scaling factor respectively to obtain multiple scaled probability distribution parameters, which constitute the above-mentioned scaled probability distribution parameter map.

[0329] Step 1360: perform entropy coding on the scaled feature map using the scaled probability distribution parameter map and write it into the bitstream.

[0330] After obtaining the scaled probability distribution parameter map and the scaled feature map, the embodiment of the present application can perform entropy coding on the scaled feature map based on the scaled probability distribution parameter map and write it into the bitstream.

[0331] As an example, if the scaled feature map includes multiple eigenvalues, the multiple eigenvalues ​​can be entropy encoded based on multiple probability distribution parameters in the scaled probability distribution parameter map and written into the bitstream.

[0332] As another example, if the scaled feature map includes multiple residual eigenvalues, the multiple residual eigenvalues ​​can be entropy encoded based on multiple probability distribution parameters in the scaled probability distribution parameter map and written into the bitstream.

[0333] In the above technical solution, by setting the threshold intervals to non-overlapping intervals, there is no need to repeat multiple times when scaling the feature map or probability distribution parameter map, thereby reducing the complexity of the algorithm without losing coding performance.

[0334] For example, the mean is calculated on the variance map of C×H×W with N×N blocks as units to obtain the variance downsampling map of C×H / N×W / N. The m thresholds divide the variance range into m+1 intervals, and different intervals correspond to different scaling factors. Each point on the variance downsampling map is compared with the m thresholds respectively, and the interval k where each point is located is obtained, thereby obtaining the corresponding scaling factor. The feature map and the scaling map are linearly weighted to obtain the final feature map.

[0335] Each point on the variance downsampling graph is compared with m thresholds respectively, and the interval where each point is located is obtained, so as to obtain the corresponding scaling factor, which specifically includes: one interval corresponds to at least one scaling factor.

[0336] At the encoding end, the feature map and the scaling map are linearly weighted to obtain the final feature map, which specifically includes: when an interval contains a scaling factor, the residual feature map is multiplied by the scaling map to obtain an updated residual feature map. When an interval contains a scaling factor, the variance feature map is multiplied by the scaling map to obtain an updated variance feature map.

[0337] At the encoding end, when an interval includes a scaling factor, the number of thresholds m, m thresholds, and m+1 scaling factors are written into the bitstream.

[0338] At the encoding end, when an interval includes two scaling factors, the number of thresholds m, m thresholds, and 2*(m+1) scaling factors are written into the bitstream.

[0339] At the decoding end, the threshold number m, m thresholds, and m+1 (or 2m+2) scaling factors are obtained by decoding the bitstream.

[0340] At the decoding end, the feature map and the scaling map are linearly weighted to obtain the final feature map, which specifically includes:

[0341] When an interval contains a scaling factor, the residual feature map is divided by the scaling map to obtain the updated residual feature map.

[0342] When an interval contains a scaling factor, the variance feature map is multiplied by the scaling map to obtain the updated variance feature map.

[0343] When an interval contains two scaling factors, the predicted feature map is multiplied by the first scaling map to obtain the first feature map, the residual feature map is multiplied by the second scaling map to obtain the second feature map, and the image feature map, the first feature map and the second feature map are multiplied. Figure 3 The sum of them is used to obtain the updated image feature map.

[0344] The technical coding end of this embodiment simplifies the adaptive variance scaling (Adptive Sigma Scale) module and / or the residual scaling (RVS) module to reduce the algorithm complexity while maintaining the coding performance. Fig. 22 shown.

[0345] The first step is to decide the number of thresholds m, thresholds T0, T1…T m , scaling factors {s00}, {s01}…{s0 m+1}.

[0346] In the second step, the threshold number m, thresholds T0, T1, ... m , scaling factors {s00}, {s01}…{s0 m+1} information is written into the bitstream, and there is no strict order restriction between this step and the subsequent steps; for example, the adaptive scaling technology syntax element table is shown in the table, where num_rvs_params represents the number of thresholds m, threshold_rvs represents the threshold T, scale_rvs represents the scaling factor s0, presice_flag_thr_rvs is 1, indicating that the threshold is encoded with 16-bit precision, otherwise the threshold is encoded with 8-bit precision; presice_flag_scale_rvs is 1, indicating that the scaling factor is encoded with 16-bit precision, otherwise the scaling factor is encoded with 8-bit precision.

[0347] The third step is to calculate the mean on the variance map of C×H×W with N×N blocks as units to obtain the variance downsampling map of C×H / N×W / N.

[0348] The fourth step is to compare each point on the variance downsampling graph with m thresholds T0, T1…T m By comparing them separately, we can get the interval k where each point is located, such as Fig.12 shown.

[0349] Step 5: According to the interval index number k, the corresponding scaling factor {S0 k ,S1 k}. Thus, the scaling map scale_rvs corresponding to the variance map is obtained. For example, when m is 3, the scaling map and the second scaling map are obtained according to the following formula:

[0350]

[0351] Step 6: Multiply the variance feature map σ′ by the scale scale_rvs to get the updated variance feature map: σ″[c,i,j]=scale_rvs[c,i,j]·σ′[c,i,j]

[0352] Step 7: Residual feature map Multiply it with the scale map scale_rvs to get the updated residual feature map: There is no strict order restriction between step 7 and step 6.

[0353] In the eighth step, according to the updated variance feature map and the updated residual feature map, subsequent steps are performed to perform entropy coding and generate a bit stream.

[0354] Fig.23 FIG. 1 is a schematic flow chart of a decoding method provided in an embodiment of the present application. Fig.23 As shown, the method may include steps 1510-1550, and steps 1510-1550 are described in detail below.

[0355] Step 1510: Decode the bitstream to obtain a probability distribution parameter map.

[0356] In the embodiment of the present application, after obtaining the bit stream, the decoding end can decode the bit stream to obtain the probability distribution parameter map. Specifically, in a possible implementation, see Fig. 9 , the decoder can decode the quantized hyper-prior information z from the bitstream, and pass the quantized hyper-prior information z through the hyper-scale decoder, sigma scale, and adaptive sigma scale modules to obtain the probability distribution parameter map. For the specific implementation process, please refer to Fig.13 The description in will not be repeated here.

[0357] Step 1520: Obtain a first matrix according to the probability distribution parameter graph.

[0358] Step 1520 corresponds to step 1330. Please refer to the description in step 1330 for details, which will not be repeated here.

[0359] Step 1530: Determine a first interval corresponding to the first matrix from a plurality of intervals, the plurality of intervals do not overlap each other, and each interval corresponds to at least one scaling factor.

[0360] Step 1530 corresponds to step 1340. Please refer to the description in step 1330 for details, which will not be repeated here.

[0361] It should be noted that if the encoder writes the value of each threshold and the number of thresholds in the bitstream, the decoder can decode the value of each threshold and the number of thresholds from the bitstream, thereby determining the above-mentioned multiple intervals based on the number of thresholds and the value of each threshold.

[0362] Specifically, when decoding, the decoding end may decode the number of thresholds and the value of each threshold based on the syntax table of the adaptive scaling technology in Table 1.

[0363] Step 1540: Scale the probability distribution parameter map corresponding to the first matrix according to the first scaling factor corresponding to the first interval to obtain a scaled probability distribution parameter map.

[0364] Before step 1540, the decoding end needs to obtain the first scaling factor corresponding to the first interval. In a possible implementation, if the encoding end also writes the first scaling factor corresponding to the first interval in the bitstream, the decoding end can decode the first scaling factor corresponding to the first interval from the bitstream. Specifically, the decoding end decodes the first scaling factor based on the syntax table of the adaptive scaling technology in Table 1.

[0365] Step 1540 is the same as the process of obtaining the scaled probability distribution parameter map according to the first scaling factor in step 1350. For details, please refer to the description in step 1350, which will not be repeated here.

[0366] Step 1550: Use the scaled probability distribution parameter map to decode from the bitstream to obtain a feature map.

[0367] After obtaining the scaled probability distribution parameter map, the embodiment of the present application can use the scaled probability distribution parameter map to decode from the bitstream to obtain a feature map.

[0368] In one example, assuming that the encoder performs entropy encoding on multiple eigenvalues ​​included in the scaled feature map based on multiple probability distribution parameters in the scaled probability distribution parameter map, then the decoder uses the scaled probability distribution parameter map to decode the feature map from the bitstream to obtain a feature map including multiple eigenvalues. The above process can be implemented in the adaptive variance scaling module of the decoder.

[0369] In another example, assuming that the encoding end performs entropy encoding on the multiple residual eigenvalues ​​included in the scaled feature map based on the multiple probability distribution parameters in the scaled probability distribution parameter map, then the feature map obtained by decoding from the bitstream using the scaled probability distribution parameter map is a feature map including multiple residual eigenvalues. The above process can be implemented in the residual inverse quantization scaling Inv RVS module of the decoding end.

[0370] In an embodiment of the present application, the decoding end may also divide the decoded feature map (feature map after scaling) by the first scaling factor corresponding to the first interval to obtain the feature map before scaling. Specifically, the multiple eigenvalues ​​or residual eigenvalues ​​included in the feature map may be divided by the first scaling factor to obtain multiple eigenvalues ​​before scaling or multiple residual eigenvalues ​​before scaling, and these multiple eigenvalues ​​before scaling or multiple residual eigenvalues ​​before scaling constitute the feature map before scaling.

[0371] It should be noted that if the feature map obtained by the above decoding includes residual feature values, the decoding end also needs to obtain feature values ​​based on the residual feature values ​​and the predicted values ​​in the prediction map. The method for obtaining the prediction map at the decoding end is the same as the method for obtaining the prediction map at the encoding end. For details, please refer to the description above and will not be repeated here.

[0372] The above method can also be applied to potential scaling modules, e.g. Fig.13 The LSBS module in is used to correct the feature map obtained above. In a possible implementation, assuming that the scaling factors corresponding to the first interval include scaling factor 1 and scaling factor 2, the predicted feature map (referred to as the predicted map) is multiplied by scaling factor 1 to obtain the feature Figure 1 , multiply the residual feature map (referred to as residual map) and the scaling factor 1 to get the feature Figure 2 , add the feature map obtained above to the feature Figure 1 and Features Figure 2 Get the updated feature map.

[0373] It should be understood that the above residual image may be a residual image before scaling, or may be a residual image after scaling, and the embodiment of the present application does not specifically limit this.

[0374] For example, the decoding process of one component (Y component or UV component) in the technical solution of this embodiment is as follows: Fig.24 As shown in the figure, the feature map z is decoded from the bitstream, and z is input into the hyper-prior variance decoding network (hyper scale decoder) to obtain Gaussian distribution parameter information σ. After scaling processing in the variance scaling (sigma scale) and adaptive variance scaling (Adptive SigmaScale) modules, it is input into the subsequent modules. According to the processed σ, the feature residual map r is decoded from the bitstream, and after scaling and dequantizing the residual map r in the inv gain unit module and the residual inverse quantization scaling (Inv RVS) module, it is input into the subsequent modules.

[0375] In the technical decoding end of this embodiment, the adaptive variance scaling (Adptive Sigma Scale) module and / or the residual scaling (Inv RVS) module are simplified, and the operation of the adaptive variance scaling (Adptive Sigma Scale) module in the decoding end is completely consistent with that in the encoding end.

[0376] The first step is to decode the code stream according to the syntax table shown in the table to obtain the threshold number m, thresholds T0, T1...T m , scaling factors {s00}, {s01}…{s0 m+1}.

[0377] Step 2: Calculate the mean on the variance map of C×H×W with N×N blocks as units to obtain the variance downsampling map of C×H / N×W / N. (Same as the third step at the encoding end).

[0378] Step 3: Each point on the variance downsampling graph is compared with m thresholds T0, T1…T m By comparing them separately, we can get the interval k where each point is located, such as Fig.21 As shown. (Same as the fourth step of the encoding end).

[0379] The fourth step is to obtain the corresponding scaling factor {S0 according to the interval index number k k ,S1 k}. Thus, the scaling map scale_rvs corresponding to the variance map is obtained. For example, when m is 3, the scaling map and the second scaling map are obtained according to the following formula: (same as the fifth step of the encoding end).

[0380]

[0381] In the fifth step, the variance feature map σ′ is multiplied by the scale scale_rvs to obtain the updated variance feature map: σ″[c,i,j]=scale_rvs[c,i,j]·σ′[c,i,j].

[0382] In the sixth step, according to the updated variance map, the residual feature map is decoded from the bitstream.

[0383] Step 7: Residual feature map Divide it by the scaled map scale_rvs to get the updated residual feature map:

[0384] In the eighth step, a reconstructed image is obtained based on the updated residual feature map and subsequent decoding network and other processing modules.

[0385] Compared with the prior art, the technology of this embodiment significantly reduces the algorithm complexity while achieving the same coding performance.

[0386] This embodiment provides a typical LSBS implementation method. The specific process of the decoding end in the technical solution of this embodiment is as follows:

[0387] Encoding side:

[0388] The first step is to obtain the threshold number m;

[0389] The second step is to obtain m+1 thresholds T0, T1…T m ;

[0390] The third step is to obtain the scaling factors of m+1 intervals {S00,S10}, {S01,S11}…{S0 m ,S1 m};

[0391] The fourth step is to set the threshold number m, thresholds T0, T1, ... m , scaling factors {S00,S10}, {S01,S11}…{S0 m ,S1 m}Write code stream.

[0392] There is no strict order restriction between this step and the subsequent steps; for example, the potential scaling technology syntax table is shown in the table, where num_lsbs_params indicates the number of thresholds m, threshold_lsbs indicates the threshold T, scale0_lsbs indicates the scaling factor s0, scale1_lsbs indicates the scaling factor s1, presice_flag_thr_lsbs is 1 to indicate that the threshold is encoded with 16-bit precision, otherwise the threshold is encoded with 8-bit precision; presice_flag_scale_lsbs is 1 to indicate that the scaling factor is encoded with 16-bit precision, otherwise the scaling factor is encoded with 8-bit precision. application_id_lsbs is 0 for Y component only, 1 for UV component only, and 2 for both Y and UV components.

[0393] Decoding side:

[0394] The first step is to decode the code stream according to the syntax table shown in the table to obtain the threshold number m, thresholds T0, T1...T m , scaling factors {s00,s10}, {s01,s11}…{s0 m+1 ,s1 m+1}.

[0395] Step 2: Calculate the mean on the variance map of C×H×W with N×N blocks as units to obtain the variance downsampling map of C×H / N×W / N.

[0396] Step 3: Each point on the variance downsampling graph is compared with m thresholds T0, T1…T m By comparing them separately, we can get the interval k where each point is located, such as Fig.21 shown.

[0397] The fourth step is to obtain the corresponding scaling factor {S0 according to the interval index number k k ,S1 k}. Thus, the first scaling map scale_lsbs 0 and the second scaling map scale_lsbs 1 corresponding to the variance map are obtained. For example, when m is 2, the first scaling map and the second scaling map are obtained according to the following formula:

[0398]

[0399] Step 5: Multiply the predicted feature map μ by the first scaled map scale_lsbs0 to obtain the first feature map and the residual feature map Multiplying with the second scaled map scale_lsbs1 obtains the second feature map, image feature map The first feature map and the second feature Figure 3 The sum of them is used to obtain the updated image feature map.

[0400] The mean is calculated on the variance map of C×H×W with N×N blocks as units to obtain the variance downsampling map of C×H / N×W / N. The m thresholds divide the variance range into m+1 intervals, and different intervals correspond to different scaling factors. Each point on the variance downsampling map is compared with the m thresholds respectively, and the interval k where each point is located is obtained, thereby obtaining the corresponding scaling factor. The feature map and the scaling map are linearly weighted to obtain the final feature map.

[0401] The key technical points of this embodiment have the following beneficial effects: while maintaining the coding performance, the complexity of the adaptive quantization technology and the potential scaling technology is reduced.

[0402] The following will be combined Fig.25 A coding device for executing the above coding method is introduced.

[0403] It is understandable that, in order to realize the above functions, the encoding device includes hardware and / or software modules corresponding to the execution of each function. In combination with the algorithm steps of each example described in the embodiments disclosed herein, the embodiments of the present application can be implemented in the form of hardware or a combination of hardware and computer software. Whether a function is executed in the form of hardware or computer software driving hardware depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application in combination with the embodiments, but such implementation should not be considered to exceed the scope of the embodiments of the present application.

[0404] The embodiment of the present application can divide the functional modules of the encoding device according to the above method example. For example, each functional module can be divided corresponding to each function, or two or more functions can be integrated into one processing module. The above integrated module can be implemented in the form of hardware. It should be noted that the division of modules in this embodiment is schematic and is only a logical function division. There may be other division methods in actual implementation.

[0405] In the case of dividing each functional module into corresponding functional modules, Fig.25 A possible schematic diagram of the composition of the encoding device involved in the above embodiment is shown. Fig.25 As shown, the encoding device 1600 may include: a transceiver unit 1601 and a processing unit 1602. The processing unit 1602 is used to obtain a feature map of an image to be encoded; the processing unit 1602 is also used to obtain a probability distribution parameter map of the feature map; the processing unit 1602 is also used to obtain a first matrix according to the probability distribution parameter map; the processing unit 1602 is also used to determine a first interval corresponding to the first matrix from a plurality of intervals, the plurality of intervals do not overlap each other, and each of the intervals corresponds to at least one scaling factor; the processing unit 1602 is also used to scale the eigenvalues ​​corresponding to the first matrix in the feature map according to the first scaling factor corresponding to the first interval to obtain a scaled feature map; the processing unit 1602 is also used to scale the probability distribution parameters corresponding to the first matrix in the probability distribution parameter map according to the first scaling factor corresponding to the first interval to obtain a scaled probability distribution parameter map; the processing unit 1602 is also used to perform entropy coding on the scaled feature map using the scaled probability distribution parameter map and write it into a bitstream; the transceiver unit 1601 is used to send the bitstream;

[0406] Optionally, the multiple intervals are divided by multiple thresholds, the number of the multiple thresholds is m, the number of the multiple intervals is m+1, and m is an integer greater than 1.

[0407] Optionally, the processing unit 1602 is specifically configured to: compare the first value of the first matrix with at least one threshold value among the multiple threshold values, and determine the first interval corresponding to the first value from the multiple intervals.

[0408] Optionally, the processing unit 1602 is further configured to write the number m of the multiple thresholds and the value of each threshold into the code stream.

[0409] Optionally, the processing unit 1602 is further configured to write the first scaling factor into the bitstream.

[0410] Optionally, the processing unit 1602 is specifically used to: multiply a first scaling factor corresponding to the first interval and an eigenvalue in the feature map corresponding to the first matrix to obtain the scaled feature map.

[0411] Optionally, the processing unit 1602 is specifically used to: multiply a first scaling factor corresponding to the first interval and a probability distribution parameter in the probability distribution parameter map corresponding to the first matrix to obtain the scaled probability distribution parameter map.

[0412] Optionally, the probability distribution parameters in the probability distribution parameter map are Gaussian distribution parameters, and the first value of the first matrix is ​​the mean or maximum value of the Gaussian distribution parameters included in the probability distribution parameter map.

[0413] Optionally, the device 1600 is applied to an adaptive variance scaling module or a residual quantization scaling module.

[0414] Optionally, the feature map is a residual feature map, and the eigenvalues ​​in the feature map corresponding to the first matrix are residual eigenvalues.

[0415] The following will be combined Fig.26 A decoding device for executing the above decoding method is introduced.

[0416] It is understandable that, in order to implement the above functions, the decoding device includes hardware and / or software modules corresponding to the execution of each function. In combination with the algorithm steps of each example described in the embodiments disclosed herein, the embodiments of the present application can be implemented in the form of hardware or a combination of hardware and computer software. Whether a function is executed in the form of hardware or computer software driving hardware depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application in combination with the embodiments, but such implementation should not be considered to exceed the scope of the embodiments of the present application.

[0417] The embodiment of the present application can divide the functional modules of the decoding device according to the above method example. For example, each functional module can be divided according to each function, or two or more functions can be integrated into one processing module. The above integrated module can be implemented in the form of hardware. It should be noted that the division of modules in this embodiment is schematic and is only a logical function division. There may be other division methods in actual implementation.

[0418] In the case of dividing each functional module into corresponding functional modules, Fig.26 A possible schematic diagram of the composition of the decoding device involved in the above embodiment is shown. Fig.26As shown, the decoding device 1700 may include: a transceiver unit 1701 and a processing unit 1702. The transceiver unit 1701 is used to obtain a code stream; the processing unit 1702 is used to decode the code stream to obtain a probability distribution parameter map; the processing unit 1702 is also used to obtain a first matrix according to the probability distribution parameter map; the processing unit 1702 is also used to determine a first interval corresponding to the first matrix from a plurality of intervals, the plurality of intervals do not overlap each other, and each of the intervals corresponds to at least one scaling factor; the processing unit 1702 is also used to scale the probability distribution parameters corresponding to the first matrix in the probability distribution parameter map according to the first scaling factor corresponding to the first interval to obtain a scaled probability distribution parameter map; the processing unit 1702 is also used to decode the code stream using the scaled probability distribution parameter map to obtain a feature map.

[0419] Optionally, the processing unit 1702 is further used to decode and obtain a number m of multiple thresholds and a value of each threshold from the code stream, the multiple intervals are divided by the multiple thresholds, the number of the multiple intervals is m+1, and m is an integer greater than 1.

[0420] Optionally, the processing unit 1702 is specifically configured to: compare the first value of the first matrix with at least one threshold value among the multiple threshold values, and determine the first interval corresponding to the first value from the multiple intervals.

[0421] Optionally, the processing unit 1702 is further configured to obtain the first scaling factor by decoding from the bit stream.

[0422] Optionally, the processing unit 1702 is specifically used to: multiply a first scaling factor corresponding to the first interval and a probability distribution parameter in the probability distribution parameter map corresponding to the first matrix to obtain the scaled probability distribution parameter map.

[0423] Optionally, the probability distribution parameters in the probability distribution parameter map are Gaussian distribution parameters, and the first value of the first matrix is ​​the mean or maximum value of the Gaussian distribution parameters included in the probability distribution parameter map.

[0424] Optionally, the processing unit 1702 is further used to divide the feature map by a first scaling factor corresponding to the first interval to obtain the feature map before scaling.

[0425] Optionally, the device 1700 is applied to an adaptive variance scaling module or a residual inverse quantization scaling module.

[0426] Optionally, the feature map is a residual feature map, which includes residual eigenvalues.

[0427] Optionally, the feature map includes eigenvalues, the device 1700 is applied to a potential scaling module, the first interval corresponds to the first scaling factor and the second scaling factor, and the processing unit 1702 is also used to obtain a prediction map and a residual map of the feature map; the processing unit 1702 is also used to scale the predicted eigenvalues ​​corresponding to the first matrix in the prediction map according to the first scaling factor corresponding to the first interval to obtain a scaled predicted feature map; the processing unit 1702 is also used to scale the residual eigenvalues ​​corresponding to the first matrix in the residual map according to the second scaling factor corresponding to the first interval to obtain a scaled residual feature map; the processing unit 1702 is also used to update the feature map according to the scaled prediction map and the scaled residual map to obtain an updated feature map.

[0428] An embodiment of the present application also provides an encoding device, which includes: at least one processor, when the at least one processor executes program code or instructions, it implements the above-mentioned related method steps to implement the encoding method in the above-mentioned embodiment.

[0429] Optionally, the device may further include at least one memory, and the at least one memory is used to store the program code or instruction.

[0430] An embodiment of the present application also provides a decoding device, which includes: at least one processor, when the at least one processor executes program code or instructions, it implements the above-mentioned related method steps to implement the decoding method in the above embodiment.

[0431] Optionally, the device may further include at least one memory, and the at least one memory is used to store the program code or instruction.

[0432] An embodiment of the present application also provides a code stream, which includes: data for entropy encoding a scaled feature map using a scaled probability distribution parameter map, wherein the scaled probability distribution parameter map scales the probability distribution parameters corresponding to the first matrix in the probability distribution parameter map according to a first scaling factor corresponding to the first interval, the scaled feature map scales the eigenvalues ​​corresponding to the first matrix in the feature map according to the first scaling factor corresponding to the first interval, the first scaling factor is a scaling factor of a first interval corresponding to the first matrix among multiple intervals, the multiple intervals do not overlap, each of the intervals corresponds to at least one scaling factor, the first matrix is ​​obtained according to the probability distribution parameter map of the feature map, and the feature map is obtained according to the image to be encoded.

[0433] An embodiment of the present application also provides a computer storage medium, in which computer instructions are stored. When the computer instructions are executed on an encoding device, the encoding device executes the above-mentioned related method steps to implement the encoding and decoding method in the above-mentioned embodiment.

[0434] The embodiment of the present application also provides a computer program product. When the computer program product is run on a computer, the computer is enabled to execute the above-mentioned related steps to implement the encoding and decoding method in the above-mentioned embodiment.

[0435] The embodiment of the present application also provides a coding and decoding device, which can be a chip, an integrated circuit, a component or a module. Specifically, the device may include a connected processor and a memory for storing instructions, or the device includes at least one processor for obtaining instructions from an external memory. When the device is running, the processor can execute instructions so that the chip executes the coding and decoding methods in the above-mentioned method embodiments.

[0436] Fig. 27 2 shows a schematic diagram of the structure of a chip 2300. The chip 2300 includes one or more processors 2301 and an interface circuit 2302. Optionally, the chip 2300 may also include a bus 2303.

[0437] The processor 2301 may be an integrated circuit chip with signal processing capability. In the implementation process, each step of the above encoding and decoding method may be completed by an integrated logic circuit of hardware in the processor 2301 or by instructions in the form of software.

[0438] Optionally, the processor 2301 may be a general purpose processor, a digital signal processing (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. The methods and steps disclosed in the embodiments of the present application may be implemented or executed. The general purpose processor may be a microprocessor or the processor may be any conventional processor, etc.

[0439] The interface circuit 2302 can be used to send or receive data, instructions or information. The processor 2301 can use the data, instructions or other information received by the interface circuit 2302 to process, and can send the processing completion information through the interface circuit 2302.

[0440] Optionally, the chip also includes a memory, which may include a read-only memory and a random access memory, and provides operation instructions and data to the processor. A portion of the memory may also include a non-volatile random access memory (NVRAM).

[0441] Optionally, the memory stores executable software modules or data structures, and the processor can perform corresponding operations by calling operation instructions stored in the memory (the operation instructions can be stored in the operating system).

[0442] Optionally, the chip can be used in the encoding device or DOP involved in the embodiment of the present application. Optionally, the interface circuit 2302 can be used to output the execution result of the processor 2301. The encoding and decoding method provided by one or more embodiments of the embodiment of the present application can refer to the aforementioned embodiments, which will not be repeated here.

[0443] It should be noted that the corresponding functions of the processor 2301 and the interface circuit 2302 can be implemented through hardware design, software design, or a combination of hardware and software, and there is no limitation here.

[0444] Among them, the device, computer storage medium, computer program product or chip provided in this embodiment is used to execute the corresponding method provided above. Therefore, the beneficial effects that can be achieved can refer to the beneficial effects in the corresponding method provided above, and will not be repeated here.

[0445] It should be understood that in various embodiments of the present application, the size of the serial numbers of the above-mentioned processes does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present application.

[0446] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the embodiments of the present application.

[0447] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0448] In the several embodiments provided in the embodiments of the present application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the above units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0449] The units described above as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0450] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0451] If the above functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the embodiment of the present application is essentially or the part that contributes to the prior art or the part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the above methods of each embodiment of the embodiment of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (Read Only Memory, ROM), random access memory (Random Access Memory, RAM), disk or optical disk and other media that can store program codes.

[0452] The above is only a specific implementation of the embodiment of the present application, but the protection scope of the embodiment of the present application is not limited thereto. Any technician familiar with the technical field can easily think of changes or replacements within the technical scope disclosed in the embodiment of the present application, which should be included in the protection scope of the embodiment of the present application. Therefore, the protection scope of the embodiment of the present application should be based on the protection scope of the claims.

Claims

1. A coding method, characterized in that: The method comprises: Obtain a feature map of an image to be encoded; Obtaining a probability distribution parameter map of the feature map; According to the probability distribution parameter graph, obtaining a first matrix; Determine a first interval corresponding to the first matrix from a plurality of intervals, the plurality of intervals do not overlap each other, and each of the intervals corresponds to at least one scaling factor; Scaling the eigenvalues ​​in the feature map corresponding to the first matrix according to a first scaling factor corresponding to the first interval to obtain a scaled feature map; Scaling the probability distribution parameters corresponding to the first matrix in the probability distribution parameter graph according to a first scaling factor corresponding to the first interval to obtain a scaled probability distribution parameter graph; The scaled probability distribution parameter map is used to perform entropy coding on the scaled feature map and write the result into a bitstream.

2. The method according to claim 1, characterized in that The multiple intervals are divided by multiple thresholds, the number of the multiple thresholds is m, the number of the multiple intervals is m+1, and m is an integer greater than 1.

3. The method according to claim 2, characterized in that The determining, from a plurality of intervals, a first interval corresponding to the first matrix comprises: The first value of the first matrix is ​​compared with at least one threshold value among the multiple threshold values, and the first interval corresponding to the first value is determined from the multiple intervals.

4. The method according to claim 2 or 3, characterized in that: The method further comprises: The number m of the multiple thresholds and the value of each threshold are written into the code stream.

5. The method according to any one of claims 1 to 4, characterized in that: The method further comprises: The first scaling factor is written into the bitstream.

6. The method according to any one of claims 1 to 5, characterized in that Scaling the eigenvalues ​​corresponding to the first matrix in the feature map according to the first scaling factor corresponding to the first interval to obtain the scaled feature map includes: A first scaling factor corresponding to the first interval is multiplied by an eigenvalue in the feature map corresponding to the first matrix to obtain the scaled feature map.

7. The method according to any one of claims 1 to 6, characterized in that Scaling the probability distribution parameter corresponding to the first matrix in the probability distribution parameter graph according to the first scaling factor corresponding to the first interval to obtain the scaled probability distribution parameter graph includes: The first scaling factor corresponding to the first interval and the probability distribution parameter in the probability distribution parameter map corresponding to the first matrix are multiplied to obtain the scaled probability distribution parameter map.

8. The method according to claim 7, characterized in that The probability distribution parameters in the probability distribution parameter map are Gaussian distribution parameters, and the first value of the first matrix is ​​the mean or maximum value of the Gaussian distribution parameters included in the probability distribution parameter map.

9. The method according to any one of claims 1 to 8, characterized in that The method is applied to an adaptive variance scaling module or a residual quantization scaling module.

10. The method according to any one of claims 1 to 9, characterized in that The feature map is a residual feature map, and the eigenvalues ​​in the feature map corresponding to the first matrix are residual eigenvalues.

11. A decoding method, characterized in that: The method comprises: Decode the bitstream to obtain a probability distribution parameter map; According to the probability distribution parameter graph, obtaining a first matrix; Determine a first interval corresponding to the first matrix from a plurality of intervals, the plurality of intervals do not overlap each other, and each of the intervals corresponds to at least one scaling factor; Scaling the probability distribution parameters corresponding to the first matrix in the probability distribution parameter graph according to a first scaling factor corresponding to the first interval to obtain a scaled probability distribution parameter graph; The scaled probability distribution parameter map is used to decode the bitstream to obtain a feature map.

12. The method according to claim 11, characterized in that The method further comprises: The number m of multiple thresholds and the value of each threshold are obtained by decoding the code stream, the multiple intervals are divided by the multiple thresholds, the number of the multiple intervals is m+1, and m is an integer greater than 1.

13. The method according to claim 12, characterized in that The determining, from a plurality of intervals, a first interval corresponding to the first matrix comprises: The first value of the first matrix is ​​compared with at least one threshold value among the multiple threshold values, and the first interval corresponding to the first value is determined from the multiple intervals.

14. The method according to any one of claims 11 to 13, characterized in that The method further comprises: The first scaling factor is obtained by decoding the code stream.

15. The method according to any one of claims 11 to 14, characterized in that Scaling the probability distribution parameter corresponding to the first matrix in the probability distribution parameter graph according to the first scaling factor corresponding to the first interval to obtain the scaled probability distribution parameter graph includes: The first scaling factor corresponding to the first interval and the probability distribution parameter in the probability distribution parameter map corresponding to the first matrix are multiplied to obtain the scaled probability distribution parameter map.

16. The method according to claim 15, characterized in that The probability distribution parameters in the probability distribution parameter map are Gaussian distribution parameters, and the first value of the first matrix is ​​the mean or maximum value of the Gaussian distribution parameters included in the probability distribution parameter map.

17. The method according to any one of claims 11 to 16, characterized in that The method further comprises: The feature map is divided by a first scaling factor corresponding to the first interval to obtain a feature map before scaling.

18. The method according to any one of claims 11 to 17, characterized in that The method is applied to an adaptive variance scaling module or a residual inverse quantization scaling module.

19. The method according to any one of claims 11 to 18, characterized in that The feature map is a residual feature map, and the residual feature map includes residual eigenvalues.

20. The method according to any one of claims 11 to 18, characterized in that The feature map includes feature values, the method is applied to a potential scaling module, the first interval corresponds to the first scaling factor and the second scaling factor, and the method further includes: Get the prediction map and residual map of the feature map; Scaling the predicted eigenvalues ​​corresponding to the first matrix in the prediction graph according to the first scaling factor corresponding to the first interval to obtain a scaled predicted eigenvalue graph; Scaling the residual eigenvalues ​​corresponding to the first matrix in the residual map according to the second scaling factor corresponding to the first interval to obtain a scaled residual feature map; The feature map is updated according to the scaled prediction map and the scaled residual map to obtain an updated feature map.

21. A coding device, characterized in that: including a transceiver unit and a processing unit; The processing unit is used to obtain a feature map of the image to be encoded and a probability distribution parameter map of the feature map; and is also used to obtain a first matrix according to the probability distribution parameter map; and determine a first interval corresponding to the first matrix from a plurality of intervals, wherein the plurality of intervals do not overlap with each other, and each of the intervals corresponds to at least one scaling factor; Scaling the eigenvalues ​​corresponding to the first matrix in the feature map according to the first scaling factor corresponding to the first interval to obtain a scaled feature map; scaling the probability distribution parameters corresponding to the first matrix in the probability distribution parameter map according to the first scaling factor corresponding to the first interval to obtain a scaled probability distribution parameter map; performing entropy coding on the scaled feature map using the scaled probability distribution parameter map and writing the scaled feature map into a bitstream; The transceiver unit is used to send the code stream.

22. The device according to claim 21, characterized in that The multiple intervals are divided by multiple thresholds, the number of the multiple thresholds is m, the number of the multiple intervals is m+1, and m is an integer greater than 1.

23. The device according to claim 22, characterized in that The processing unit is specifically used for: The first value of the first matrix is ​​compared with at least one threshold value among the multiple threshold values, and the first interval corresponding to the first value is determined from the multiple intervals.

24. The device according to claim 22 or 23, characterized in that The processing unit is further configured to write the number m of the multiple thresholds and the value of each threshold into the code stream.

25. The device according to any one of claims 21 to 24, characterized in that The processing unit is further configured to write the first scaling factor into the code stream.

26. The device according to any one of claims 21 to 25, characterized in that The processing unit is specifically used for: A first scaling factor corresponding to the first interval is multiplied by an eigenvalue in the feature map corresponding to the first matrix to obtain the scaled feature map.

27. The device according to any one of claims 21 to 26, characterized in that The processing unit is specifically used for: The first scaling factor corresponding to the first interval and the probability distribution parameter in the probability distribution parameter map corresponding to the first matrix are multiplied to obtain the scaled probability distribution parameter map.

28. The device according to claim 27, characterized in that The probability distribution parameters in the probability distribution parameter map are Gaussian distribution parameters, and the first value of the first matrix is ​​the mean or maximum value of the Gaussian distribution parameters included in the probability distribution parameter map.

29. The device according to any one of claims 21 to 28, characterized in that The device is applied to an adaptive variance scaling module or a residual quantization scaling module.

30. The device according to any one of claims 21 to 29, characterized in that The feature map is a residual feature map, and the eigenvalues ​​in the feature map corresponding to the first matrix are residual eigenvalues.

31. A decoding device, characterized in that: including a transceiver unit and a processing unit; The transceiver unit is used to obtain a code stream; The processing unit is used to decode the bitstream to obtain a probability distribution parameter map; and is also used to obtain a first matrix according to the probability distribution parameter map; and determine a first interval corresponding to the first matrix from a plurality of intervals, wherein the plurality of intervals do not overlap with each other, and each of the intervals corresponds to at least one scaling factor; According to a first scaling factor corresponding to the first interval, the probability distribution parameters corresponding to the first matrix in the probability distribution parameter map are scaled to obtain a scaled probability distribution parameter map; and the scaled probability distribution parameter map is used to decode the bitstream to obtain a feature map.

32. The device according to claim 31, characterized in that The processing unit is further used to decode from the code stream to obtain the number m of multiple thresholds and the value of each threshold, the multiple intervals are divided by the multiple thresholds, the number of the multiple intervals is m+1, and m is an integer greater than 1.

33. The device according to claim 32, characterized in that The processing unit is specifically used for: The first value of the first matrix is ​​compared with at least one threshold value among the multiple threshold values, and the first interval corresponding to the first value is determined from the multiple intervals.

34. The device according to any one of claims 31 to 33, characterized in that The processing unit is further configured to obtain the first scaling factor by decoding the bit stream.

35. The device according to any one of claims 31 to 34, characterized in that The processing unit is specifically used for: The first scaling factor corresponding to the first interval and the probability distribution parameter in the probability distribution parameter map corresponding to the first matrix are multiplied to obtain the scaled probability distribution parameter map.

36. The device according to claim 35, characterized in that The probability distribution parameters in the probability distribution parameter map are Gaussian distribution parameters, and the first value of the first matrix is ​​the mean or maximum value of the Gaussian distribution parameters included in the probability distribution parameter map.

37. The device according to any one of claims 31 to 36, characterized in that The processing unit is further configured to divide the feature map by a first scaling factor corresponding to the first interval to obtain a feature map before scaling.

38. The device according to any one of claims 31 to 37, characterized in that The device is applied to an adaptive variance scaling module or a residual inverse quantization scaling module.

39. The device according to any one of claims 31 to 38, characterized in that The feature map is a residual feature map, and the residual feature map includes residual eigenvalues.

40. The device according to any one of claims 31 to 38, characterized in that The feature map includes feature values, the device is applied to a potential scaling module, the first interval corresponds to the first scaling factor and the second scaling factor, The processing unit is further used to obtain a prediction map and a residual map of the feature map; The processing unit is further configured to scale the predicted feature value corresponding to the first matrix in the prediction graph according to the first scaling factor corresponding to the first interval to obtain a scaled predicted feature graph; The processing unit is further configured to scale the residual eigenvalues ​​corresponding to the first matrix in the residual map according to the second scaling factor corresponding to the first interval to obtain a scaled residual feature map; The processing unit is further used to update the feature map according to the scaled prediction map and the scaled residual map to obtain an updated feature map.

41. A coding device, comprising at least one processor and a memory, characterized in that: The at least one processor executes a program or instruction stored in the memory so that the encoding device implements the method according to any one of claims 1 to 10.

42. A decoding device, comprising at least one processor and a memory, characterized in that: The at least one processor executes a program or instruction stored in the memory so that the decoding device implements the method according to any one of claims 11 to 20.

43. A code stream, characterized in that It includes data for entropy encoding a scaled feature map using a scaled probability distribution parameter map, wherein the scaled probability distribution parameter map scales the probability distribution parameters corresponding to the first matrix in the probability distribution parameter map according to a first scaling factor corresponding to the first interval, the scaled feature map scales the eigenvalues ​​corresponding to the first matrix in the feature map according to the first scaling factor corresponding to the first interval, the first scaling factor is a scaling factor of a first interval corresponding to the first matrix among multiple intervals, the multiple intervals do not overlap each other, each of the intervals corresponds to at least one scaling factor, the first matrix is ​​obtained according to the probability distribution parameter map of the feature map, and the feature map is obtained according to the image to be encoded.

44. A computer-readable storage medium for storing a computer program, characterized in that: When the computer program is executed on a computer or a processor, the computer or the processor is enabled to implement the method of any one of claims 1 to 10 or any one of claims 11 to 20.

45. A computer program product, comprising instructions, characterized in that: When the instructions are executed on a computer or a processor, the computer or the processor is enabled to implement the method of any one of claims 1 to 10 or any one of claims 11 to 20.

Images