Encoding method and system, decoding method and system, device, and storage medium

Abstract

Embodiments of the present application provide an encoding method and system, a decoding method and system, a device, and a storage medium. The method comprises the following steps: on the basis of an image and a reconstructed image of the image in a base layer, determining a residual image of the image in an enhancement layer; on the basis of the reconstructed image, using a trained probability model to predict a probability distribution corresponding to the residual image; and on the basis of the probability distribution corresponding to the residual image, performing entropy encoding on the residual image to obtain a code stream of the image in the enhancement layer. The encoding and decoding solution provided by the embodiments of the present application can improve the compression efficiency of the image in the enhancement layer, thereby improving the overall compression efficiency of the image.

Description

Coding and decoding method, system, device and storage medium

[0001] This application claims priority to the Chinese patent application filed with the China Patent Office on August 25, 2023, with application number 202311085789.0 and application name “Encoding and decoding method, system, device and storage medium”, the entire contents of which are incorporated by reference into this application. Technical Field

[0002] The present application relates to the field of coding and decoding technology, and in particular to a coding and decoding method, system, device and storage medium. Background Art

[0003] Video coding and decoding technology is an effective means to reduce video redundant information and lower video transmission bandwidth. It is widely used in cloud desktops, live video broadcasts, video conferencing and other scenarios.

[0004] In order to further enhance the user's viewing experience and recognition ability, a solution commonly used in video coding systems is to introduce layered coding technology. The so-called layered coding is to encode the base layer and the enhancement layer separately to obtain the base layer code stream and the enhancement layer code stream. Among them, the base layer code stream can enable the decoding end to decode the basic video content completely normally, but the video image obtained by the base layer code stream may have a low resolution or poor quality. When the channel is limited or the channel environment is complex, the base layer code stream can ensure that the decoding end can receive a smooth video image that can be watched. When the channel environment is good or the channel resources are abundant, the enhancement layer code stream can be transmitted additionally to improve the resolution or video quality. However, in the existing technology, the compression efficiency of the enhancement layer is poor.

[0005] Summary of the Invention

[0006] In view of the above problems, the present application is proposed to provide a coding and decoding method, system, device and storage medium that solve the above problems or at least partially solve the above problems.

[0007] Therefore, in one embodiment of the present application, an encoding method is provided, comprising:

[0008] Determining a residual image of the image at an enhancement layer based on the image and a reconstructed image thereof at a base layer;

[0009] According to the reconstructed image, using a trained probability model to predict the probability distribution corresponding to the residual image;

[0010] The residual image is entropy-coded according to the probability distribution corresponding to the residual image to obtain a code stream of the image at the enhancement layer.

[0011] In another embodiment of the present application, a decoding method is provided, comprising:

[0012] Obtaining a code stream of an image at a base layer and a code stream of the image at an enhancement layer;

[0013] Decoding a code stream of the image at a base layer to obtain a reconstructed image;

[0014] According to the reconstructed image, using a trained probability model to predict a probability distribution corresponding to a residual image between the image and the reconstructed image;

[0015] performing entropy decoding on a code stream of the image at an enhancement layer according to a probability distribution corresponding to the residual image to obtain a reconstructed residual image;

[0016] The reconstructed image is corrected according to the reconstructed residual image to obtain a corrected reconstructed image.

[0017] In yet another embodiment of the present application, a coding and decoding system is provided, comprising: a coding end and a decoding end;

[0018] The encoder is configured to: determine a residual image of the image in an enhancement layer based on an image and a reconstructed image of the image in a base layer; predict a probability distribution corresponding to the residual image using a trained probability model based on the reconstructed image; perform entropy coding on the residual image based on the probability distribution corresponding to the residual image to obtain a code stream of the image in the enhancement layer; and send the code stream of the image in the base layer and the code stream of the image in the enhancement layer to the decoding device;

[0019] The decoding end is used to: receive a code stream of the image on a base layer and a code stream of the image on an enhancement layer; decode the code stream of the image on the base layer to obtain a reconstructed image; predict, based on the reconstructed image, a probability distribution corresponding to a residual image between the image and the reconstructed image using a trained probability model; perform entropy decoding on the code stream of the image on the enhancement layer based on the probability distribution corresponding to the residual image to obtain a reconstructed residual image; and correct the reconstructed image based on the reconstructed residual image to obtain a corrected reconstructed image.

[0020] In another embodiment of the present application, an electronic device is provided. The electronic device includes: a memory and a processor, wherein:

[0021] The memory is used to store programs;

[0022] The processor is coupled to the memory and is configured to execute the program stored in the memory to implement any of the above methods.

[0023] In another embodiment of the present application, a computer-readable storage medium storing a computer program is provided. When the computer program is executed by a computer, any of the above-mentioned methods can be implemented.

[0024] The technical solution provided by the embodiments of this application utilizes the correlation between the reconstructed image of the base layer and the residual image of the enhancement layer during the enhancement layer encoding process. The reconstructed image of the base layer is reused as prior data to predict the probability distribution of the residual image features. This eliminates the need to store or transmit additional prior data, improves the compression efficiency of the image in the enhancement layer, and thus improves the overall image compression efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, a brief introduction will be given below to the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0026] FIG1 is a structural block diagram of a coding and decoding system provided by an embodiment of the present application;

[0027] FIG2a is a schematic diagram of a flow chart of an encoding method provided in an embodiment of the present application;

[0028] FIG2 b is a schematic diagram of a priori data preprocessing process according to an embodiment of the present application;

[0029] FIG3 is a schematic diagram of a flow chart of a decoding method provided in an embodiment of the present application;

[0030] FIG4 is a flowchart illustrating a model training method according to an embodiment of the present application;

[0031] FIG5 is a structural block diagram of a cloud desktop system provided in one embodiment of the present application;

[0032] FIG6 is a schematic diagram of a flow chart of a cloud desktop image encoding and decoding method provided in an embodiment of the present application;

[0033] FIG7 is an experimental comparison diagram of the PSNR indicator provided by an embodiment of the present application;

[0034] FIG8 is an experimental comparison diagram of the SSIM index provided in an embodiment of the present application;

[0035] FIG9 is an experimental comparison diagram of LPIPS indicators provided in one embodiment of the present application;

[0036] FIG10 is an experimental comparison diagram of the DISTS indicator provided in an embodiment of the present application;

[0037] FIG11 is a structural block diagram of an electronic device provided in one embodiment of the present application. DETAILED DESCRIPTION

[0038] Typically, an image consists of two chromaticity maps and a luminance map. For example, in the YCbCr format, Y represents luminance (or luma), or grayscale values, while Cb and Cr represent chromaticity, reflecting the offset in blue and red concentrations. YCbCr is commonly used in the video field.

[0039] In order to reduce bit rate consumption, at the base layer, it is necessary to first convert the original image from YCbCr4:4:4 to YCbCr4:2:0 through chroma downsampling, and then encode the image in YCbCr4:2:0 format to obtain the base layer code stream. Among them, the chroma downsampling process of converting the original data from YCbCr4:4:4 to YCbCr4:2:0 will cause information loss, resulting in obvious artifacts and blur in the image or video screen obtained by subsequent decoding. At the enhancement layer, the residual image between the original image and the reconstructed image of the base layer is encoded to obtain the enhancement layer code stream. At the decoding end, combining the base layer code stream and the enhancement layer code stream can improve the objective and subjective quality of the image or video screen at the decoding end.

[0040] How to compress residual images efficiently and with high quality has high research significance and application value.

[0041] The present embodiment proposes a residual image coding scheme. Specifically, during the enhancement layer coding process, the correlation between the reconstructed image of the base layer and the residual image of the enhancement layer is exploited, and the reconstructed image of the base layer is reused as prior data to predict the probability distribution of residual image features. This eliminates the need to store or transmit additional prior data, improves image compression efficiency in the enhancement layer, and thereby improves overall image compression efficiency.

[0042] In order to enable those skilled in the art to better understand the present invention, the technical solutions in the embodiments of the present invention will be clearly and completely described below based on the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments in the present invention, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present invention.

[0043] In addition, some of the processes described in the specification, claims and the above-mentioned figures of this application include multiple operations that appear in a specific order. These operations may not be executed in the order in which they appear in this document or may be executed in parallel. The serial numbers of the operations, such as 101, 102, etc., are only used to distinguish between different operations, and the serial numbers themselves do not represent any order of execution. In addition, these processes may include more or fewer operations, and these operations may be executed in sequence or in parallel. It should be noted that the descriptions of "first", "second", etc. in this document are used to distinguish different messages, devices, modules, etc., and do not represent the order of precedence, nor do they limit "first" and "second" to be different types.

[0044] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with the relevant laws, regulations and standards of relevant countries and regions, and provide corresponding operation entrances for users to choose to authorize or refuse.

[0045] Before introducing the encoding method and decoding method provided by the embodiment of the present application, the system architecture involved in the present solution is introduced. As shown in Figure 1, the encoding and decoding system includes: an encoding end 1 and a decoding end 2; wherein,

[0046] The encoding end is used to perform the following steps:

[0047] 11. Determine a residual image of the image in the enhancement layer based on the image and its reconstructed image in the base layer.

[0048] 12. Based on the reconstructed image, use a trained probability model to predict the probability distribution corresponding to the residual image.

[0049] 13. Perform entropy coding on the residual image according to the probability distribution corresponding to the residual image to obtain a code stream of the image in the enhancement layer.

[0050] 14. Send the code stream of the image at the base layer and the code stream of the image at the enhancement layer to the decoding end.

[0051] The decoding end is used to perform the following steps:

[0052] 21. Receive a code stream of the image at a base layer and a code stream of the image at an enhancement layer.

[0053] 22. Decode the code stream of the image at the base layer to obtain a reconstructed image.

[0054] 23. Based on the reconstructed image, use the probability model to predict the probability distribution corresponding to the residual image between the image and the reconstructed image.

[0055] 24. Perform entropy decoding on the code stream of the image at the enhancement layer according to the probability distribution corresponding to the residual image to obtain a reconstructed residual image.

[0056] 25. Correct the reconstructed image according to the reconstructed residual image to obtain a corrected reconstructed image.

[0057] The above image can be an ordinary image or a video frame in a video.

[0058] The encoding and decoding system provided in the embodiments of the present application can be applied in scenarios such as cloud desktop, live video broadcast, and video conferencing. Taking the cloud desktop as an example, the above image can be a video frame in the cloud desktop video stream, the above encoding end is also the cloud desktop server end; the above decoding end is also the cloud desktop client end.

[0059] The technical solution provided by the embodiments of this application utilizes the correlation between the reconstructed image of the base layer and the residual image of the enhancement layer during the enhancement layer encoding process. The reconstructed image of the base layer is reused as prior data to predict the probability distribution of the residual image features. This eliminates the need to store or transmit additional prior data, improves the compression efficiency of the image in the enhancement layer, and thus improves the overall image compression efficiency.

[0060] The specific processing process of the encoding end device and the decoding end device in the above encoding and decoding system and the interaction process between the two will be described in detail in the following embodiments.

[0061] FIG2a shows a schematic flow chart of the encoding method provided in an embodiment of the present application. The execution subject of the method may be the encoding end in the above-mentioned encoding and decoding system. As shown in FIG2a, the encoding method includes:

[0062] 201. Determine a residual image of the image in an enhancement layer according to the image and its reconstructed image in a base layer;

[0063] 202. Predicting a probability distribution corresponding to the residual image using a trained probability model based on the reconstructed image;

[0064] 203. Perform entropy coding on the residual image according to the probability distribution corresponding to the residual image to obtain a code stream of the image in the enhancement layer.

[0065] In the above 201, the image is a video frame in the cloud desktop video stream.

[0066] A residual image between the image and the reconstructed image may be calculated as the residual image of the image in the enhancement layer. The residual image is used to represent the difference between the image and the reconstructed image.

[0067] When the resolution of the reconstructed image differs from the resolution of the image, the reconstructed image may be resampled to obtain a resampled reconstructed image; the resolution of the resampled reconstructed image is the same as the resolution of the image. A residual image is then calculated between the image and the resampled reconstructed image to serve as the residual image of the image in the enhancement layer.

[0068] In a coding and decoding scenario, generally speaking, the resolution of a reconstructed image is smaller than the resolution of the image. Therefore, the above resampling is specifically upsampling.

[0069] In the above 202, the probability model can also be referred to as an entropy model. The probability model can be selected according to actual needs and is not specifically limited in the embodiments of the present application. The above probability model can be: a Gaussian probability model, a Laplace probability model, etc. The above probability model is based on a neural network and can be trained based on training samples.

[0070] In practical applications, the reconstructed image can be input into a trained probability model, and the probability model will output the probability distribution corresponding to its predicted residual image.

[0071] In one example, the probability distribution corresponding to the residual image may include the probability distribution of the data values ​​of each data bit in the residual image. Generally, any probability distribution can be represented by distribution parameters. For example, a Gaussian probability distribution can be represented by two distribution parameters: the mean and the variance. Therefore, the probability model predicts the distribution parameters of the probability distribution corresponding to the residual image.

[0072] In 203 above, entropy coding refers to coding that uses the entropy principle to avoid losing any information during the encoding process. Information entropy is the average amount of information in the source. Common entropy coding methods include Shannon coding, Huffman coding, and arithmetic coding. Entropy coding is a lossless encoding / lossless compression method.

[0073] In one example, the distribution parameters of the probability distribution of the data value of each data bit in the residual image and the residual image may be input into an entropy encoder for entropy encoding to obtain a code stream of the image at the enhancement layer.

[0074] The residual image is obtained by taking the difference between the image and the reconstructed image. Therefore, the residual image itself is related to the reconstructed image. Generally speaking, the low-frequency and high-frequency signals in an image are also called low-frequency components and high-frequency components. The high-frequency component in an image refers to areas where the image intensity (brightness / grayscale) changes dramatically, that is, edges (outlines, textures); the low-frequency component in an image refers to areas where the image intensity (brightness / grayscale) changes smoothly, that is, large blocks of color. Compared to the original image, the reconstructed image has more distortion in the high-frequency signal and less distortion in the low-frequency signal.

[0075] Therefore, the location of high-frequency information in the reconstructed image can reveal the location of high-frequency signals in the residual image. High-frequency signals in the residual image are important signals and can be compressed at a higher bit rate. The location of low-frequency information in the reconstructed image can reveal the location of low-frequency signals in the residual image. Low-frequency signals in the residual image are unimportant information and can be compressed at a lower bit rate. Therefore, when entropy coding the residual image, it is possible to use the reconstructed image related to the residual image to more accurately predict the probability distribution corresponding to the residual image, thereby reducing the overall bit rate consumption.

[0076] The technical solution provided by the embodiments of this application utilizes the correlation between the reconstructed image of the base layer and the residual image of the enhancement layer during the enhancement layer encoding process. The reconstructed image of the base layer is reused as prior data to predict the probability distribution of the residual image features. This eliminates the need to store or transmit additional prior data, improves the compression efficiency of the image in the enhancement layer, and thus improves the overall image compression efficiency.

[0077] In practical applications, the luminance component is crucial information, and resolution reduction is generally not permitted. Only the two chrominance components are allowed to reduce resolution to improve compression efficiency. Resolution represents the number of pixels in the width and height of an image. For example, if the resolution of image A is 100 x 50, it means that image A has 100 pixels in the width and 50 pixels in the height.

[0078] In one example, the resolution of the luminance map of the reconstructed image is the same as the resolution of the luminance map of the image; the resolution of the chrominance map of the reconstructed image is less than the resolution of the chrominance map of the image. That is, the resolution is not reduced on the luminance component, but the resolution is reduced on the chrominance component. Therefore, the coding loss on the luminance component is negligible. Therefore, in the enhancement layer, only the loss information on the chrominance component needs to be encoded. In the above 201, "determining the residual image of the image in the enhancement layer based on the image and its reconstructed image in the base layer" can be implemented by the following steps:

[0079] 2011. Calculate a chromaticity residual image between the chromaticity map of the image and the chromaticity map of the reconstructed image.

[0080] 2012. Determine the chroma residual image as the residual image of the image in the enhancement layer.

[0081] In the above 2011, the chromaticity residual image is used to represent the difference between the chromaticity diagram of the image and the chromaticity diagram of the reconstructed image.

[0082] Because the resolution of the reconstructed image's chromaticity map is lower than that of the image's chromaticity map, the reconstructed image's chromaticity map needs to be upsampled to obtain an upsampled chromaticity map. The upsampling algorithm used for this upsampling may include, but is not limited to, nearest neighbor interpolation and bilinear interpolation. The resolution of the upsampled chromaticity map is consistent with that of the image's chromaticity map.

[0083] The chromaticity map of the image and the corresponding pixel points in the upsampled chromaticity map are subtracted to obtain a chromaticity residual image.

[0084] In the above 2012, the chroma residual image is used as the residual image of the image in the enhancement layer.

[0085] Since the resolution of the chromaticity map of the reconstructed image is different from that of the luminance map of the reconstructed image, scale alignment is required before they can be used as input to the probability model. In the above 202, "based on the reconstructed image, using the trained probability model to predict the probability distribution corresponding to the residual image" can be achieved by the following steps:

[0086] 2021. Upsample the chromaticity map of the reconstructed image to obtain an upsampled chromaticity map.

[0087] The resolution of the upsampled chromaticity map is the same as the resolution of the luminance map of the reconstructed image.

[0088] 2022. Determine input data of the probability model based on the upsampled chromaticity map and the luminance map of the reconstructed image.

[0089] In the above 2021, the upsampling process can refer to the corresponding content in the above embodiment and will not be repeated here.

[0090] In one example, in step 2022, the upsampled chromaticity map and the luminance map of the reconstructed image may be used as different channel maps in the input data. In this embodiment, the input data includes two channel maps, one being the upsampled chromaticity map and the other being the luminance map of the reconstructed image.

[0091] In this embodiment, not only the chrominance component of the reconstructed image but also the luminance component of the reconstructed image is used to preset the probability distribution corresponding to the chrominance residual image, which has a good prediction accuracy.

[0092] In another example, the reconstructed image has less distortion in the luma component and more distortion in the chroma component than the original image. This is because, during image encoding, the chroma downsampling operation was performed on the image, but not the luma downsampling operation. Chroma downsampling is the most significant cause of significant chroma distortion. To improve the entropy model's probability distribution prediction accuracy, the luma image of the reconstructed image can be subjected to the same downsampling operation to simulate the chroma downsampling process. The luma residual image between the reconstructed image's luma image and the downsampled luma image can then be used to improve the entropy model's probability distribution prediction accuracy. The luma residual image carries luma downsampling distortion. Since the downsampling algorithm used for luma downsampling is the same as that used for chroma downsampling, the luma downsampling distortion is similar to the chroma downsampling distortion. Therefore, using the luma residual image, which carries luma downsampling distortion, can improve the entropy model's prediction accuracy for the probability distribution corresponding to the chroma residual image. Specifically, the step of “determining the input data of the probability model according to the upsampled chromaticity map and the luminance map of the reconstructed image” in 2022 can be implemented by the following steps:

[0093] S11. Determine a downsampling algorithm based on which a chroma downsampling operation is performed on the image at the base layer.

[0094] S12. Perform brightness downsampling on the brightness map of the reconstructed image according to the downsampling algorithm to obtain a downsampled brightness map.

[0095] S13: Determine a brightness residual image between the brightness image of the reconstructed image and the brightness image after downsampling.

[0096] The resolution of the brightness residual image is the same as the resolution of the brightness image of the reconstructed image.

[0097] S14: using the upsampled chromaticity map, the luminance map of the reconstructed image, and the luminance residual map as different channel maps in the input data of the probability model respectively.

[0098] In the above S11, the downsampling algorithm used in the chroma downsampling operation performed on the image at the base layer can be selected according to actual needs, and this embodiment of the application does not specifically limit this. The downsampling algorithm may include but is not limited to: average pooling algorithm, maximum pooling algorithm.

[0099] In the above S12, the luminance map of the reconstructed image is luminance downsampled according to the downsampling algorithm based on the chroma downsampling operation to obtain a downsampled luminance map.

[0100] For example, if the downsampling algorithm used in the chroma downsampling operation is an average pooling algorithm, the luminance map of the reconstructed image is downsampled according to the average pooling algorithm to obtain a downsampled luminance map.

[0101] In the above S13, the resolution of the downsampled luminance map is smaller than that of the reconstructed image luminance map. Therefore, before calculating the luminance residual map, the downsampled luminance map needs to be upsampled to obtain the upsampled luminance map. The resolution of the upsampled luminance map is consistent with the resolution of the luminance map of the reconstructed image. The difference between the corresponding pixels in the luminance map of the reconstructed image and the upsampled luminance map is calculated to obtain the luminance residual map. The upsampling algorithm based on which the upsampling of the downsampled luminance map is performed can be selected according to actual needs and is not specifically limited in this embodiment of the present application. In a specific example, the upsampling algorithm based on which the upsampling of the downsampled luminance map is performed can be consistent with the upsampling algorithm based on which the upsampling operation performed on the chrominance map of the reconstructed image in the above step 201 is performed. In this way, the downsampling process and upsampling process corresponding to the luminance residual map and the chrominance residual map are completely consistent, ensuring that the luminance distortion carried by the luminance residual map can more accurately reflect the chrominance distortion.

[0102] In S14 above, the upsampled chromaticity map, the luminance map of the reconstructed image, and the luminance residual map are used as different channel maps in the input data of the probability model. In other words, the input data includes three channel maps: the upsampled chromaticity map, the luminance map of the reconstructed image, and the luminance residual map. This input data is the final required prior data.

[0103] In this scheme, the accuracy of probability distribution estimation can be improved by using multi-channel information joint input.

[0104] Figure 2b shows an example of the processing flow from reconstructed image to prior data. As shown in Figure 2b, the luminance map of the reconstructed image is subjected to average pooling 2001 and nearest neighbor interpolation 2002, and then subtracted from the luminance map of the reconstructed image to obtain a luminance residual image. Nearest neighbor interpolation 2003 is performed on the chrominance map of the reconstructed image to obtain an upsampled chrominance map. The luminance map of the reconstructed image, the luminance residual image, and the upsampled chrominance map are used as the final prior data.

[0105] In a feasible solution, the probability distribution of the data value of each data bit in the residual image can be directly predicted by the probability model, and then the residual image is directly input into the entropy encoder for encoding.

[0106] In another feasible solution, the probability distribution corresponding to the residual image includes the probability distribution of the residual image features of the residual image. In the above 202, "predicting the probability distribution corresponding to the residual image using a trained probability model based on the reconstructed image" may include:

[0107] According to the reconstructed image, a probability distribution of the residual image features is predicted using a trained probability model.

[0108] Accordingly, the step of “performing entropy coding on the residual image according to the probability distribution corresponding to the residual image to obtain a code stream of the image in the enhancement layer” in 203 may include:

[0109] According to the probability distribution of the residual image features, entropy coding is performed on the residual image features to obtain a code stream of the image at the enhancement layer.

[0110] In this embodiment, rather than directly entropy encoding the residual image, feature extraction is first performed on the residual image to obtain residual image features, which are then entropy encoded. Because the residual image contains a large amount of redundant information, feature extraction extracts important information before entropy encoding, effectively improving the overall compression efficiency of the residual image.

[0111] Optionally, the above method may further include:

[0112] 204. Perform feature extraction on the residual image using the trained coding model to obtain the residual image features.

[0113] The encoding model may be based on a neural network and may be trained in advance based on training samples.

[0114] In one example, the residual image is the chroma residual image described above, which generally includes two chroma components. These two chroma components can be input into the encoding model as two channel maps.

[0115] In practical applications, the residual image features output by the coding model are of floating-point type, whereas entropy coding requires integer input data. Therefore, the residual image can be input into a trained coding model to obtain initial residual image features output by the coding model. These initial residual image features are then quantized to obtain residual image features.

[0116] The above steps 201 to 204 are all performed on the enhancement layer. The following will introduce the processing flow of the base layer. Specifically, the above method may also include:

[0117] 205. At the base layer, perform chroma downsampling on the image to obtain a downsampled image.

[0118] 206. Compress the downsampled image to obtain a code stream of the image at a base layer.

[0119] 207. Decode the code stream of the image at the base layer to obtain the reconstructed image.

[0120] The specific implementation of the above step 205 can refer to the corresponding content in the above corresponding embodiment.

[0121] In the above 206, the compression process of the downsampled image may include: feature extraction, entropy coding, etc., which is not specifically limited in this embodiment of the present application.

[0122] In the above 207, the decoding process of the code stream of the image in the base layer may include: entropy decoding, feature decoding, etc., which is not specifically limited in this embodiment of the present application.

[0123] FIG3 shows a schematic flow chart of a decoding method provided by an embodiment of the present application. The decoding method may be performed by a decoding terminal in the above-mentioned encoding and decoding system. As shown in FIG3 , the method may include:

[0124] 301. Obtain a code stream of an image at a base layer and a code stream of the image at an enhancement layer.

[0125] 302. Decode the code stream of the image at the base layer to obtain a reconstructed image.

[0126] 303. Based on the reconstructed image, use a trained probability model to predict a probability distribution corresponding to a residual image between the image and the reconstructed image.

[0127] 304. Perform entropy decoding on the code stream of the image at the enhancement layer according to the probability distribution corresponding to the residual image to obtain a reconstructed residual image.

[0128] 305. Correct the reconstructed image according to the reconstructed residual image to obtain a corrected reconstructed image.

[0129] In the above 302, the decoding process of the code stream of the image in the base layer may include: entropy decoding, feature decoding, etc., which is not specifically limited in this embodiment of the present application.

[0130] In the above 303, the probability model here is the same as the probability model used by the encoding end. The specific implementation process of the above 303 can be found in the corresponding content of the above embodiments, and will not be repeated here.

[0131] The output of the probability model is: the distribution parameters of the probability distribution corresponding to the residual image.

[0132] In the above 304, the distribution parameters of the probability distribution corresponding to the residual image and the code stream of the image in the enhancement layer can be input into the entropy decoder; and the reconstructed residual image is determined according to the output result of the entropy encoder.

[0133] In one example, the probability distribution corresponding to the residual image includes: a probability distribution of data values ​​of each data bit in the residual image, then the output of the entropy encoder is: a reconstructed residual image.

[0134] In another example, the probability distribution corresponding to the residual image includes: the probability distribution corresponding to the residual image includes the probability distribution of the residual image features of the residual image, that is, the probability distribution of the data value of each data bit in the residual image features. Then, the entropy encoder outputs the residual image features. Then, the reconstructed residual image can be determined based on the residual image features. In one feasible solution, the residual image features can be feature decoded using a trained decoding model to obtain a reconstructed residual image. The encoding model can be based on a neural network. The decoding model and the encoding model can be trained together to improve the training effect of both. The specific training process will be introduced in the following embodiments.

[0135] In the above 305 , the reconstructed residual image may be superimposed on the reconstructed image to obtain a corrected reconstructed image.

[0136] In one embodiment, the reconstructed residual image is a reconstructed chroma residual image, and the resolution of the reconstructed chroma residual image is greater than the resolution of the chroma map of the reconstructed image. Therefore, the reconstructed image can be chroma upsampled to obtain an upsampled reconstructed image; the resolution of the chroma map of the upsampled reconstructed image is the same as the resolution of the reconstructed chroma residual image; in the upsampled reconstructed image, the reconstructed chroma residual image is superimposed on the chroma map of the upsampled reconstructed image to obtain a corrected reconstructed image.

[0137] The technical solution provided by the embodiments of this application utilizes the correlation between the reconstructed image of the base layer and the residual image of the enhancement layer during the enhancement layer encoding process. The reconstructed image of the base layer is reused as prior data to predict the probability distribution of the residual image features. This eliminates the need to store or transmit additional prior data, improves the compression efficiency of the image in the enhancement layer, and thus improves the overall image compression efficiency.

[0138] The following will introduce the training method of the model involved in the embodiment of the present application in conjunction with Figure 4. As shown in Figure 4, the method may include:

[0139] 401. Determine a sample residual image of the sample image in an enhancement layer according to the sample image and its reconstructed sample image in a base layer.

[0140] 402. Perform feature extraction on the sample residual image using a coding model to obtain sample residual image features.

[0141] 403. Perform feature decoding on the sample residual image features using a decoding model to obtain a reconstructed sample residual image.

[0142] 404. Correct the reconstructed sample image according to the reconstructed sample residual image to obtain a corrected reconstructed sample image.

[0143] 405. Optimize parameters of the encoding model and the decoding model with minimizing the first loss function as an optimization goal.

[0144] The first loss function is determined according to the difference between the sample image and the corrected reconstructed sample image.

[0145] In the above 401, the sample residual image between the sample image and the reconstructed sample image is determined as the sample residual image of the sample image in the enhancement layer.

[0146] The specific calculation process can refer to the calculation process of the "residual image" in the above embodiments, and will not be described in detail here.

[0147] In the above 402 and 403, the internal structures of the encoding model and the decoding model can be designed according to actual needs, and the embodiments of the present application do not specifically limit this.

[0148] In the above 404 , the correction process of the reconstructed sample image can refer to the correction process of the “reconstructed image” in the above embodiments, and will not be described in detail here.

[0149] In the above 405, a gradient descent algorithm can be used to optimize the parameters of the encoding model and the decoding model. The difference between the sample image and the modified reconstructed sample image can be characterized by mean square error (MSE), structural similarity (SSIM), etc.

[0150] Optionally, the above training method may further include:

[0151] 406. Determine a predicted probability distribution of the sample residual image features using the probability model according to the reconstructed sample image.

[0152] 407. Optimize parameters of the probability model with minimizing the second loss function as the optimization goal.

[0153] The second loss function is determined according to the difference between the predicted probability distribution and the true probability distribution of the sample residual image feature.

[0154] The specific implementation of the above 406 can refer to the specific implementation of the above 202, which will not be described in detail here.

[0155] In the above 407, a gradient descent algorithm may be used to optimize the parameters of the probability model. The second loss function may be specifically Shannon cross entropy.

[0156] In practical applications, the encoding model, decoding model, and probability model can be jointly trained. Specifically, a total loss function can be determined based on the first and second loss functions. With minimizing the total loss function as the optimization goal, the parameters of the encoding model, decoding model, and probability model are optimized. The total loss function is the sum of the first and second loss functions.

[0157] The solution provided in the embodiment of the present application utilizes a data-driven approach to improve feature extraction of residual images, and is more targeted in processing chroma residual images of different contents.

[0158] The following will take the cloud desktop scenario as an example to introduce the embodiments of this application in detail:

[0159] As shown in Figure 5:

[0160] The cloud desktop server 501 sends the code stream of the cloud desktop image at the base layer and the code stream of the cloud desktop image at the enhancement layer to the cloud desktop client 502. The cloud desktop image is a video frame in the cloud desktop video stream.

[0161] As shown in Figure 6:

[0162] The encoding process on the cloud desktop server side is as follows: the chroma residual image 1 between the cloud desktop image and the reconstructed cloud desktop image 9 is input into the trained encoding model 2 for feature extraction to obtain the initial chroma residual image features; the initial chroma residual image features are input into the quantizer for quantization processing to obtain the chroma residual image features; the chroma residual image features are input into the entropy encoder 5, which entropy encodes the chroma residual image features according to the probability distribution N(μ, θ) of the chroma residual image features output by the probability model to obtain the cloud desktop image code stream at the enhancement layer. Where μ is the mean of the Gaussian distribution, and θ is the variance of the Gaussian distribution.

[0163] Note: Since the dynamic range of the residual image is relatively small and mostly concentrated near 0, it is not normalized to avoid the dynamic range of the latent variable features learned later (that is, the initial chroma residual image features mentioned above) being too small.

[0164] The decoding process of the cloud desktop client: the code stream of the cloud desktop image in the enhancement layer is input into the entropy decoder 6. The upper decoder 6 performs entropy decoding on the code stream of the cloud desktop image in the enhancement layer according to the probability distribution N(μ, θ) of the chroma residual image features output by the probability model to obtain the chroma residual image features; the chroma residual image features are input into the decoding model 7 for feature decoding to obtain the reconstructed chroma residual image.

[0165] In addition, the cloud desktop server can also perform chroma downsampling on the cloud desktop image at the basic layer to obtain a downsampled image; encode the downsampled image to obtain a code stream of the cloud desktop image at the basic layer; and decode the code stream of the cloud desktop image at the basic layer to obtain the above-mentioned reconstructed cloud desktop image.

[0166] The cloud desktop client also includes: decoding the code stream of the cloud desktop image at the basic layer to obtain a reconstructed cloud desktop image; and correcting the reconstructed cloud desktop image at the basic layer according to the reconstructed chroma residual image to obtain a corrected reconstructed cloud desktop image.

[0167] The probability distribution process that both the cloud desktop server and the cloud desktop client need to execute: the prior preprocessing model 10 preprocesses the reconstructed cloud desktop image to obtain prior data. The preprocessing may include: the size alignment mentioned above and the acquisition of the luminance residual image. The preprocessing process can refer to the corresponding content in the above embodiments and will not be repeated here. The prior data is input into the prior feature extraction module 1001 of the probability model 11 for feature extraction to obtain the prior features. Among them, the prior feature extraction module also adopts the structure of a neural network, retains a relatively large receptive field, and extracts the global characteristics of the prior data. The prior data also contains high-frequency information such as edges and textures in the original image, which needs to be decomposed and extracted by the prior encoder to screen out high-frequency features close to the chroma residual to improve the probability prediction accuracy. The prior features are input into the probability prediction module 1002 of the probability model 11 to predict the probability distribution N (μ, θ) of the chroma residual image features.

[0168] The encoding module can extract the effective features of the chroma residual image and obtain the latent variable features. The encoding module and decoding module are responsible for the conversion between the image space and the latent variable space of the chroma residual image. The encoding module and decoding module based on neural networks can more effectively adapt to the content characteristics of the chroma residual, and the representation ability of the latent variable features is more powerful. The probability model takes the reconstructed image as input, extracts the global prior information of the chroma residual image, establishes an independent Gaussian distribution for each position of the latent variable feature, and uses the global prior to predict the Gaussian distribution parameters. The independent Gaussian model combined with the prior information can more effectively adapt to the residual data distribution with higher sparsity and improve the prediction accuracy of the probability model. Entropy encoder / decoder: An arithmetic encoder / decoder can be used to encode / decode the quantized latent variable features using the Gaussian probability distribution parameters provided by the probability model to achieve conversion between features and code streams. The solution provided in the embodiment of the present application can effectively improve the compression efficiency of the chroma residual image and achieve better compression quality at the same bit rate.

[0169] Experimental comparison:

[0170] The following uses subjective and objective performance indicators, such as Peak signal-to-noise ratio (PSNR), SSIM, Learned Perceptual Image Patch Similarity (LPIPS), and DISTS (Differentiable Image Saliency Transform for Improved Scalability and Portability of Image Quality Assessment), to demonstrate the technical effectiveness of each scheme. The chroma residual images obtained using this scheme and several existing compression schemes are superimposed on the reconstructed image of the base layer, and the final performance results are compared. It can be seen that compared with existing compression methods, this scheme achieves superior results in all indicators.

[0171] As shown in Figure 7, this solution performs well in terms of PSNR. Note: The higher the PSNR, the better.

[0172] As shown in Figure 8, this solution performs well in terms of the SSIM indicator. Note: The higher the SSIM indicator, the better.

[0173] As shown in Figure 9, this solution performs well in terms of the LPIPS indicator. Note: The lower the LPIPS indicator, the better.

[0174] As shown in Figure 10, this solution performs well in terms of the DISTS indicator. Note: The lower the DISTS indicator, the better.

[0175] It should be noted that the multiple unlabeled curves in FIG7 , FIG8 , FIG9 and FIG10 respectively correspond to multiple existing compression schemes.

[0176] In summary, the neural network-based compression scheme provided by this solution supports dual-chroma channel residuals as input to the coding model, and inputs dual-chroma information as different channel images of the coding model for processing. In addition, the coding model supports input of pixels of various bit depths, such as 16-bit depth. To address the problem of probability / entropy estimation, a probability model is introduced to predict the probability distribution of the residual image based on the reconstructed image. The distribution of the residual information is estimated by using the reconstructed data of the base layer shared by the encoder and decoder. Compared with traditional compression algorithms, the encoding and transmission of super-prior information is omitted. In addition, the probability model uses multi-channel information joint input to improve the accuracy of probability estimation. To address the problem of inconsistent sizes of the luminance / chrominance components of the reconstructed image, methods including nearest neighbor sampling are introduced to align the sizes before inputting them into the probability model.

[0177] Figure 11 shows a schematic diagram of the structure of an electronic device provided in an embodiment of the present application. As shown in Figure 11, the electronic device includes a memory 1101 and a processor 1102. The memory 1101 can be configured to store various other data to support operations on the electronic device. Examples of such data include instructions for any application or method operating on the electronic device. The memory 1101 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read only memory (EEPROM), erasable programmable read only memory (EPROM), programmable read only memory (PROM), read only memory (ROM), magnetic memory, flash memory, magnetic disk or optical disk.

[0178] The memory 1101 is used to store programs;

[0179] The processor 1102 is coupled to the memory 1101 and is configured to execute the program stored in the memory 1101 to implement the methods provided in the above method embodiments.

[0180] 11 , the electronic device further includes other components such as a communication component 1103, a display 1104, a power supply component 1105, and an audio component 1106. FIG11 only schematically illustrates some components, which does not mean that the electronic device only includes the components shown in FIG11 .

[0181] Accordingly, an embodiment of the present application further provides a computer-readable storage medium storing a computer program, which, when executed by a computer, can implement the steps or functions of the methods provided in the above-mentioned method embodiments.

[0182] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, i.e., they may be located in one location or distributed across multiple network units. Some or all of the modules may be selected based on actual needs to achieve the objectives of the present embodiment. Persons of ordinary skill in the art will be able to understand and implement the present invention without inventive effort.

[0183] Through the description of the above embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus a necessary general hardware platform, or of course, by hardware. Based on this understanding, the above technical solution, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. The computer software product can be stored in a computer-readable storage medium, such as ROM (Read Only Memory) / RAM (Random Access Memory), a disk, an optical disk, etc., and includes a number of instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or certain parts of the embodiments.

[0184] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A coding method, characterized in that: include: Determining a residual image of the image at an enhancement layer according to the image and its reconstructed image at a base layer; According to the reconstructed image, using a trained probability model to predict the probability distribution corresponding to the residual image; According to the probability distribution corresponding to the residual image, entropy coding is performed on the residual image to obtain a code stream of the image at the enhancement layer.

2. The method according to claim 1, characterized in that Determining a residual image of the image at an enhancement layer according to the image and its reconstructed image at a base layer includes: Calculating a chromaticity residual image between the chromaticity map of the image and the chromaticity map of the reconstructed image; The chrominance residual image is determined as the residual image of the image in the enhancement layer.

3. The method according to claim 2, characterized in that According to the reconstructed image, using a trained probability model to predict the probability distribution corresponding to the residual image, comprising: Upsampling the chromaticity map of the reconstructed image to obtain an upsampled chromaticity map; the resolution of the upsampled chromaticity map is the same as the resolution of the luminance map of the reconstructed image; Input data of the probability model is determined according to the upsampled chromaticity map and the luminance map of the reconstructed image.

4. The method according to claim 3, characterized in that: The resolution of the brightness map of the reconstructed image is the same as the resolution of the brightness map of the image; Determining input data of the probability model according to the upsampled chromaticity map and the brightness map of the reconstructed image includes: determining a downsampling algorithm according to which a chroma downsampling operation performed on the image at the base layer is performed; According to the downsampling strategy, the brightness map of the reconstructed image is downsampled to obtain a downsampled brightness map; Determine a brightness residual map between the brightness map of the reconstructed image and the brightness map after downsampling; the resolution of the brightness residual map is the same as the resolution of the brightness map of the reconstructed image; The upsampled chromaticity map, the brightness map of the reconstructed image, and the brightness residual map are respectively used as different channel maps in the input data of the probability model.

5. The method according to any one of claims 1 to 4, characterized in that The probability distribution corresponding to the residual image includes the probability distribution of the residual image features of the residual image; According to the reconstructed image, using a trained probability model to predict the probability distribution corresponding to the residual image, comprising: According to the reconstructed image, predicting the probability distribution of the residual image features using a trained probability model; According to the probability distribution corresponding to the residual image, entropy coding is performed on the residual image to obtain a code stream of the image at the enhancement layer, including: According to the probability distribution of the residual image features, the residual image features are entropy encoded to obtain the The code stream of the image in the enhancement layer.

6. The method according to claim 5, characterized in that Also includes: The trained coding model is used to perform feature extraction on the residual image to obtain the residual image features.

7. The method according to claim 6, characterized in that Extracting features from the residual image using the trained coding model to obtain features of the residual image includes: Inputting the residual image into a trained coding model to obtain initial residual image features output by the coding model; The initial residual image features are quantized to obtain residual image features.

8. The method according to any one of claims 1 to 4, characterized in that The image is a video frame in the cloud desktop video stream.

9. The method according to any one of claims 1 to 4, characterized in that Also includes: At the base layer, chroma downsampling is performed on the image to obtain a downsampled image; Encoding the downsampled image to obtain a code stream of the image at a base layer; The code stream of the image at the base layer is decoded to obtain the reconstructed image.

10. A decoding method, characterized in that: include: Acquire a code stream of an image at a base layer and a code stream of the image at an enhancement layer; Decoding a code stream of the image at a base layer to obtain a reconstructed image; According to the reconstructed image, predicting the probability distribution corresponding to the residual image between the image and the reconstructed image using a trained probability model; According to the probability distribution corresponding to the residual image, entropy decoding is performed on the code stream of the image at the enhancement layer to obtain a reconstructed residual image; The reconstructed image is corrected according to the reconstructed residual image to obtain a corrected reconstructed image.

11. A coding and decoding system, characterized in that: include: Encoding end and decoding end; The encoding end is used to: determine a residual image of the image at the enhancement layer according to the image and its reconstructed image at the base layer; According to the reconstructed image, predict the probability distribution corresponding to the residual image by using a trained probability model; according to the probability distribution corresponding to the residual image, entropy encode the residual image to obtain a code stream of the image at the enhancement layer; send the code stream of the image at the base layer and the code stream of the image at the enhancement layer to the decoding end device; The decoding end is used to: receive the code stream of the image at the basic layer and the code stream of the image at the enhanced layer; decode the code stream of the image at the basic layer to obtain the reconstructed image; predict the probability distribution corresponding to the residual image between the image and the reconstructed image using the probability model according to the reconstructed image; perform entropy decoding on the code stream of the image at the enhanced layer according to the probability distribution corresponding to the residual image to obtain the reconstructed residual image; and correct the reconstructed image according to the reconstructed residual image to obtain a corrected reconstructed image. image.

12. An electronic device, characterized in that: include: A memory and a processor, wherein The memory is used to store programs; The processor is coupled to the memory, and is configured to execute the program stored in the memory to implement the method according to any one of claims 1 to 10.

13. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a computer, the method according to any one of claims 1 to 10 can be implemented.

Images