Coding and decoding method, system, equipment and storage medium

Abstract

The embodiment of the invention provides a coding and decoding method and system, equipment and a storage medium. The method comprises the following steps: determining a residual image of an image in an enhancement layer according to the image and a reconstructed image of the image in a basic layer; according to the reconstructed image, utilizing a trained probability model to predict probability distribution corresponding to the residual image; and performing entropy coding on the residual image according to the probability distribution corresponding to the residual image to obtain a code stream of the image in an enhancement layer. According to the encoding and decoding scheme provided by the embodiment of the invention, the compression efficiency of the image in the enhancement layer can be improved, and the overall compression efficiency of the image is further improved.

Description

Technical Field

[0001] The present application relates to the field of coding and decoding technology, and in particular to a coding and decoding method, system, device and storage medium. Background Art

[0002] As an effective means to reduce redundant video information and lower video transmission bandwidth, video coding and decoding technology is widely used in cloud desktop, live video, video conferencing and other scenarios.

[0003] In order to further enhance the user's viewing experience and recognition ability, a solution commonly used in video coding systems is to introduce layered coding technology. The so-called layered coding is to encode the basic layer and the enhancement layer respectively to obtain the code stream of the basic layer and the code stream of the enhancement layer. Among them, the code stream of the basic layer can enable the decoding end to decode the basic video content completely normally, but the video image obtained by the code stream of the basic layer may have a low resolution or poor quality. When the channel is limited or the channel environment is complex, the code stream of the basic layer can ensure that the decoding end can receive a smooth video image that can be watched. When the channel environment is good or the channel resources are abundant, the code stream of the enhancement layer can be transmitted additionally to improve the resolution or video quality. However, in the prior art, the compression efficiency of the enhancement layer is poor. Summary of the invention

[0004] In view of the above problems, the present application is proposed to provide a coding and decoding method, system, device and storage medium that solve the above problems or at least partially solve the above problems.

[0005] Therefore, in one embodiment of the present application, a coding method is provided, comprising:

[0006] Determining a residual image of the image at an enhancement layer according to the image and its reconstructed image at a base layer;

[0007] According to the reconstructed image, using a trained probability model to predict the probability distribution corresponding to the residual image;

[0008] According to the probability distribution corresponding to the residual image, entropy coding is performed on the residual image to obtain a code stream of the image at the enhancement layer.

[0009] In another embodiment of the present application, a decoding method is provided, comprising:

[0010] Acquire a code stream of an image at a base layer and a code stream of the image at an enhancement layer;

[0011] Decoding a code stream of the image at a base layer to obtain a reconstructed image;

[0012] According to the reconstructed image, predicting a probability distribution corresponding to a residual image between the image and the reconstructed image using a trained probability model;

[0013] According to the probability distribution corresponding to the residual image, entropy decoding is performed on the code stream of the image at the enhancement layer to obtain a reconstructed residual image;

[0014] The reconstructed image is corrected according to the reconstructed residual image to obtain a corrected reconstructed image.

[0015] In yet another embodiment of the present application, a coding and decoding system is provided, comprising: a coding end and a decoding end;

[0016] The encoding end is used to: determine a residual image of the image in an enhancement layer according to an image and a reconstructed image of the image in a base layer; predict a probability distribution corresponding to the residual image by using a trained probability model according to the reconstructed image; perform entropy coding on the residual image according to the probability distribution corresponding to the residual image to obtain a code stream of the image in an enhancement layer; and send the code stream of the image in the base layer and the code stream of the image in the enhancement layer to the decoding end device;

[0017] The decoding end is used to: receive a code stream of the image at a basic layer and a code stream of the image at an enhanced layer; decode the code stream of the image at the basic layer to obtain a reconstructed image; predict, according to the reconstructed image, a probability distribution corresponding to a residual image between the image and the reconstructed image using a trained probability model; perform entropy decoding on the code stream of the image at the enhanced layer according to the probability distribution corresponding to the residual image to obtain a reconstructed residual image; and correct the reconstructed image according to the reconstructed residual image to obtain a corrected reconstructed image.

[0018] In another embodiment of the present application, an electronic device is provided. The electronic device includes: a memory and a processor, wherein:

[0019] The memory is used to store programs;

[0020] The processor is coupled to the memory, and is used to execute the program stored in the memory to implement any of the methods described above.

[0021] In yet another embodiment of the present application, a computer-readable storage medium storing a computer program is provided, and when the computer program is executed by a computer, any of the above-mentioned methods can be implemented.

[0022] In the technical solution provided by the embodiment of the present application, during the enhancement layer encoding process, the correlation between the reconstructed image of the base layer and the residual image of the enhancement layer is used to reuse the reconstructed image of the base layer and use it as prior data to predict the probability distribution of the residual image features. In this way, there is no need to store or transmit additional prior data, which can improve the compression efficiency of the image in the enhancement layer, thereby improving the overall compression efficiency of the image. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, a brief introduction will be given below to the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0024] Figure 1 A structural block diagram of a coding and decoding system provided in an embodiment of the present application;

[0025] Figure 2a A schematic diagram of a flow chart of an encoding method provided in an embodiment of the present application;

[0026] Figure 2b A schematic diagram of a priori data preprocessing process provided in an embodiment of the present application;

[0027] Figure 3 A flowchart of a decoding method provided in an embodiment of the present application;

[0028] Figure 4 An example flow chart of a model training method provided in one embodiment of the present application;

[0029] Figure 5 A structural block diagram of a cloud desktop system provided in one embodiment of the present application;

[0030] Figure 6 A schematic diagram of a process flow of a cloud desktop image encoding and decoding method provided in an embodiment of the present application;

[0031] Figure 7 An experimental comparison chart of the PSNR index provided in an embodiment of the present application;

[0032] Figure 8 An experimental comparison chart of the SSIM index provided in an embodiment of the present application;

[0033] Fig. 9 An experimental comparison chart of the LPIPS index provided in one embodiment of the present application;

[0034] Fig.10An experimental comparison chart of the DISTS index provided in an embodiment of the present application;

[0035] Fig.11 A structural block diagram of an electronic device provided in one embodiment of the present application. DETAILED DESCRIPTION

[0036] Usually, an image consists of two chromaticity graphs and one luminance graph. Take the YCbCr format image as an example, where Y represents luminance (Luminance or luma), that is, the grayscale value; while Cb and Cr represent chromaticity, which is used to reflect the concentration offset of blue and red. Among them, YCbCr is often used in the video field.

[0037] In order to reduce the bit rate consumption, at the base layer, it is necessary to first convert the original image from YCbCr4:4:4 to YCbCr4:2:0 through chroma downsampling, and then encode the image in YCbCr4:2:0 format to obtain the base layer bitstream. Among them, the chroma downsampling process of converting the original data from YCbCr4:4:4 to YCbCr4:2:0 will cause information loss, resulting in obvious artifacts and blur in the image or video obtained by subsequent decoding. At the enhancement layer, the residual image between the original image and the reconstructed image of the base layer is encoded to obtain the bitstream of the enhancement layer. At the decoding end, combining the bitstream of the base layer and the bitstream of the enhancement layer can improve the objective and subjective quality of the image or video at the decoding end.

[0038] How to compress residual images efficiently and with high quality has high research significance and application value.

[0039] The embodiment of the present application proposes a residual image coding scheme. That is, in the process of enhancing layer coding, the correlation between the reconstructed image of the base layer and the residual image of the enhancing layer is utilized, and the reconstructed image of the base layer is reused as prior data to predict the probability distribution of the residual image features. In this way, there is no need to store or transmit additional prior data, which can improve the compression efficiency of the image in the enhancing layer, thereby improving the overall compression efficiency of the image.

[0040] In order to enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below according to the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present application.

[0041] In addition, some of the processes described in the specification, claims and the above-mentioned figures of the present application include multiple operations that appear in a specific order, and these operations may not be executed in the order in which they appear in this article or may be executed in parallel. The serial numbers of the operations, such as 101, 102, etc., are only used to distinguish between different operations, and the serial numbers themselves do not represent any execution order. In addition, these processes may include more or fewer operations, and these operations may be executed in sequence or in parallel. It should be noted that the descriptions of "first", "second", etc. in this article are used to distinguish different messages, devices, modules, etc., do not represent the order of precedence, and do not limit the "first" and "second" to be different types.

[0042] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with the relevant laws, regulations and standards of the relevant countries and regions, and provide corresponding operation entrances for users to choose to authorize or refuse.

[0043] Before introducing the encoding method and decoding method provided in the embodiments of the present application, the system architecture involved in the present solution is introduced. Figure 1 As shown, the encoding and decoding system includes: an encoding end 1 and a decoding end 2; wherein,

[0044] The encoding end is used to perform the following steps:

[0045] 11. Determine a residual image of the image at an enhancement layer based on the image and its reconstructed image at a base layer.

[0046] 12. Based on the reconstructed image, use a trained probability model to predict the probability distribution corresponding to the residual image.

[0047] 13. Perform entropy coding on the residual image according to the probability distribution corresponding to the residual image to obtain a code stream of the image at the enhancement layer.

[0048] 14. Send the code stream of the image at the base layer and the code stream of the image at the enhancement layer to the decoding end.

[0049] The decoding end is used to perform the following steps:

[0050] 21. Receive a code stream of the image at a base layer and a code stream of the image at an enhancement layer.

[0051] 22. Decode the code stream of the image at the base layer to obtain a reconstructed image.

[0052] 23. Based on the reconstructed image, use the probability model to predict the probability distribution corresponding to the residual image between the image and the reconstructed image.

[0053] 24. Perform entropy decoding on the code stream of the image at the enhancement layer according to the probability distribution corresponding to the residual image to obtain a reconstructed residual image.

[0054] 25. According to the reconstructed residual image, correct the reconstructed image to obtain a corrected reconstructed image.

[0055] The above image can be a normal image or a video frame in a video.

[0056] The encoding and decoding system provided in the embodiment of the present application can be applied in cloud desktop, video live broadcast, video conferencing and other scenarios. Taking cloud desktop as an example, the above image can be a video frame in the cloud desktop video stream, the above encoding end is also the cloud desktop server end; the above decoding end is also the cloud desktop client end.

[0057] In the technical solution provided by the embodiment of the present application, during the enhancement layer encoding process, the correlation between the reconstructed image of the base layer and the residual image of the enhancement layer is used to reuse the reconstructed image of the base layer and use it as prior data to predict the probability distribution of the residual image features. In this way, there is no need to store or transmit additional prior data, which can improve the compression efficiency of the image in the enhancement layer, thereby improving the overall compression efficiency of the image.

[0058] The specific processing process of the encoding end device and the decoding end device in the above encoding and decoding system and the interaction process between the two will be described in detail in the following embodiments.

[0059] Figure 2a FIG. 1 is a flow chart of a coding method provided by an embodiment of the present application. The execution subject of the method may be a coding end in the above coding and decoding system. Figure 2a As shown, the encoding method includes:

[0060] 201. Determine a residual image of the image in an enhancement layer according to the image and its reconstructed image in a base layer;

[0061] 202. Predicting a probability distribution corresponding to the residual image using a trained probability model according to the reconstructed image;

[0062] 203. Perform entropy coding on the residual image according to the probability distribution corresponding to the residual image to obtain a code stream of the image at the enhancement layer.

[0063] In the above 201, the image is a video frame in the cloud desktop video stream.

[0064] A residual image between the image and the reconstructed image may be calculated as the residual image of the image at the enhancement layer. The residual image is used to represent the difference between the image and the reconstructed image.

[0065] When the resolution of the reconstructed image is different from the resolution of the image, the reconstructed image may be resampled to obtain a resampled reconstructed image; the resolution of the resampled reconstructed image is the same as the resolution of the image. Then, a residual image between the image and the resampled reconstructed image is calculated as the residual image of the image in the enhancement layer.

[0066] In the encoding and decoding scenario, generally speaking, the resolution of the reconstructed image is smaller than the resolution of the image, so the above resampling is specifically upsampling.

[0067] In the above 202, the probability model can also be referred to as an entropy model. The probability model can be selected according to actual needs, and the embodiment of the present application does not specifically limit this. The above probability model can be: a Gaussian probability model, a Laplace probability model, etc. The above probability model is based on a neural network and can be trained according to training samples.

[0068] In practical applications, the reconstructed image can be input into a trained probability model, and the probability model will output the probability distribution corresponding to its predicted residual image.

[0069] In one example, the probability distribution corresponding to the residual image may include: the probability distribution of the data value of each data bit in the residual image. Generally, any probability distribution can be represented by a distribution parameter. Taking the Gaussian probability distribution as an example, the Gaussian probability distribution can be represented by two distribution parameters, the mean and the variance. Therefore, the probability model predicts the distribution parameters of the probability distribution corresponding to the residual image.

[0070] In the above 203, entropy coding is coding that does not lose any information during the coding process according to the entropy principle. Information entropy is the average amount of information in the source. Common entropy coding methods include Shannon coding, Huffman coding, and arithmetic coding. Entropy coding is a lossless coding / lossless compression method.

[0071] In one example, the distribution parameters of the probability distribution of the data value of each data bit in the residual image and the residual image may be input into an entropy encoder for entropy encoding to obtain a code stream of the image at the enhancement layer.

[0072] The residual image is obtained by taking the difference between the image and the reconstructed image. Therefore, the residual image itself is related to the reconstructed image. Generally speaking, the low-frequency signal and high-frequency signal in the image are also called low-frequency component and high-frequency component. The high-frequency component in the image refers to the place where the image intensity (brightness / grayscale) changes dramatically, that is, the edge (contour, texture); the low-frequency component in the image refers to the place where the image intensity (brightness / grayscale) changes smoothly, that is, the large color blocks. Compared with the image, the reconstructed image has more high-frequency signal distortion and less low-frequency signal distortion.

[0073] Then, the position of high-frequency information in the reconstructed image can reveal the position of high-frequency signals in the residual image. High-frequency signals in the residual image are important signals and can be compressed at a higher bit rate. The position of low-frequency information in the reconstructed image can reveal the position of low-frequency signals in the residual image. Low-frequency signals in the residual image are unimportant information and can be compressed at a lower bit rate. It can be seen that when entropy coding the residual image, the reconstructed image related to the residual image can be used to more accurately predict the probability distribution corresponding to the residual image, so as to reduce the overall bit rate consumption.

[0074] In the technical solution provided by the embodiment of the present application, during the enhancement layer encoding process, the correlation between the reconstructed image of the base layer and the residual image of the enhancement layer is used to reuse the reconstructed image of the base layer and use it as prior data to predict the probability distribution of the residual image features. In this way, there is no need to store or transmit additional prior data, which can improve the compression efficiency of the image in the enhancement layer, thereby improving the overall compression efficiency of the image.

[0075] In practical applications, the brightness component is very important information, and resolution reduction is generally not allowed. Only the resolution reduction on the two chrominance components is allowed to improve compression efficiency. Resolution is used to indicate the number of pixels in the width and height of an image. For example, if the resolution of image A is 100*50, it means that image A has 100 pixels in the width direction and 50 pixels in the height direction.

[0076] In one example, the resolution of the luminance map of the reconstructed image is the same as the resolution of the luminance map of the image; the resolution of the chrominance map of the reconstructed image is less than the resolution of the chrominance map of the image. That is, the resolution is not reduced on the luminance component, but the resolution is reduced on the chrominance component. Therefore, the coding loss on the luminance component is negligible. Therefore, in the enhancement layer, only the loss information on the chrominance component needs to be encoded. In the above 201, "determining the residual image of the image on the enhancement layer based on the image and its reconstructed image on the base layer" can be implemented by the following steps:

[0077] 2011. Calculate a chromaticity residual image between the chromaticity map of the image and the chromaticity map of the reconstructed image.

[0078] 2012. Determine the chrominance residual image as the residual image of the image in the enhancement layer.

[0079] In the above 2011, the chromaticity residual image is used to represent the difference between the chromaticity diagram of the image and the chromaticity diagram of the reconstructed image.

[0080] Since the resolution of the chromaticity diagram of the reconstructed image is smaller than the resolution of the chromaticity diagram of the image, it is necessary to upsample the chromaticity diagram of the reconstructed image to obtain an upsampled chromaticity diagram. The upsampling algorithm based on the upsampling may include but is not limited to: nearest neighbor interpolation method, bilinear interpolation method. The resolution of the upsampled chromaticity diagram is consistent with the resolution of the chromaticity diagram of the image.

[0081] The chromaticity map of the image and the corresponding pixel points in the upsampled chromaticity map are subtracted to obtain a chromaticity residual image.

[0082] In the above 2012, the chrominance residual image is used as the residual image of the image in the enhancement layer.

[0083] Since the resolution of the chromaticity map of the reconstructed image is different from that of the brightness map of the reconstructed image, scale alignment is required before it can be used as the input of the probability model. In the above 202, "according to the reconstructed image, using the trained probability model to predict the probability distribution corresponding to the residual image" can be implemented by the following steps:

[0084] 2021. Upsample the chromaticity map of the reconstructed image to obtain an upsampled chromaticity map.

[0085] The resolution of the up-sampled chromaticity map is the same as the resolution of the luminance map of the reconstructed image.

[0086] 2022. Determine input data of the probability model according to the upsampled chromaticity map and the brightness map of the reconstructed image.

[0087] In the above 2021, the upsampling process can refer to the corresponding content in the above embodiment, and will not be repeated here.

[0088] In one example, in the above 2022, the upsampled chromaticity map and the brightness map of the reconstructed image can be used as different channel maps in the input data. In this embodiment, the input data includes two channel maps, one is the upsampled chromaticity map, and the other is the brightness map of the reconstructed image.

[0089] In this embodiment, not only the chrominance component of the reconstructed image but also the luminance component of the reconstructed image is used to preset the probability distribution corresponding to the chrominance residual image, which has a good prediction accuracy.

[0090] In another example, the reconstructed image has a smaller distortion in the luminance component and a larger distortion in the chrominance component than the image. This is because, when encoding the image, the chrominance downsampling operation is performed on the image, but the luminance downsampling operation is not performed. Among them, the chrominance downsampling operation is the most important reason for the large amount of chrominance distortion. In order to improve the probability distribution prediction accuracy of the entropy model, the same downsampling operation can be performed on the luminance map of the reconstructed image to simulate the chrominance downsampling process, and the luminance residual image between the luminance map of the reconstructed image and the downsampled luminance map is used to improve the probability distribution prediction accuracy of the entropy model. Among them, the luminance residual image carries the luminance downsampling distortion. Since the downsampling algorithm used for luminance downsampling is the same as the downsampling algorithm used for chrominance downsampling, the luminance downsampling distortion is similar to the chrominance downsampling distortion. Therefore, using the luminance residual image carrying the luminance downsampling distortion can improve the prediction accuracy of the entropy model for the probability distribution corresponding to the chrominance residual image. Specifically, in the above 2022, “determining the input data of the probability model according to the upsampled chromaticity map and the brightness map of the reconstructed image” can be implemented by the following steps:

[0091] S11. Determine a downsampling algorithm based on which a chroma downsampling operation performed on the image at the base layer is performed.

[0092] S12. Perform brightness downsampling on the brightness map of the reconstructed image according to the downsampling algorithm to obtain a downsampled brightness map.

[0093] S13: Determine a brightness residual image between the brightness image of the reconstructed image and the brightness image after downsampling.

[0094] The resolution of the brightness residual map is the same as the resolution of the brightness map of the reconstructed image.

[0095] S14, using the upsampled chromaticity map, the brightness map of the reconstructed image, and the brightness residual map as different channel maps in the input data of the probability model respectively.

[0096] In the above S11, the downsampling algorithm based on which the chroma downsampling operation is performed on the image at the base layer can be selected according to actual needs, and the embodiment of the present application does not specifically limit this. The downsampling algorithm may include but is not limited to: average pooling algorithm, maximum pooling algorithm.

[0097] In the above S12, according to the downsampling algorithm based on the chroma downsampling operation, the luminance map of the reconstructed image is downsampled to obtain a downsampled luminance map.

[0098] For example: if the downsampling algorithm used for the chroma downsampling operation is an average pooling algorithm, the brightness map of the reconstructed image is downsampled according to the average pooling algorithm to obtain a downsampled brightness map.

[0099] In the above S13, the resolution of the downsampled brightness map is smaller than the brightness map of the reconstructed image. Therefore, before calculating the brightness residual map, the downsampled brightness map needs to be upsampled to obtain the upsampled brightness map. The resolution of the upsampled brightness map is consistent with the resolution of the brightness map of the reconstructed image. The brightness map of the reconstructed image and the corresponding pixel points in the upsampled brightness map are subtracted to obtain the brightness residual map. The upsampling algorithm based on which the upsampling of the downsampled brightness map is performed can be selected according to actual needs, and the embodiment of the present application does not make specific restrictions on this. In a specific example, the upsampling algorithm based on which the upsampling of the downsampled brightness map is performed can be consistent with the upsampling algorithm based on which the upsampling operation performed on the chromaticity map of the reconstructed image in the above step 201 is performed. In this way, the downsampling process and upsampling process corresponding to the brightness residual map and the chromaticity residual map are completely consistent, ensuring that the brightness distortion carried by the brightness residual map can more accurately reflect the chromaticity distortion.

[0100] In the above S14, the upsampled chromaticity map, the brightness map of the reconstructed image, and the brightness residual map are respectively used as different channel maps in the input data of the probability model. In other words, the input data includes three channel maps, namely: the upsampled chromaticity map, the brightness map of the reconstructed image, and the brightness residual map. The input data is the prior data ultimately required.

[0101] In this scheme, the accuracy of probability distribution estimation can be improved by using multi-channel information joint input.

[0102] Figure 2b An example diagram of the processing flow from reconstructed image to prior data is shown. Figure 2b As shown: the brightness map of the reconstructed image is subjected to average pooling 2001 and nearest neighbor interpolation 2002 in sequence, and then the brightness map of the reconstructed image is subtracted to obtain a brightness residual image; the chromaticity map of the reconstructed image is subjected to nearest neighbor interpolation 2003 to obtain an upsampled chromaticity map; the brightness map of the reconstructed image, the brightness residual image and the upsampled chromaticity map are used as the final prior data.

[0103] In a feasible solution, the probability distribution of the data value of each data bit in the residual image can be directly predicted by the probability model, and then the residual image is directly input into the entropy encoder for encoding.

[0104] In another feasible solution, the probability distribution corresponding to the residual image includes the probability distribution of the residual image features of the residual image. In the above 202, "predicting the probability distribution corresponding to the residual image using the trained probability model according to the reconstructed image" may include:

[0105] According to the reconstructed image, the probability distribution of the residual image features is predicted using a trained probability model.

[0106] Accordingly, in the above 203, “performing entropy coding on the residual image according to the probability distribution corresponding to the residual image to obtain a code stream of the image at the enhancement layer” may include:

[0107] According to the probability distribution of the residual image features, entropy coding is performed on the residual image features to obtain a code stream of the image at the enhancement layer.

[0108] In this embodiment, the residual image is not directly entropy encoded, but the residual image is first feature extracted to obtain the residual image features of the residual image, and then the residual image features are entropy encoded. Since there is a large amount of redundant information in the residual image, after extracting important information through feature extraction and then performing entropy encoding, the overall compression efficiency of the residual image can be effectively improved.

[0109] Optionally, the above method may further include:

[0110] 204. Perform feature extraction on the residual image using the trained coding model to obtain features of the residual image.

[0111] The above coding model may be based on a neural network. The coding model may be trained in advance based on training samples.

[0112] In one example, the residual image is the above-mentioned chrominance residual image, which generally includes: two chrominance components. The two chrominance components can be input into the encoding model as two channel maps.

[0113] In practical applications, the data type of the residual image features output by the coding model is a floating point type, while entropy coding requires the data type of the input data to be an integer. Therefore, the residual image can be input into a trained coding model to obtain the initial residual image features output by the coding model; the initial residual image features are quantized to obtain the residual image features.

[0114] The above steps 201 to 204 are all performed on the enhancement layer, and the processing flow of the base layer will be described below. Specifically, the above method may also include:

[0115] 205. At a basic layer, perform chroma downsampling on the image to obtain a downsampled image.

[0116] 206. Compress the downsampled image to obtain a code stream of the image at a base layer.

[0117] 207. Decode the code stream of the image at the base layer to obtain the reconstructed image.

[0118] The specific implementation of the above step 205 can refer to the corresponding content in the above corresponding embodiment.

[0119] In the above 206, the compression process of the downsampled image may include: feature extraction, entropy coding, etc., which is not specifically limited in the embodiment of the present application.

[0120] In the above 207, the decoding process of the code stream of the image at the basic layer may include: entropy decoding, feature decoding, etc., which is not specifically limited in this embodiment of the present application.

[0121] Figure 3 FIG. 1 is a flow chart of a decoding method provided by an embodiment of the present application. The decoding method may be performed by a decoding end in the above encoding and decoding system. Figure 3 As shown, the method may include:

[0122] 301. Obtain a code stream of an image at a base layer and a code stream of the image at an enhancement layer.

[0123] 302. Decode the code stream of the image at the base layer to obtain a reconstructed image.

[0124] 303. According to the reconstructed image, use a trained probability model to predict a probability distribution corresponding to a residual image between the image and the reconstructed image.

[0125] 304. Perform entropy decoding on a code stream of the image at an enhancement layer according to a probability distribution corresponding to the residual image to obtain a reconstructed residual image.

[0126] 305. Correct the reconstructed image according to the reconstructed residual image to obtain a corrected reconstructed image.

[0127] In the above 302, the decoding process of the code stream of the image at the basic layer may include: entropy decoding, feature decoding, etc., which is not specifically limited in the embodiment of the present application.

[0128] In the above 303, the probability model here is the same as the probability model used by the encoding end. The specific implementation process of the above 303 can refer to the corresponding content in the above embodiments, which will not be repeated here.

[0129] The output of the probability model is: the distribution parameters of the probability distribution corresponding to the residual image.

[0130] In the above 304, the distribution parameters of the probability distribution corresponding to the residual image and the code stream of the image in the enhancement layer can be input into the entropy decoder; and the reconstructed residual image is determined according to the output result of the entropy encoder.

[0131] In one example, the probability distribution corresponding to the residual image includes: the probability distribution of the data value of each data bit in the residual image, then, the output of the entropy encoder is: the reconstructed residual image.

[0132] In another example, the probability distribution corresponding to the residual image includes: the probability distribution corresponding to the residual image includes the probability distribution of the residual image features of the residual image, that is, the probability distribution of the data value of each data bit in the residual image features, then, the entropy encoder outputs the residual image features. Then, the reconstructed residual image can be determined based on the residual image features. In an achievable scheme, the residual image features can be feature decoded using a trained decoding model to obtain a reconstructed residual image. Among them, the encoding model can be based on a neural network. Among them, the decoding model and the encoding model can be trained together to improve the training effect of both. The specific training process will be introduced in the following embodiments.

[0133] In the above 305, the reconstructed residual image may be superimposed on the reconstructed image to obtain a corrected reconstructed image.

[0134] In one example, the reconstructed residual image is a reconstructed chroma residual image, and the resolution of the reconstructed chroma residual image is greater than the resolution of the chroma diagram of the reconstructed image. Therefore, the reconstructed image can be chroma upsampled to obtain an upsampled reconstructed image; the resolution of the chroma diagram of the upsampled reconstructed image is the same as the resolution of the reconstructed chroma residual image; in the upsampled reconstructed image, the reconstructed chroma residual image is superimposed on the chroma diagram of the upsampled reconstructed image to obtain a corrected reconstructed image.

[0135] In the technical solution provided by the embodiment of the present application, during the enhancement layer encoding process, the correlation between the reconstructed image of the base layer and the residual image of the enhancement layer is used to reuse the reconstructed image of the base layer and use it as prior data to predict the probability distribution of the residual image features. In this way, there is no need to store or transmit additional prior data, which can improve the compression efficiency of the image in the enhancement layer, thereby improving the overall compression efficiency of the image.

[0136] The following will be combined Figure 4 The training method of the model involved in the embodiment of the present application is introduced. Figure 4 As shown, the method may include:

[0137] 401. Determine a sample residual image of the sample image at an enhancement layer according to the sample image and its reconstructed sample image at a base layer.

[0138] 402. Perform feature extraction on the sample residual image using a coding model to obtain sample residual image features.

[0139] 403. Perform feature decoding on the sample residual image features using a decoding model to obtain a reconstructed sample residual image.

[0140] 404. Correct the reconstructed sample image according to the reconstructed sample residual image to obtain a corrected reconstructed sample image.

[0141] 405. Optimize parameters of the encoding model and the decoding model by taking minimizing the first loss function as an optimization goal.

[0142] The first loss function is determined according to the difference between the sample image and the corrected reconstructed sample image.

[0143] In the above 401, the sample residual image between the sample image and the reconstructed sample image is determined as the sample residual image of the sample image in the enhancement layer.

[0144] The specific calculation process may refer to the calculation process of the “residual image” in the above embodiments, which will not be described in detail here.

[0145] In the above 402 and 403, the internal structure of the encoding model and the decoding model can be designed according to actual needs, and the embodiment of the present application does not make any specific limitation on this.

[0146] In the above 404, the correction process of the reconstructed sample image can refer to the correction process of the "reconstructed image" in the above embodiments, which will not be described in detail here.

[0147] In the above 405, a gradient descent algorithm may be used to optimize the parameters of the encoding model and the decoding model. The difference between the sample image and the modified reconstructed sample image may be characterized by mean square error (MSE), structural similarity (SSIM), and the like.

[0148] Optionally, the above training method may further include:

[0149] 406. Determine, according to the reconstructed sample image, a predicted probability distribution of the sample residual image features using the probability model.

[0150] 407. Optimize parameters of the probability model by minimizing the second loss function as an optimization goal.

[0151] The second loss function is determined according to the difference between the predicted probability distribution and the true probability distribution of the sample residual image feature.

[0152] The specific implementation of the above 406 can refer to the specific implementation of the above 202, which will not be described in detail here.

[0153] In the above 407, a gradient descent algorithm may be used to optimize the parameters of the probability model. The second loss function may be Shannon cross entropy.

[0154] In practical applications, the encoding model, decoding model and probability model can be jointly trained. Specifically, the total loss function can be determined according to the first loss function and the second loss function. The parameters of the encoding model, decoding model and probability model are optimized with minimizing the total loss function as the optimization goal. The total loss function is the sum of the first loss function and the second loss function.

[0155] The solution provided in the embodiment of the present application utilizes a data-driven approach to improve feature extraction of residual images, and is more targeted in processing chroma residual images of different contents.

[0156] The following will take the cloud desktop scenario as an example to introduce the embodiments of the present application in detail:

[0157] like Figure 5 As shown:

[0158] The cloud desktop server 501 sends the code stream of the cloud desktop image at the basic layer and the code stream of the cloud desktop image at the enhanced layer to the cloud desktop client 502. The cloud desktop image is a video frame in the cloud desktop video stream.

[0159] like Figure 6 As shown:

[0160] The encoding process of the cloud desktop server: the chroma residual image 1 between the cloud desktop image and the reconstructed cloud desktop image 9 is input into the trained encoding model 2 for feature extraction to obtain the initial chroma residual image features; the initial chroma residual image features are input into the quantizer for quantization processing to obtain the chroma residual image features; the chroma residual image features are input into the entropy encoder 5, and the entropy encoder 5 performs entropy encoding on the chroma residual image features according to the probability distribution N(μ, θ) of the chroma residual image features output by the probability model to obtain the code stream of the cloud desktop image in the enhancement layer. Wherein, μ is the mean of the Gaussian distribution, and θ is the variance of the Gaussian distribution.

[0161] Note: Since the dynamic range of the residual image is relatively small and mostly concentrated near the value of 0, it is not normalized to avoid the dynamic range of the latent variable features learned later (that is, the initial chrominance residual image features mentioned above) being too small.

[0162] The decoding process of the cloud desktop client: the code stream of the cloud desktop image in the enhancement layer is input into the entropy decoder 6, and the upper decoder 6 performs entropy decoding on the code stream of the cloud desktop image in the enhancement layer according to the probability distribution N(μ, θ) of the chrominance residual image features output by the probability model to obtain the chrominance residual image features; the chrominance residual image features are input into the decoding model 7 for feature decoding to obtain the reconstructed chrominance residual image.

[0163] In addition, the cloud desktop server can also perform chromaticity downsampling on the cloud desktop image at the basic layer to obtain a downsampled image; encode the downsampled image to obtain a code stream of the cloud desktop image at the basic layer; and decode the code stream of the cloud desktop image at the basic layer to obtain the above-mentioned reconstructed cloud desktop image.

[0164] The cloud desktop client also includes: decoding the code stream of the cloud desktop image at the basic layer to obtain a reconstructed cloud desktop image; and correcting the reconstructed cloud desktop image of the basic layer according to the reconstructed chroma residual image to obtain a corrected reconstructed cloud desktop image.

[0165] The probability distribution process that both the cloud desktop server and the cloud desktop client need to execute: the prior preprocessing model 10 preprocesses the reconstructed cloud desktop image to obtain prior data. The preprocessing may include: the size alignment mentioned above and the acquisition of the brightness residual image. The preprocessing process can refer to the corresponding content in the above embodiments, which will not be repeated here. The prior data is input into the prior feature extraction module 1001 of the probability model 11 for feature extraction to obtain the prior features. Among them, the prior feature extraction module also adopts the structure of a neural network, retains a relatively large receptive field, and extracts the global characteristics of the prior data. The prior data also contains high-frequency information such as edges and textures in the original image, which needs to be decomposed and extracted by the prior encoder to screen out high-frequency features close to the chroma residual to improve the accuracy of probability prediction. The prior features are input into the probability prediction module 1002 of the probability model 11 to predict the probability distribution N (μ, θ) of the chroma residual image features.

[0166] The encoding module can extract the effective features of the chroma residual image and obtain the latent variable features. The encoding module and the decoding module are responsible for the conversion between the image space and the latent variable space of the chroma residual image. The encoding module and the decoding module based on the neural network can more effectively adapt to the content characteristics of the chroma residual, and the characterization ability of the latent variable features is more powerful. The probability model takes the reconstructed image as input, extracts the global prior information of the chroma residual image, establishes an independent Gaussian distribution for each position of the latent variable feature, and uses the global prior to predict the Gaussian distribution parameters. The independent Gaussian model combined with the prior information can more effectively adapt to the residual data distribution with higher sparsity and improve the prediction accuracy of the probability model. Entropy encoder / decoder: an arithmetic encoder / decoder can be used to encode / decode the quantized latent variable features using the Gaussian probability distribution parameters provided by the probability model to realize the conversion between features and code streams. The scheme provided in the embodiment of the present application can effectively improve the compression efficiency of the chroma residual image and achieve better compression quality at the same bit rate.

[0167] Experimental comparison:

[0168] The following uses subjective and objective performance indicators, such as Peak signal-to-noise ratio (PSNR), SSIM, Learned Perceptual Image Patch Similarity (LPIPS), and DISTS (Differentiable Image Saliency Transform for Improved Scalability and Portability of Image Quality Assessment, an image quality evaluation indicator based on differentiable image saliency transform) to demonstrate the technical effects of each scheme. The chrominance residual image obtained under this scheme and several existing compression schemes is superimposed with the reconstructed image of the base layer, and the final performance results are compared. It can be seen that compared with existing compression methods, this scheme can achieve better results in all indicators.

[0169] like Figure 7 As shown in the figure: This solution performs well in terms of PSNR index. Note: The higher the PSNR index, the better.

[0170] like Figure 8 As shown in the figure, this solution performs well in terms of the SSIM index. Note: The higher the SSIM index, the better.

[0171] like Fig. 9 As shown in the figure, this solution performs well in terms of LPIPS indicator. Note: The lower the LPIPS indicator, the better.

[0172] like Fig.10 As shown in the figure, this scheme performs better in terms of the DISTS index. Note: The lower the DISTS index, the better.

[0173] It should be noted that Figure 7 , Figure 8 , Fig. 9 as well as Fig.10 The multiple unlabeled curves correspond to multiple existing compression schemes.

[0174] In summary, the neural network-based compression scheme provided by this scheme supports dual-chromaticity channel residuals as input of the coding model, and inputs dual-chromaticity information as different channel images of the coding model into the coding model for processing; and the coding model supports the input of pixels of various bit depths, such as: 16-bit depth. In view of the probability / entropy estimation problem, a probability model based on the reconstructed image to predict the probability distribution of the residual image is introduced; the distribution of the residual information is estimated by using the reconstructed data of the basic layer shared by the encoder / decoder. Compared with the traditional compression algorithm, the encoding and transmission of the super-prior information is omitted; and the probability model adopts the joint input of multi-channel information to improve the accuracy of the probability estimation; in view of the problem of inconsistent sizes of the brightness / chromaticity components of the reconstructed image, methods including nearest neighbor sampling are introduced to align the sizes before inputting them into the probability model.

[0175] Fig.11 FIG. 1 is a schematic diagram showing the structure of an electronic device provided by an embodiment of the present application. Fig.11 As shown, the electronic device includes a memory 1101 and a processor 1102. The memory 1101 can be configured to store various other data to support operations on the electronic device. Examples of these data include instructions for any application or method for operating on the electronic device. The memory 1101 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as a static random access memory (SRAM), an electrically erasable programmable read only memory (EEPROM), an erasable programmable read only memory (EPROM), a programmable read only memory (PROM), a read only memory (ROM), a magnetic memory, a flash memory, a magnetic disk or an optical disk.

[0176] The memory 1101 is used to store programs;

[0177] The processor 1102 is coupled to the memory 1101 and is used to execute the program stored in the memory 1101 to implement the methods provided by the above-mentioned method embodiments.

[0178] Further, if Fig.11 As shown, the electronic device also includes: a communication component 1103, a display 1104, a power component 1105, an audio component 1106 and other components. Fig.11 Only some components are shown schematically, which does not mean that the electronic device only includes Fig.11 Components shown.

[0179] Accordingly, an embodiment of the present application further provides a computer-readable storage medium storing a computer program, and when the computer program is executed by a computer, the steps or functions of the methods provided in the above-mentioned method embodiments can be implemented.

[0180] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.

[0181] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM (Read Only Memory) / RAM (Random Access Memory), a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0182] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit it. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A coding method, characterized in that: include: Determining a residual image of the image at an enhancement layer according to the image and its reconstructed image at a base layer; According to the reconstructed image, using a trained probability model to predict the probability distribution corresponding to the residual image; According to the probability distribution corresponding to the residual image, entropy coding is performed on the residual image to obtain a code stream of the image at the enhancement layer.

2. The method according to claim 1, characterized in that Determining a residual image of the image at an enhancement layer according to the image and its reconstructed image at a base layer includes: Calculating a chromaticity residual image between the chromaticity map of the image and the chromaticity map of the reconstructed image; The chrominance residual image is determined as the residual image of the image in the enhancement layer.

3. The method according to claim 2, characterized in that According to the reconstructed image, using a trained probability model to predict the probability distribution corresponding to the residual image, including: Upsampling the chromaticity map of the reconstructed image to obtain an upsampled chromaticity map; the resolution of the upsampled chromaticity map is the same as the resolution of the luminance map of the reconstructed image; Input data of the probability model is determined according to the upsampled chromaticity map and the luminance map of the reconstructed image.

4. The method according to claim 3, characterized in that The resolution of the brightness map of the reconstructed image is the same as the resolution of the brightness map of the image; Determining input data of the probability model according to the upsampled chromaticity map and the brightness map of the reconstructed image includes: determining a downsampling algorithm according to which a chroma downsampling operation performed on the image at the base layer is performed; According to the downsampling strategy, the brightness map of the reconstructed image is downsampled to obtain a downsampled brightness map; Determine a brightness residual map between the brightness map of the reconstructed image and the brightness map after downsampling; the resolution of the brightness residual map is the same as the resolution of the brightness map of the reconstructed image; The up-sampled chromaticity map, the brightness map of the reconstructed image, and the brightness residual map are respectively used as different channel maps in the input data of the probability model.

5. The method according to any one of claims 1 to 4, characterized in that The probability distribution corresponding to the residual image includes the probability distribution of the residual image features of the residual image; According to the reconstructed image, using a trained probability model to predict the probability distribution corresponding to the residual image, including: According to the reconstructed image, predicting the probability distribution of the residual image features using a trained probability model; According to the probability distribution corresponding to the residual image, entropy coding is performed on the residual image to obtain a code stream of the image at the enhancement layer, including: According to the probability distribution of the residual image features, entropy coding is performed on the residual image features to obtain a code stream of the image at the enhancement layer.

6. The method according to claim 5, characterized in that Also includes: The trained coding model is used to perform feature extraction on the residual image to obtain the residual image features.

7. The method according to claim 6, characterized in that Extracting features from the residual image using the trained coding model to obtain features of the residual image includes: Inputting the residual image into a trained coding model to obtain initial residual image features output by the coding model; The initial residual image features are quantized to obtain residual image features.

8. The method according to any one of claims 1 to 4, characterized in that The image is a video frame in the cloud desktop video stream.

9. The method according to any one of claims 1 to 4, characterized in that Also includes: At the base layer, chroma downsampling is performed on the image to obtain a downsampled image; Encoding the downsampled image to obtain a code stream of the image at a base layer; The code stream of the image at the base layer is decoded to obtain the reconstructed image.

10. A decoding method, characterized in that: include: Acquire a code stream of an image at a base layer and a code stream of the image at an enhancement layer; Decoding a code stream of the image at a base layer to obtain a reconstructed image; According to the reconstructed image, predicting the probability distribution corresponding to the residual image between the image and the reconstructed image using a trained probability model; According to the probability distribution corresponding to the residual image, entropy decoding is performed on the code stream of the image at the enhancement layer to obtain a reconstructed residual image; The reconstructed image is corrected according to the reconstructed residual image to obtain a corrected reconstructed image.

11. A coding and decoding system, characterized in that: include: Encoding end and decoding end; The encoding end is used to: determine a residual image of the image at the enhancement layer according to the image and its reconstructed image at the base layer; According to the reconstructed image, predict the probability distribution corresponding to the residual image by using a trained probability model; according to the probability distribution corresponding to the residual image, entropy encode the residual image to obtain a code stream of the image at the enhancement layer; send the code stream of the image at the base layer and the code stream of the image at the enhancement layer to the decoding end device; The decoding end is used to: receive a code stream of the image at a basic layer and a code stream of the image at an enhanced layer; decode the code stream of the image at the basic layer to obtain the reconstructed image; predict the probability distribution corresponding to the residual image between the image and the reconstructed image using the probability model according to the reconstructed image; perform entropy decoding on the code stream of the image at the enhanced layer according to the probability distribution corresponding to the residual image to obtain a reconstructed residual image; and correct the reconstructed image according to the reconstructed residual image to obtain a corrected reconstructed image.

12. An electronic device, characterized in that: include: A memory and a processor, wherein The memory is used to store programs; The processor is coupled to the memory, and is configured to execute the program stored in the memory to implement the method according to any one of claims 1 to 10.

13. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a computer, the method according to any one of claims 1 to 10 can be implemented.

Images