Image encoding and decoding method and apparatus

Abstract

This application relates to the field of image compression technology, and particularly to an image encoding and decoding method and apparatus. The image decoding method includes: acquiring the side information bitstream of the image to be decoded; acquiring image side information features based on the side information bitstream; and acquiring a preview image based on the image side information features. The image side information features are a second image feature set obtained by feature extraction from a first image feature set corresponding to the image to be decoded based on a super-prior coding neural network. Based on the above decoding scheme, the decoding end can quickly acquire the preview image based on the side information bitstream, effectively avoiding the situation in the prior art where the preview image cannot be acquired in a timely manner when the user browses the image.

Description

Technical Field

[0001] This application relates to the field of image compression technology, and in particular to an image encoding and decoding method and an encoding and decoding device. Background Technology

[0002] Image compression technology refers to a technique that utilizes image data characteristics such as spatial redundancy, visual redundancy, and statistical redundancy to represent image information with as few bits as possible, with or without loss. Image compression technology enables the efficient transmission and storage of image information. Currently, the most commonly used image compression technology is image coding technology, which specifically includes... Figure 1 As shown, at the encoding end, the image is encoded into a corresponding bitstream for storage or transmission to the decoding end. When the image needs to be displayed, the decoding end receives the bitstream, decodes it to obtain the original image, and then scales the original image to obtain a lower-resolution thumbnail for easy previewing. Figure 2 As shown, when browsing images on electronic device 001 via a web application, or in applications such as video players and photo albums, the decoding end, such as the web application, fully decodes the stored or received bitstream to obtain the original resolution image. Then, through image scaling, it converts the original resolution image into a lower resolution preview image for easier user viewing. After the user selects the image they want to view, they can view the reconstructed image corresponding to the original image by clicking on the thumbnail or other operations.

[0003] During the decompression process described above, regardless of the thumbnail size required by the application, the original resolution image must first be decoded. This approach places a significant demand on the computing power of electronic devices. If the device's computing power is insufficient, it may fail to refresh the thumbnail interface in a timely manner, causing noticeable lag for the user. For example, when a user scrolls through images in a web application, the thumbnail interface may not refresh promptly, resulting in a noticeable lag.

[0004] Some existing encoding schemes employ image encoding and decoding schemes based on recurrent neural networks, specifically as follows: Figure 3 As shown, after each iteration, the encoding end will generate an iterative bitstream and reconstruction error data for the next iteration, for example... Figure 3 The bitstreams obtained through three iterations shown are iteration bitstream 1, iteration bitstream 2, and iteration bitstream 3. After receiving these bitstreams, the decoding end decodes each bitstream sequentially to extract the basic reconstructed image and residual image for use in the next image reconstruction, thus gradually improving the quality of the reconstructed image. However, the computational complexity of the above scheme with multiple iterations is high, and since each decoding is performed on a per-bitstream basis, its scalability is poor.

[0005] Some encoding schemes employ additional encoding for transmitting thumbnails, such as... Figure 4 As shown, to obtain additional thumbnail bitstreams at the encoding end using compression and other methods, the thumbnail bitstreams and the full image bitstreams corresponding to the regular images are merged for transmission. At the decoding end, the thumbnail bitstreams are decoded first for previewing. However, this approach increases the overall bitstream file size, reduces the compression ratio, and only yields thumbnails of the same quality as those compressed at the encoding end during decoding. Summary of the Invention

[0006] To address the problems of high computational complexity and poor decoding scalability in existing encoding schemes, which can only obtain decoded images of a set quality, embodiments of this application provide an image encoding and decoding method and an encoding and decoding apparatus.

[0007] In a first aspect, embodiments of this application provide an image decoding method, comprising: acquiring a side information bitstream of an image to be decoded; acquiring image side information features based on the side information bitstream, wherein the image side information features are a second image feature set obtained by feature extraction of a first image feature set corresponding to the image to be decoded based on a super-prior coding neural network, wherein the data volume of the second image feature set is less than the data volume of the first image feature set, and the super-prior coding neural network is obtained by training constraints based on bit rate loss, reconstructed image distortion loss and preview image distortion loss, wherein the preview image distortion loss represents the difference in similarity between the original training image and the preview image of the original training image; and acquiring a preview image of the image to be decoded based on the image side information features.

[0008] Based on the above decoding scheme, the decoding end can quickly obtain the preview image based on the side information bitstream with a small data volume. In this way, it is not necessary to obtain the original image during decoding and then perform scaling to obtain a lower resolution thumbnail, which reduces the computing power requirements of electronic devices. It can also effectively avoid the situation in the prior art where users cannot obtain the preview image in time because the prior art needs to decode to obtain the original resolution image and then perform scaling to obtain a lower resolution thumbnail, which takes a long time.

[0009] Furthermore, based on the above decoding method, the corresponding encoding method only needs to obtain the side information bitstream and the image bitstream. Compared with the iterative image encoding scheme based on recurrent neural networks in the prior art, it can effectively reduce the computational complexity. Specifically, as in the background art, the existing iterative image encoding and decoding scheme based on recurrent neural networks requires multiple iterations of encoding to obtain multiple encoding end bitstreams in the encoding scheme, so as to reconstruct multiple preview images from multiple bitstreams during decoding to achieve progressive decoding, which has high computational complexity. In contrast, the encoding scheme in this application only needs to perform one-time row encoding on the image bitstream and the side information bitstream to obtain the corresponding encoding end bitstream. Moreover, during decoding, multiple preview images can be gradually reconstructed based on the side information features obtained from decoding the side information bitstream and the gradually increasing image features obtained from gradually decoding the image bitstream, thus achieving progressive decoding. In summary, the decoding scheme provided by the embodiments of this application can effectively reduce the computational complexity of the entire encoding and decoding process.

[0010] Furthermore, based on the aforementioned decoding method, the corresponding encoding method only needs to acquire the side information bitstream and the image bitstream. Compared to the existing technology's scheme of additionally encoding and transmitting thumbnails, this reduces the overall bitstream file size and improves the compression ratio. Specifically, in the existing technology, the bitstream corresponding to the additionally encoded and transmitted thumbnails is an extra bitstream, while in the embodiments of this application, the side information bitstream and image bitstream are both necessary for acquiring the final reconstructed image. Therefore, no extra bitstream is added, which reduces the overall bitstream file size and improves the compression ratio.

[0011] It is understood that the first image feature set mentioned in the embodiments of this application can be the image features obtained by the feature extraction module in the following text.

[0012] In one possible implementation, obtaining a preview image of the image to be decoded based on image side information features includes: obtaining a first preview image of the image to be decoded based on image side information features.

[0013] In one possible implementation, the decoding method further includes: acquiring an image bitstream of the image to be decoded, the image bitstream including data from multiple channels of the image to be decoded; acquiring first decoding data based on the image bitstream, the first decoding data being at least a portion of image features of a first image feature set; correspondingly, acquiring a preview image of the image to be decoded based on image side information features, including: acquiring a second preview image based on the image side information features and the first decoding data, the second preview image having a greater similarity to the image to be decoded than the first preview image having a greater similarity to the image to be decoded.

[0014] It is understood that the image bitstream in this application embodiment includes encoded data corresponding to all image features of the image to be decoded. In this application embodiment, the first decoding data may be decoded data obtained by decoding the data of some channels in the image bitstream that have high output response importance.

[0015] It is understandable that reconstructing high-quality preview images based on edge information features and the first decoded data facilitates user previewing. Secondly, the aforementioned decoding scheme can combine edge information features with image features of different channel numbers to acquire preview images of varying qualities, meeting diverse preview image quality requirements. Furthermore, after decoding some image features, the current preview image can be obtained based on the decoded image features and edge information features. As more image features are decoded, the next preview image can be obtained based on the decoded image features and edge information features to refresh the current preview image, allowing users to gradually see higher-quality preview images, providing a progressive image preview effect and enhancing the user experience.

[0016] In one possible implementation, the first decoded data is obtained based on the image bitstream, including:

[0017] Determine the data of at least one channel in the image bitstream; decode the data of at least one channel to obtain the first decoded data.

[0018] In one possible implementation, determining data for at least one channel of the image bitstream includes:

[0019] According to the data volume of each channel in the image bitstream, obtain the data of at least one channel.

[0020] Obtaining data for at least one channel based on the amount of data in each channel of the image bitstream can include: sorting the channels in descending order of the amount of data in each channel of the image bitstream and obtaining the data for the first predetermined number of channels in the sort; or sorting the channels in ascending order of the amount of data in each channel of the image bitstream and obtaining the data for the last predetermined number of channels in the sort.

[0021] It is understood that in this embodiment, the importance of the channel output response can be determined by the amount of information contained in each channel. When a channel contains a large amount of information, the importance of its output response is high; when a channel contains a small amount of information, the importance of its output response is low. The amount of information contained in each channel can be determined by the amount of data in the corresponding bitstream.

[0022] It is understood that the decoding time for each channel is the same during decoding. Therefore, the channel with higher output response importance is selected for decoding to obtain partial image features. Decoding only part of the channel data not only greatly saves decoding time, but also ensures the quality of the preview image as much as possible because the channel with higher output response importance is selected. Therefore, the embodiment of this application reduces the encoding time while ensuring the quality of the preview image.

[0023] It is understandable that the above decoding scheme can combine side information features with image features of different number of channels to acquire preview images of various qualities, thus meeting the requirements of various preview image quality.

[0024] In one possible implementation, the decoding method further includes: acquiring an image bitstream of the image to be decoded, the image bitstream including data from multiple channels of the image to be decoded; correspondingly, acquiring a preview image of the image to be decoded based on image side information features; including: decoding a first predetermined number of channel data in the image bitstream to obtain second decoded data, and acquiring a third preview image based on image side information features and the second decoded data;

[0025] A third decoded data is obtained by decoding a second set number of channel data in the image bitstream. A fourth preview image is obtained based on the image edge information features and the third decoded data. The second set number is greater than the first set number, and the second set number of channel data includes the first set number of channel data.

[0026] It is understood that in this embodiment, after acquiring the first preview image, the electronic device will display the first acquired preview image. After each subsequent acquisition of a new preview image, the new preview image will replace the previous preview image. For example, when acquiring the third preview image, the electronic device will display the third preview image. After acquiring the fourth preview image, the fourth preview image will replace (or overwrite) the third preview image, that is, the electronic device will display the fourth preview image.

[0027] Based on the above scheme, preview images can be quickly obtained according to the decoded image features and edge information features, and the currently displayed preview image can be continuously refreshed, so that users can gradually see higher quality preview images, with a progressive image preview effect, thus improving the user experience.

[0028] In one possible implementation, each channel can be decoded sequentially according to its importance from high to low in the output response of each channel data in the image features, so as to sequentially acquire a first set number of channel data and a second set number of channel data, etc.

[0029] In one possible implementation, the method further includes: in response to a user clicking on a preview image, obtaining the reconstructed image corresponding to the image to be decoded based on the image bitstream.

[0030] It's understandable that the reconstructed image can be a larger version of the preview image.

[0031] In one possible implementation, a preview image is obtained based on image side information features; this includes: obtaining the mean of each data point in the image features based on the image side information features, and obtaining the preview image based on the mean of each data point in the image features.

[0032] In one possible implementation, at least some image features are obtained based on the image bitstream; this includes determining the probability corresponding to each data in the image features based on image side information features; and obtaining at least some image features based on the image bitstream and the probability corresponding to each data in the image features.

[0033] Specifically, determining the probability corresponding to each data point in the image features based on image side information features can include: determining the mean and variance corresponding to each data point in the image features based on image side information features, and determining the probability corresponding to each data point in the image features based on the mean and variance corresponding to each data point in the image features.

[0034] Secondly, embodiments of this application provide an image encoding method, comprising: acquiring an image to be encoded; acquiring a first image feature set of the image to be encoded; acquiring side information features of the image to be encoded based on the first image feature set; the side information features are a second image feature set obtained by feature extraction of the first image feature set based on a super-prior coding neural network, wherein the amount of data in the second image feature set is less than the amount of data in the first image feature set, and the super-prior coding neural network is obtained by training constraints based on bit rate loss, reconstruction image distortion loss, and preview image distortion loss, wherein the preview image distortion loss represents the difference in similarity between the original training image and the preview image of the original training image; acquiring an image bitstream based on the side information features of the image to be encoded and the first image feature set, and acquiring a side information bitstream based on the side information features of the image to be encoded, wherein the side information bitstream is used by the decoding end to acquire the preview image of the image to be encoded.

[0035] It is understandable that, since the amount of side information bitstream data is small, the side information features are encoded to obtain the side information bitstream, which facilitates the rapid acquisition of a lower resolution preview image based on the side information bitstream during decoding. In this way, it is not necessary to obtain the original image during decoding and then obtain a lower resolution thumbnail through scaling. It should be understood that a thumbnail can also be called a preview image. The solution provided by the embodiments of this application can effectively avoid the situation in the prior art where users cannot obtain a preview image in a timely manner when browsing images.

[0036] In one possible implementation, the image bitstream is obtained based on the side information features and image features of the image to be encoded; this includes: obtaining the distribution information of each data point in a first image feature set based on the side information features of the image to be encoded; and encoding the image features based on the distribution information of each data point in the first image feature set to obtain the image bitstream. It can be understood that the distribution information of each data point in the image features can refer to the mean and variance of each data point in each channel of the image features.

[0037] In one possible implementation, obtaining the side information bitstream based on the side information features of the image to be encoded includes: estimating the probability of each data in the side information features according to preset distribution information, and obtaining the probability of each data in the side information features; encoding the side information features based on the probability of each data in the side information features, and obtaining the side information bitstream.

[0038] Thirdly, embodiments of this application provide an image decoding method, comprising: acquiring a side information bitstream of an image to be decoded; acquiring image side information features based on the side information bitstream, wherein the image side information features are a second image feature set obtained by feature extraction of a first image feature set corresponding to the image to be decoded based on a super-prior coding neural network, wherein the data volume of the second image feature set is less than the data volume of the first image feature set, and the super-prior coding neural network is obtained by training constraints based on bit rate loss, reconstructed image distortion loss, and preview image distortion loss, wherein the preview image distortion loss represents the difference in similarity between the original training image and the preview image of the original training image; acquiring an image bitstream of the image to be decoded, wherein the image bitstream includes data from multiple channels of the image to be decoded; decoding a first predetermined number of channel data in the image bitstream to obtain first decoded data; acquiring a first reconstructed image based on the image side information features and the first decoded data; decoding a second predetermined number of channel data in the image bitstream to obtain second decoded data; and acquiring a second reconstructed image based on the image side information features and the second decoded data.

[0039] It is understood that in this embodiment, after the first reconstructed image is acquired, the electronic device will display the first acquired reconstructed image. After each subsequent acquisition of a new reconstructed image, the new reconstructed image will replace the previous reconstructed image for display. For example, after acquiring the first reconstructed image, the first reconstructed image will be displayed. After acquiring the second reconstructed image, the second reconstructed image will replace the first reconstructed image, that is, the electronic device will display the second reconstructed image.

[0040] Based on the above method, electronic devices can gradually display reconstructed images of different qualities, achieving a progressive decoding effect. This prevents stuttering when users view the decoded image corresponding to the original image, thus improving the user experience.

[0041] In one possible implementation, the decoding method further includes decoding all channel data of the image bitstream to obtain third decoded data, and obtaining a third reconstructed image based on side information features and the third decoded data.

[0042] It is understandable that the third reconstructed image can be the final reconstructed image, and after acquiring the final reconstructed image, the electronic device can display the final reconstructed image.

[0043] In one possible implementation, the decoding method includes: after acquiring a first reconstructed image, sending the first reconstructed image to a display for display; and after acquiring a second reconstructed image, sending the second reconstructed image to a display for display to cover the first reconstructed image.

[0044] Fourthly, embodiments of this application provide an image decoding apparatus, comprising: an acquisition module for acquiring a side information bitstream of an image to be decoded; an acquisition module for acquiring image side information features based on the side information bitstream, wherein the image side information features are a second image feature set obtained by feature extraction of a first image feature set corresponding to the image to be decoded based on a super-prior coding neural network, wherein the data volume of the second image feature set is less than the data volume of the first image feature set, and the super-prior coding neural network is trained under constraints based on bitrate loss, reconstructed image distortion loss, and preview image distortion loss; and a pre-decoding module for acquiring a preview image of the image to be decoded based on the image side information features.

[0045] In one possible implementation, the acquisition module is further configured to acquire an image bitstream of the image to be decoded, the image bitstream including data from multiple channels of the image to be decoded; and acquire first decoded data based on the image bitstream, the first decoded data being at least a portion of image features of a first image feature set.

[0046] In one possible implementation, the acquisition module includes a priori decoding module and an entropy decoding module; the priori decoding module is used to determine the mean and variance of each data in the first image feature set based on the image side information features; the entropy decoding module is used to determine the probability corresponding to each data in the image features based on the mean and variance of each data in the first image feature set; and the entropy decoding module is used to acquire the first decoded data based on the image bitstream and the probability corresponding to each data in the first image feature set.

[0047] In one possible implementation, the acquisition module further includes a priori entropy estimation module; and the priori entropy estimation module is used to obtain the probability corresponding to each data in the image side information features based on the side information bitstream; the entropy decoding module is used to decode the side information bitstream according to the probability corresponding to each data in the image side information features to obtain the image side information features.

[0048] In one possible implementation, the pre-decoding module is used to obtain a first preview image based on image side information features; or, it is used to obtain a second preview image based on image side information features and first decoding data; the similarity between the second preview image and the image to be decoded is greater than the similarity between the first preview image and the image to be decoded.

[0049] In one possible implementation, the pre-decoding module is a pre-decoding network, which includes an image scaling operator;

[0050] Alternatively, a pre-decoding network can be used to obtain a preview image of a set size.

[0051] Fifthly, embodiments of this application provide an image encoding apparatus, including a feature extraction module for acquiring an image to be encoded and acquiring a first image feature set of the image to be encoded; and a super-prior encoding module for acquiring side information features of the image to be encoded based on the first image feature set; the side information features are a second image feature set obtained by feature extraction from the first image feature set by the super-prior encoding module, the data size of the second image feature set is smaller than the data size of the first image feature set, and the super-prior encoding module is obtained by training constraints based on bit rate loss, reconstructed image distortion loss and preview image distortion loss, wherein the preview image distortion loss represents the difference in similarity between the original training image and the preview image of the original training image;

[0052] The feature encoding module obtains the image bitstream based on the side information features of the image to be encoded and the first image feature set, and obtains the side information bitstream based on the side information features of the image to be encoded.

[0053] In one possible implementation, the feature encoding module includes a priori decoding module and an entropy encoding module; the priori decoding module is used to obtain the mean and variance of each data in the first image feature set based on the side information features of the image to be encoded; the entropy encoding module is used to entropy encode the first image feature set based on the mean and variance of each data in the image features to obtain the image bitstream.

[0054] In one possible implementation, the feature encoding module includes a prior entropy estimation module; the prior entropy estimation module is used to estimate the probability of each data in the side information feature based on preset distribution information, and obtain the probability of each data in the side information feature; the entropy encoding module is used to encode the side information feature based on the probability of each data in the side information feature, and obtain the side information code stream.

[0055] In one possible implementation, the super-prior coding module includes a super-prior coding neural network; during training, the super-prior coding neural network is trained based on bit rate loss, reconstructed image distortion loss, and preview image distortion loss.

[0056] It is understandable that in some embodiments, in order to achieve progressive preview images, the aforementioned preview image distortion loss can be a variable preview image loss, taking into account the distortion loss in all progressive states. For example, with N progressive states, the distortion loss can have N forms. After the super-prior coding neural network is trained, the quality of the preview images in the N progressive states, that is, the quality of the preview images obtained by sequential decoding, can be guaranteed. However, the distortion loss in the prior art is a non-variable distortion loss, which can only be used to guarantee the final decoding quality.

[0057] In some embodiments, to achieve progressive image reconstruction, the aforementioned image distortion loss can be a variable image loss, taking into account the distortion loss in all progressive states. For example, with N progressive states, the distortion loss can have N forms. After the super-prior coding neural network is trained, the quality of the preview images in the N progressive states, i.e., the quality of the images obtained by sequential decoding, can be guaranteed.

[0058] In other embodiments, to achieve progressive image reconstruction, the super-prior coding neural network for acquiring side information features can also be trained under constraints based on two losses during training: one is the bitrate loss, and the other is the image distortion loss. The image distortion loss is a variable distortion loss.

[0059] In a sixth aspect, embodiments of this application provide an image decoding apparatus, comprising: an acquisition module for acquiring a side information bitstream of an image to be decoded; an acquisition module for acquiring image side information features based on the side information bitstream, wherein the image side information features are a second image feature set obtained by feature extraction from a first image feature set corresponding to the image to be decoded based on a super-prior coding neural network, the data size of the second image feature set being smaller than the data size of the first image feature set, and the super-prior coding neural network being obtained by training constraints based on bitrate loss, reconstructed image distortion loss, and preview image distortion loss, wherein the preview image distortion loss represents the difference in similarity between the original training image and the preview image of the original training image; an acquisition module for acquiring an image bitstream of the image to be decoded, the image bitstream including data from multiple channels of the image to be decoded; an acquisition module for decoding a first predetermined number of channel data in the image bitstream to obtain first decoded data; a decoding module for acquiring a first reconstructed image based on the image side information features and the first decoded data; and a decoding module for acquiring a second reconstructed image based on the image side information features and the second decoded data.

[0060] In one possible implementation, the decoding module is used to obtain a third reconstructed image based on side information features and the third decoded data after decoding all channels of the image bitstream to obtain the third decoded data. The decoding module is also used to send the first reconstructed image to a display for display after obtaining the first reconstructed image.

[0061] In one possible implementation, a decoding module is used to send the second reconstructed image to a display for display after acquiring the second reconstructed image, thereby overwriting the first reconstructed image.

[0062] In a seventh aspect, embodiments of this application provide an electronic device, including the image decoding device and / or image encoding device mentioned in embodiments of this application.

[0063] Eighthly, embodiments of this application provide a decoding device, which is an encoding device or a decoding device. The decoding device includes one or more processors and a memory. The memory is used to store program instructions. When the program instructions are executed by one or more processors, the image decoding method or image encoding method mentioned in the embodiments of this application is implemented.

[0064] Ninthly, embodiments of this application provide a readable storage medium storing instructions that, when executed on an electronic device, cause the electronic device to perform the image decoding method or image encoding method mentioned in the embodiments of this application.

[0065] In a tenth aspect, embodiments of this application provide a computer program product, including instructions that, when executed on an electronic device, cause the electronic device to perform the image decoding method or image encoding method mentioned in embodiments of this application.

[0066] Eleventhly, embodiments of this application provide a computer-readable storage medium storing an image bitstream and a side information bitstream obtained by an encoding method as mentioned in the embodiments of this application, executed by one or more processors.

[0067] In a twelfth aspect, embodiments of this application provide a decoding apparatus, including a memory and a decoder, wherein the memory is used to store an image bitstream and a side information bitstream; and the decoder is used to execute an image decoding method.

[0068] In a thirteenth aspect, embodiments of this application provide an artificial intelligence neural network encoding / decoding architecture, comprising: an encoding / decoding network and a super-prior encoding / decoding network, wherein the encoding / decoding network includes an encoding neural network, a first entropy encoder, a first entropy decoder, and a decoding neural network; the super-prior encoding / decoding network includes a super-prior encoding neural network, a second entropy encoder, a second entropy decoder, and a pre-decoding neural network; the encoding neural network processes the image to be encoded to obtain a first image feature set of the image to be encoded; the super-prior encoding neural network extracts features from the first image feature set to obtain side information features; the first entropy encoder entropy-encodes the first image feature set to obtain an image bitstream; the second entropy encoder entropy-encodes the side information features to obtain a side information bitstream; the second entropy decoder entropy-decodes the side information bitstream to obtain decoded side information features; the pre-decoding neural network processes the decoded side information features to obtain a preview image of the image to be encoded; the first entropy decoder entropy-decodes the image bitstream to obtain an entropy decoding result corresponding to the image bitstream; and the decoding neural network processes the entropy decoding result corresponding to the image bitstream to obtain a reconstructed image of the image to be encoded.

[0069] It should be understood that the entropy decoding result corresponding to the edge information feature can also be called the decoded edge information feature; the entropy decoding result corresponding to the image bitstream can also be called the decoded image feature.

[0070] In one possible implementation, the super-prior encoder-decoder network further includes a super-prior decoder neural network. The super-prior decoder neural network processes the entropy decoding results corresponding to the edge information features to obtain the distribution information of the first image feature set. This distribution information is used by the first entropy encoder to entropy encode the first image feature set and to entropy decode the image bitstream. The output result or a part of the output result of the super-prior decoder neural network is processed by the pre-decoding neural network to obtain a preview image of the image to be encoded.

[0071] In a fourteenth aspect, embodiments of this application provide an artificial intelligence neural network decoding architecture, comprising: an encoding / decoding network and a super-prior encoding / decoding network, wherein the encoding / decoding network includes a first entropy decoder and a decoding neural network; the super-prior encoding / decoding network includes a second entropy decoder and a pre-decoding neural network; the second entropy decoder acquires the side information bitstream of the image to be decoded, and performs entropy decoding on the side information bitstream to obtain the entropy decoding result corresponding to the side information features; the pre-decoding neural network processes the entropy decoding result corresponding to the side information features to obtain a first preview image of the image to be decoded; the first entropy decoder acquires the image bitstream of the image to be decoded, and performs entropy decoding on the image bitstream to obtain the entropy decoding result corresponding to the image bitstream; the decoding neural network processes the entropy decoding result corresponding to the image bitstream to obtain a reconstructed image of the image to be decoded.

[0072] In a fifteenth aspect, embodiments of this application provide an artificial intelligence neural network coding architecture, including: a coding-decoding network and a super-prior coding-decoding network, wherein the coding-decoding network includes a coding neural network and a first entropy encoder; the super-prior coding-decoding network includes a super-prior coding neural network and a second entropy encoder. The coding neural network processes the image to be encoded to obtain a first image feature set of the image to be encoded; the super-prior coding neural network extracts features from the first image feature set to obtain side information features; the first entropy encoder entropy-encodes the first image feature set to obtain an image bitstream; and the second entropy encoder entropy-encodes the side information features to obtain a side information bitstream.

[0073] In a sixteenth aspect, embodiments of this application provide an artificial intelligence neural network decoding architecture, including: an encoding / decoding network and a super-prior encoding / decoding network, wherein the encoding / decoding network includes a first entropy decoder and a decoding neural network; the super-prior encoding / decoding network includes a second entropy decoder and a pre-decoding neural network; the side information bitstream of the image to be decoded is obtained through the second entropy decoder, and the side information bitstream is entropy decoded to obtain the entropy decoding result corresponding to the side information features;

[0074] The first entropy decoder obtains the image bitstream of the image to be decoded, and decodes a first set number of channel data in the image bitstream to obtain the first decoded data; the entropy decoding result corresponding to the edge information features and the first decoded data are processed by the pre-decoding neural network to obtain the second preview image of the image to be decoded.

[0075] In a seventeenth aspect, embodiments of this application provide an artificial intelligence neural network decoding architecture, including: an encoding / decoding network and a super-prior encoding / decoding network, wherein the encoding / decoding network includes a first entropy decoder and a decoding neural network; the super-prior encoding / decoding network includes a second entropy decoder and a pre-decoding neural network; the side information bitstream of the image to be decoded is obtained through the second entropy decoder, and the side information bitstream is entropy decoded to obtain the entropy decoding result corresponding to the side information features;

[0076] The image bitstream of the image to be decoded is obtained through a first entropy decoder. A second set number of channel data in the image bitstream are decoded to obtain second decoded data. A pre-decoding neural network processes the entropy decoding result corresponding to the edge information features and the second decoded data to obtain a third preview image of the image to be decoded. The first entropy decoder then decodes a third set number of channel data in the image bitstream to obtain third decoded data. A pre-decoding neural network then processes the entropy decoding result corresponding to the edge information features and the third decoded data to obtain a third preview image of the image to be decoded.

[0077] In an eighteenth aspect, embodiments of this application provide an artificial intelligence neural network decoding architecture, comprising: an encoding / decoding network and a priori encoding / decoding network, wherein the encoding / decoding network includes a first entropy decoder and a decoding neural network; the priori encoding / decoding network includes a second entropy decoder and a pre-decoding neural network; the second entropy decoder acquires the side information bitstream of the image to be decoded, and performs entropy decoding on the side information bitstream to obtain the entropy decoding result corresponding to the side information features; the pre-decoding neural network processes the entropy decoding result corresponding to the side information features to obtain a preview image of the image to be decoded; the first entropy decoder acquires the image bitstream of the image to be decoded, and decodes a first predetermined number of channel data in the image bitstream to obtain first decoded data; the decoding neural network processes the entropy decoding result corresponding to the side information features and the first decoded data to obtain a first reconstructed image of the image to be decoded; the first entropy decoder decodes the first predetermined number of channel data in the image bitstream to obtain second decoded data; the decoding neural network processes the entropy decoding result corresponding to the side information features and the second decoded data to obtain a second reconstructed image of the image to be decoded. Attached Figure Description

[0078] Figure 1 According to some embodiments of this application, a schematic diagram of an encoding and decoding method is shown;

[0079] Figure 2 According to some embodiments of this application, a schematic diagram of an application scenario for an encoding and decoding method is shown;

[0080] Figure 3 According to some embodiments of this application, a schematic diagram of an encoding and decoding method is shown;

[0081] Figure 4 According to some embodiments of this application, a schematic diagram of an encoding and decoding method is shown;

[0082] Figures 5a-5c According to some embodiments of this application, schematic diagrams of encoding and decoding methods are shown respectively;

[0083] Figure 6 A block diagram of an electronic device is shown according to some embodiments of this application;

[0084] Figure 7 According to some embodiments of this application, a flowchart of an encoding and decoding scheme is shown;

[0085] Figure 8 According to some embodiments of this application, a flowchart of an encoding and decoding scheme is shown;

[0086] Figure 9According to some embodiments of this application, a flowchart of an encoding and decoding scheme is shown;

[0087] Figure 10 According to some embodiments of this application, a flowchart of an encoding and decoding scheme in a photo album scenario is shown;

[0088] Figure 11 According to some embodiments of this application, a flowchart of an encoding and decoding scheme in an edge-cloud collaborative scenario is shown;

[0089] Figure 12 According to some embodiments of this application, a flowchart of an encoding method is shown;

[0090] Figure 13 According to some embodiments of this application, a flowchart of a decoding method is shown;

[0091] Figure 14a According to some embodiments of this application, a method demonstrating the use of Figure 13 The RD curves are obtained by decoding 24 PNG images with a resolution of 768x512 or 512x768 from the Kodak test set.

[0092] Figure 14b According to some embodiments of this application, a method is shown that... Figure 13 The diagram shows a list of thumbnails obtained by decoding 24 PNG images with a resolution of 768x512 or 512x768 from the Kodak test set.

[0093] Figure 15 According to some embodiments of this application, a flowchart of a decoding method is shown;

[0094] Figure 16a According to some embodiments of this application, a schematic diagram of the relationship between the number of channels used and the quality of the preview image for image features measured on the Kodak test set is shown.

[0095] Figure 16b According to some embodiments of this application, a schematic diagram of the relationship between the number of channels used and the time consumed for image features measured on the Kodak test set is shown;

[0096] Figure 17 According to some embodiments of this application, a flowchart of a decoding method is shown;

[0097] Figure 18 According to some embodiments of this application, a flowchart of a decoding method is shown;

[0098] Figure 19According to some embodiments of this application, a method of using Figure 18 The diagram illustrates the progressively reconstructed image obtained by decoding a 768x512 PNG image on the Kodak test set.

[0099] Figure 20 According to some embodiments of this application, a schematic diagram of the hardware structure of an electronic device is shown;

[0100] Figure 21 A schematic block diagram of an encoding device is shown according to some embodiments of this application;

[0101] Figure 22 According to some embodiments of this application, a schematic diagram of a cloud application device system architecture is shown. Detailed Implementation

[0102] The illustrative embodiments of this application include, but are not limited to, an image encoding and decoding method and an encoding and decoding apparatus.

[0103] To better understand the solutions in the embodiments of this application, some terms involved in the embodiments of this application will be explained below.

[0104] Image features: These can be feature maps, which may include multi-dimensional data from the outputs of convolutional layers, activation layers, pooling layers, batch normalization layers, etc., in a convolutional neural network. Generally, they contain at least three dimensions: width, height, and channels. That is, image features can include multiple channels, each containing width * height (H * W) data points. The data in each channel can represent a portion of the corresponding features of the image.

[0105] Side information features: Side information features are features further extracted from image features. They are a type of feature map that contains fewer feature elements than the image features themselves. In some embodiments, side information features are partial or approximate information of the image features obtained by transforming the image features through a correspondence function. For example, side information features can contain three-dimensional data with width, height, and channels; they are typically used to assist entropy coding or entropy decoding, i.e., to assist in image reconstruction.

[0106] Entropy coding: This refers to coding that does not lose any information during the encoding process according to the entropy principle. Common entropy coding methods include Huffman coding and arithmetic coding.

[0107] Entropy decoding: a technique for restoring the original image from image data that has undergone entropy encoding. Common entropy decoding methods include Huffman coding and arithmetic coding.

[0108] Image bitstream: A bitstream obtained by entropy coding of image features or quantized image features. It includes coded data corresponding to multiple channels of data in the image features.

[0109] Bitrate: In image compression tasks, it refers to the average coding length required to encode a unit of pixel. Generally, the higher the bitrate, the better the image reconstruction quality.

[0110] As described in the background section, some existing decoding schemes require decoding the original resolution image first, regardless of the thumbnail size needed by the application. This approach places a high demand on the computing power of electronic devices. If the computing power of the electronic device is insufficient, it may result in the inability to refresh the thumbnail interface in a timely manner, causing noticeable lag for the user. For example, when a user scrolls through images in a web application, the thumbnail interface may not refresh in a timely manner, resulting in noticeable lag.

[0111] Other encoding schemes employ image encoding / decoding schemes based on recurrent neural networks, as detailed above. Figure 3 As shown, after each iteration, the encoding end will generate an iterative bitstream and reconstruction error data for the next iteration, for example... Figure 3 The bitstreams obtained through three iterations shown are iteration bitstream 1, iteration bitstream 2, and iteration bitstream 3. After receiving these bitstreams, the decoding end decodes each bitstream sequentially to extract the basic reconstructed image and residual image for use in the next image reconstruction, thus gradually improving the quality of the reconstructed image. However, the computational complexity of the above scheme with multiple iterations is high, and since each decoding is performed on a per-bitstream basis, its scalability is poor.

[0112] Some encoding schemes also employ additional encoding for transmitting thumbnails, as detailed above. Figure 4 As shown, this method involves using compression and other techniques at the encoding end to additionally obtain the thumbnail bitstream. The thumbnail bitstream and the full image bitstream corresponding to the regular image are then merged for transmission. At the decoding end, the thumbnail bitstream is decoded first for previewing. However, this approach increases the overall bitstream file size, reduces the compression ratio, and only yields thumbnails of the same quality as those compressed at the encoding end during decoding.

[0113] Therefore, existing encoding and decoding schemes suffer from slow thumbnail acquisition, high computational complexity, poor decoding scalability, and the inability to obtain decoded images of a set quality.

[0114] To address the aforementioned issues, embodiments of this application provide an image encoding and decoding method applied to electronic devices. It is understood that the electronic devices provided in these embodiments include, but are not limited to, smartphones, in-vehicle devices, personal computers, artificial intelligence devices, tablets, computers, personal digital assistants, smart wearable devices (e.g., smartwatches or bracelets, smart glasses), smart voice devices (e.g., smart speakers), and network access devices (e.g., gateways), servers, etc.

[0115] Among them, such as Figure 5a As shown, the image encoding method may include: first, acquiring the original image, i.e., the image to be encoded; then, performing feature extraction on the original image to obtain the image features corresponding to the original image, and further extracting features from the image features to obtain the side information features corresponding to the original image. Subsequently, encoding the side information features to obtain the side information bitstream, and encoding the image features to obtain the image feature bitstream, and storing the side information bitstream and the image bitstream together or sending them to the decoding end.

[0116] It is understandable that, since the amount of side information bitstream data is small, the side information features are encoded to obtain the side information bitstream, which facilitates the rapid acquisition of a lower resolution preview image based on the side information bitstream during decoding. In this way, it is not necessary to obtain the original image during decoding and then obtain a lower resolution thumbnail through scaling. It should be understood that a thumbnail can also be called a preview image. The solution provided by the embodiments of this application can effectively avoid the situation in the prior art where users cannot obtain a preview image in a timely manner when browsing images.

[0117] Furthermore, the aforementioned scheme based on edge information features and image feature encoding effectively reduces computational complexity compared to existing iterative image encoding schemes based on recurrent neural networks. Specifically, as described in the background section, existing iterative image encoding and decoding schemes based on recurrent neural networks require multiple iterations to obtain multiple encoded bitstreams. During decoding, multiple preview images are reconstructed from these bitstreams to achieve progressive decoding, resulting in high computational complexity. In contrast, the encoding scheme in this application only requires one-time row encoding of the image bitstream and the edge information bitstream to obtain the corresponding encoded bitstreams. During decoding, multiple preview images are gradually reconstructed based on the edge information features obtained from decoding the edge information bitstream and the increasing number of image features obtained from gradually decoding the image bitstream, achieving progressive decoding. Therefore, the encoding scheme provided in this application effectively reduces computational complexity.

[0118] Furthermore, the above encoding method only requires acquiring the side information bitstream and image bitstream. Compared to existing technologies that additionally encode and transmit thumbnails, this reduces the overall bitstream file size and improves the compression ratio. Specifically, in existing technologies, the bitstream corresponding to the additionally encoded thumbnails is an extra bitstream, while in this embodiment, the side information bitstream and image bitstream are both necessary for acquiring the final reconstructed image. Therefore, no extra bitstream is added, reducing the overall bitstream file size and improving the compression ratio.

[0119] It is understandable that methods for encoding image features to obtain an image bitstream may include: obtaining the probability of each feature element in the image features based on side information features, and encoding the image features based on the probability of each feature element in the image features to obtain an image bitstream.

[0120] Corresponding to the above image encoding method, embodiments of this application provide an image decoding method such as... Figure 5a As shown, it includes: receiving side information bitstream and image bitstream. The side information bitstream is decoded to obtain side information features, and a preview image is obtained based on these features for user preview.

[0121] It is understood that a neural network can be used to decode the edge information bitstream in this embodiment. Furthermore, a scaling operator can be set in the neural network to adjust the size of the preview image.

[0122] In one embodiment, such as Figure 5b As shown, this application can also partially decode the image stream to obtain partial image features, and then reconstruct a higher-quality preview image based on the side information features and partial image features to facilitate user preview. It is understood that, as mentioned above, the image stream includes data from multiple channels; therefore, partial image features can be decoded data obtained by decoding the data of channels with higher output response importance in the image stream.

[0123] It is understandable that the importance of a channel's output response can be judged by the amount of information contained in each channel. When a channel contains a large amount of information, the importance of its output response is high; when a channel contains a small amount of information, the importance of its output response is low. The amount of information contained in each channel can be determined by the size of the corresponding bitstream data.

[0124] It is understood that the decoding time for each channel is the same during decoding. Therefore, the channel with higher output response importance is selected for decoding to obtain partial image features. Decoding only part of the channel data not only greatly saves decoding time, but also ensures the quality of the preview image as much as possible because the channel with higher output response importance is selected. Therefore, the embodiment of this application reduces the encoding time while ensuring the quality of the preview image.

[0125] It is understood that the image quality mentioned in the embodiments of this application may include the similarity between the reconstructed image or preview image obtained by decoding at the decoding end and the original image to be compressed. The higher the similarity, the higher the image quality, and the lower the similarity, the lower the image quality.

[0126] One method for partially decoding an image bitstream may include: determining the data of a subset of channels with high output response importance in the image features; decoding the data of that subset of channels to obtain first image feature data; and obtaining a preview image based on the side information features and the first image feature data.

[0127] In some embodiments, this application may also perform step-by-step (channel-by-channel) decoding of the image bitstream, progressively refreshing the current preview image based on the decoded image features and edge information features. For example, such as Figure 5c As shown, after the first part of the image features is decoded, preview image one can be obtained based on the edge information features and the first part of the image features; then, after the second part of the image features is decoded, preview image two can be obtained based on the edge information features, the first part of the image features, and the second part of the image features.

[0128] It is understood that in this embodiment of the application, after the first preview image is acquired, the electronic device will display the first acquired preview image. After each subsequent acquisition of a new preview image, the new preview image will replace the previous preview image. For example, after acquiring preview image one, the electronic device will display preview image one. After acquiring preview image two, preview image two will replace preview image one, that is, the electronic device will display preview image two.

[0129] Specifically, methods for progressively decoding the image stream and updating the current preview image based on the decoded image features and edge information features can include:

[0130] The channels in the image features are sorted from high to low according to the importance of the output response to obtain a first sequence, and the data of each channel in the first sequence are decoded in sequence. When the number of channels that have been decoded reaches a first set number, a preview image one is obtained based on the image edge information features and the decoded data corresponding to the first set number of channel data. When the number of channels that have been decoded reaches a second set number, a preview image two is obtained based on the image edge information features and the decoded data corresponding to the second set number of channel data.

[0131] It is understandable that in the above decoding scheme, the decoding end can quickly obtain the preview image based on the side information bitstream, effectively avoiding the situation in the existing technology where the preview image cannot be obtained in a timely manner when the user browses the image.

[0132] Secondly, the aforementioned decoding scheme can combine side information features with image features of different channel numbers to acquire preview images of various qualities, meeting the needs of different preview image quality requirements. Furthermore, the current preview image can be refreshed based on the decoded image features and side information features, allowing users to gradually see higher quality preview images, providing a progressive image preview effect and improving the user experience.

[0133] It is understandable that the above image encoding and decoding scheme can be applied to various application scenarios such as web browsing and photo album browsing.

[0134] For example, in web browsing application scenarios, web servers can use the above method to extract image features and side information features from each photo, and then encode the image features and side information features to obtain the corresponding side information bitstream and image bitstream for storage.

[0135] In some embodiments, when a web server needs to send a preview image of a corresponding photo to the display interface for the user to browse in response to a user's search or other operations, the web server will decode the side information bitstream corresponding to each photo to obtain side information features, obtain a preview image based on the side information features, and send it to the display interface for preview.

[0136] In other embodiments, the web server can decode the edge information bitstream corresponding to each photo to obtain edge information features, and partially decode the image bitstream corresponding to each photo to obtain partial image features. For example, it can decode the data of three channels with high output response importance in the image bitstream to obtain image features; based on the edge information features and the image features corresponding to the data of the three channels, a preview image is reconstructed and generated, and sent to the display interface for preview.

[0137] In other embodiments, the web server may first obtain a preview image 1 based on the side information features for previewing, and then decode the data of each channel sequentially according to the first sequence determined by the importance of the output response of each channel in the image features. Assuming that the image bitstream includes channel 1, channel 2, channel 3, channel 4 and channel 5, and the channel order in the determined first sequence is channel 2-channel 3-channel 4-channel 1-channel 5, if the first set quantity is 1, then after channel 2 is decoded, a preview image 2 is obtained based on the side information features and the decoded data corresponding to channel 2 to replace preview image 1; if the second set quantity is 2, then after channel 2 and channel 3 are decoded, a preview image 3 is obtained based on the side information features and the decoded data corresponding to channel 2 and channel 3 to replace preview image 2.

[0138] It is understandable that after browsing multiple preview images, when a user wants to view the decoded image corresponding to any target preview image, they can do so by clicking on the target preview image or performing other operations to view the decoded image corresponding to the original image. To avoid lag when viewing the decoded image, this application provides a decoding method, including:

[0139] The process involves acquiring the side information bitstream and image bitstream corresponding to the image to be decoded, and then decoding the side information bitstream to obtain side information features. The image bitstream is then decoded progressively (channel-by-channel) as before, with the current image being refreshed progressively based on the decoded image features and side information features. For example, after the first part of the image features is decoded, a first reconstructed image can be obtained based on the side information features and the first part of the image features. It is understood that in this embodiment, the first reconstructed image can also be called the first decoded image. Then, after the second part of the image features is decoded, a second reconstructed image can be obtained based on the side information features, the first part of the image features, and the second part of the image features. It is understood that in this embodiment, the second reconstructed image can also be called the second decoded image. This progressive decoding is completed in the above manner until all image features are decoded. After all image features are decoded, the final reconstructed image is obtained based on the side information features and all image features. It is understood that each reconstructed image in the reconstruction process in this embodiment can be a reconstructed image matching the size of the original image. It is understood that in this embodiment, a reconstructed image matching the size of the original image can also be called a decoded image matching the size of the original image.

[0140] It is understood that in this embodiment, after the first reconstructed image is acquired, the electronic device will display the first acquired reconstructed image. After each subsequent acquisition of a new reconstructed image, the new reconstructed image will replace the previous reconstructed image for display. For example, after acquiring the first reconstructed image, the first reconstructed image will be displayed. After acquiring the second reconstructed image, the second reconstructed image will replace the first reconstructed image, that is, the electronic device will display the second reconstructed image.

[0141] In this way, electronic devices can gradually display reconstructed images of different qualities, achieving a progressive decoding effect. This prevents stuttering when users view the decoded image corresponding to the original image, thus improving the user experience.

[0142] Before detailing the encoding and decoding methods of this application, a brief description of the structure of the electronic device provided in the embodiments of this application will be given first. Figure 6 A block diagram of an electronic device according to an embodiment of this application is shown. Figure 7 A flowchart illustrating an encoding / decoding scheme according to an embodiment of this application is shown. Now, in conjunction with... Figure 6 and Figure 7 The structure of the electronic device provided in the embodiments of this application will be briefly described.

[0143] like Figure 6 As shown, the electronic device provided in this application embodiment may include an encoding device and a decoding device, a storage module and a loading module;

[0144] The encoding device may include an AI encoding unit and an entropy encoding module; the Artificial Intelligence (AI) encoding unit may include a feature extraction module, a priori encoding module, a priori decoding module, and a priori entropy estimation module. The feature extraction module is used to acquire the image to be encoded and to obtain the image features of the image to be encoded.

[0145] In some embodiments, the feature extraction module can be as follows: Figure 7 The encoding neural network shown is used to extract image features Q1 from the image to be compressed, taking the image as input. Compared to the original image, the image features Q1 output by the encoding neural network may have changed in size to some extent, and the removal of redundant information makes it easier to entropy encode.

[0146] It is understood that in some embodiments, the feature extraction module can mark several channels with higher output importance in the extracted image features. In this way, when decoding at the decoding end, the data in the marked channels can be directly decoded to obtain some image features, and the partial image features and side information features can be combined to obtain a preview image.

[0147] The advanced prior coding module is used to obtain the side information features of the image to be encoded based on the image features of the image to be encoded;

[0148] In some embodiments, the advanced prior coding module can be as follows: Figure 7 The super-prior coding neural network shown is used to further extract brief information from image features, namely side information features Q2. Compared to image features Q1, the side information features Q2 output by the super-prior coding network generally have a smaller size.

[0149] It is understood that the side information features involved in the embodiments of this application are substantially different from the side information in some embodiments that can only be used to assist in encoding image features.

[0150] Furthermore, the super-prior coding neural network used in this embodiment to obtain the side information features that can generate a preview image is substantially different from the neural network used to obtain side information in some embodiments.

[0151] For ease of description of the above differences, the super-prior coding neural network used to obtain side information features that can generate preview images in the embodiments of this application is defined as the first super-prior coding neural network, and the neural network used to obtain side information that can only be used to assist in feature encoding of images in some embodiments is defined as the second super-prior coding neural network.

[0152] During training, the second super-prior coding neural network typically constrains the overall model through two losses: rate loss and image distortion loss. That is, during training, the parameter updates of the second super-prior coding network are only affected by the gradients propagated from the image distortion loss and rate distortion loss. Therefore, the extracted side information features can only be used for entropy estimation of image features.

[0153] Specifically, during the encoding and decoding process, the various neural networks involved, such as the encoding neural network, the super-prior encoding neural network, the super-prior entropy estimation neural network, the super-prior decoding neural network, and the decoding network, calculate the bit rate loss and the reconstructed image distortion loss through forward propagation (that is, the original image x is processed by the various neural networks involved in the encoding and decoding process, such as the encoding neural network, the super-prior encoding neural network, the super-prior entropy estimation neural network, the super-prior decoding neural network, and the decoding network). Then, the total loss obtained by weighting the two losses is used to update the gradient of the corresponding values ​​of all learnable parameters in all neural networks through backpropagation. The updated values ​​of the parameters are determined based on the current value, the updated gradient, and the learning rate. This iterative process makes the total loss tend to decrease with each iteration. The trained super-prior decoding neural network can minimize the bit rate overhead and the reconstruction distortion loss.

[0154] It can be understood that bit rate loss refers to the minimum number of bits (encoding overhead) required to encode the quantized image features and side information features into a bitstream; image reconstruction distortion loss refers to the difference in similarity between the reconstructed image and the original image (the image to be encoded), where the difference can be mean square error (MSE), mean absolute error (MAE), or other loss functions that can characterize the distance between the reconstructed image and the original image.

[0155] The overall loss function of the second hyperprior coding neural network during training is expressed as follows:

[0156] Loss = Loss r +α*Loss d

[0157] Loss r Represents bitrate loss, Loss d This represents the distortion loss of the reconstructed image. By controlling the value of α, the ratio between the bitrate distortion loss and the reconstruction distortion loss can be controlled, thereby achieving the purpose of controlling the compression quality, i.e., the compression ratio.

[0158] In the embodiments of this application, the first super-prior coding neural network is trained based on the overall model constrained by three losses: bit rate loss, reconstruction image distortion loss, and preview image distortion loss. That is, during the training of the first super-prior coding network, the parameter update is affected by the gradient of the backpropagation of the three losses: bit rate loss, reconstruction image distortion loss, and preview image distortion loss. Therefore, the side information features extracted by the first super-prior coding neural network can be used for entropy estimation of image features on the one hand, and can also be used to extract the preview image through the pre-decoding network on the other hand.

[0159] Specifically, during network training, the super-prior decoding neural network calculates bitrate loss, reconstruction image distortion loss, and preview image distortion loss through forward propagation computation (that is, the process by which the original image x is processed by various neural networks involved in the encoding and decoding process, such as the encoding neural network, super-prior encoding neural network, super-prior entropy estimation neural network, super-prior decoding neural network, and decoding network). Then, the total loss obtained by weighting the three losses is used to update the gradients of the corresponding values ​​of all learnable parameters in the super-prior decoding neural network through backpropagation. The updated values ​​of the parameters are determined based on the current value, the update gradient, and the learning rate. This iterative process makes the total loss tend to decrease with each iteration. The trained super-prior decoding neural network can minimize the sum of bitrate loss, reconstruction distortion loss, and preview image distortion loss.

[0160] Here, bitrate loss refers to the minimum number of bits (encoding overhead) required to encode the quantized image features and side information features into a bitstream; reconstructed image distortion loss refers to the difference in similarity between the reconstructed image and the original image (the image to be encoded), which can be mean square error (MSE), mean absolute error (MAE), or other loss functions that can characterize the distance between the reconstructed image and the original image; preview image distortion loss refers to the difference in similarity between the preview image and the original image, which can be the mean square error, mean absolute error, or other loss functions calculated by scaling the preview image and the original image to the same size.

[0161] The overall loss function is expressed as follows:

[0162] Loss = Loss r +α*Loss d +β*Loss pd

[0163] Among them, Loss r Represents bitrate loss, Loss d Loss represents the distortion loss of the reconstructed image. pd The values ​​of α and β represent the preview image distortion loss. By controlling the values ​​of α and β, the ratio between the bit rate, the reconstructed image distortion loss, and the preview image distortion loss can be controlled, thereby controlling the compression quality, i.e., the compression ratio and the preview image quality.

[0164] In some embodiments, to achieve progressive preview images, the aforementioned preview image distortion loss can be a variable preview image loss, taking into account the distortion loss in all progressive states. For example, if there are N progressive states, the distortion loss can have N forms. After the super-prior coding neural network is trained, the preview image quality in the N progressive states, i.e., the quality of the preview images obtained by sequential decoding, can be guaranteed.

[0165] In some embodiments, to achieve progressive image reconstruction, the aforementioned image distortion loss can be a variable image loss, taking into account the distortion loss in all progressive states. For example, with N progressive states, the distortion loss can have N forms. After the super-prior coding neural network is trained, the quality of the reconstructed image in the N progressive states, i.e., the quality of the images obtained by sequential decoding, can be guaranteed.

[0166] In some embodiments, the encoding device may further include a quantization module for quantizing image features or side information features to obtain quantized feature data.

[0167] The super-prior decoding module is used to obtain the mean and variance of each data in the image features based on the side information features of the image to be encoded.

[0168] The entropy coding module is used to entropy code image features based on the mean and variance of each data point in the image features to obtain the image bitstream.

[0169] The prior entropy estimation module is used to estimate the probability of each data in the edge information feature based on the preset distribution information, and to obtain the probability of each data in the edge information feature.

[0170] The entropy coding module is used to encode the side information features based on the probability of each data in the side information features to obtain the side information code stream.

[0171] Among them, the entropy coding module can be Figure 7 The Arithmetic Encoding (AE) module shown.

[0172] It is understood that in the embodiments of this application, the super-prior decoding module, the super-prior entropy estimation module, and the entropy coding module can be used as modules in the feature coding unit.

[0173] The decoding device may include an AI decoding unit and an entropy decoding module; the AI ​​decoding unit may include a priori decoding module, a priori entropy estimation module, a pre-decoding module, and a decoding module.

[0174] The entropy decoding module is used to obtain the side information bitstream and image bitstream of the image to be decoded.

[0175] The super-prior entropy estimation module is used to obtain the probability of each data in the image side information features based on the side information bitstream.

[0176] The entropy decoding module is also used to decode the side information bitstream based on the probabilities corresponding to each data point in the image side information features output by the prior entropy estimation module, thereby obtaining the image side information features. The entropy decoding module can... Figure 7 The Arithmetic Decoding (AD) module shown is illustrated.

[0177] In some embodiments, the entropy decoding module can decode each channel sequentially according to the importance of the output response of each channel in the image bitstream.

[0178] The advanced prior decoding module is used to determine the mean and variance of each data point in the image features based on the image side information features;

[0179] In some embodiments, the priori decoding module can be Figure 7 The super-prior decoding neural network shown.

[0180] The pre-decoding module is used to obtain a preview image based on image side information features, or based on image side information features and at least some image features.

[0181] In some embodiments, the pre-decoding module can be Figure 7 The pre-decoding neural network shown is an example. The pre-decoding network can reuse the parameters of the super-prior decoding network, reducing the overall number of parameters and computational cost of the model. Parameter reuse is specifically described as follows: the super-prior decoding network is used as part of the pre-decoding neural network; that is, the super-prior decoding network can first be used to obtain the mean of each data point in the image features, and then the pre-decoding network can be used to obtain a preview image based on the mean of each data point in the image features.

[0182] In some embodiments, the pre-decoding neural network can directly decode the preview image by taking the side information features output by the entropy decoding module as input.

[0183] In some embodiments, such as Figure 8 As shown, the pre-decoding neural network can also use the side information features output by the entropy decoding module and at least some image features as input to decode a higher quality preview image.

[0184] Specifically, in some embodiments, the distribution information (mean and variance) can be extracted from the side information features through a super prior decoding network, and the distribution information and some image features can be jointly input into a pre-decoding neural network to decode the preview image.

[0185] In other embodiments, distribution information (mean and variance) can be extracted from the side information features through a super-prior decoding network, and then some image features are used to replace the corresponding part of the mean in the distribution information. That is, the distribution information after replacement is input into the pre-decoding neural network to decode the preview image.

[0186] In some embodiments, in order to make the output preview image size meet a predetermined size, an image scaling operator that implements feature map scaling operation can be introduced into the pre-decoding neural network to control the output size of the pre-decoding neural network.

[0187] In some embodiments, the pre-decoding neural network may also be configured as a neural network capable of outputting the target size.

[0188] In some embodiments, after the entropy decoding module decodes a first predetermined number of channel data, the pre-decoding neural network can be used to obtain a first preview image based on the image side information features and the decoded data corresponding to the first predetermined number of channel data. After the entropy decoding module decodes a second predetermined number of channel data, the pre-decoding neural network can be used to obtain a second preview image based on the image side information features and the decoded data corresponding to the second predetermined number of channel data. After the entropy decoding module decodes all image features, the pre-decoding neural network can be used to obtain the original image based on the image side information features and all image features. That is, the embodiments of this application can achieve a progressive presentation effect of the preview image through the pre-decoding network, meaning the quality of the preview image continuously improves.

[0189] In some embodiments, this application may include an image scaling module to scale the preview image to a matching preview size after the pre-decoding network.

[0190] The decoding module can be used for, for example Figure 7 The decoding neural network shown is used to take the image features output by the entropy decoding module as input and inversely map the image features into a reconstructed image.

[0191] In some embodiments, such as Figure 9 As shown, after the entropy decoding module decodes a first predetermined number of channel data, the decoding module can obtain a first reconstructed image based on the image side information features and the decoded data corresponding to the first predetermined number of channel data. After the entropy decoding module decodes a second predetermined number of channel data, the decoding module can obtain a second reconstructed image based on the image side information features and the decoded data corresponding to the second predetermined number of channel data. After the entropy decoding module decodes all image features, the decoding module can obtain a final quality image based on the image side information features and all image features.

[0192] In scenarios where users view the original image by clicking on a preview image, the original image can be presented to the user in a progressive manner, with the image quality continuously improving.

[0193] Storage module: Used to save the data files generated by the entropy encoding module to the corresponding storage location on the terminal;

[0194] Loading module: Used to load data files from the corresponding storage location on the terminal and input them into the entropy decoding module.

[0195] The following is based on Figure 7 Taking an example, a method for encoding and decoding according to an embodiment of this application will be briefly described. Figure 7 As shown, when encoding the original image, the encoding neural network extracts features from the original image to obtain the image features Q1 corresponding to the original image.

[0196] When it is necessary to obtain the side information bitstream, image feature Q1 can be input into a super-prior coding neural network, which can further extract side information feature Q2 from image feature Q1. The super-prior entropy estimation module can estimate the probability of each data in side information feature Q2 based on preset distribution information to obtain the probability of each data in the side information feature; the arithmetic encoding (AE) module can perform entropy encoding on the side information feature based on the probability of each data in the side information feature Q2 to obtain the side information bitstream.

[0197] When an image bitstream needs to be acquired, image feature Q1 can be input into the AE module; a super-prior decoding neural network is used to determine the mean and variance of each data in image feature Q1 based on the side information feature Q2; the super-prior entropy estimation module can obtain the probability of each data in image feature Q1 based on the mean and variance of each data in image feature Q1; the AE module can entropy encode image feature Q1 based on the probability of each data in image feature Q1 to acquire the image bitstream.

[0198] The super-prior entropy estimation module can obtain the probability information corresponding to each data in the image side information feature. The second entropy encoder AE performs entropy encoding on the side information feature Q2 based on the probability information to obtain the side information bitstream.

[0199] The second Arithmetic Decoding (AD) module can perform entropy decoding on the side information bitstream based on the probabilities corresponding to each data point in the image side information features output by the super-prior entropy estimation module, thereby obtaining the decoded image side information features. It should be understood that... Figure 7 Although the pre-decoding neural network and the super-prior decoding neural network are two separate networks, the decoded side information features output by the second AD are processed by the super-prior decoding neural network and then by the pre-decoding neural network to obtain a preview image of the original image. In this case, the input of the pre-decoding neural network is the output or a part of the output of the super-prior decoding neural network. The output of the super-prior decoding neural network is also used to send to the AE (referred to as the first AE) of the encoding and decoding network for entropy encoding of image features Q1, and to send to the AD (referred to as the first AD) of the encoding and decoding network for entropy decoding of the image bitstream.

[0200] In an optional configuration, the pre-decoding neural network and the super-prior decoding neural network can be merged into a single network called the pre-decoding neural network. The decoded side information features output by the second AD serve as the input to the pre-decoding neural network, which can be used to reconstruct a preview image of the original image based on these decoded image side information features. In this case, the pre-decoding neural network is equivalent to adding some extra layers to the existing super-prior decoding neural network. The pre-decoding neural network also outputs the distribution information (or probability information) of the image features for the first AE and the first AD to perform entropy encoding and entropy decoding, respectively.

[0201] It is understandable that the pre-decoding network can reuse the parameters of the super-prior decoding network, which can reduce the overall number of parameters and computational cost of the model. The parameter reuse is specifically described as follows: the super-prior decoding network is used as part of the pre-decoding neural network. That is, the super-prior decoding network can be used first to obtain the mean of each data in the image features, and then the pre-decoding network can be used to obtain the preview image based on the mean of each data in the side information features.

[0202] In some embodiments, such as Figure 8 As shown, the pre-decoding neural network can also use the side information features output by the AD module and at least some of the image features of the module as input to decode a higher quality preview image.

[0203] like Figure 7 and Figure 8 As shown, when decoding an image bitstream to obtain a reconstructed image, the image features output by the AD module can be used as input to a decoding neural network to inversely map the image features into a reconstructed image.

[0204] In some embodiments, the method for obtaining the reconstructed image may be as follows: Figure 9 As shown, the decoding neural network can progressively decode part of the channel data, i.e. part of the image features, and the image edge information features based on the AD module to obtain multiple reconstructed images as described above, thereby realizing progressive image reconstruction.

[0205] In some embodiments, such as browsing a local photo album, the general process of image encoding and decoding based on the aforementioned electronic device can be as follows: Figure 10 As shown, firstly, images are acquired through applications such as cameras. After processing by the AI ​​encoding unit and entropy encoding module, corresponding side information bitstreams and image bitstreams are obtained. These bitstreams are then stored through a storage module. When the system detects that a user needs to view or browse a corresponding image, the electronic device can retrieve the stored side information bitstream and image bitstream through the loading module. The entropy decoding module and AI decoding unit then decode these bitstreams to obtain the corresponding preview image.

[0206] In other embodiments, the electronic device may further include a JPEG encoding module and a JPEG decoding module. These are used to compress and decompress digital images at the device side or cloud side, respectively, to reduce the amount of data transmitted between the local and cloud sides.

[0207] For example, in an edge-cloud collaborative scenario, the general process of image encoding and decoding based on the aforementioned electronic devices can be as follows: Figure 11 As shown, firstly, the terminal electronic device acquires images through applications such as a camera. Then, the images are compressed using a JPEG encoding module, and the compressed images are uploaded to a cloud server. The cloud server's JPEG decoding module then retrieves the decompressed images. The decompressed images are then processed by an AI encoding unit and an entropy encoding module to obtain the corresponding side information bitstream and image bitstream. These bitstreams are then stored using a storage module. When a user needs to view or browse a corresponding image, the cloud server can use a loading module to retrieve the stored side information bitstream and image bitstream of the corresponding image. The entropy decoding module and AI decoding unit then decode these bitstreams to obtain the corresponding preview image. The preview image is then compressed using the JPEG encoding module and sent to the terminal electronic device. The terminal electronic device decompresses the compressed preview image using the JPEG decoding module and displays it.

[0208] The encoding and decoding methods provided in the embodiments of this application will be described in detail below based on the aforementioned electronic device. Figure 12 The diagram shows a flowchart of an encoding method according to an embodiment of this application. Figure 12 As shown, the encoding method may include:

[0209] 1201: Obtain the image to be encoded.

[0210] It is understood that in this embodiment of the application, the image to be encoded can be any image, such as an image acquired through an application like a camera. It can also be an image stored in an electronic device or acquired from a cloud server.

[0211] 1202: Obtain the image features of the image to be encoded.

[0212] In this embodiment of the application, the electronic device can perform feature extraction on the image to be encoded in order to obtain the image features corresponding to the image to be encoded.

[0213] It is understandable that image features can be feature maps, which are multidimensional data output from convolutional layers, activation layers, pooling layers, batch normalization layers, etc. in a convolutional neural network. They generally contain at least three dimensions: width, height, and channel.

[0214] It is understood that in this embodiment of the application, after obtaining image features, the image features can be quantized to obtain quantized image features. In subsequent processes, encoding can be performed based on the quantized image features to obtain an image bitstream.

[0215] 1203: Obtaining the side information features of the image to be encoded based on its image features.

[0216] It is understood that in the embodiments of this application, the edge information feature is also a type of feature map, which contains fewer feature elements than the image feature. For example, it can be part or approximate information of the image feature obtained by transforming the image feature through a corresponding function. For example, the edge information feature can be three-dimensional data containing three dimensions: width, height, and channel.

[0217] 1204: Obtain the image bitstream based on the side information features and image features of the image to be encoded, and obtain the side information bitstream based on the side information features of the image to be encoded.

[0218] It can be understood that obtaining the image bitstream based on the side information features and image features of the image to be encoded includes:

[0219] Feature transformation is performed based on the side information features of the image to be encoded to obtain the distribution information of the data in the image features; entropy encoding is then performed on the image features based on the distribution information of each data point in the image features to obtain the image bitstream. The distribution information of each data point in the image features can be the mean and variance used to determine the numerical distribution of the image features.

[0220] In some embodiments, obtaining the side information bitstream based on the side information features of the image to be encoded includes:

[0221] The probability of each data in the edge information feature is estimated based on the preset distribution information to obtain the probability of each data in the edge information feature; the edge information feature is encoded based on the probability of each data in the edge information feature to obtain the edge information code stream.

[0222] In this embodiment of the application, the electronic device can store the side information bitstream and the image bitstream together or send them to the decoding end.

[0223] For example, in some embodiments, such as browsing a local photo album, image encoding and decoding need to be implemented within the same electronic device. Specifically, the electronic device can jointly store the side information bitstream and image bitstream corresponding to the image. When it detects that a user needs to view or browse a corresponding image, the electronic device can call the stored side information bitstream and image bitstream of the corresponding image for decoding.

[0224] In other embodiments, image encoding and decoding can be implemented in different electronic devices. For example, in an image transmission scenario, the sending electronic device can transmit the side information bitstream and image bitstream corresponding to the image to the receiving electronic device. The receiving electronic device can decode the received side information bitstream and image bitstream to obtain the corresponding preview image.

[0225] It is understandable that, since the amount of side information bitstream data is relatively small, encoding the side information features to obtain the side information bitstream makes it easier to quickly obtain a low-resolution preview image based on the side information bitstream at the decoding end. This eliminates the need to obtain a thumbnail or preview image of the lower resolution after decoding and scaling the original image, effectively avoiding the situation in existing technologies where users cannot obtain a preview image in a timely manner when browsing images.

[0226] Furthermore, the above-mentioned encoding and decoding scheme based on side information features and image features effectively reduces computational complexity compared to the existing iterative image encoding and decoding scheme based on recurrent neural networks.

[0227] The decoding method provided in the embodiments of this application will be described in detail below based on the above-mentioned electronic device. Figure 13 The diagram shows a flowchart of a decoding method according to an embodiment of this application. Figure 13 As shown, the decoding method may include:

[0228] 1301: Obtain the side information bitstream of the image to be decoded.

[0229] 1302: Obtaining image side information features based on side information bitstream;

[0230] In this embodiment of the application, the probability corresponding to each data in the image features can be obtained based on the edge information bitstream, and the edge information bitstream can be entropy decoded according to the probability corresponding to each data in the image edge information features to obtain the image edge information features.

[0231] Among them, entropy decoding of edge information bitstreams can include Huffman decoding and arithmetic decoding, etc.

[0232] 1303: Obtaining preview images based on image edge information features.

[0233] It can be understood that obtaining a preview image based on image side information features can include: obtaining the mean of each data point in the image features based on the image side information features, and obtaining a preview image based on the mean of each data point in the image features.

[0234] It is understood that in the embodiments of this application, each data point in the image features refers to each data point in each channel. Each channel may include h*w data points.

[0235] Understandable. In the above decoding scheme, the decoding end can quickly obtain the preview image based on the side information bitstream, effectively avoiding the situation in the existing technology where the preview image cannot be obtained in time when the user is browsing the image.

[0236] Figure 14a The text demonstrates the use of... Figure 13 The RD curves (relationship curves between pixel depth and image quality) obtained by decoding 24 PNG images with a resolution of 768x512 or 512x768 in the Kodak test set are shown. It can be seen from the RD curves that the preview images generated by this scheme using the side information bitstream can have an image quality of more than 20dB in the bitrate range of 0.2 to 1.5 pixel depth (bits per pixel, bpp).

[0237] Figure 14b To pass Figure 13 The scheme shown decodes 24 PNG images with resolutions of 768x512 or 512x768 from the Kodak test set to obtain a list of thumbnails. Both the thumbnail list and the RD curve show that the thumbnails generated using the side information bitstream are of higher quality.

[0238] Furthermore, in a test environment with a Tesla V100 image processor and an Intel(R) xeon(R) Gold6152 CPU, the time for regular image reconstruction of a single Kodak test image was 560ms, and the time for thumbnail generation was 307ms. From the thumbnail generation time perspective, through... Figure 13 The decoding scheme shown enables a preview function, and the time for generating a thumbnail from the start of decoding to display is reduced by about 45% compared to the time required for conventional image reconstruction.

[0239] In summary, the embodiments of this application can be derived. Figure 13 The decoding scheme shown can quickly obtain high-quality preview images.

[0240] Figure 15 The diagram shows a flowchart of a decoding method according to an embodiment of this application. Figure 15 As shown, the decoding method may include:

[0241] 1501: Obtain the side information bitstream and image bitstream of the image to be decoded.

[0242] 1502: Obtain image side information features based on the side information bitstream, and obtain at least some image features based on the image bitstream.

[0243] In this embodiment of the application, the probability corresponding to each data in the image side information feature can be obtained based on the side information bitstream, and the side information bitstream can be entropy decoded according to the probability corresponding to each data in the image side information feature to obtain the image side information feature.

[0244] It is understood that the image feature data mentioned in the embodiments of this application refers to each data point in each channel. Each channel may include h*w data points.

[0245] The probabilities corresponding to each data point of the image features mentioned in the embodiments of this application refer to the integral of the probability density function over the quantization interval where the value of each data point lies. It can be understood that in some embodiments, the probability density function corresponding to each data point can be determined based on the mean and variance of each data point in the image features, and the integral of the probability density function over the quantization interval where the value of each data point in the image features lies can be used as the probability corresponding to each data point in the image features.

[0246] It is understood that in the embodiments of this application, the method of obtaining image features may be as follows: determining the mean and variance of each data in the image features based on the image edge information features; determining the probability corresponding to each data in the image features based on the mean and variance of each data in the image features; and obtaining image features based on the image bitstream and the probability corresponding to each data in the image features.

[0247] In some embodiments, partial image features can be obtained based on the image bitstream to facilitate the subsequent reconstruction of a high-quality preview image based on side information features and partial image features, making it easier for users to preview. It can be understood that partial image features can be decoded data obtained by decoding data from channels in the image bitstream that have high output response importance.

[0248] Specifically, in some embodiments, the electronic device can sort the channels in the image features according to the importance of the output response from high to low, obtain a first sequence, and decode the first set of channels in the first sequence to obtain first image feature data. That is, it decodes the first set of channels with higher output response importance in the image features to obtain the corresponding partial image features.

[0249] In some embodiments, the electronic device may sort the channels in the image features from low to high according to the importance of the output response, obtain a first sequence, and decode the last set of channels in the first sequence to obtain first image feature data.

[0250] In some embodiments, the electronic device may select a set number of channels with higher output response importance from each channel of the image features for decoding to obtain first image feature data.

[0251] It is understandable that the number of channels used for decoding can be set based on actual needs, such as the quality requirements of the preview image or the resolution speed requirements.

[0252] 1503: Obtain a preview image based on image edge information features and at least some image features.

[0253] In this embodiment of the application, the side information bitstream is decoded to obtain side information features, and the preview image generated by reconstructing based on the side information features and the aforementioned partial image features can be a high-quality preview image.

[0254] Specifically, obtaining a preview image based on image edge information features and at least some image features may include: obtaining the mean of each data in the image features based on image edge information features, and obtaining a preview image based on the mean of each data in the image features and at least some image features.

[0255] Understandable. Figure 15 The decoding scheme shown can combine side information features with image features of different channel numbers to acquire preview images of various qualities, meeting the needs of various preview image quality.

[0256] Figure 16a A schematic diagram illustrating the relationship between the number of channels and the quality of the preview image for image features measured on the Kodak test set. Figure 16b This is a schematic diagram illustrating the relationship between the number of channels and the time required to generate a preview image for image features measured on the Kodak test set. Figure 16a and Figure 16b It can be seen that using edge information and some image features can achieve a fast preview function. Compared with the scheme that only uses edge information to generate a preview image, the combination of edge information and some image features can generate a higher quality preview image. The more image feature channels used, the higher the quality of the preview image, but the longer it takes to generate the preview image.

[0257] In some embodiments, this application may also perform stepwise (channel-by-channel) decoding of the image bitstream, and gradually refresh the current preview image based on the decoded image features and side information features. Figure 17 The diagram shows a flowchart of a decoding method according to an embodiment of this application. Figure 17 As shown, the decoding method may include:

[0258] 1701: Obtain the side information bitstream and image bitstream of the image to be decoded.

[0259] 1702: Obtaining image side information features based on side information bitstream.

[0260] In this embodiment of the application, the probability corresponding to each data in the image side information feature can be obtained based on the side information bitstream, and the side information bitstream can be entropy decoded according to the probability corresponding to each data in the image side information feature to obtain the image side information feature.

[0261] It can be understood that each data point in an image feature refers to each data point in each channel. Each channel can contain h*w data points.

[0262] 1703: When the number of channels in the decoded image bitstream reaches a first set number, the first preview image is obtained based on the image side information features and the decoded data corresponding to the first set number of channel data.

[0263] It is understood that, in the embodiments of this application, the method for decoding the image bitstream can be as follows: determining the mean and variance of each data in the image features based on the image side information features; determining the probability corresponding to each data in the image features based on the mean and variance of each data in the image features; and obtaining the image features based on the image bitstream and the probability corresponding to each data in the image features.

[0264] Specifically, in this embodiment, the channels in the image bitstream can be sorted from high to low according to the importance of the output response to obtain a first sequence, and the data of each channel in the first sequence can be decoded sequentially. That is, the channels in the image bitstream are decoded sequentially according to the importance of the output response from high to low.

[0265] 1704: When the number of channels that have been decoded reaches the second set number, the second preview image is obtained based on the image edge information features and the decoded data corresponding to the second set number of channel data.

[0266] In some embodiments, the first preview image can be obtained based on the edge information features, and then the current preview image can be refreshed step by step based on the decoded image features and edge information features.

[0267] It is understood that the above description only uses the first and second preview images as examples to illustrate the refresh process of the preview images. This application does not limit the number of refreshes, that is, there may be a third preview image, a fourth preview image, etc.

[0268] For example, as mentioned above, the electronic device can first obtain a preview image 1 based on the side information features for previewing, and then decode the data of each channel in sequence according to the first sequence determined by the importance of the output response of each channel in the image bitstream. Assuming that the channel order in the first sequence is channel 2-channel 3-channel 4-channel 1-channel 5, if the first set quantity is 1, then after channel 2 is decoded, a preview image 2 is obtained based on the side information features and the decoded data corresponding to channel 2 to replace preview image 1. If the second set quantity is 2, then after channel 2 and channel 3 are decoded, a preview image 3 is obtained based on the side information features and the decoded data corresponding to channel 2 and channel 3 to replace preview image 2.

[0269] based on Figure 17 The decoding scheme shown can refresh the current preview image based on the decoded image features and edge information features, allowing users to gradually see higher quality preview images, providing a progressive image preview effect and improving the user experience.

[0270] In this embodiment, the image decoding method may further include reconstructing a regular image based on image characteristics. This allows the user to view the original image when performing actions such as clicking on a preview image.

[0271] It is understandable that after browsing multiple preview images, if a user wants to view the reconstructed image corresponding to any target preview image, they can view the original image by clicking on the target preview image or performing other operations.

[0272] To avoid lag when users view reconstructed images, this application provides a decoding method, such as... Figure 18 As shown, it includes:

[0273] 1801: Obtain the side information bitstream and image bitstream of the image to be decoded.

[0274] 1802: Obtaining image side information features based on side information bitstream.

[0275] In this embodiment of the application, the probability corresponding to each data in the image side information feature can be obtained based on the side information bitstream, and the side information bitstream can be entropy decoded according to the probability corresponding to each data in the image side information feature to obtain the image side information feature.

[0276] 1803: When the number of channels in the decoded image bitstream reaches a first set number, the first reconstructed image is obtained based on the image side information features and the decoded data corresponding to the first set number of channel data.

[0277] Specifically, it can be understood that in the embodiments of this application, the method for decoding the image bitstream can be as follows: determining the mean and variance of each data in the image features based on the image side information features; determining the probability corresponding to each data in the image features based on the mean and variance of each data in the image features; and obtaining the image features based on the image bitstream and the probability corresponding to each data in the image features.

[0278] In this embodiment, the channels in the image bitstream can be sorted from high to low according to the importance of their output responses to obtain a first sequence, and the data of each channel in the first sequence can be decoded sequentially. That is, the channels in the image bitstream are decoded sequentially according to the importance of their output responses from high to low.

[0279] 1804: When the number of channels that have been decoded reaches the second set number, the second reconstructed image is obtained based on the image edge information features and the decoded data corresponding to the second set number of channel data.

[0280] 1805: After all channels have been decoded, the final reconstructed image is obtained based on the side information features and the decoded data corresponding to all channels.

[0281] In this way, reconstructed images of different qualities can be presented gradually, achieving a progressive decoding effect. This prevents users from experiencing lag when viewing the original image, thus improving the user experience.

[0282] It is understood that the above is only an explanation of the original image refresh process using the first and second reconstructed images as examples. This application does not limit the number of refreshes, that is, there may be a third reconstructed image, a fourth reconstructed image, etc.

[0283] Figure 19 It shows the use of Figure 18 The diagram illustrates the progressively reconstructed image obtained by decoding a 768x512 PNG image on the Kodak test set. As can be seen from the diagram, the quality of the reconstructed image gradually increases, achieving the effect of progressive decoding.

[0284] The hardware structure of the electronic device mentioned in this application will be briefly introduced below, taking mobile phone 10 as an example.

[0285] like Figure 20 As shown, the mobile phone 10 may include a processor 110, a power module 140, a memory 180, a mobile communication module 130, a wireless communication module 120, a sensor module 190, an audio module 150, a camera 170, an interface module 160, buttons 101, and a display screen 102, etc.

[0286] It is understood that the structures illustrated in the embodiments of the present invention do not constitute a specific limitation on the mobile phone 10. In other embodiments of this application, the mobile phone 10 may include more or fewer components than illustrated, or combine some components, or split some components, or have different component arrangements. The illustrated components may be implemented in hardware, software, or a combination of software and hardware.

[0287] Processor 110 may include one or more processing units, such as processing modules or circuits of a Central Processing Unit (CPU), Graphics Processing Unit (GPU), Digital Signal Processor (DSP), Micro-programmed Control Unit (MCU), Artificial Intelligence (AI) processor, or Field Programmable Gate Array (FPGA). Different processing units may be independent devices or integrated into one or more processors. Processor 110 may include storage units for storing instructions and data. In some embodiments, the storage unit in processor 110 is a cache memory 180.

[0288] It is understood that the above-described encoding and decoding methods of this application can be executed by the processor 110.

[0289] The power module 140 may include a power supply, a power management component, etc. The power supply may be a battery. The power management component manages the charging of the power supply and the power supply to other modules. In some embodiments, the power management component includes a charging management module and a power management module. The charging management module receives charging input from a charger; the power management module connects to the power supply and the processor 110. The power management module receives input from the power supply and / or the charging management module to supply power to the processor 110, the display 102, the camera 170, and the wireless communication module 120, etc.

[0290] The mobile communication module 130 may include, but is not limited to, an antenna, a power amplifier, a filter, and an LNA (Low Noise Amplifier). The mobile communication module 130 can provide wireless communication solutions, including 2G / 3G / 4G / 5G, for use on the mobile phone 10. The mobile communication module 130 can receive electromagnetic waves via the antenna, filter and amplify the received electromagnetic waves, and then transmit them to a modem processor for demodulation. The mobile communication module 130 can also amplify the signal modulated by the modem processor and convert it into electromagnetic waves for radiation via the antenna. In some embodiments, at least some functional modules of the mobile communication module 130 may be housed in the processor 110. In some embodiments, at least some functional modules of the mobile communication module 130 and at least some modules of the processor 110 may be housed in the same device.

[0291] The wireless communication module 120 may include an antenna, which enables the transmission and reception of electromagnetic waves. The wireless communication module 120 can provide solutions for wireless communication applications on the mobile phone 10, including wireless local area networks (WLAN) (such as wireless fidelity (Wi-Fi) networks), Bluetooth (BT), global navigation satellite system (GNSS), frequency modulation (FM), near field communication (NFC), and infrared (IR) technologies. The mobile phone 10 can communicate with networks and other devices through wireless communication technologies.

[0292] In some embodiments, the mobile communication module 130 and the wireless communication module 120 of the mobile phone 10 may also be located in the same module.

[0293] The display screen 102 is used to display human-computer interaction interfaces, images, videos, etc. The display screen 102 includes a display panel. The display panel can be a liquid crystal display (LCD), an organic light-emitting diode (OLED), an active-matrix organic light-emitting diode (AMOLED), a flexible light-emitting diode (FLED), a Mini LED, a MicroLED, a Micro-OLED, a quantum dot light-emitting diode (QLED), etc.

[0294] The sensor module 190 may include proximity sensors, pressure sensors, gyroscope sensors, barometric pressure sensors, magnetic sensors, accelerometers, distance sensors, fingerprint sensors, temperature sensors, touch sensors, ambient light sensors, bone conduction sensors, etc.

[0295] The audio module 150 is used to convert digital audio information into analog audio signals for output, or to convert analog audio input into digital audio signals. The audio module 150 can also be used for encoding and decoding audio signals. In some embodiments, the audio module 150 may be located in the processor 110, or some functional modules of the audio module 150 may be located in the processor 110. In some embodiments, the audio module 150 may include a speaker, a handset, a microphone, and a headphone jack.

[0296] Camera 170 is used to capture still images or videos. An object passes through the lens to generate an optical image that is projected onto a photosensitive element. The photosensitive element converts the light signal into an electrical signal, which is then passed to the ISP (Image Signal Processing) to be converted into a digital image signal. Mobile phone 10 can achieve its shooting function through the ISP, camera 170, video codec, GPU (Graphics Processing Unit), display 102, and application processor.

[0297] Interface module 160 includes an external memory interface, a universal serial bus (USB) interface, and a subscriber identification module (SIM) card interface. The external memory interface can be used to connect an external memory card, such as a Micro SD card, to expand the storage capacity of the mobile phone 10. The external memory card communicates with the processor 110 through the external memory interface to perform data storage. The USB interface is used for communication between the mobile phone 10 and other electronic devices. The SIM card interface is used to communicate with the SIM card installed in the mobile phone 10, for example, to read or write phone numbers stored in the SIM card.

[0298] In some embodiments, the mobile phone 10 further includes buttons 101, a motor, and indicators. The buttons 101 may include volume buttons, a power button, etc. The motor is used to generate a vibration effect in the mobile phone 10, for example, vibrating when the user's mobile phone 10 is called to prompt the user to answer the call. The indicators may include laser indicators, radio frequency indicators, LED indicators, etc.

[0299] Figure 21 This is a schematic block diagram illustrating an example of an encoding device 20 used to implement the technology of this application. Figure 21 In the example, the encoding device 20 includes an input terminal (or input interface) 202, an encoding network 204, a rounding unit 206, an entropy encoding module 208, a priori encoding module 209, a priori entropy estimation module 210, and an output terminal (or output interface) 212. Figure 21 The encoding device 20 shown can also be called an end-to-end encoding device 20.

[0300] The encoding device 20 can be used to receive an image to be encoded via an input terminal 202, etc. The received image to be encoded or the image data to be encoded can also be a preprocessed image to be encoded (or preprocessed image data to be encoded). For simplicity, the following description uses the image to be encoded.

[0301] A digital image is, or can be viewed as, a two-dimensional array or matrix of pixels with intensity values. Pixels in an array are also called pixels (short for image element). The number of pixels in the array or image along the horizontal and vertical directions (or axes) determines the image size and / or resolution. To represent color, three color components are typically used, meaning an image can be represented as or comprise an array of three pixels. In RBG format or color space, an image includes corresponding arrays of red, green, and blue pixels. However, in video or image encoding, each pixel is typically represented in a luma / chroma format or color space, such as YCbCr, which includes the luma component indicated by Y (sometimes also represented by L) and two chroma components represented by Cb and Cr. The luma component Y represents the brightness or grayscale level intensity (e.g., both are the same in grayscale images), while the two chroma components Cb and Cr represent the chroma or color information components. Accordingly, a YCbCr format image consists of a luminance pixel array for the luminance pixel value (Y) and two chrominance pixel arrays for the chrominance values ​​(Cb and Cr). An RGB format image can be converted or transformed to YCbCr format, and vice versa; this process is also known as color conversion or transformation. If the image is black and white, it may only include the luminance pixel array. Accordingly, the image can be, for example, a monochrome format luminance pixel array or a 4:2:0, 4:2:2, and 4:4:4 color format luminance pixel array and two corresponding chrominance pixel arrays.

[0302] In one possibility, embodiments of the encoding device 20 may include an image segmentation unit for segmenting an image into multiple (typically non-overlapping) image blocks. These blocks may also be referred to as root blocks, macroblocks (H.264 / AVC), coding tree blocks (CTBs), or coding tree units (CTUs) in the H.265 / HEVC and VVC standards. The segmentation unit may be used to apply the same block size and a corresponding grid with defined block sizes to all images in a video sequence, or to vary the block size between images, subsets of images, or groups of images, segmenting each image into corresponding blocks.

[0303] In another possibility, the encoding device can be used to directly receive image blocks of the image to be encoded, for example, one, several, or all of the blocks that make up the image to be encoded. An image block can also be referred to as the current image block or the image block to be encoded.

[0304] Like an image, an image patch is also a two-dimensional array or matrix composed of pixels with intensity values ​​(pixel values), but an image patch is smaller than an image. In other words, an image patch can include a single pixel array (e.g., a luminance array in the case of a monochrome image or a luminance or chrominance array in the case of a color image) or a three-pixel array (e.g., a luminance array and two chrominance arrays in the case of a color image) or any other number and / or type of array depending on the color format used. The number of pixels in the horizontal and vertical directions (or axes) of an image patch defines the size of the patch. Accordingly, a patch can be an M×N (M columns × N rows) pixel array, or an M×N transform coefficient array, etc.

[0305] In another possibility, Figure 21 The encoding device 20 shown is used to encode the image block by block, for example, to perform encoding, rounding and entropy encoding on each image block.

[0306] In another possibility, Figure 21 The encoding device 20 shown is used to encode an image, for example, to perform encoding, quantization and entropy encoding on the image.

[0307] In another possibility, Figure 21 The encoding device 20 shown is used to encode audio data, for example, to perform encoding, quantization and entropy encoding on the audio data.

[0308] In another possibility, Figure 21 The encoding device 20 shown can also be used to segment and encode images using slices (also called video slices), wherein an image can be segmented or encoded using one or more slices (typically non-overlapping). Each slice may include one or more blocks (e.g., coding tree units, CTUs) or one or more groups of blocks (e.g., coded tiles in the H.265 / HEVC / VVC standard and subpictures in the VVC standard).

[0309] In another possibility, Figure 21 The encoding device 20 shown can also be used to segment audio using segments, wherein the audio can be segmented or encoded using one or more segments (typically non-overlapping).

[0310] In another possibility, Figure 21The encoding device 20 shown can also be used to segment and / or encode an image using slice / encoding block groups (also known as video encoding block groups) and / or encoding blocks (also known as video encoding blocks), wherein the image can be segmented or encoded using one or more slice / encoding block groups (typically non-overlapping), each slice / encoding block group may include one or more blocks (e.g., CTUs) or one or more encoding blocks, wherein each encoding block may be rectangular or the like, and may include one or more complete or partial blocks (e.g., CTUs).

[0311] Encoding Network 204

[0312] like Figure 21 The encoding network 204 is used to obtain the output feature maps 205 of each feature layer based on the input data. Each encoding network unit outputs at least two output feature maps corresponding to two or more feature layers.

[0313] In one possibility, the encoding network contains K sub-encoding networks, each corresponding to its own feature layer and its corresponding output feature map. In this case, there are K output feature maps, where K >= 2.

[0314] In one possibility, the input to the encoding network is either the image to be encoded or a block of images to be encoded.

[0315] In one possibility, the encoding network unit consists of T network layers, where M, L, T, and K are positive integers. The Mth and Kth output feature maps are both outputs of the encoding network, and the Kth output feature map is output after layer L of the encoding network unit, while the Mth output feature map is output after layer T. It is understood that multiple output feature maps can be output after different network layer positions within the encoding network unit; this is not limited here.

[0316] It is understood that in the embodiments of this application, the coding network may refer to the feature extraction module used to extract image features in the embodiments of this application.

[0317] Rounding unit 206

[0318] The rounding unit 206 is used to round the output feature map 205 by, for example, scalar quantization or vector quantization, to obtain a rounded feature map.

[0319] Entropy coding module 208

[0320] The entropy coding module 208 is used to apply an entropy coding algorithm or scheme (e.g., variable length coding (VLC), context adaptive VLC (CALVC), arithmetic coding scheme, binarization algorithm, context adaptive binary arithmetic coding (CABAC), syntax-based context-adaptive binary arithmetic coding (SBAC), probability interval partitioning entropy (PIPE) coding, or other entropy coding methods or techniques) to the rounding feature map in at least two feature layers to obtain coded image data that can be output through the output terminal 212 in the form of a coded bitstream or the like.

[0321] In one possibility, since entropy coding does not know the true character probabilities of the rounded feature map, it may be necessary to collect such or related information and add it to the entropy coding module, and then transmit this information to the decoding end.

[0322] The advanced prior coding module 209 is used to obtain the side information features of the image to be encoded based on the image features of the image to be encoded;

[0323] Super-prior entropy estimation module 210

[0324] The super-prior entropy estimation module 210 is used to perform probability estimation on the rounded feature map to achieve bitrate estimation for model compression. The dashed input is optional. The super-prior entropy estimation module corresponds to N entropy estimation modules for N feature layers.

[0325] The prior entropy estimation module 210 corresponds to the N entropy estimation modules corresponding to the N feature layers and the lossless entropy estimation module corresponding to the residual layer.

[0326] In one possibility, the prior entropy estimation module 210 is a convolutional network, which includes convolutional layers and nonlinear activation layers.

[0327] In one embodiment, Figure 22 This diagram illustrates a possible system architecture for a cloud application device, including:

[0328] The local and cloud-side JPEG encoding / decoding modules are used to compress / decompress digital images on the device side or cloud side to reduce the amount of data transmitted between the local and cloud sides.

[0329] The AI ​​encoding unit transforms the image into output features with lower redundancy and generates probability estimates for each point in the output features. The AI ​​encoding unit includes a feature extraction module, a super-prior encoding module, a super-prior decoding module, and a super-prior entropy estimation module. The AI ​​decoding unit performs an inverse transformation on the output features, parsing them back into an image. The AI ​​decoding unit may include a super-prior decoding module, a super-prior entropy estimation module, a pre-decoding module, and a decoding module.

[0330] The entropy coding / decoding module (taking arithmetic coding as an example) is used to reduce the coding redundancy of output features, further reducing the amount of data transmitted during image compression.

[0331] The storage module is used to save data files such as image bitstreams or edge information bitstreams generated by the entropy coding (taking arithmetic coding as an example) module to the corresponding storage location on the cloud side;

[0332] The loading module is used to load data files from the corresponding storage location on the cloud side and input them into the entropy decoding module.

[0333] This application provides a decoding device, which is an encoding device or a decoding device. The decoding device includes one or more processors and a memory. The memory is used to store program instructions. When the program instructions are executed by one or more processors, the above-mentioned image encoding method or image decoding method is implemented.

[0334] This application provides a readable storage medium storing instructions that, when executed on an electronic device, cause the machine to perform the aforementioned image encoding or image decoding method.

[0335] This application provides a computer program product, including instructions that, when executed on an electronic device, cause the machine to perform the aforementioned image encoding method or image decoding method.

[0336] This application provides a computer-readable storage medium storing an image bitstream and a side information bitstream obtained by an encoding method as mentioned in the embodiments of this application, executed by one or more processors.

[0337] This application provides a decoding device, including a memory and a decoder. The memory is used to store image bitstreams and side information bitstreams; the decoder is used to execute image decoding methods.

[0338] This application provides an artificial intelligence neural network encoding / decoding architecture, including: an encoding / decoding network and a super-prior encoding / decoding network. The encoding / decoding network includes an encoding neural network, a first entropy encoder, a first entropy decoder, and a decoding neural network. The super-prior encoding / decoding network includes a super-prior encoding neural network, a second entropy encoder, a second entropy decoder, and a pre-decoding neural network. The encoding neural network processes the image to be encoded to obtain a first image feature set. The super-prior encoding neural network extracts features from the first image feature set to obtain side information features. The first entropy encoder entropy-encodes the first image feature set to obtain an image bitstream. The second entropy encoder entropy-encodes the side information features to obtain a side information bitstream. The second entropy decoder entropy-decodes the side information bitstream to obtain an entropy decoding result corresponding to the side information features. The pre-decoding neural network processes the entropy decoding result corresponding to the side information features to obtain a preview image of the image to be encoded. The first entropy decoder entropy-decodes the image bitstream to obtain an entropy decoding result corresponding to the image bitstream. The decoding neural network processes the entropy decoding result corresponding to the image bitstream to obtain a reconstructed image of the image to be encoded.

[0339] It is understandable that the super-prior encoding / decoding network also includes a super-prior decoding neural network. The super-prior decoding neural network processes the entropy decoding results corresponding to the edge information features to obtain the reconstructed features of the edge information features. The pre-decoding neural network processes the reconstructed features of the edge information features to obtain a preview image of the image to be encoded.

[0340] This application provides an artificial intelligence neural network decoding architecture, including: an encoding / decoding network and a super-prior encoding / decoding network, wherein the encoding / decoding network includes a first entropy decoder and a decoding neural network; the super-prior encoding / decoding network includes a second entropy decoder and a pre-decoding neural network; the second entropy decoder acquires the side information bitstream of the image to be decoded, and performs entropy decoding on the side information bitstream to obtain the entropy decoding result corresponding to the side information features; the pre-decoding neural network processes the entropy decoding result corresponding to the side information features to obtain a first preview image of the image to be decoded; the first entropy decoder acquires the image bitstream of the image to be decoded, and performs entropy decoding on the image bitstream to obtain the entropy decoding result corresponding to the image bitstream; the decoding neural network processes the entropy decoding result corresponding to the image bitstream to obtain the reconstructed image of the image to be decoded.

[0341] This application provides an artificial intelligence neural network coding architecture, including: an encoding / decoding network and a super-prior encoding / decoding network. The encoding / decoding network includes an encoding neural network and a first entropy encoder; the super-prior encoding / decoding network includes a super-prior encoding neural network and a second entropy encoder. The encoding neural network processes the image to be encoded to obtain a first image feature set; the super-prior encoding neural network extracts features from the first image feature set to obtain side information features; the first entropy encoder entropy-encodes the first image feature set to obtain an image bitstream; and the second entropy encoder entropy-encodes the side information features to obtain a side information bitstream.

[0342] This application provides an artificial intelligence neural network decoding architecture, including: an encoding / decoding network and a super-prior encoding / decoding network, wherein the encoding / decoding network includes a first entropy decoder and a decoding neural network; the super-prior encoding / decoding network includes a second entropy decoder and a pre-decoding neural network; the side information bitstream of the image to be decoded is obtained through the second entropy decoder, and the side information bitstream is entropy decoded to obtain the entropy decoding result corresponding to the side information features;

[0343] The first entropy decoder obtains the image bitstream of the image to be decoded, and decodes a first set number of channel data in the image bitstream to obtain the first decoded data; the entropy decoding result corresponding to the edge information features and the first decoded data are processed by the pre-decoding neural network to obtain the second preview image of the image to be decoded.

[0344] This application provides an artificial intelligence neural network decoding architecture, including: an encoding / decoding network and a super-prior encoding / decoding network, wherein the encoding / decoding network includes a first entropy decoder and a decoding neural network; the super-prior encoding / decoding network includes a second entropy decoder and a pre-decoding neural network; the side information bitstream of the image to be decoded is obtained through the second entropy decoder, and the side information bitstream is entropy decoded to obtain the entropy decoding result corresponding to the side information features;

[0345] The image bitstream of the image to be decoded is obtained through a first entropy decoder. A second set number of channel data in the image bitstream are decoded to obtain second decoded data. A pre-decoding neural network processes the entropy decoding result corresponding to the edge information features and the second decoded data to obtain a third preview image of the image to be decoded. The first entropy decoder then decodes a third set number of channel data in the image bitstream to obtain third decoded data. A pre-decoding neural network then processes the entropy decoding result corresponding to the edge information features and the third decoded data to obtain a third preview image of the image to be decoded.

[0346] This application provides an artificial intelligence neural network decoding architecture, including: an encoding / decoding network and a super-prior encoding / decoding network. The encoding / decoding network includes a first entropy decoder and a decoding neural network; the super-prior encoding / decoding network includes a second entropy decoder and a pre-decoding neural network. The second entropy decoder acquires the side information bitstream of the image to be decoded and performs entropy decoding on the side information bitstream to obtain entropy decoding results corresponding to the side information features. The pre-decoding neural network processes the entropy decoding results corresponding to the side information features to obtain a preview image of the image to be decoded. The first entropy decoder acquires the image bitstream of the image to be decoded and decodes a first predetermined number of channel data in the image bitstream to obtain first decoded data. The decoding neural network processes the entropy decoding results corresponding to the side information features and the first decoded data to obtain a first reconstructed image of the image to be decoded. The first entropy decoder decodes the first predetermined number of channel data in the image bitstream to obtain second decoded data. The decoding neural network processes the entropy decoding results corresponding to the side information features and the second decoded data to obtain a second reconstructed image of the image to be decoded.

[0347] The embodiments disclosed in this application can be implemented in hardware, software, firmware, or a combination of these implementation methods. Embodiments of this application can be implemented as computer programs or program code executable on a programmable system, the programmable system including at least one processor, a storage system (including volatile and non-volatile memory and / or storage elements), at least one input device, and at least one output device.

[0348] Program code can be applied to input instructions to execute the functions described in this application and generate output information. The output information can be applied to one or more output devices in a known manner. For the purposes of this application, the processing system includes any system having a processor such as, for example, a digital signal processor (DSP), a microcontroller, an application-specific integrated circuit (ASIC), or a microprocessor.

[0349] The program code can be implemented using a high-level procedural language or an object-oriented programming language to communicate with the processing system. Assembly language or machine language can also be used when needed. In fact, the mechanisms described in this application are not limited to any particular programming language. In either case, the language can be a compiled language or an interpreted language.

[0350] In some cases, the disclosed embodiments may be implemented in hardware, firmware, software, or any combination thereof. The disclosed embodiments may also be implemented as instructions carried or stored thereon on one or more temporary or non-temporary machine-readable (e.g., computer-readable) storage media, which may be read and executed by one or more processors. For example, the instructions may be distributed via a network or through other computer-readable media. Therefore, machine-readable media may include any mechanism for storing or transmitting information in a machine-readable (e.g., computer-readable) form, including but not limited to floppy disks, optical disks, CD-ROMs, magneto-optical disks, read-only memory (ROM), random access memory (RAM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), magnetic cards or optical cards, flash memory, or tangible machine-readable storage for transmitting information (e.g., carrier waves, infrared signals, digital signals, etc.) using the Internet in the form of electrical, optical, acoustic, or other propagation signals. Therefore, machine-readable media include any type of machine-readable medium suitable for storing or transmitting electronic instructions or information in a machine-readable (e.g., computer-readable) form.

[0351] In the accompanying drawings, some structural or methodological features may be shown in a specific arrangement and / or order. However, it should be understood that such a specific arrangement and / or order may not be necessary. Rather, in some embodiments, these features may be arranged in a manner and / or order different from that shown in the illustrative drawings. Furthermore, the inclusion of structural or methodological features in a particular figure does not imply that such features are required in all embodiments, and in some embodiments, these features may be omitted or may be combined with other features.

[0352] It should be noted that all units / modules mentioned in the device embodiments of this application are logical units / modules. Physically, a logical unit / module can be a physical unit / module, a part of a physical unit / module, or a combination of multiple physical units / modules. The physical implementation of these logical units / modules themselves is not the most important factor; the combination of functions implemented by these logical units / modules is the key to solving the technical problems proposed in this application. Furthermore, to highlight the innovative aspects of this application, the above-described device embodiments of this application have not introduced units / modules that are not closely related to solving the technical problems proposed in this application. This does not mean that the above-described device embodiments do not contain other units / modules.

[0353] It should be noted that in the examples and description of this patent, relational terms such as "first" and "second" are used merely to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one" does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes the element.

[0354] Although this application has been illustrated and described with reference to certain preferred embodiments thereof, those skilled in the art will understand that various changes in form and detail may be made thereto without departing from the scope of this application.

Claims

1. An image decoding method, characterized in that, include: Obtain the edge information bitstream of the image to be decoded; Image side information features are obtained based on the side information bitstream. The image side information features are a second image feature set obtained by extracting features from the first image feature set corresponding to the image to be decoded based on a super prior coding neural network. The data volume of the second image feature set is less than that of the first image feature set. The super prior coding neural network is obtained by training constraints based on bit rate loss, reconstruction image distortion loss and preview image distortion loss. The preview image distortion loss represents the difference in similarity between the original training image and the preview image of the original training image. A preview image of the image to be decoded is obtained based on the image side information features.

2. The method according to claim 1, characterized in that, The step of obtaining a preview image of the image to be decoded based on the image side information features includes: The first preview image of the image to be decoded is obtained based on the image side information features.

3. The method according to claim 2, characterized in that, Also includes: Obtain the image bitstream of the image to be decoded, wherein the image bitstream includes data from multiple channels of the image to be decoded; First decoded data is obtained based on the image bitstream, wherein the first decoded data is at least a portion of the image features of the first image feature set; correspondingly, The step of obtaining a preview image of the image to be decoded based on the image side information features includes: A second preview image is obtained based on the image edge information features and the first decoding data, wherein the similarity between the second preview image and the image to be decoded is greater than the similarity between the first preview image and the image to be decoded.

4. The method according to claim 3, characterized in that, The step of obtaining the first decoded data based on the image bitstream includes: Determine the data of at least one channel in the image bitstream; Decode the data from the at least one channel to obtain the first decoded data.

5. The method according to claim 4, characterized in that, The step of determining data for at least one channel in the image bitstream includes: Data for at least one channel is obtained according to the amount of data in each channel of the image bitstream.

6. The method according to claim 1, characterized in that, Also includes: Obtain the image bitstream of the image to be decoded, the image bitstream including data from multiple channels of the image to be decoded; accordingly, The step of obtaining a preview image of the image to be decoded based on the image side information features includes: Decode a first set number of channel data in the image bitstream to obtain second decoded data, and obtain a third preview image based on the image edge information features and the second decoded data; A third decoded data is obtained by decoding a second set number of channel data in the image bitstream. A fourth preview image is obtained based on the image edge information features and the third decoded data. The second set number is greater than the first set number, and the second set number of channel data includes the first set number of channel data.

7. The method according to any one of claims 3-6, characterized in that, Also includes: In response to the user clicking on the preview image, the reconstructed image corresponding to the image to be decoded is obtained based on the image bitstream.

8. The method according to any one of claims 1-6, characterized in that, Obtaining a preview image based on the image edge information features; including: The mean value of each data in the image features is obtained based on the image edge information features, and the preview image is obtained based on the mean value of each data in the image features.

9. The method according to claim 3, characterized in that, The step of obtaining the first decoded data based on the image bitstream includes: The probability corresponding to each data in the image feature is determined based on the image edge information feature; At least some image features are obtained based on the probability corresponding to each data in the image bitstream and the image features.

10. An image encoding method, characterized in that, include: Obtain the image to be encoded; Obtain the first image feature set of the image to be encoded; The side information features of the image to be encoded are obtained based on the first image feature set of the image to be encoded; the side information features are a second image feature set obtained by feature extraction of the first image feature set based on the super prior coding neural network, the data volume of the second image feature set is less than the data volume of the first image feature set, the super prior coding neural network is obtained by training constraints based on bit rate loss, reconstruction image distortion loss and preview image distortion loss, wherein the preview image distortion loss represents the difference in similarity between the original training image and the preview image of the original training image; An image bitstream is obtained based on the side information features of the image to be encoded and the first image feature set, and a side information bitstream is obtained based on the side information features of the image to be encoded. The side information bitstream is used by the decoding end to obtain a preview image of the image to be encoded.

11. The method according to claim 10, characterized in that, The step of obtaining the image bitstream based on the side information features of the image to be encoded and the first image feature set includes: Based on the side information features of the image to be encoded, obtain the distribution information of each data in the first image feature set; The image features are encoded based on the distribution information of each data in the first image feature set to obtain the image bitstream.

12. The method according to claim 11, characterized in that, The step of obtaining the side information bitstream based on the side information features of the image to be encoded includes: The probability of each data in the edge information feature is estimated based on the preset distribution information to obtain the probability of each data in the edge information feature. The edge information features are encoded based on the probability of each data point in the edge information features to obtain the edge information code stream.

13. An image decoding method, characterized in that, include: Obtain the edge information bitstream of the image to be decoded; Image side information features are obtained based on the side information bitstream. The image side information features are a second image feature set obtained by extracting features from the first image feature set corresponding to the image to be decoded based on a super prior coding neural network. The data volume of the second image feature set is less than that of the first image feature set. The super prior coding neural network is obtained by training constraints based on bit rate loss, reconstruction image distortion loss and preview image distortion loss. The preview image distortion loss represents the difference in similarity between the original training image and the preview image of the original training image. Obtain the image bitstream of the image to be decoded, wherein the image bitstream includes data from multiple channels of the image to be decoded; Decode the first predetermined number of channel data in the image bitstream to obtain the first decoded data; The first reconstructed image is obtained based on the image side information features and the first decoded data; Decode the second predetermined number of channel data in the image bitstream to obtain the second decoded data; The second reconstructed image is obtained based on the image side information features and the second decoded data.

14. The method according to claim 13, characterized in that, It also includes obtaining third decoded data by decoding all channel data of the image bitstream, and obtaining a third reconstructed image based on the side information features and the third decoded data.

15. The method according to claim 13, characterized in that, include: After acquiring the first reconstructed image, the first reconstructed image is sent to the display for display; After acquiring the second reconstructed image, the second reconstructed image is sent to the display for display to cover the first reconstructed image.

16. An image decoding apparatus, characterized in that, include: The acquisition module is used to acquire the side information bitstream of the image to be decoded; The acquisition module is used to acquire image side information features based on the side information bitstream. The image side information features are a second image feature set obtained by extracting features from the first image feature set corresponding to the image to be decoded based on the super prior coding neural network. The data volume of the second image feature set is less than the data volume of the first image feature set. The super prior coding neural network is trained based on bit rate loss, reconstruction image distortion loss and preview image distortion loss. The pre-decoding module is used to obtain a preview image of the image to be decoded based on the image side information features.

17. The apparatus according to claim 16, characterized in that, The acquisition module is further configured to acquire the image bitstream of the image to be decoded, the image bitstream including data from multiple channels of the image to be decoded; First decoded data is obtained based on the image bitstream, wherein the first decoded data is at least a portion of the image features of the first image feature set.

18. The apparatus according to claim 17, characterized in that, The acquisition module includes a priori decoding module and an entropy decoding module; and... The super-prior decoding module is used to determine the mean and variance of each data in the first image feature set based on the image side information features; The entropy decoding module is used to determine the probability of each data in the image feature based on the mean and variance of each data in the first image feature set. The entropy decoding module is used to obtain the first decoded data based on the probability corresponding to each data in the first image feature set and the image bitstream.

19. The apparatus according to claim 18, characterized in that, The acquisition module further includes a priori entropy estimation module; and... The super-prior entropy estimation module is used to obtain the probability corresponding to each data in the image side information features based on the side information bitstream; The entropy decoding module is used to decode the edge information bitstream according to the probability corresponding to each data in the image edge information features to obtain the image edge information features.

20. The apparatus according to any one of claims 17-19, characterized in that, The pre-decoding module is used to obtain a first preview image based on the image edge information features; or, it is used to obtain a second preview image based on the image edge information features and the first decoding data. The similarity between the second preview image and the image to be decoded is greater than the similarity between the first preview image and the image to be decoded.

21. The apparatus according to any one of claims 16-19, characterized in that, The pre-decoding module is a pre-decoding network, which includes an image scaling operator; Alternatively, the pre-decoding network can be used to obtain a preview image of a set size.

22. An image encoding device, characterized in that, include The feature extraction module is used to acquire the image to be encoded and to acquire a first set of image features of the image to be encoded. A super-prior coding module is used to obtain the side information features of the image to be encoded based on a first image feature set of the image to be encoded; the side information features are a second image feature set obtained by feature extraction of the first image feature set by the super-prior coding module, the data size of the second image feature set is smaller than that of the first image feature set, and the super-prior coding module is obtained by training constraints based on bit rate loss, reconstruction image distortion loss and preview image distortion loss, wherein the preview image distortion loss represents the difference in similarity between the original training image and the preview image of the original training image; The feature encoding module obtains an image bitstream based on the side information features of the image to be encoded and the first image feature set, and obtains a side information bitstream based on the side information features of the image to be encoded.

23. The apparatus according to claim 22, characterized in that, The feature encoding module includes a super-prior decoding module and an entropy encoding module; The advanced prior decoding module is used to obtain the mean and variance of each data in the first image feature set based on the side information features of the image to be encoded; The entropy coding module is used to perform entropy coding on the first image feature set based on the mean and variance of each data in the image features to obtain the image bitstream.

24. The apparatus according to claim 23, characterized in that, The feature encoding module includes a priori entropy estimation module; The prior entropy estimation module is used to estimate the probability of each data in the edge information feature based on the preset distribution information, and to obtain the probability of each data in the edge information feature. The entropy encoding module is used to encode the side information features based on the probability of each data in the side information features to obtain the side information code stream.

25. An image decoding device, characterized in that, include: The acquisition module is used to acquire the side information bitstream of the image to be decoded; The acquisition module is used to acquire image side information features based on the side information bitstream. The image side information features are a second image feature set obtained by extracting features from a first image feature set corresponding to the image to be decoded based on a super-prior coding neural network. The data volume of the second image feature set is less than that of the first image feature set. The super-prior coding neural network is obtained by training constraints based on bit rate loss, reconstructed image distortion loss, and preview image distortion loss. The preview image distortion loss represents the difference in similarity between the original training image and the preview image of the original training image. The acquisition module is used to acquire the image bitstream of the image to be decoded, the image bitstream including data from multiple channels of the image to be decoded; The acquisition module is used to decode a first set number of channel data in the image bitstream to obtain first decoded data; The decoding module is used to obtain a first reconstructed image based on the image side information features and the first decoded data; The acquisition module is used to decode a second set number of channel data in the image bitstream to obtain second decoded data; The decoding module is used to obtain a second reconstructed image based on the image side information features and the second decoding data.

26. The apparatus according to claim 25, characterized in that... The decoding module is used to obtain a third reconstructed image based on the side information features and the third decoded data after decoding all channel data of the image bitstream to obtain the third decoded data.

27. The apparatus according to claim 25, characterized in that, The decoding module is used to send the first reconstructed image to the display for display after acquiring the first reconstructed image; The decoding module is used to send the second reconstructed image to the display after acquiring the second reconstructed image to cover the first reconstructed image.

28. An electronic device, characterized in that, The image decoding apparatus includes any one of claims 16-21 and / or any one of claims 22-24 and / or any one of claims 25-27.

29. A decoding apparatus, wherein the decoding apparatus is an encoding apparatus or a decoding apparatus, the decoding apparatus comprising: One or more processors and a memory, the memory being used to store program instructions that, when executed by the one or more processors, cause the image decoding method of any one of claims 1-9, the image encoding method of any one of claims 10-12, or the image decoding method of any one of claims 13-14 to be implemented.

30. A readable storage medium, characterized in that, The readable storage medium stores instructions that, when executed on an electronic device, cause the electronic device to perform the image decoding method of any one of claims 1-9, the image encoding method of any one of claims 10-12, or the image decoding method of any one of claims 13-15.

31. A computer program product, characterized in that, The method includes instructions that, when executed on an electronic device, cause the electronic device to perform the image decoding method of any one of claims 1-9, the image encoding method of any one of claims 10-12, or the image decoding method of any one of claims 13-15.

32. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores an image bitstream and a side information bitstream obtained by an encoding method as described in any one of claims 10 to 12, executed by one or more processors.

33. A decoding device, characterized in that, Including memory and decoder, The memory is used to store image bitstreams and side information bitstreams; The decoder is used to perform the image decoding method as described in any one of claims 1-9 or 13-15.

Images