Image compression model training, image encoding, image decoding method and device

Abstract

The application relates to an image compression model training, image coding and image decoding method and device, which is applied to the field of image compression and comprises the following steps: acquiring a training sample set comprising multiple training images; constructing an initial image compression model, which comprises an analysis transformation network, a quantization compression module and a synthesis transformation network; the quantization compression module performs soft quantization processing, entropy coding processing, entropy decoding processing and quantization compensation processing on the spatial features after analysis transformation, so that quantized and compressed spatial features are obtained; the quantization compensation processing is used to reduce the difference between the soft quantization processing in the training process and the hard quantization processing in the test process; each training image in the training sample set is input into the initial image compression model, so that a reconstructed image corresponding to each training image is obtained; and the initial image compression model is trained based on each training image and the reconstructed image corresponding to each training image, so that a target image compression model is obtained.

Description

Technical Field

[0001] Some embodiments of this application relate to the field of image compression. More specifically, they relate to an image compression model training, image encoding, image decoding method, apparatus, and terminal device. Background Technology

[0002] Deep learning-based end-to-end image coding techniques involve three steps: transform, quantization, and entropy coding. In this end-to-end learning-based image compression framework, quantization, as a core element, has a crucial impact on coding performance. Quantization converts the transformed feature activation values ​​from floating-point numbers to regular fixed-point numbers, which serve as input for subsequent entropy coding.

[0003] Currently, hard quantization is used for testing end-to-end compressed networks, with the most common hard quantization function being the rounding function, which is non-differentiable. However, the target distortion function primarily uses gradient descent to optimize rate-distortion in end-to-end encoding, requiring the quantization function to be globally differentiable during backpropagation. Therefore, research on image compression based on end-to-end learning has focused on solving the problem of non-differentiability of quantization (quantization is discontinuous, its derivative is zero or infinite everywhere). Therefore, to avoid affecting the backpropagation of gradients during the training of end-to-end compressed networks, a soft quantization function, which is equivalent to and differentiable than the hard quantization function, is used during the training process (i.e., soft quantization).

[0004] However, although soft quantization functions and hard quantization functions are equivalent and approximate, they are not completely identical and there are still some differences. These differences inevitably lead to a mismatch between the training and testing processes, which in turn results in the encoding and decoding performance of the end-to-end compressed network during testing being inconsistent with the encoding and decoding performance of the end-to-end compressed network obtained during training. Summary of the Invention

[0005] To address or at least partially address the aforementioned technical problems, some embodiments of this application provide an image compression model training, image encoding, and image decoding method, apparatus, and terminal device, which can reduce the mismatch between the training and testing processes, improve the encoding performance of the end-to-end compression network during the testing process, and ensure consistency with the encoding and decoding performance of the trained end-to-end compression network.

[0006] In a first aspect, some embodiments of this application provide an image compression model training method, comprising: acquiring a training sample set, the training sample set including multiple training images; constructing an initial image compression model, the initial image compression model including an analysis transform network, a quantization compression module, and a synthesis transform network; the analysis transform network being used to perform analysis transform processing on the input training images to obtain analyzed transform spatial features; the quantization compression module performing soft quantization processing, entropy encoding processing, entropy decoding processing, and quantization compensation processing on the analyzed transform spatial features to obtain quantized compressed spatial features; the quantization compensation processing being used to reduce the difference between the soft quantization processing in the training process and the hard quantization processing in the testing process; the synthesis transform network being used to perform synthesis transform processing on the quantized compressed spatial features to obtain a reconstructed image corresponding to the training image; inputting each training image in the training sample set into the initial image compression model to obtain a reconstructed image corresponding to each training image; and training the initial image compression model based on each training image and the reconstructed image corresponding to each training image to obtain a trained target image compression model.

[0007] Secondly, some embodiments of this application provide an image encoding method applied to an encoding device, comprising: acquiring an image to be encoded; inputting the image to be encoded into a trained analysis and transformation network for analysis and transformation processing to obtain analyzed and transformed spatial features; inputting the analyzed and transformed spatial features into a hard quantization module for hard quantization processing to obtain quantized spatial features; inputting the quantized spatial features into a trained entropy encoding module for entropy encoding processing to obtain an entropy-encoded binary bitstream; and sending the entropy-encoded binary bitstream to a decoding device; wherein the analysis and transformation network and the entropy encoding module are respectively modules corresponding to the analysis and transformation network and the entropy encoding processing of the quantization compression module in the image compression model trained by any image compression model training method in the first aspect.

[0008] Thirdly, some embodiments of this application provide an image decoding method applied to a decoding device, comprising: receiving an entropy-encoded binary code stream from an encoding device, wherein the entropy-encoded binary code stream is obtained by the encoding device after performing analysis and transformation processing, hard quantization processing, and entropy encoding processing on an image to be encoded; inputting the entropy-encoded binary code stream into a trained entropy decoding module for entropy decoding processing to obtain entropy-decoded spatial features; and inputting the entropy-decoded spatial features into a trained synthesis transformation network for synthesis transformation processing to obtain a reconstructed image corresponding to the image to be encoded; wherein the entropy decoding module and the synthesis transformation network are respectively the module corresponding to the entropy decoding processing of the quantization compression module and the synthesis transformation network in the image compression model trained by any of the image compression model training methods in the first aspect.

[0009] Fourthly, some embodiments of this application provide an image compression model training apparatus, including: a memory configured to store a training sample set; and one or more processors configured to: acquire the training sample set, which includes multiple training images; construct an initial image compression model, which includes an analysis transform network, a quantization compression module, and a synthesis transform network; the analysis transform network is used to perform analysis transform processing on the input training images to obtain encoded spatial features; the quantization compression module performs soft quantization processing, entropy encoding processing, entropy decoding processing, and quantization compensation processing on the encoded spatial features to obtain quantized compressed spatial features; the quantization compensation processing is used to reduce the difference between the soft quantization processing in the training process and the hard quantization processing in the testing process; the synthesis transform network is used to perform synthesis transform processing on the quantized compressed spatial features to obtain a reconstructed image corresponding to the training image; input each training image in the training sample set into the initial image compression model to obtain a reconstructed image corresponding to each training image; and train the initial image compression model based on each training image and the reconstructed image corresponding to each training image to obtain a trained target image compression model.

[0010] Fifthly, some embodiments of this application provide an image encoding apparatus, including: a memory configured to store an image to be encoded; and one or more processors configured to: acquire the image to be encoded; input the image to be encoded into a trained analysis and transformation network for analysis and transformation processing to obtain analyzed and transformed spatial features; input the analyzed and transformed spatial features into a hard quantization module for hard quantization processing to obtain quantized spatial features; input the quantized spatial features into a trained entropy encoding module for entropy encoding processing to obtain an entropy-encoded binary bitstream; and send the entropy-encoded binary bitstream to a decoding device; wherein the analysis and transformation network and the entropy encoding module are respectively modules corresponding to the analysis and transformation network and the entropy encoding processing of the quantization compression module in the image compression model trained by the image compression model training method described in the first aspect.

[0011] In a sixth aspect, some embodiments of this application provide an image decoding apparatus, including: a memory configured to store a reconstructed image; and one or more processors configured to: receive an entropy-encoded binary bitstream from an encoding device, the entropy-encoded binary bitstream being obtained by the encoding device after performing analysis and transformation processing, hard quantization processing, and entropy encoding processing on the image to be encoded; input the entropy-encoded binary bitstream into a trained entropy decoding module for entropy decoding processing to obtain entropy-decoded spatial features; and input the entropy-decoded spatial features into a trained synthesis transformation network for synthesis transformation processing to obtain a reconstructed image corresponding to the image to be encoded; wherein the entropy decoding module and the synthesis transformation network are respectively the module corresponding to the entropy decoding processing of the quantization compression module and the synthesis transformation network in the image compression model trained by the image compression model training method described in the first aspect.

[0012] In a seventh aspect, some embodiments of this application provide a terminal device, including: an image compression model training apparatus as described in the fourth aspect, or an image encoding apparatus as described in the fifth aspect, or an image decoding apparatus as described in the sixth aspect.

[0013] Eighthly, some embodiments of this application provide a computer-readable storage medium, including: storing a computer program on the computer-readable storage medium, wherein when the computer program is executed by a processor, it implements the image compression model training method as shown in the first aspect, the image encoding method as shown in the second aspect, or the image decoding method as shown in the third aspect.

[0014] Ninthly, some embodiments of this application provide a computer program product, including: when the computer program product is run on a computer, causing the computer to implement the image compression model training method as shown in the first aspect, the image encoding method as shown in the second aspect, or the image decoding method as shown in the third aspect.

[0015] Compared with related technologies, the technical solutions provided in some embodiments of this application have the following advantages: In some embodiments of this application, by adding quantization compensation processing to the quantization compression module to reduce the difference between the soft quantization processing in the training process and the hard quantization processing in the testing process, the difference between the soft quantization processing and the hard quantization processing can be compensated through quantization compensation processing, thereby reducing the mismatch between the training process and the testing process caused by the difference between the soft quantization processing and the hard quantization processing, improving the encoding and decoding performance of the end-to-end compression network in the testing process, and the consistency with the encoding and decoding performance of the end-to-end compression network obtained from training, thereby improving the encoding and decoding performance of the end-to-end compression network.

[0016] In some embodiments of this application, the analysis and transformation network and the entropy coding processing module of the image compression model trained by the image compression model training method are used to encode the image to be encoded, which can improve the encoding performance of the image.

[0017] In some embodiments of this application, the quantization compression module in the image compression model trained by the image compression model training method is used to perform entropy decoding processing, and the synthesis transform network is used to decode the entropy-encoded binary code stream, which can improve the decoding performance of the image. Attached Figure Description

[0018] To more clearly illustrate the implementation methods in some embodiments or related technologies of this application, the accompanying drawings used in the description of the embodiments or related technologies will be briefly introduced below. Obviously, the accompanying drawings described below are some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings.

[0019] Figure 1 A block diagram of a video decoding and encoding system according to some embodiments is shown;

[0020] Figure 2 A flowchart illustrating an image compression model training method according to some embodiments is shown;

[0021] Figure 3 This shows one of the structural schematic diagrams of an image compression model according to some embodiments;

[0022] Figure 4 A second schematic diagram of the structure of an image compression model according to some embodiments is shown;

[0023] Figure 5 The third schematic diagram of the structure of an image compression model according to some embodiments is shown;

[0024] Figure 6 The fourth schematic diagram of the structure of an image compression model according to some embodiments is shown;

[0025] Figure 7 The fifth schematic diagram of the structure of an image compression model according to some embodiments is shown;

[0026] Figure 8 Sixth schematic diagram of the structure of an image compression model according to some embodiments is shown;

[0027] Figure 9 The seventh schematic diagram of the structure of an image compression model according to some embodiments is shown;

[0028] Figure 10Eighth schematic diagram of the structure of an image compression model according to some embodiments is shown;

[0029] Figure 11 A structural schematic diagram of an image compression model according to some embodiments is shown as diagram nine;

[0030] Figure 12 A schematic diagram of the structure of an image compression model according to some embodiments is shown.

[0031] Figure 13 Eleventh of the structural schematic diagrams of an image compression model according to some embodiments is shown;

[0032] Figure 14 A schematic diagram of the hard quantization function and the quantization error function according to some embodiments is shown;

[0033] Figure 15 A schematic diagram of the first K harmonics of the Taylor series expansion of the quantization error function according to some embodiments is shown.

[0034] Figure 16 A flowchart illustrating an image encoding / decoding test method according to some embodiments is shown;

[0035] Figure 17 A flowchart illustrating an image encoding method according to some embodiments is shown;

[0036] Figure 18 A flowchart illustrating an image decoding method according to some embodiments is shown;

[0037] Figure 19 A framework diagram of an image compression model training apparatus, an image encoding apparatus, or an image decoding apparatus according to some embodiments is shown. Detailed Implementation

[0038] To make the objectives and implementation methods of this application clearer, the exemplary implementation methods of this application will be clearly and completely described below with reference to the accompanying drawings of the exemplary embodiments of this application. Obviously, the exemplary embodiments described are only some embodiments of this application, and not all embodiments.

[0039] It should be noted that the brief descriptions of terms in this application are only for the convenience of understanding the embodiments described below, and are not intended to limit the embodiments of this application. Unless otherwise stated, these terms should be understood in their ordinary and common meaning.

[0040] The terms "first," "second," "third," etc., used in the specification, claims, and accompanying drawings of this application are used to distinguish similar or related objects or entities, and do not necessarily imply a specific order or sequence, unless otherwise specified. It should be understood that such terms are interchangeable where appropriate.

[0041] The terms “comprising” and “having”, and any variations thereof, are intended to cover but not exclude inclusion, for example, a product or device that includes a range of components is not necessarily limited to all of the components that are clearly listed, but may include other components that are not clearly listed or that are inherent to such product or device.

[0042] Uncompressed raw image data is massive and difficult to use later. Therefore, it needs to be compressed into a binary bitstream through image encoding for subsequent transmission and processing. Currently, traditional image encoding frameworks have evolved over decades, forming hybrid encoding frameworks that include modules such as prediction, transformation, quantization, and entropy coding. Each module is typically designed manually, requiring separate optimization, making joint tuning of the entire encoding framework impossible and limiting performance improvements. In recent years, end-to-end image encoding technology based on deep learning has attracted widespread attention from academia and industry. Compared to traditional methods, end-to-end image compression methods not only jointly optimize network parameters but also avoid complex manual algorithm design. By intelligently learning to remove redundant information from the image, the network achieves better results in terms of compression ratio and visual quality.

[0043] With the explosive growth of internet data, image data compression is essential under limited network bandwidth. The compression process is essentially a rate-distortion optimization problem, which uses R+λD to achieve a trade-off between bit rate R and distortion D (λ is the Lagrange multiplier that adjusts the trade-off). The goal is to minimize the coding bit rate while maintaining a certain level of quality, or to minimize coding distortion within a certain coding bit rate limit.

[0044] Figure 1 This is a block diagram illustrating an example video encoding and decoding system 10 that can utilize the techniques described in this application. Figure 1 As shown, system 10 includes source device 12 (corresponding to...) Figure 1 The encoding device 200 in the middle, whose generation is to be subsequently handled by the destination device 14 (corresponding to Figure 1The decoding device 300 decodes the encoded video data. The source device 12 and destination device 14 can include any of a wide variety of devices, including desktop computers, laptop computers, tablet computers, set-top boxes, telephones such as so-called "smart" phones, so-called "smart" tablets, televisions, cameras, display devices, digital media players, video game consoles, video streaming devices, and so on. In some cases, the source device 12 and destination device 14 may be equipped with wireless communication.

[0045] Destination device 14 can receive encoded video data to be decoded via link 16. Link 16 may include any type of medium or device capable of moving encoded video data from source device 12 to destination device 14. In one example, link 16 may include a communication medium enabling source device 12 to transmit encoded video data directly to destination device 14 in real time. The encoded video data may be modulated according to communication standards such as wireless communication protocols and transmitted to destination device 14. The communication medium may include any wireless or wired communication medium, such as radio frequency (RF) spectrum or one or more physical transmission lines. The communication medium may form part of a packet-based network, such as a local area network, a wide area network, or a global network such as the Internet. The communication medium may include a router, switch, base station, or any other device that may be used to facilitate communication from source device 12 to destination device 14.

[0046] In another example, the encoded data can be output from output interface 22 to storage device 26. Similarly, the encoded data can be accessed from storage device 26 via input interface. Storage device 26 can include any of a variety of distributed or locally accessed data storage media, such as hard disk drives, Blu-ray discs, DVDs, CD-ROMs, flash memory, volatile or non-volatile memory, or any other suitable digital storage medium for storing encoded video data. In another example, storage device 26 can correspond to a file server or another intermediate storage device that can hold the encoded video generated by source device 12. Destination device 14 can access the stored video data from storage device 26 via streaming or downloading. The file server can be any type of server capable of storing and sending the encoded video data to destination device 14. Example file servers include web servers (e.g., for websites), FTP servers, network attached storage (NAS) devices, or local disk drives. Destination device 14 can access the encoded video data via any standard data connection including an Internet connection. This may include a wireless channel (e.g., Wi-Fi connection), a wired connection (e.g., DSL, cable modem, etc.), or a combination of both, suitable for accessing encoded video data stored on a file server. The transfer of encoded video data from storage device 26 may be streaming, downloading, or a combination of both.

[0047] The technology described in this application is not necessarily limited to wireless applications or setups. It can be applied to video encoding that supports any of a variety of multimedia applications, such as over-the-air television broadcasting, cable television transmission, satellite television transmission, streaming video transmission via the Internet, encoding digital video stored on data storage media, decoding digital video stored on data storage media, or other applications. In some examples, system 10 can be configured to support one-way or two-way video transmission to support applications such as video streaming, video playback, video broadcasting, and / or video telephony.

[0048] exist Figure 1In the example, source device 12 includes a video source 18, a video encoder 20, and an output interface 22. In some cases, the output interface 22 may include a modulator / demodulator (modem) and / or a transmitter. In source device 12, the video source 18 may include, for example, a video capture device (e.g., a camera), a video feed interface including a pre-captured video archive, a video feed interface for receiving video from a video content provider, and / or a computer graphics system for generating computer graphics data as source video, or a combination of these sources. As an example, if the video source 18 is a camera, then source device 12 and destination device 14 may form a so-called videophone or video call. However, the techniques described in this application are generally applicable to video encoding and can be applied to wireless and / or wired applications.

[0049] Captured, pre-captured, or computer-generated video can be encoded by video encoder 20. The encoded video data can be sent directly to destination device 14 via output interface 22 of source device 12. The encoded video data can also (or alternatively) be stored on storage device 26 for later access by destination device 14 or other devices for decoding and / or playback.

[0050] Destination device 14 includes an input interface 28, a video decoder 30, and a display device 32. In some cases, the input interface 28 may include a receiver and / or a modem. The input interface 28 of destination device 14 receives encoded video data via link 16. The encoded video data transmitted via link 16 or provided on storage device 26 may include various syntax elements generated by video encoder 20 for use by video decoder (such as video decoder 30) when decoding the video data. Such syntax elements may be included in the encoded video data transmitted on a communication medium, stored on a storage medium, or stored on a file server.

[0051] Display device 32 may be integrated with destination device 14 or located outside of destination device 14. In some examples, destination device 14 may include an integrated display device and be configured to interface with an external display device. In other examples, destination device 14 may be a display device. In general, display device 32 displays decoded video data to a user and may include any of a variety of display devices, such as a liquid crystal display (LCD), a plasma display, an organic light-emitting diode (OLED) display, or another type of display device.

[0052] This application does not limit the video compression standards applied to the video encoder 20 and the video decoder 30, the encoding circuit or encoding model corresponding to the video encoder 20, and the encoding circuit or encoding model corresponding to the video decoder 30, etc., in some embodiments.

[0053] Quantization: This involves the ratio of the variable to be quantized to the quantization step size, followed by rounding. This process is also known as hard quantization. In deep neural network image coding techniques, the quantization step size is 1, so hard quantization is essentially a rounding process (i.e., hard quantization uses the round function). The round function is not differentiable, making it impossible to train an end-to-end compressed network based on hard quantization.

[0054] In the training process of end-to-end compression networks, it is necessary to ensure that the entire process is differentiable and continuous to achieve end-to-end training. However, hard quantization is not differentiable. Therefore, if hard quantization is used for model training, it will affect the backpropagation of gradients during the network learning process, thus bringing great difficulties to end-to-end training. Therefore, soft quantization, which is equivalent to hard quantization, is used in the training process of image compression models. Thus, the primary task of current end-to-end image compression algorithms is to design an efficient and trainable quantization scheme, that is, how to obtain a soft quantization scheme that is closest to and differentiable than the hard quantization scheme.

[0055] Although some soft quantization schemes have been proposed, such as additive uniform noise, pass-through estimators, or soft-to-hard algorithms, which are equivalent to hard quantization, there are still some differences between soft quantization functions and hard quantization functions. They are not completely consistent, and the differences between the two inevitably lead to a mismatch between the training and testing processes. This results in the encoding and decoding performance of the end-to-end compressed network during the testing process being inconsistent with the encoding and decoding performance of the end-to-end compressed network obtained during training.

[0056] To address the aforementioned technical issues, one current approach employs a two-stage training method. First, an end-to-end compressed network is trained using soft quantization. After obtaining the trained compressed network, the parameters of the analytical transform network within the compressed network are fixed. Then, the soft quantization process is replaced with hard quantization. Based on this hard quantization, the network parameters of the entropy encoding module, entropy decoding module, and synthetic transform network in the compressed network are trained again. (At this point, since the parameters of the analytical transform network are fixed, further training is unnecessary, and gradients from the network learning process no longer need to propagate to the analytical transform network. Therefore, quantization differentiability is not required, and the network parameters of the entropy encoding module, entropy decoding module, and synthetic transform network in the compressed network can be trained again based on hard quantization.) Ultimately, this ensures that the training and testing processes of the trained compressed network match. However, this two-stage training method requires two training phases to obtain a compressed network that matches the training and testing processes, making the training process complex and time-consuming.

[0057] To address the aforementioned technical problems, in some embodiments of this application, a quantization compensation process is added to the quantization compression module to reduce the difference between the soft quantization process during training and the hard quantization process during testing. Therefore, the quantization compensation process can compensate for the difference between the soft and hard quantization processes, thereby reducing the mismatch between the training and testing processes caused by this difference. This improves the consistency between the end-to-end compression network encoding / decoding performance during testing and the training-obtained end-to-end compression network encoding / decoding performance, thus enhancing the overall encoding / decoding performance of the end-to-end compression network.

[0058] In some embodiments of this application, the executing entity of the image compression model training method can be a server or an electronic device, or it can be a functional module or functional entity in the server or electronic device that can implement the image compression model training method, which can be determined according to the time situation.

[0059] The aforementioned servers can be a single server cluster or multiple server clusters, and can include one or more types of servers. Electronic devices can be televisions, smart televisions, laser projectors, monitors, electronic bulletin boards, electronic tables, mobile phones, tablets, laptops, PDAs, in-vehicle electronic devices, etc., without specific limitations.

[0060] Figure 2 The flowchart illustrates the steps of an image compression model training method according to one or more embodiments of this application. The image compression model training method may include the following steps S401 to S404.

[0061] S401. Obtain the training sample set.

[0062] The training sample set includes multiple training images. The training sample set can be an existing training sample set, such as the MS COCO dataset (from Microsoft COCO: Common objects in context) and / or the ImageNet2017 dataset (from ImageNet: A large-scale hierarchical image database), with images cropped to a preset size (e.g., 256×256) as the training sample set; alternatively, a newly collected training sample set can be used, depending on the specific circumstances, and is not limited here.

[0063] S402. Construct the initial image compression model.

[0064] The initial image compression model includes an analysis transform network, a quantization compression module, and a synthesis transform network. The analysis transform network performs analysis transform processing on the input training image to obtain the analyzed transform spatial features. The quantization compression module performs soft quantization, entropy encoding, entropy decoding, and quantization compensation processing on the analyzed transform spatial features to obtain quantized compressed spatial features. The quantization compensation processing is used to reduce the difference between the soft quantization processing during training and the hard quantization processing during testing. The synthesis transform network performs synthesis transform processing on the quantized compressed spatial features to obtain the reconstructed image corresponding to the training image.

[0065] It should be noted that in some embodiments of this application, the specific neural network structure used by the analysis transformation network, quantization compression module and synthesis transformation network is not limited. For example, the analysis transformation network, the number of layers of convolutional networks used by the quantization compression module and synthesis transformation network, whether to use a bidirectional long short-term memory network (LSTM), whether to use a self-attention mechanism, etc., can be designed according to actual needs.

[0066] For example, such as Figure 3 The diagram shows a schematic of the structure of the initial image compression model provided in this application embodiment. The initial image compression model includes an analysis transform network 501, a quantization compression module 502, and a synthesis transform network 503. The quantization compression module 502 includes quantization processing, compression processing, and quantization compensation processing.

[0067] The analytical transform network 501 maps an image from the data domain to the coding domain, generating lower-dimensional latent spatial features. These lower-dimensional latent spatial features are then quantized to obtain discrete spatial features. The analytical transform network is used to transform an image from pixel space to a low-redundancy feature space (i.e., from the data domain to the coding domain), removing spatial correlations between pixels and thus reducing spatial redundancy in the image, resulting in the analyzed spatial features.

[0068] The quantization compression module 502 includes quantization processing, entropy encoding processing, entropy decoding processing, and quantization compensation processing. Quantization processing discretizes the analyzed spatial features, effectively reducing the value space of the spatial features and achieving better compression performance, resulting in quantized spatial features. This operation is irreversible and is a major source of image distortion. Entropy encoding processing, based on the principle of information entropy, converts the quantized spatial features into an entropy-encoded binary code stream for transmission or storage. Entropy decoding processing performs entropy decoding on the entropy-encoded binary code stream to obtain entropy-decoded spatial features, which correspond to the discrete spatial features. Quantization compensation processing compensates for the difference between soft quantization and hard quantization, making soft quantization closer to hard quantization and thus reducing the difference between the two.

[0069] The synthetic transform network 503 reinterprets discrete spatial features as continuous spatial features, that is, it maps from the coding domain back to the data domain through a synthetic transform function (analyzing the inverse process of the transform) to obtain the reconstructed image.

[0070] It is understood that in some embodiments of this application, quantization compensation processing is used in the quantization compression module, which can reduce the difference between soft quantization processing in the training process of the end-to-end image compression model and hard quantization processing in the testing process of the end-to-end image compression model, thereby reducing the mismatch between the training process and the testing process and improving the encoding and decoding performance of the end-to-end compression network.

[0071] S403. Input each training image in the training sample set into the initial image compression model to obtain the reconstructed image corresponding to each training image.

[0072] S404. Based on each training image and the corresponding reconstructed image, train the initial image compression model to obtain the trained target image compression model.

[0073] In some embodiments of this application, the initial image compression model can be trained based on each training image, the corresponding reconstructed image, and the target loss function to obtain a trained target image compression model. The target loss function can be determined according to actual circumstances and is not limited here.

[0074] For example, the target loss function is constructed based on a rate-distortion function, which is a weighted sum of bit rate and distortion, where distortion is determined based on mean squared error (MSE). For instance, the target loss function is:

[0075]

[0076] Where L is the target loss function, L rate For bitrate, The distortion is determined based on the mean square error (MSE), and λ is the Lagrange daily number for adjusting the bit rate and distortion.

[0077] In some embodiments of this application, based on each training image and the corresponding reconstructed image, as well as the target loss function, the initial image compression model is trained using an optimizer and a learning decay strategy to obtain a target image compression model trained on the training sample set.

[0078] The optimizer can be determined according to the actual situation and is not limited here. For example, it can be the Stochastic Gradient Descent (SGD) optimizer, the Stochastic Gradient Descent with Momentum (SGDM) optimizer, the Adagrad optimizer, the RMSProp optimizer, the Adam optimizer, etc.

[0079] The learning rate is a crucial hyperparameter in neural network optimization. In gradient descent, the value of the learning rate α is critical; too large a value will prevent convergence, while too small a value will result in slow convergence. A common strategy in neural network training is to use a learning rate update strategy, where the learning rate gradually decays with each iteration of model training. This balances learning efficiency with later-stage stability: initially, a large learning rate is used for rapid searching to find a suitable (more likely globally minimum) local region; later, a smaller learning rate is used to converge within this local region. The main learning rate update strategies include: piecewise constant decay, exponential decay, natural exponential decay, polynomial decay, cosine decay, and reciprocal decay. The specific strategy can be chosen based on the specific circumstances and is not limited here.

[0080] In some embodiments of this application, based on each training image and the corresponding reconstructed image, a target loss function is used to calculate the reconstruction distortion. The initial image compression model is iteratively trained according to the reconstruction distortion, and the network parameters of the image compression model are corrected one by one until the reconstruction distortion reaches the convergence condition, thus obtaining the trained target image compression model.

[0081] In some embodiments of this application, by adding a quantization compensation process to the quantization compression module to reduce the difference between the soft quantization process during training and the hard quantization process during testing, the difference between the soft and hard quantization processes can be compensated, thereby reducing the mismatch between the training and testing processes caused by the difference between the soft and hard quantization processes, improving the encoding and decoding performance of the end-to-end compressed network during testing, and ensuring consistency with the encoding and decoding performance of the end-to-end compressed network obtained during training, thus improving the encoding and decoding performance of the end-to-end compressed network.

[0082] In some embodiments of this application, the quantization compensation processing included in the quantization compression module can be placed before the quantization processing to compensate for the difference between soft quantization processing during model training and hard quantization processing during model testing or practical application. Alternatively, the quantization compensation processing included in the quantization compression module can be placed after the entropy decoding processing to compensate for the difference between soft quantization processing during model training and hard quantization processing during model testing or practical application. Quantization compensation processing can also be performed before the quantization processing and after the entropy decoding processing in the quantization compression module to compensate for the difference between soft quantization processing during model training and hard quantization processing during model testing or practical application. This makes the impact of soft quantization processing on each module of the image compression model during training more closely resemble the impact of hard quantization processing on each module of the image compression model during testing or practical application, thereby improving the encoding and decoding performance of the image compression model.

[0083] In some embodiments of this application, the quantization compression module includes a quantization compensation module, a quantization module, an entropy encoding module, and an entropy decoding module. The quantization compensation module is used to perform quantization compensation processing on the spatial features after analysis and transformation to obtain compensated spatial features. The quantization module is used to perform quantization processing on the compensated spatial features to obtain quantized spatial features. The entropy encoding module is used to perform entropy encoding processing on the quantized spatial features to obtain an entropy-encoded binary code stream. The entropy decoding module is used to perform entropy decoding processing on the entropy-encoded binary code stream to obtain entropy-decoded spatial features, which are the quantized and compressed spatial features.

[0084] For example, combined Figure 3 ,like Figure 4As shown, the quantization compression module 502 includes a quantization compensation module 502a, a quantization module 502b, an entropy encoding module 502c, and an entropy decoding module 502d. The input of the quantization compensation module 502a is the input of the quantization compression module 502, and the output of the quantization compensation module 502a is the input of the quantization module 502b. The output of the quantization module 502b is the input of the entropy encoding module 502c, the output of the entropy encoding module 502c is the input of the entropy decoding module 502d, and the output of the entropy decoding module 502d is the output of the quantization compression module 502. That is, before quantizing the transformed spatial features, quantization compensation processing is performed on the transformed spatial features. Through quantization compensation processing, the difference between the soft quantization processing during training and the hard quantization processing during testing is reduced, thereby improving the encoding and decoding performance of the end-to-end compression network.

[0085] In some embodiments of this application, the quantization compression module includes a quantization compensation module, a quantization module, an entropy encoding module, and an entropy decoding module. The quantization module is used to quantize the spatial features after analysis and transformation to obtain quantized spatial features. The entropy encoding module is used to entropy encode the quantized spatial features to obtain an entropy-encoded binary code stream. The entropy decoding module is used to entropy decode the entropy-encoded binary code stream to obtain entropy-decoded spatial features. The quantization compensation module is used to quantize and compensate the entropy-decoded spatial features to obtain the quantized and compressed spatial features.

[0086] For example, combined Figure 3 ,like Figure 5 As shown, the quantization compression module 502 includes a quantization module 502b, an entropy encoding module 502c, an entropy decoding module 502d, and a quantization compensation module 502e. The input of the quantization module 502b is the input of the quantization compression module 502, the output of the quantization module 502b is the input of the entropy encoding module 502c, the output of the entropy encoding module 502c is the input of the entropy decoding module 502d, the output of the entropy decoding module 502d is the input of the quantization compensation module 502e, and the output of the quantization compensation module 502e is the output of the quantization compression module 502. That is, after sequentially performing quantization, entropy encoding, and entropy decoding on the analyzed and transformed spatial features, quantization compensation is then performed on the entropy-decoded spatial features. Through quantization compensation, the difference between the soft quantization process during training and the hard quantization process during testing is reduced, thereby improving the encoding and decoding performance of the end-to-end compression network.

[0087] In some embodiments of this application, the quantization compression module includes a first quantization compensation module, a second quantization compensation module, a quantization module, an entropy encoding module, and an entropy decoding module. The first quantization compensation module is used to perform quantization compensation processing on the spatial features after analysis and transformation to obtain compensated spatial features. The quantization module is used to perform quantization processing on the compensated spatial features to obtain quantized spatial features. The entropy encoding module is used to perform entropy encoding processing on the quantized spatial features to obtain an entropy-encoded binary code stream. The entropy decoding module is used to perform entropy decoding processing on the entropy-encoded binary code stream to obtain entropy-decoded spatial features. The second quantization compensation module is used to perform quantization compensation processing on the entropy-decoded spatial features to obtain the quantized compressed spatial features.

[0088] For example, combined Figure 3 ,like Figure 6 As shown, the quantization compression module 502 includes a quantization compensation module 502a (corresponding to the first quantization compensation module), a quantization module 502b, an entropy encoding module 502c, an entropy decoding module 502d, and a quantization compensation module 502e (corresponding to the second quantization compensation module). The input of the quantization compensation module 502a is the input of the quantization compression module 502, the output of the quantization compensation module 502a is the input of the quantization module 502b, the output of the quantization module 502b is the input of the entropy encoding module 502c, the output of the entropy encoding module 502c is the input of the entropy decoding module 502d, the output of the entropy decoding module 502d is the input of the quantization compensation module 502e, and the output of the quantization compensation module 502e is the output of the quantization compression module 502. That is, before quantizing the spatial features after analysis and transformation, quantization compensation processing is performed on the spatial features after analysis and transformation. Then, after quantization, entropy encoding and entropy decoding are performed in sequence, quantization compensation processing is performed on the spatial features after entropy decoding. Through two quantization compensation processes, the difference between the soft quantization process in the training process and the hard quantization process in the testing process is reduced, thereby improving the encoding and decoding performance of the end-to-end compression network.

[0089] In some embodiments of this application, the analysis transformation network, quantization compression module, and synthesis transformation network in the initial image compression model are used as the basic encoding and decoding network. In order to further remove the correlation in the latent feature space, a more accurate entropy encoding module and entropy decoding module are established in the basic encoding and decoding network. A super-prior encoding and decoding network is introduced into the initial image compression model, and then the super-prior encoding and decoding network is used to assist in training the entropy encoding module and entropy decoding module in the basic encoding and decoding network.

[0090] In some embodiments of this application, the initial image compression model includes a basic encoding / decoding network and a super-prior encoding / decoding network; the basic encoding / decoding network includes the analysis transform network, the quantization compression module, and the synthesis transform network; the super-prior encoding / decoding network includes a super-prior analysis transform network, a super-prior quantization compression module, a super-prior synthesis transform network, a super-prior context model network, and a super-prior entropy parameter network; wherein, the super-prior analysis transform network is used to perform analysis transform processing on the spatial features after analysis transform to obtain the spatial features after super-prior analysis transform; the super-prior quantization compression module is used to process the super-prior quantization transformation network... The spatial features after the prior analysis and transformation are quantized and compressed to obtain the super-prior quantized and compressed spatial features. The super-prior synthesis transformation network is used to synthesize and transform the super-prior quantized and compressed spatial features to obtain the super-prior synthesized and transformed spatial features. The super-prior context model network is used to reduce the spatial redundancy of the quantized features based on the quantized spatial features and the entropy-decoded spatial features. The super-prior entropy parameter network is used to determine the model parameters of the entropy encoding module and the entropy decoding module based on the output of the super-prior context model network and the super-prior synthesized and transformed spatial features.

[0091] The super-prior context model network is an autoregressive model based on quantized spatial features and entropy-decoded spatial features. It uses mask convolution to reduce spatial redundancy and can access the decoded latent feature representation (entropy-decoded spatial features) during decoding. The probability distribution parameters of the entropy model (entropy encoding module and entropy decoding module) are obtained through the super-prior entropy parameter network by using the outputs of the super-prior context model network and the super-prior synthesis transformation network.

[0092] For example, combined Figures 4 to 6 The analysis transform network 501, quantization compression module 502, and synthesis transform network 503 shown constitute the basic encoding and decoding network of the image compression model, combined with... Figure 4 ,like Figure 7 As shown, or, in combination Figure 5 ,like Figure 8 As shown, or, in combination Figure 6 ,like Figure 9As shown, the image compression model also includes a priori encoding / decoding network, which includes a priori analysis transform network 504, a priori quantization compression module 505, a priori synthesis transform network 506, a priori context model network 507, and a priori entropy parameter network 510. The input to the priori analysis transform network 504 is the output of the analysis transform network 501, the output of the priori analysis transform network 504 is the input of the priori quantization compression module 505, the output of the priori quantization compression module 505 is the input of the priori synthesis transform network 506, the input of the priori context model network 507 is the output of the quantization module 502b and the entropy decoding module 502d, and the outputs of the priori context model network 507 and the priori synthesis transform network 506 are the inputs of the priori entropy parameter network 510. The output of the priori entropy parameter network 510 is used to adjust the model parameters of the entropy encoding module 502c and the entropy decoding module 502d.

[0093] In some embodiments of this application, a super-prior codec network can be used to assist in training the basic codec network, thereby improving the encoding and decoding performance of the end-to-end compression network.

[0094] In some embodiments of this application, in order to better assist the training of the entropy coding module and entropy decoding module in the basic coding network through the advanced prior coding network, quantization compensation processing can be added to the advanced prior quantization compression module. Specifically, the quantization compensation processing included in the super-prior quantization compression module can be placed before the super-prior quantization processing to compensate for the difference between soft quantization processing during model training and hard quantization processing during model testing or practical application. Alternatively, the quantization compensation processing included in the super-prior quantization compression module can be placed after the super-prior entropy decoding processing to compensate for the difference between soft quantization processing during model training and hard quantization processing during model testing or practical application. Quantization compensation processing can also be performed before and after the super-prior quantization processing in the super-prior quantization compression module to compensate for the difference between soft quantization processing during model training and hard quantization processing during model testing or practical application. This makes the impact of soft quantization processing on each module of the image compression model during training more closely resemble the impact of hard quantization processing on each module of the image compression model during testing or practical application, thereby improving the encoding and decoding performance of the image compression model. The super-prior encoding and decoding network can be used to assist in training the entropy encoding and entropy decoding modules in the basic encoding and decoding network.

[0095] In some embodiments of this application, the super-prior quantization compression module includes a super-prior quantization module, a super-prior entropy encoding module, a super-prior entropy decoding module, and a super-prior quantization compensation module. The super-prior quantization compensation module is used to perform quantization compensation processing on the spatial features after the super-prior analysis transformation to obtain super-prior compensated spatial features. The super-prior quantization module is used to perform quantization processing on the super-prior compensated spatial features to obtain super-prior quantized spatial features. The super-prior entropy encoding module is used to perform entropy encoding processing on the super-prior quantized spatial features to obtain a super-prior entropy encoded binary code stream. The super-prior entropy decoding module is used to perform entropy decoding processing on the super-prior entropy encoded binary code stream to obtain the super-prior quantized compressed spatial features.

[0096] For example, such as Figure 10 As shown, this is for the above Figures 7 to 9 A schematic diagram showing the unfolded structure of any one of the priori quantization compression modules 505, wherein the priori quantization compression module 505 may include, for example: Figure 10 The diagram shows a super-prior quantization compensation module 505a, a super-prior quantization module 505b, a super-prior entropy encoding module 505c, and a super-prior entropy decoding module 505d. The input of the super-prior quantization compensation module 505a is the input of the super-prior quantization compression module 505. The output of the super-prior quantization compensation module 505a is the input of the super-prior quantization module 505b, the output of the super-prior quantization module 505b is the input of the super-prior entropy encoding module 505c, the output of the super-prior entropy encoding module 505c is the input of the super-prior entropy decoding module 505d, and the output of the super-prior entropy decoding module 505d is the output of the super-prior quantization compression module 505. That is, before quantizing the spatial features after the super-prior analysis transformation, quantization compensation processing is performed on the spatial features after the super-prior analysis transformation. Through quantization compensation processing, the difference between the soft quantization processing in the training process and the hard quantization processing in the testing process is reduced, thereby improving the encoding and decoding performance of the end-to-end compression network.

[0097] In some embodiments of this application, the super-prior quantization compression module includes a super-prior quantization module, a super-prior entropy encoding module, a super-prior entropy decoding module, and a super-prior quantization compensation module. The super-prior quantization module is used to quantize the spatial features after the super-prior analysis transformation to obtain super-prior quantized spatial features. The super-prior entropy encoding module is used to entropy entropy entropy entropy encoding the spatial features to obtain a super-prior entropy entropy encoded binary code stream. The super-prior entropy decoding module is used to entropy decode the super-prior entropy entropy encoded binary code stream to obtain super-prior entropy decoded spatial features. The super-prior quantization compensation module is used to quantize and compensate the super-prior entropy decoded spatial features to obtain super-prior quantized compressed spatial features.

[0098] For example, such as Figure 11 As shown, this is for the above Figures 7 to 9 A schematic diagram showing another specific structure of the super-prior quantization compression module 505 in any one of the above. The super-prior quantization compression module 505 may include, for example: Figure 11 The illustrated super-prior quantization compression module includes a super-prior quantization module 505b, a super-prior entropy encoding module 505c, a super-prior entropy decoding module 505d, and a super-prior quantization compensation module 505e. The input of the super-prior quantization module 505b is the input of the super-prior quantization compression module 505, the output of the super-prior quantization module 505b is the input of the super-prior entropy encoding module 505c, the output of the super-prior entropy encoding module 505c is the input of the super-prior entropy decoding module 505d, the output of the super-prior entropy decoding module 505d is the input of the super-prior quantization compensation module 505e, and the output of the super-prior quantization compensation module 505e is the output of the super-prior quantization compression module 505. That is, after performing quantization, entropy encoding and entropy decoding on the spatial features after the prior analysis transformation, quantization compensation is then performed on the spatial features after the prior entropy decoding. Through quantization compensation, the difference between the soft quantization process in the training process and the hard quantization process in the testing process is reduced, thereby improving the encoding and decoding performance of the end-to-end compression network.

[0099] In some embodiments of this application, the super-prior quantization compression module includes a third quantization compensation module, a fourth quantization compensation module, a super-prior quantization module, a super-prior entropy encoding module, and a super-prior entropy decoding module. The third quantization compensation module is used to perform quantization compensation processing on the spatial features after the super-prior analysis transformation to obtain super-prior compensated spatial features. The super-prior quantization module is used to perform quantization processing on the super-prior compensated spatial features to obtain super-prior quantized spatial features. The super-prior entropy encoding module is used to perform entropy encoding processing on the super-prior quantized spatial features to obtain a super-prior entropy encoded binary code stream. The super-prior entropy decoding module is used to perform entropy decoding processing on the super-prior entropy encoded binary code stream to obtain super-prior entropy decoded spatial features. The fourth quantization compensation module is used to perform quantization compensation processing on the super-prior entropy decoded spatial features to obtain super-prior quantized compressed spatial features.

[0100] For example, such as Figure 12 As shown, this is for the above Figures 7 to 9 A schematic diagram showing another specific structure of the priori quantization compression module 505 in any of the above-mentioned modules. The priori quantization compression module 505 may include, for example: Figure 12The diagram shows a super-prior quantization compensation module 505a (corresponding to the third quantization compensation module), a super-prior quantization module 505b, a super-prior entropy encoding module 505c, a super-prior entropy decoding module 505d, and a super-prior quantization compensation module 505e (corresponding to the fourth quantization compensation module). The input of super-prior quantization compensation module 505a is the input of super-prior quantization compression module 505. The output of super-prior quantization compensation module 505a is the input of super-prior quantization module 505b. The output of super-prior quantization module 505b is the input of super-prior entropy encoding module 505c. The output of super-prior entropy encoding module 505c is the input of super-prior entropy decoding module 505d. The output of super-prior entropy decoding module 505d is the input of super-prior quantization compensation module 505e. The output of super-prior quantization compensation module 505e is the output of super-prior quantization compression module 505. That is, before quantizing the spatial features after the analysis transformation of the super-prior, the spatial features after the analysis transformation of the super-prior are first quantized and compensated. Then, after quantization, entropy encoding and entropy decoding are performed in sequence, the spatial features after entropy decoding of the super-prior are quantized and compensated. Through two quantization compensation processes, the difference between the soft quantization process in the training process and the hard quantization process in the testing process is reduced, thereby improving the encoding and decoding performance of the end-to-end compression network.

[0101] In some embodiments of this application, the initial image compression model includes a basic encoder-decoder network and a super-prior encoder-decoder network. Quantization compensation processing can be added to the quantization compression module of the super-prior encoder-decoder network to help train the basic entropy encoding module and basic entropy decoding module in the basic encoder-decoder network better, thereby narrowing the gap between soft quantization and hard quantization.

[0102] In some embodiments of this application, the initial image compression model includes a basic encoding / decoding network and a super-prior encoding / decoding network; the basic encoding / decoding network includes a basic analysis transform network, a basic quantization module, a basic entropy encoding module, a basic entropy decoding module, and a basic synthesis transform network; wherein, the basic analysis transform network is used to perform analysis and transform processing on the input training image to obtain spatial features after basic analysis transform; the basic quantization module is used to perform soft quantization processing on the spatial features after basic analysis transform to obtain spatial features after basic quantization; the basic entropy encoding module is used to perform entropy encoding processing on the spatial features after basic quantization to obtain a binary bitstream after basic entropy encoding; the basic entropy decoding module is used to perform entropy decoding processing on the binary bitstream after basic entropy encoding to obtain spatial features after basic entropy decoding; and the basic synthesis transform network is used to process the basic entropy... The decoded spatial features are subjected to a synthetic transformation process to obtain the reconstructed image corresponding to the training image. The advanced prior encoding / decoding network includes an analysis transformation network, a quantization compression module, a synthetic transformation network, a context model network, and an entropy parameter network. Specifically, the analysis transformation network is used to perform analysis transformation processing on the basic quantized spatial features to obtain the analyzed transformed spatial features. The synthetic transformation network is used to perform synthetic transformation processing on the quantized compressed spatial features to obtain the synthetic transformed spatial features. The context model network is used to reduce the spatial redundancy of the quantized features based on the basic quantized spatial features and the basic entropy decoded spatial features. The entropy parameter network is used to determine the model parameters of the basic entropy encoding module and the basic entropy decoding module based on the output of the context model network and the synthetic transformed spatial features.

[0103] For example, combined Figure 3 ,like Figure 13 As shown, the analysis includes the transform network 501 and the quantization compression module 502 (the quantization compression module 502 can be referred to...). Figures 4 to 6 The initial image compression model consists of a super-prior encoder-decoder network comprising any pair of quantization compression modules 502, a synthesis transformation network 503, a context model network 509, and an entropy parameter network 510. The initial image compression model also includes a basic encoder-decoder network, which includes a basic analysis transformation network 511, a basic quantization module 512, a basic entropy encoding module 513, a basic entropy decoding module 514, and a basic synthesis transformation network 515.

[0104] In some embodiments of this application, a priori codec network can assist in training the basic codec network, thereby improving the encoding and decoding performance of the end-to-end compression network; moreover, through the quantization compensation processing in the quantization compression module of the priori codec network, the difference between the soft quantization processing in the training process and the hard quantization processing in the testing process can be reduced, thereby improving the encoding and decoding performance of the end-to-end compression network.

[0105] In some embodiments of this application, quantization compensation processing is added to the quantization compression module. This can be done by adding quantization compensation processing before quantization processing, or after entropy decoding processing, or by adding quantization compensation processing both before and after quantization processing. All of these methods can compensate for the differences between soft quantization processing during training and hard quantization processing during testing, thereby reducing inconsistencies between training processes and improving the encoding and decoding performance of the model.

[0106] In some embodiments of this application, for an image compression model including a basic codec network and a super-prior codec network, adding quantization compensation processing to the quantization processing module of the basic codec network, or adding quantization compensation processing to the quantization processing module of the super-prior codec network, or adding quantization compensation processing to both the quantization processing modules of the basic codec network and the super-prior codec network, can compensate for the difference between soft quantization processing during training and hard quantization processing during testing, reduce inconsistencies between training processes, and thus improve the encoding and decoding performance of the model.

[0107] In some embodiments of this application, the quantization compensation process is obtained based on the Fourier series expansion, Fourier transform function, cosine series expansion, sine series expansion, or Taylor series expansion of the quantization error function; wherein, the quantization error function is determined based on the error generated by the hard quantization process.

[0108] Currently, for hard quantization, the Round(x) function with rounding is used for quantization. For the hard quantization process, the quantization error function err(x) is the difference before and after the quantization process, that is: err(x) = x - round(x). Figure 14 The relationship between quantization error and the quantization function is illustrated. It can be observed that the curve of the quantization function Round(x) exhibits a stepped distribution (shown by the bold curve in the figure), while the quantization error is a periodic function, its graph displaying a sawtooth wave shape. Therefore, in this scheme, based on existing methods, quantization compensation processing is added to the quantization compression module during the training process of the end-to-end image compression model. This quantization compensation processing reduces the difference between soft and hard quantization, eliminates the training-test mismatch problem, and thus improves the encoding and decoding performance of the end-to-end compression network.

[0109] In some embodiments of this application, the quantization error err(x) is expanded using a Fourier series and approximated using the first k harmonics to obtain the quantization compensation function s(x) corresponding to the quantization compensation processing, so as to reduce the difference between soft quantization and hard quantization caused during training and testing (or application).

[0110] The quantization error function is:

[0111] Where m is an integer.

[0112] The Fourier series expansion of the quantization error function is:

[0113] The quantization compensation function is the sum of the first k harmonics of the Fourier series expansion of the quantization error function, specifically:

[0114]

[0115] Where n is an integer, n = 1, 2, ..., k, the maximum amplitude is A / 2, the period is T1, and ω = (2π) / T1.

[0116] n represents the nth harmonic; different values ​​of k result in different approximate sawtooth wave effects on the quantization compensation function curve, such as... Figure 15 As shown, the approximate effects of the quantization compensation function for the first 2 harmonics, the first 3 harmonics, and the first 10 harmonics are respectively displayed. It can be found that the higher k is, the more similar the curve of the quantization compensation function is to the curve of the quantization error function.

[0117] In some embodiments of this application, if quantization compensation processing is performed before the quantization processing of the basic codec network, such as... Figure 4 , Figure 6 , Figure 7 and Figure 9 As shown, the input to the quantization compensation module is y, and the output is y. qc , then y qc =ys(y); If quantization compensation is performed before quantization processing in the advanced priori encoder-decoder network, such as Figure 10 and Figure 12 As shown, the input of the quantization compensation module is z, and the output is z. qc Then z qc =zs(z); If quantization compensation is performed after entropy decoding of the basic encoder-decoder network, such as Figure 5 and Figure 8 As shown, the input to the quantization compensation module is y. n The output is but If quantization compensation is performed before quantization processing in the advanced priori encoder-decoder network, such as Figure 10 and Figure 12 As shown, the input to the quantization compensation module is z. n The output is but

[0118] In some embodiments of this application, s(x) can also be obtained by performing Fourier transform, Taylor series expansion, sine series expansion or cosine series expansion on err(x). The specific method can be determined according to the actual situation and is not limited here.

[0119] In some embodiments of this application, various methods for determining quantization compensation processing are provided. The specific method can be determined according to the actual situation and is not limited here. The method of adding quantization compensation processing to the quantization compression module provided in some embodiments of this application can be extended to the training process of any image compression model to compensate for the difference between soft quantization processing in the training process and hard quantization processing in the testing process, reduce the inconsistency between the training process and the training process, and thus improve the encoding and decoding performance of the model.

[0120] In some embodiments of this application, the soft quantization process is determined based on additive uniform noise, a pass-through estimator, or a soft-to-hard algorithm. The soft quantization process can also be determined based on other soft quantization methods, which are not limited here. Specific soft quantization methods such as additive uniform noise, pass-through estimators, or soft-to-hard algorithms can be found in existing related technologies, and are not limited here.

[0121] For example, hard quantization uses a rounding function, such as y o =round(y i ), where y o For hard-quantized output, y i For hard quantization input, `round()` is the rounding function. The corresponding soft quantization process can be determined based on additive uniform noise, such as soft quantization for y... o =y i +Δy, where y o For the output of soft quantization, y i For soft quantization, Δy ~ U(-0.5, 0.5), which means that Δy is noise uniformly distributed on (-0.5, 0.5).

[0122] In some embodiments of this application, various methods for determining soft quantization processing are provided. The method of adding quantization compensation processing to the quantization compression module provided in some embodiments of this application can be extended to the training process of any image compression model to compensate for the difference between soft quantization processing in the training process and hard quantization processing in the testing process, reduce the inconsistency between the training process and the training process, and thus improve the encoding and decoding performance of the model.

[0123] For example, with an image compression model including a basic encoder-decoder network and a super-prior encoder-decoder network, soft quantization is applied as additive uniform noise, and quantization compensation is added before the quantization processing of the basic encoder-decoder network and the super-prior encoder-decoder network, or quantization compensation is added after the entropy decoding processing of the basic encoder-decoder network and the super-prior encoder-decoder network. The compression quality of the image is significantly improved, with the peak signal-to-noise ratio (PSNR) improving by about 0.2 dB and the multi-scale-structural similarity index (MS-SSIM) improving by about 0.002.

[0124] Figure 16 This is a flowchart illustrating the steps of implementing an image encoding / decoding test method according to one or more embodiments of this application. The executing entity of the image encoding / decoding test method can be a server or electronic device, or a functional module or entity within the server or electronic device capable of implementing the image encoding method; no limitation is made here. The image encoding / decoding test method is applied to a test device and may include the following steps S601 to S606.

[0125] S601. Obtain the image to be encoded.

[0126] S602. Input the image to be encoded into the trained analysis and transformation network for analysis and transformation processing to obtain the spatial features after analysis and transformation.

[0127] S603. Input the spatial features after analysis and transformation into the hard quantization module for hard quantization processing to obtain the quantized spatial features.

[0128] S604. Input the quantized spatial features into the trained entropy coding module for entropy coding processing to obtain the entropy-coded binary code stream.

[0129] S605. Input the entropy-encoded binary code stream into the trained entropy decoding module for entropy decoding processing to obtain the entropy-decoded spatial features.

[0130] S606. Input the spatial features after the entropy decoding into the trained synthesis transformation network for synthesis transformation processing to obtain the reconstructed image corresponding to the image to be encoded.

[0131] Wherein, the analysis transformation network, the entropy encoding module, the entropy decoding module, and the synthesis transformation network are respectively the analysis transformation network, the module corresponding to the entropy encoding processing of the quantization compression module, the module corresponding to the entropy decoding processing of the quantization compression module, and the synthesis transformation network in the image compression model trained in the above-mentioned image compression model training method embodiment.

[0132] The model training device can obtain a trained image compression model by executing an image compression model training method. The model testing device can obtain an image compression test model by removing the quantization compensation processing module from the trained image compression model and replacing the soft quantization processing with hard quantization processing. The image compression test model is then tested based on a large number of sample images. After passing the test, the analysis transform network, the hard quantization processing of the quantization compression module, and the entropy coding processing modules from the image compression test model are sent to the encoding device. The entropy decoding processing module and the synthesis transform network from the trained image compression model are sent to the decoding device. The encoding device constructs an image encoding model including an analysis transform network and an entropy coding module based on the analysis transform network, hard quantization processing of the quantization compression module, and entropy coding processing modules sent by the model testing device. It then encodes the image to be encoded and sends the resulting entropy-encoded binary bitstream to the decoding device. The decoding device is used to construct an image decoding model, including an entropy decoding module and a synthesis transformation network, based on the entropy decoding processing module and the synthesis transformation network corresponding to the quantization compression module issued by the model testing device. It decodes the entropy-encoded binary code stream received from the encoding device to obtain the reconstructed image corresponding to the image to be encoded.

[0133] Among them, model training equipment, model testing equipment, encoding equipment, and decoding equipment can all be servers or electronic devices.

[0134] Figure 17 The flowchart illustrates the steps of implementing an image encoding method according to one or more embodiments of this application. The executing entity of the image encoding method can be a server or an electronic device, or a functional module or entity within the server or electronic device capable of implementing the image encoding method; no limitation is made here. The image encoding method is applied to an encoding device and may include the steps S701 to S705 described below.

[0135] S701. Obtain the image to be encoded.

[0136] S702. Input the image to be encoded into the trained analysis and transformation network for analysis and transformation processing to obtain the spatial features after analysis and transformation.

[0137] S703. Input the spatial features after analysis and transformation into the hard quantization module for hard quantization processing to obtain the quantized spatial features.

[0138] S704. Input the quantized spatial features into the trained entropy coding module for entropy coding processing to obtain the entropy-coded binary code stream.

[0139] S705: Send the entropy-encoded binary code stream to the decoding device.

[0140] In this context, the analysis transformation network and the entropy coding module are respectively the modules corresponding to the entropy coding processing of the analysis transformation network and the quantization compression module in the image compression model trained in the above-mentioned image compression model training method embodiment.

[0141] In some embodiments of this application, the analysis and transformation network and the entropy coding processing module of the image compression model trained by the image compression model training method are used to encode the image to be encoded, which can improve the encoding performance of the image.

[0142] Figure 18 The flowchart illustrates the steps of implementing an image decoding method according to one or more embodiments of this application. The executing entity of the image decoding method can be a server or an electronic device, or a functional module or entity within the server or electronic device capable of implementing the image decoding method; no limitation is made here. The image decoding method is applied to a decoding device and may include the steps S801 to S803 described below.

[0143] S801: Receive the entropy-encoded binary code stream from the encoding device.

[0144] The entropy-encoded binary bitstream is obtained by the encoding device after performing analysis, transformation, hard quantization, and entropy encoding on the image to be encoded.

[0145] S802. Input the entropy-encoded binary code stream into the trained entropy decoding module for entropy decoding processing to obtain the entropy-decoded spatial features.

[0146] S803. Input the spatial features after the entropy decoding into the trained synthesis transformation network for synthesis transformation processing to obtain the reconstructed image corresponding to the image to be encoded.

[0147] Wherein, the entropy decoding module and the synthetic transformation network are respectively the module corresponding to the entropy decoding processing of the quantization compression module in the image compression model trained in the above-mentioned image compression model training method embodiment and the synthetic transformation network.

[0148] In some embodiments of this application, the quantization compression module in the image compression model trained by the image compression model training method is used to perform entropy decoding processing, and the synthesis transform network is used to decode the entropy-encoded binary code stream, which can improve the decoding performance of the image.

[0149] Figure 19 This is a structural block diagram of an image compression model training device shown in an embodiment of this application, such as... Figure 19As shown, the device includes: a memory 901 configured to store a training sample set; and one or more processors 902 (one processor is shown as an example in the figure), configured to: acquire the training sample set, which includes multiple training images; construct an initial image compression model, which includes an analysis transform network, a quantization compression module, and a synthesis transform network; the analysis transform network is used to perform analysis transform processing on the input training images to obtain encoded spatial features; the quantization compression module performs soft quantization processing, entropy encoding processing, entropy decoding processing, and quantization compensation processing on the encoded spatial features to obtain quantized compressed spatial features; the quantization compensation processing is used to reduce the difference between the soft quantization processing in the training process and the hard quantization processing in the testing process; the synthesis transform network is used to perform synthesis transform processing on the quantized compressed spatial features to obtain a reconstructed image corresponding to the training image; input each training image in the training sample set into the initial image compression model to obtain a reconstructed image corresponding to each training image; and train the initial image compression model based on each training image and the reconstructed image corresponding to each training image to obtain a trained target image compression model.

[0150] In some embodiments of this application, the quantization compression module includes a quantization compensation module, a quantization module, an entropy encoding module, and an entropy decoding module. The quantization compensation module is used to perform quantization compensation processing on the spatial features after analysis and transformation to obtain compensated spatial features. The quantization module is used to perform quantization processing on the compensated spatial features to obtain quantized spatial features. The entropy encoding module is used to perform entropy encoding processing on the quantized spatial features to obtain an entropy-encoded binary code stream. The entropy decoding module is used to perform entropy decoding processing on the entropy-encoded binary code stream to obtain entropy-decoded spatial features, which are the quantized and compressed spatial features.

[0151] In some embodiments of this application, the quantization compression module includes a quantization compensation module, a quantization module, an entropy encoding module, and an entropy decoding module. The quantization module is used to quantize the spatial features after analysis and transformation to obtain quantized spatial features. The entropy encoding module is used to entropy encode the quantized spatial features to obtain an entropy-encoded binary code stream. The entropy decoding module is used to entropy decode the entropy-encoded binary code stream to obtain entropy-decoded spatial features. The quantization compensation module is used to quantize and compensate the entropy-decoded spatial features to obtain the quantized and compressed spatial features.

[0152] In some embodiments of this application, the quantization compression module includes a first quantization compensation module, a second quantization compensation module, a quantization module, an entropy encoding module, and an entropy decoding module. The first quantization compensation module is used to perform quantization compensation processing on the spatial features after analysis and transformation to obtain compensated spatial features. The quantization module is used to perform quantization processing on the compensated spatial features to obtain quantized spatial features. The entropy encoding module is used to perform entropy encoding processing on the quantized spatial features to obtain an entropy-encoded binary code stream. The entropy decoding module is used to perform entropy decoding processing on the entropy-encoded binary code stream to obtain entropy-decoded spatial features. The second quantization compensation module is used to perform quantization compensation processing on the entropy-decoded spatial features to obtain the quantized compressed spatial features.

[0153] In some embodiments of this application, the initial image compression model includes a basic encoding / decoding network and a super-prior encoding / decoding network; the basic encoding / decoding network includes the analysis transform network, the quantization compression module, and the synthesis transform network; the super-prior encoding / decoding network includes a super-prior analysis transform network, a super-prior quantization compression module, a super-prior synthesis transform network, a super-prior context model network, and a super-prior entropy parameter network; wherein, the super-prior analysis transform network is used to perform analysis transform processing on the spatial features after analysis transform to obtain the spatial features after super-prior analysis transform; the super-prior quantization compression module is used to process the super-prior quantization transformation network... The spatial features after the prior analysis and transformation are quantized and compressed to obtain the super-prior quantized and compressed spatial features. The super-prior synthesis transformation network is used to synthesize and transform the super-prior quantized and compressed spatial features to obtain the super-prior synthesized and transformed spatial features. The super-prior context model network is used to reduce the spatial redundancy of the quantized features based on the quantized spatial features and the entropy-decoded spatial features. The super-prior entropy parameter network is used to determine the model parameters of the entropy encoding module and the entropy decoding module based on the output of the super-prior context model network and the super-prior synthesized and transformed spatial features.

[0154] In some embodiments of this application, the super-prior quantization compression module includes a super-prior quantization module, a super-prior entropy encoding module, a super-prior entropy decoding module, and a super-prior quantization compensation module. The super-prior quantization compensation module performs quantization compensation processing on the spatial features after the super-prior analysis transformation to obtain super-prior compensated spatial features. The super-prior quantization module performs quantization processing on the super-prior compensated spatial features to obtain super-prior quantized spatial features. The super-prior entropy encoding module performs entropy encoding processing on the super-prior quantized spatial features to obtain a super-prior entropy encoded binary code stream. The super-prior entropy decoding module performs entropy decoding processing on the super-prior entropy encoded binary code stream to obtain the super-prior quantized compressed spatial features.

[0155] In some embodiments of this application, the super-prior quantization compression module includes a super-prior quantization module, a super-prior entropy encoding module, a super-prior entropy decoding module, and a super-prior quantization compensation module. The super-prior quantization module is used to quantize the spatial features after the super-prior analysis transformation to obtain super-prior quantized spatial features. The super-prior entropy encoding module is used to entropy entropy entropy entropy encoding the spatial features to obtain a super-prior entropy entropy encoded binary code stream. The super-prior entropy decoding module is used to entropy decode the super-prior entropy entropy encoded binary code stream to obtain super-prior entropy decoded spatial features. The super-prior quantization compensation module is used to quantize and compensate the super-prior entropy decoded spatial features to obtain super-prior quantized compressed spatial features.

[0156] In some embodiments of this application, the super-prior quantization compression module includes a third quantization compensation module, a fourth quantization compensation module, a super-prior quantization module, a super-prior entropy encoding module, and a super-prior entropy decoding module. The third quantization compensation module is used to perform quantization compensation processing on the spatial features after the super-prior analysis transformation to obtain super-prior compensated spatial features. The super-prior quantization module is used to perform quantization processing on the super-prior compensated spatial features to obtain super-prior quantized spatial features. The super-prior entropy encoding module is used to perform entropy encoding processing on the super-prior quantized spatial features to obtain a super-prior entropy encoded binary code stream. The super-prior entropy decoding module is used to perform entropy decoding processing on the super-prior entropy encoded binary code stream to obtain super-prior entropy decoded spatial features. The fourth quantization compensation module is used to perform quantization compensation processing on the super-prior entropy decoded spatial features to obtain super-prior quantized compressed spatial features.

[0157] In some embodiments of this application, the initial image compression model includes a basic encoding / decoding network and a super-prior encoding / decoding network; the basic encoding / decoding network includes a basic analysis transform network, a basic quantization module, a basic entropy encoding module, a basic entropy decoding module, and a basic synthesis transform network; wherein, the basic analysis transform network is used to perform analysis and transform processing on the input training image to obtain spatial features after basic analysis transform; the basic quantization module is used to perform soft quantization processing on the spatial features after basic analysis transform to obtain spatial features after basic quantization; the basic entropy encoding module is used to perform entropy encoding processing on the spatial features after basic quantization to obtain a binary bitstream after basic entropy encoding; the basic entropy decoding module is used to perform entropy decoding processing on the binary bitstream after basic entropy encoding to obtain spatial features after basic entropy decoding; and the basic synthesis transform network is used to process the basic entropy... The decoded spatial features are subjected to a synthetic transformation process to obtain the reconstructed image corresponding to the training image. The advanced prior encoding / decoding network includes an analysis transformation network, a quantization compression module, a synthetic transformation network, a context model network, and an entropy parameter network. Specifically, the analysis transformation network is used to perform analysis transformation processing on the basic quantized spatial features to obtain the analyzed transformed spatial features. The synthetic transformation network is used to perform synthetic transformation processing on the quantized compressed spatial features to obtain the synthetic transformed spatial features. The context model network is used to reduce the spatial redundancy of the quantized features based on the basic quantized spatial features and the basic entropy decoded spatial features. The entropy parameter network is used to determine the model parameters of the basic entropy encoding module and the basic entropy decoding module based on the output of the context model network and the synthetic transformed spatial features.

[0158] In some embodiments of this application, the quantization compensation process is obtained based on the Fourier series expansion, Fourier transform function, cosine series expansion, sine series expansion, or Taylor series expansion of the quantization error function;

[0159] The quantization error function is determined based on the error generated by the hard quantization process.

[0160] In some embodiments of this application, the soft quantization process is determined based on additive uniform noise, a pass-through estimator, or a soft-to-hard algorithm.

[0161] In some embodiments of this application, each module of the image compression model training device can implement the image compression model training method provided in the above method embodiments and achieve the same technical effect. To avoid repetition, it will not be described again here.

[0162] Figure 19 This is a structural block diagram of an image encoding apparatus shown in an embodiment of this application, such as... Figure 19As shown, the device includes: a memory 901 configured to store an image to be encoded; and one or more processors 902 (one processor is shown as an example in the figure), configured to: acquire the image to be encoded; input the image to be encoded into a trained analysis and transformation network for analysis and transformation processing to obtain spatial features after analysis and transformation; input the spatial features after analysis and transformation into a hard quantization module for hard quantization processing to obtain quantized spatial features; input the quantized spatial features into a trained entropy encoding module for entropy encoding processing to obtain an entropy-encoded binary bitstream; and send the entropy-encoded binary bitstream to a decoding device; wherein the analysis and transformation network and the entropy encoding module are respectively the modules corresponding to the analysis and transformation network and the entropy encoding processing of the quantization compression module in the image compression model trained by the image compression model training method described in the first aspect.

[0163] In some embodiments of this application, each module of the image encoding device can implement the image encoding method provided in the above method embodiments and achieve the same technical effect. To avoid repetition, it will not be described again here.

[0164] Figure 19 This is a structural block diagram of an image decoding device shown in an embodiment of this application, such as... Figure 19 As shown, the device includes: a memory 901 configured to store reconstructed images; and one or more processors 902 (one processor is shown as an example in the figure), configured to: receive an entropy-encoded binary bitstream from an encoding device, the entropy-encoded binary bitstream being obtained by the encoding device after performing analysis and transformation processing, hard quantization processing, and entropy encoding processing on the image to be encoded; input the entropy-encoded binary bitstream into a trained entropy decoding module for entropy decoding processing to obtain entropy-decoded spatial features; input the entropy-decoded spatial features into a trained synthesis transformation network for synthesis transformation processing to obtain the reconstructed image corresponding to the image to be encoded; wherein, the entropy decoding module and the synthesis transformation network are respectively the module corresponding to the entropy decoding processing of the quantization compression module and the synthesis transformation network in the image compression model trained by the image compression model training method described in the first aspect.

[0165] In some embodiments of this application, each module of the image decoding device can implement the image decoding method provided in the above method embodiments and achieve the same technical effect. To avoid repetition, it will not be described again here.

[0166] Some embodiments of this application also provide a terminal device, including: the image compression model training device described above, or the image encoding device described above, or the image decoding device described above.

[0167] This application also provides a computer-readable storage medium storing a computer program. When the computer program is executed by a processor, it implements the various processes executed in the above-described method embodiments and achieves the same technical effects. To avoid repetition, it will not be described again here.

[0168] The computer-readable storage medium can be a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk, etc.

[0169] This application provides a computer program product, including: when the computer program product is run on a computer, causing the computer to perform various processes executed in the above-described method embodiments.

[0170] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features therein. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of this application.

[0171] For ease of explanation, the above description has been provided in conjunction with specific embodiments. However, the above exemplary discussion is not intended to be exhaustive or to limit the embodiments to the specific forms disclosed above. Various modifications and variations can be obtained based on the above teachings. The selection and description of the above embodiments are for the purpose of better explaining the principles and practical applications, thereby enabling those skilled in the art to better utilize the described embodiments and various different variations of embodiments suitable for specific use considerations.

Claims

1. A method for training an image compression model, characterized in that, include: Obtain a training sample set, which includes multiple training images; An initial image compression model is constructed, comprising: an analysis transform network, a quantization compression module, and a synthesis transform network. The analysis transform network performs analysis transform processing on the input training image to obtain the spatial features after analysis transform. The quantization compression module performs soft quantization, entropy encoding, entropy decoding, and quantization compensation processing on the spatial features after analysis transform to obtain quantized compressed spatial features. The quantization compensation processing is used to reduce the difference between the soft quantization processing during training and the hard quantization processing during testing. The synthesis transform network performs synthesis transform processing on the quantized compressed spatial features to obtain the reconstructed image corresponding to the training image. Each training image in the training sample set is input into the initial image compression model to obtain the reconstructed image corresponding to each training image; Based on each training image and the corresponding reconstructed image, the initial image compression model is trained to obtain the trained target image compression model.

2. The method according to claim 1, characterized in that, The quantization compression module includes a quantization compensation module, a quantization module, an entropy encoding module, and an entropy decoding module; The quantization compensation module is used to perform quantization compensation processing on the spatial features after the analysis transformation to obtain the compensated spatial features. The quantization module is used to quantize the compensated spatial features to obtain quantized spatial features; The entropy encoding module is used to perform entropy encoding processing on the quantized spatial features to obtain an entropy-encoded binary code stream; the entropy decoding module is used to perform entropy decoding processing on the entropy-encoded binary code stream to obtain entropy-decoded spatial features, which are the quantized and compressed spatial features. or, The quantization module is used to quantize the spatial features after the analysis transformation to obtain quantized spatial features; the entropy encoding module is used to entropy encode the quantized spatial features to obtain an entropy-encoded binary code stream; the entropy decoding module is used to entropy decode the entropy-encoded binary code stream to obtain entropy-decoded spatial features. The quantization compensation module is used to perform quantization compensation processing on the spatial features after entropy decoding to obtain the quantized compressed spatial features.

3. The method according to claim 1, characterized in that, The quantization compression module includes a first quantization compensation module, a second quantization compensation module, a quantization module, an entropy encoding module, and an entropy decoding module; The first quantization compensation module is used to perform quantization compensation processing on the spatial features after the analysis transformation to obtain the compensated spatial features; The quantization module is used to quantize the compensated spatial features to obtain quantized spatial features; The entropy encoding module is used to perform entropy encoding processing on the quantized spatial features to obtain an entropy-encoded binary code stream; the entropy decoding module is used to perform entropy decoding processing on the entropy-encoded binary code stream to obtain entropy-decoded spatial features. The second quantization compensation module is used to perform quantization compensation processing on the spatial features after entropy decoding to obtain the quantized compressed spatial features.

4. The method according to claim 2 or 3, characterized in that, The initial image compression model includes a basic encoder-decoder network and a super-prior encoder-decoder network; The basic encoding / decoding network includes the analysis transform network, the quantization compression module, and the synthesis transform network; The super-prior encoding / decoding network includes a super-prior analysis transform network, a super-prior quantization compression module, a super-prior synthesis transform network, a super-prior context model network, and a super-prior entropy parameter network. The super-prior analysis transformation network is used to perform analysis transformation processing on the spatial features after analysis transformation to obtain super-prior analysis transformation spatial features; the super-prior quantization compression module is used to quantize and compress the spatial features after super-prior analysis transformation to obtain super-prior quantized compressed spatial features; the super-prior synthesis transformation network is used to perform synthesis transformation processing on the super-prior quantized compressed spatial features to obtain super-prior synthesized transformed spatial features; the super-prior context model network is used to reduce the spatial redundancy of quantized features based on the quantized spatial features and the entropy-decoded spatial features; and the super-prior entropy parameter network is used to determine the model parameters of the entropy encoding module and the entropy decoding module according to the output of the super-prior context model network and the super-prior synthesized transformed spatial features.

5. The method according to claim 4, characterized in that, The super-prior quantization compression module includes a super-prior quantization module, a super-prior entropy encoding module, a super-prior entropy decoding module, and a super-prior quantization compensation module. The super-prior quantization compensation module is used to perform quantization compensation processing on the spatial features after the super-prior analysis transformation to obtain the super-prior compensated spatial features. The super-prior quantization module is used to quantize the super-prior compensated spatial features to obtain super-prior quantized spatial features. The super-prior entropy coding module is used to perform entropy coding processing on the super-prior quantized spatial features to obtain a super-prior entropy encoded binary code stream. The super-prior entropy decoding module is used to perform entropy decoding processing on the binary code stream encoded by the super-prior entropy to obtain the spatial features after super-prior quantization compression. or, The advanced prior quantization module is used to quantize the spatial features after the advanced prior analysis transformation to obtain the advanced prior quantized spatial features. The super-prior entropy encoding module is used to perform entropy encoding processing on the super-prior quantized spatial features to obtain a super-prior entropy encoded binary code stream; the super-prior entropy decoding module is used to perform entropy decoding processing on the super-prior entropy encoded binary code stream to obtain super-prior entropy decoded spatial features; the super-prior quantization compensation module is used to perform quantization compensation processing on the super-prior entropy decoded spatial features to obtain super-prior quantized compressed spatial features.

6. The method according to claim 4, characterized in that, The super-prior quantization compression module includes a third quantization compensation module, a fourth quantization compensation module, a super-prior quantization module, a super-prior entropy encoding module, and a super-prior entropy decoding module. The third quantization compensation module is used to perform quantization compensation processing on the spatial features after the a priori analysis transformation to obtain the spatial features after a priori compensation. The super-prior quantization module is used to quantize the super-prior compensated spatial features to obtain super-prior quantized spatial features. The super-prior entropy coding module is used to perform entropy coding processing on the super-prior quantized spatial features to obtain a super-prior entropy encoded binary code stream. The super-prior entropy decoding module is used to perform entropy decoding processing on the binary code stream encoded by super-prior entropy to obtain the spatial features after super-prior entropy decoding. The fourth quantization compensation module is used to perform quantization compensation processing on the spatial features after decoding the hyper-prior entropy to obtain the spatial features after hyper-prior quantization compression.

7. The method according to any one of claims 1 to 3, characterized in that, The initial image compression model includes a basic encoder-decoder network and a super-prior encoder-decoder network; The basic encoding / decoding network includes a basic analysis and transformation network, a basic quantization module, a basic entropy encoding module, a basic entropy decoding module, and a basic synthesis and transformation network. The basic analysis and transformation network analyzes and transforms the input training image to obtain spatial features after the basic analysis and transformation. The basic quantization module performs soft quantization on the spatial features after the basic analysis and transformation to obtain quantized spatial features. The basic entropy encoding module performs entropy encoding on the quantized spatial features to obtain a binary bitstream with basic entropy encoding. The basic entropy decoding module performs entropy decoding on the binary bitstream with basic entropy encoding to obtain spatial features with basic entropy decoding. The basic synthesis and transformation network performs synthesis and transformation on the spatial features with basic entropy decoding to obtain the reconstructed image corresponding to the training image. The advanced priori encoding / decoding network includes the analysis transformation network, the quantization compression module, the synthesis transformation network, the context model network, and the entropy parameter network. Specifically, the analysis transformation network performs analysis transformation processing on the spatial features after the basic analysis transformation to obtain the analyzed transformed spatial features. The synthesis transformation network performs synthesis transformation processing on the quantized compressed spatial features to obtain synthesized transformed spatial features. The context model network reduces spatial redundancy of quantized features based on the basic quantized spatial features and the basic entropy decoded spatial features. The entropy parameter network determines the model parameters of the basic entropy encoding module and the basic entropy decoding module based on the output of the context model network and the synthesized transformed spatial features.

8. The method according to any one of claims 1 to 3, characterized in that, The quantization compensation process is obtained based on the Fourier series expansion, Fourier transform function, cosine series expansion, sine series expansion, or Taylor series expansion of the quantization error function; The quantization error function is determined based on the error generated by the hard quantization process.

9. The method according to any one of claims 1 to 3, characterized in that, The soft quantization process is determined based on additive uniform noise, a pass-through estimator, or a soft-to-hard algorithm.

10. An image encoding method, characterized in that, Applied to encoding devices, including: Obtain the image to be encoded; The image to be encoded is input into a trained analysis and transformation network for analysis and transformation processing to obtain the spatial features after analysis and transformation. The spatial features after analysis and transformation are input into the hard quantization module for hard quantization processing to obtain the quantized spatial features. The quantized spatial features are input into the trained entropy coding module for entropy coding processing to obtain the entropy-coded binary code stream. Send the entropy-encoded binary code stream to the decoding device; Wherein, the analysis transformation network and the entropy coding module are respectively the modules corresponding to the entropy coding processing of the analysis transformation network and the quantization compression module in the image compression model trained by any one of the image compression model training methods in claims 1 to 9.

11. An image decoding method, characterized in that, Applied to decoding devices, including: The encoding device receives an entropy-encoded binary bitstream, which is obtained by the encoding device after performing analysis, transformation, hard quantization, and entropy encoding on the image to be encoded. The entropy-encoded binary code stream is input into the trained entropy decoding module for entropy decoding processing to obtain the entropy-decoded spatial features. The spatial features after entropy decoding are input into a trained synthesis transformation network for synthesis transformation processing to obtain the reconstructed image corresponding to the image to be encoded. Wherein, the entropy decoding module and the synthetic transformation network are respectively the module corresponding to the entropy decoding processing of the quantization compression module and the synthetic transformation network in the image compression model trained by any one of the image compression model training methods in claims 1 to 9.

12. An image compression model training device, characterized in that, include: The memory is configured to store the training sample set; and One or more processors, configured as follows: Obtain a training sample set, which includes multiple training images; An initial image compression model is constructed, comprising: an analysis transform network, a quantization compression module, and a synthesis transform network. The analysis transform network performs analysis transform processing on the input training image to obtain encoded spatial features. The quantization compression module performs soft quantization, entropy encoding, entropy decoding, and quantization compensation processing on the encoded spatial features to obtain quantized compressed spatial features. The quantization compensation processing is used to reduce the difference between the soft quantization processing during training and the hard quantization processing during testing. The synthesis transform network performs synthesis transform processing on the quantized compressed spatial features to obtain a reconstructed image corresponding to the training image. Each training image in the training sample set is input into the initial image compression model to obtain the reconstructed image corresponding to each training image; Based on each training image and the corresponding reconstructed image, the initial image compression model is trained to obtain the trained target image compression model.

13. An image encoding device, characterized in that, include: Memory, configured to store images to be encoded; and One or more processors, configured as follows: Obtain the image to be encoded; The image to be encoded is input into a trained analysis and transformation network for analysis and transformation processing to obtain the spatial features after analysis and transformation. The spatial features after analysis and transformation are input into the hard quantization module for hard quantization processing to obtain the quantized spatial features. The quantized spatial features are input into the trained entropy coding module for entropy coding processing to obtain the entropy-coded binary code stream. Send the entropy-encoded binary code stream to the decoding device; Wherein, the analysis transformation network and the entropy coding module are respectively the modules corresponding to the entropy coding processing of the analysis transformation network and the quantization compression module in the image compression model trained by any one of the image compression model training methods in claims 1 to 9.

14. An image decoding device, characterized in that, include: Memory, configured to store reconstructed images; and One or more processors, configured as follows: The encoding device receives an entropy-encoded binary bitstream, which is obtained by the encoding device after performing analysis, transformation, hard quantization, and entropy encoding on the image to be encoded. The entropy-encoded binary code stream is input into the trained entropy decoding module for entropy decoding processing to obtain the entropy-decoded spatial features. The spatial features after entropy decoding are input into a trained synthesis transformation network for synthesis transformation processing to obtain the reconstructed image corresponding to the image to be encoded. Wherein, the entropy decoding module and the synthetic transformation network are respectively the module corresponding to the entropy decoding processing of the quantization compression module and the synthetic transformation network in the image compression model trained by any one of the image compression model training methods in claims 1 to 9.

15. A terminal device, characterized in that, include: The image compression model training apparatus as described in claim 12, or the image encoding apparatus as described in claim 13, or the image decoding apparatus as described in claim 14.

Images