Image compression method and electronic device based on neural network
By using semantic-level text descriptions instead of pixel-level residual representations in image compression methods, a bitstream containing image features and text information is generated. This solves the problems of long image reconstruction time and unstable quality in diffusion models at extremely low bitrates, and achieves efficient and real-time image reconstruction.
Patent Information
- Application Number
- CN202611344739.3
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2026-09-01
- Publication Date
- 2026-11-17
AI Technical Summary
Existing image compression methods based on diffusion models suffer from excessively long inference times and unstable reconstruction quality when reconstructing images at extremely low bit rates. The lack of effective semantic guidance leads to significant differences between the reconstructed image and the original image, making it difficult to meet the needs of real-time applications.
The pixel-level residual representation in the image compression method is replaced with a semantic-level text description. The encoding end network generates a bitstream containing image feature representation and text information, and the decoding end uses the text information as a guiding condition for image reconstruction.
It achieves realism and fidelity in image reconstruction at extremely low bitrates, while solving the problem of excessive inference time, ensuring consistency between the reconstructed image and the original image and real-time performance.
Smart Images

Figure CN123123579A_ABST
Abstract
Description
Technical Field
[0001] This specification relates to one or more embodiments in the field of artificial intelligence, and more particularly to an image compression method and electronic device based on neural networks. Background Technology
[0002] In today's rapidly developing information age, the demand for image data transmission and storage continues to grow. Compressed images are widely disseminated on the Internet through various multimedia applications, and the massive amount of data puts continuous pressure on communication bandwidth and storage space.
[0003] In practical applications, the purpose of image compression is usually to convert the original image into the most compact bitstream possible for subsequent transmission and processing. Effective image compression methods must balance controlling the bitstream size with ensuring the quality of the reconstructed image; striking this balance has long been a core challenge in the field of image compression.
[0004] Over the past few decades, image compression technology has developed along two main lines.
[0005] On the one hand, traditional encoding and decoding standards, represented by JPEG, HEVC, and VVC, are widely used in real-world scenarios due to their mature technical frameworks; On the other hand, deep learning-based neural network image compression methods, through the joint optimization of nonlinear transformation and rate distortion, are gradually showing the potential to compete with or even surpass traditional standards.
[0006] However, both of the above methods take rate-distortion performance as the core optimization objective, and their common limitation is: When the bitrate is compressed to a low level, the reconstructed image is prone to visual degradation problems such as blurring and loss of detail, making it difficult to meet the requirements for image perception quality in ultra-low bitrate scenarios.
[0007] To improve the visual quality of reconstructed images at low bitrates, generative image compression methods oriented towards perceptual quality have gained increasing attention. These methods prioritize a comprehensive trade-off between rate, distortion, and perception, with one important direction being the introduction of diffusion models into image compression tasks. Diffusion models possess powerful image generation capabilities, enabling them to replenish details through iterative denoising processes even at extremely low bitrates and with significant loss of image information, thereby generating reconstructed images with better visual quality.
[0008] However, existing image compression methods based on diffusion models generally face two difficulties.
[0009] First, the denoising process of diffusion models typically requires dozens of iterations to achieve acceptable reconstruction quality, resulting in lengthy inference times that are difficult to meet the needs of real-time applications. Although the number of denoising steps can be compressed to a single step to improve speed, single-step denoising is highly dependent on guiding conditions; without effective semantic guidance information, the reconstruction quality will drop significantly.
[0010] Secondly, the diffusion model itself has a certain degree of randomness, making it difficult to guarantee the consistency of the reconstructed image with the original image in terms of content. The generated result may deviate significantly from the original image, which restricts its application value in real-world scenarios where high-fidelity reconstruction is required. Summary of the Invention
[0011] This specification aims to disclose a technical solution that replaces the pixel-level residual representation in existing image compression methods with semantic-level text descriptions, and uses these text descriptions as guiding conditions for image reconstruction, thereby simultaneously ensuring the realism and fidelity of the reconstructed image under extremely low bitrate conditions, and guaranteeing the real-time performance of encoding and decoding.
[0012] According to a first aspect of the embodiments of this specification, a neural network-based image compression method is proposed, wherein the encoding network includes a first encoding network and a deep compression network; wherein the first encoding network is used to encode the target image into an image feature representation; and the deep compression network is used to compress the image feature representation into a latent feature representation, and to perform compressed encoding on the latent feature representation; The target image is input to the encoding network, whereby the encoding network compresses and encodes the target image to generate a bitstream corresponding to the target image, including: The target image is input into the first encoding network so that the first encoding network encodes the target image into an image feature representation; The image feature representation output by the first encoding network is obtained, and the image feature representation is further input into the deep compression network so that the deep compression network compresses the image feature representation into a latent feature representation, and the latent feature representation is compressed and encoded to obtain a bitstream corresponding to the target image.
[0013] Optionally, the first encoding network includes multiple network branches; wherein the multiple network branches include a main encoder for encoding the target image into the first detail feature representation; and an auxiliary encoder for encoding the target image into the first structural feature representation; The target image is input into the first encoding network so that the first encoding network encodes the target image into an image feature representation; The target image is input to the main encoder and the auxiliary encoder respectively, so that the main encoder encodes the target image into a first detail feature representation, and the auxiliary encoder encodes the target image into a first structural feature representation; The first detail feature representation output by the main encoder and the first structural feature representation output by the auxiliary encoder are obtained, and feature fusion is performed on the first detail feature representation and the first structural feature representation to obtain an image feature representation corresponding to the target image.
[0014] Optionally, at least one convolutional layer is deployed in both the main encoder and the auxiliary encoder; wherein the at least one convolutional layer is used to adjust the resolution of the first detail feature representation or the first structural feature representation to a first target value; and to adjust the number of channels of the first detail feature representation or the first structural feature representation to a second target value; Obtain the first detail feature representation output by the main encoder and the first structural feature representation output by the auxiliary encoder, and perform feature fusion on the first detail feature representation and the first structural feature representation, including: Obtain the first detail feature representation of the main encoder outputting through the at least one convolutional layer with a resolution of the first target value and a channel number of the second target value, and obtain the first structural feature representation of the auxiliary encoder outputting through the at least one convolutional layer with a resolution of the first target value and a channel number of the second target value; The obtained first detail feature representation and first structural feature representation are superimposed on pixel values along the channel dimension to obtain an image feature representation corresponding to the target image.
[0015] Optionally, the deep compression network includes a compression network and a second encoding network; wherein, the compression network is used to compress the image feature representation into a latent feature representation in a downsampling manner; the second encoding network is used to compress and encode the latent feature representation to obtain a bitstream corresponding to the target image; The image feature representation is further input into the deep compression network, which compresses the image feature representation into a latent feature representation, and then performs compression encoding on the latent feature representation to obtain a bitstream corresponding to the target image, including: The image feature representation is further input into the compression network, so that the compression network compresses the image feature representation into a latent feature representation in a downsampling manner; The latent feature representation output by the compression network is obtained, and the obtained latent feature representation is further input into the second encoding network so that the latent feature representation is compressed and encoded by the second encoding network to obtain a bitstream corresponding to the target image.
[0016] Optionally, the encoding network further includes a first decoding network corresponding to the first encoding network; the first decoding network includes multiple network branches; wherein the multiple network branches include a main decoder corresponding to the main encoder; and an auxiliary decoder corresponding to the auxiliary encoder; The method further includes: The acquired latent feature representation is input into the auxiliary decoder, so that the auxiliary decoder decodes the latent feature representation into a second structural feature representation; The second structural feature representation output by the auxiliary decoder is obtained, and the obtained second structural feature representation is further input into the main decoder so that the main decoder decodes the second structural feature representation into a second detail feature representation; The second detail feature representation output by the main decoder is obtained, and the obtained second detail feature representation and the target image are input to the LLM accessed by the encoding end network, so that the LLM can perform inference calculation on the second detail feature representation and the target image to generate text information describing the semantic differences between the second detail feature representation and the target image; The text information output by the LLM is obtained, the text information is encoded, and the encoded text information is written into the bitstream corresponding to the target image.
[0017] Optionally, the main encoder includes the encoder contained in the VAE part of the diffusion model that has been fine-tuned based on LoRA technology; the auxiliary encoder includes a compression network contained in the encoder end of a pre-trained and parameter-frozen deep compression codec; wherein, the compression network supports downsampling to compress the input feature representation into a latent feature representation. The main decoder includes the decoder contained in the VAE part of the diffusion model, which is fine-tuned based on LoRA technology; the auxiliary decoder includes a decompression network contained in the decoder end of a pre-trained and parameter-frozen deep compression codec; wherein, the decompression network supports decompressing the input latent feature representation into an image feature representation corresponding to the target image by oversampling.
[0018] Optionally, the diffusion model includes SD-Turbo; the deep compression codec includes the ELIC model; the compression network includes the analysis transform network in the ELIC model; and the decompression network includes the synthesis transform network in the ELIC model.
[0019] Optionally, the decoding network includes a deep decompression network, a generation network, and a second decoding network; The deep decompression network is used to perform compression decoding on the latent feature representation contained in the bitstream in accordance with the compression encoding, and to decompress the latent feature representation after compression decoding to obtain the image feature representation corresponding to the target image. The generating network is used to take the text information contained in the bitstream as a guiding condition and process the image feature representation output by the deep decompression network into a third detail feature representation for reconstructing the target image; The second decoding network is used to decode the latent feature representation after compression and decoding output by the deep decompression network into a third structural feature representation for reconstructing the target image, and to fuse the third structural feature representation and the third detail feature representation to obtain a reconstructed feature representation for reconstructing the target image, and to reconstruct the target image based on the reconstructed feature representation.
[0020] Optionally, the decoding network performs image reconstruction based on the text information and image feature representation contained in the bitstream to obtain the target image, including: The latent feature representation contained in the bitstream is input into the deep decompression network, so that the deep decompression network performs compression decoding on the latent feature representation corresponding to the compression encoding, and decompresses the latent feature representation after compression decoding to obtain the image feature representation corresponding to the target image; The latent feature representation and the image feature representation after compression and decoding are obtained from the deep decompression network. The text information contained in the bitstream and the obtained image feature representation are then input into the generator network. The generator network uses the text information as a guiding condition to process the image feature representation into a third detail feature representation for reconstructing the target image. The third detail feature representation output by the generating network is obtained. The latent feature representation after compression and decoding and the obtained third detail feature representation are input into the second decoding network. The second decoding network decodes the latent feature representation after compression and decoding into a third structural feature representation for reconstructing the target image. The third structural feature representation and the third detail feature representation are fused to obtain a reconstructed feature representation for reconstructing the target image. The target image is then reconstructed based on the reconstructed feature representation.
[0021] Optionally, the deep decompression network includes a third decoding network and a decompression network; wherein, the third decoding network is used to perform compression decoding on the latent feature representation corresponding to the compression encoding; and the decompression network is used to decompress the latent feature representation into an image feature representation corresponding to the target image by means of upsampling. The latent feature representation contained in the bitstream is input into the deep decompression network, which performs compression and decoding on the latent feature representation corresponding to the compression encoding, and then decompresses the latent feature representation after compression and decoding to obtain the image feature representation corresponding to the target image, including: The latent feature representation contained in the bitstream is input to the third decoding network, so that the third decoding network performs compression decoding on the latent feature representation corresponding to the compression encoding; The latent feature representation after compression and decoding output by the third decoding network is obtained, and the obtained latent feature representation after compression and decoding is further input into the decompression network so that the latent feature representation is decompressed into an image feature representation corresponding to the target image by the third decoding network in an upsampling manner.
[0022] Optionally, the generation network includes a learning network and an image generation network; wherein, the learning network is used to learn the image semantic features described by the text information contained in the bitstream; the image generation network is used to use the image semantic features as guiding conditions to generate a third detail feature representation for reconstructing the target image based on the image feature representation output by the deep decompression network; The text information contained in the bitstream and the obtained image feature representation are further input into the generator network, so that the generator network uses the text information as a guiding condition to process the image feature representation into a third detail feature representation for reconstructing the target image, including: The text information contained in the bitstream is input into the learning network so that the learning network can learn the image semantic features described by the text information from the text information; The image semantic features output by the learning network are obtained, and the image semantic features and the obtained image feature representation are further input into the image generation network, so that the image generation network uses the image semantic features as a guiding condition to generate a third detail feature representation for reconstructing the target image based on the image feature representation.
[0023] Optionally, the learning network includes a CLIP network; the image generation network includes a denoising network in a diffusion model fine-tuned based on LoRA technology. The image semantic features and the acquired image feature representation are further input into the image generation network, so that the image generation network uses the image semantic features as a guiding condition to generate a third detail feature representation for reconstructing the target image based on the image feature representation, including: The image semantic features and the obtained image feature representation are further input into the denoising network, so that the denoising network uses the image semantic features as a guiding condition to denoise the image feature representation, thereby generating a third detail feature representation for reconstructing the target image.
[0024] Optionally, the diffusion model includes SD-Turbo; the denoising network includes the U-net network in the SD-Turbo model.
[0025] Optionally, the second decoding network includes multiple network branches; wherein the multiple network branches include an auxiliary decoder for decoding the latent feature representation after compression decoding into a third structural feature representation for reconstructing the target image; and a main decoder for decoding the reconstructed feature representation into the target image; The second decoding network decodes the compressed latent feature representation into a third structural feature representation for reconstructing the target image, and fuses the third structural feature representation and the third detail feature representation to obtain a reconstructed feature representation for reconstructing the target image, and reconstructs the target image based on the reconstructed feature representation, including: The latent feature representation after compression and decoding is input into the auxiliary decoder, so that the auxiliary decoder decodes the latent feature representation after compression and decoding into a third structural feature representation for reconstructing the target image; The third structural feature representation output by the auxiliary decoder is obtained, and the obtained third structural feature representation and the third detail feature representation are fused to obtain a reconstructed feature representation for reconstructing the target image; The reconstructed feature representation is further input into the main decoder, which decodes the reconstructed feature representation into the target image to complete the reconstruction of the target image.
[0026] Optionally, the deep compression network includes an encoder-side network of a pre-trained and parameter-frozen deep compression codec; the deep decompression network includes at least a decoder-side network of a pre-trained and parameter-frozen deep compression codec.
[0027] Optionally, the deep compression codec includes the ELIC model; The encoder-side network is a network composed of the analytical transformation network, the advanced prior analytical transformation network, the SCCTX network, and the arithmetic encoder in the ELIC model. The decoder-side network is a network composed of the synthesis transform network, the super-prior synthesis transform network, the SCCTX network, and the decoder in the ELIC model. Accordingly, the compression network includes the analysis transformation network in the ELIC model; the second encoding network includes a network composed of the super-prior analysis transformation network, the SCCTX network, and the encoder in the ELIC model; The third decoding network includes a network composed of the super-prior synthesis transform network, the SCCTX network, and the arithmetic decoder in the ELIC model; the decompression network includes the synthesis transform network in the ELIC model.
[0028] Optionally, the neural network further includes a discriminant network; wherein the discriminant network is used to determine the image type of the input target image; the image type includes the original target image and the reconstructed target image; The training loss used to train the neural network includes: Rate-distortion loss associated with compression and image reconstruction of the target image; wherein the rate-distortion loss includes bitrate loss and distortion loss; Perceptual loss is used to describe the semantic loss of the reconstructed target image compared to the original target image; The discriminant network calculates the discriminant loss for the input target image.
[0029] Optionally, the perceptual loss is represented by LPIPS loss; The discriminant network includes a classification network for classifying the input target image; the image type of the target image output by the classification network includes the original target image and the reconstructed target image; correspondingly, the discriminant loss is represented by the cross-entropy loss for classifying the target image.
[0030] Optionally, training the neural network includes: The first stage of training involves training the neural network using the weighted sum of the bitrate loss, the distortion loss, and the perceptual loss as the target training loss; wherein, in the first stage of training, the weight value of the bitrate loss is set to a first weight value. The second stage of training involves using the weighted sum of the bitrate loss, the distortion loss, the perceptual loss, and the discriminative loss as the target training loss to train the neural network. In this second stage, the weight value of the bitrate loss is set as a second weight value, which is greater than the first weight value.
[0031] Optionally, the encoding method used to compress the target image includes entropy coding.
[0032] According to a second aspect of the embodiments of this specification, a neural network is also provided, comprising: The encoding network compresses and encodes the input target image to generate a bitstream corresponding to the target image; wherein the bitstream includes an image feature representation obtained by compressing and encoding the target image; and text information describing the semantic differences between the reconstructed image based on the image feature representation and the target image. A decoding network is used to perform image reconstruction based on the text information and image feature representation contained in the bitstream transmitted to the local network, so as to obtain the target image.
[0033] According to a third aspect of the embodiments of this specification, an electronic device is also provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the method as described in any one of the first aspects.
[0034] According to a fourth aspect of the embodiments of this specification, a computer program product is also provided, comprising a computer program / instructions that, when executed by a processor, implement the method as described in any one of the first aspects.
[0035] The above technical solutions have the following technical effects: The scheme disclosed in the embodiments of this specification can package the image feature representation obtained by compressing and encoding the target image together with the text information describing the semantic differences between the reconstructed image and the target image into the same bitstream, so that the decoding end can use the text information as a guiding condition with clear semantic content, and combine it with the image feature representation to complete the image reconstruction of the target image.
[0036] This design fundamentally changes the way traditional image compression schemes handle reconstruction errors.
[0037] In traditional solutions, if auxiliary information needs to be added to the bitstream to compensate for reconstruction loss, pixel-level residual images are typically used. This involves encoding the pixel-by-pixel difference between the original and reconstructed images and writing it into the bitstream. However, the pixel-level residual is itself an image, with a data volume on the same order of magnitude as the original image. In low bitrate scenarios, this overhead is often unbearable. The above-mentioned technical solution replaces the pixel-level residual with semantic-level text descriptions. The data volume of text is much smaller than that of the residual image. Therefore, without significantly increasing the bitrate burden, the bitstream can carry effective information describing the semantic differences of the image content.
[0038] It is evident that, compared to the randomness inherent in image reconstruction without guidance in related technologies, the above technical solutions, by introducing explicit semantic guidance, not only address the problem of excessively long summarization and reasoning time in the image reconstruction process, making it difficult to meet the needs of real-time applications, but also avoid the predicament of low reconstruction quality due to the lack of effective guidance from noise factors. Thus, under extremely low bitrate conditions, it is possible to achieve a synergistic balance between the realism, fidelity, and real-time performance of the reconstructed image.
[0039] For example, in related technologies, image reconstruction is achieved by using a diffusion model to denoise images based on pixel-level residuals. However, due to the lack of accurate semantic guidance, the diffusion model typically requires dozens of iterative denoising steps to complete image reconstruction. The above-mentioned solution, by introducing a more lightweight semantic guidance, can effectively restore image details in a single step of denoising at extremely low bitrates, eliminating the need for dozens of iterations. This ensures both the realism and fidelity of the reconstructed image while maintaining real-time performance. Attached Figure Description
[0040] Figure 1 This is a schematic diagram of the overall system architecture of a neural network provided in an exemplary embodiment.
[0041] Figure 2 This is a schematic diagram of the overall process of an image compression method provided in an exemplary embodiment.
[0042] Figure 3A This is a detailed network architecture diagram of an encoding end network provided in an exemplary embodiment.
[0043] Figure 3B This is a schematic diagram of a detailed network architecture for a decoding end network, provided in an exemplary embodiment.
[0044] Figure 4This is a schematic diagram of the data flow of a local decoding branch and semantic residual extraction process within an encoding end network, provided in an exemplary embodiment.
[0045] Figure 5 This is an exemplary embodiment of a data flow diagram between modules within a decoding network.
[0046] Figure 6 This is a schematic diagram illustrating the loss function structure of a two-stage training strategy provided in an exemplary embodiment.
[0047] Figure 7 This is a schematic diagram of the structure of an electronic device provided in an exemplary embodiment.
[0048] Figure 8 This is a block diagram of an image compression device based on a neural network, provided in an exemplary embodiment. Detailed Implementation
[0049] To enable those skilled in the art to better understand the technical solutions in this specification, the technical solutions in the embodiments of this specification will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this specification, and not all embodiments. Based on the embodiments in this specification, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of this specification.
[0050] The user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this manual are all information and data authorized by the user or fully authorized by all parties. The collection, use and processing of related data shall comply with the relevant laws, regulations and standards of the relevant countries and regions, and corresponding operation portals shall be provided for users to choose to authorize or refuse.
[0051] The essence of image compression is to preserve the visual information of an image as completely as possible within the constraints of limited storage or transmission resources. In related technologies, the core metric for evaluating the quality of an image compression method is usually called rate-distortion performance, which is the degree of distortion between the reconstructed image and the original image at a given bitrate (unit: bpp, bits per pixel). The lower the distortion, the better the compression method.
[0052] Over the past few decades, the field of image compression has developed a traditional encoding and decoding standard system represented by JPEG, HEVC (High Efficiency Video Coding), and VVC (Various Functional Video Coding).
[0053] The common technical approach of these standards is to transform the image from pixel space to frequency space (such as discrete cosine transform), quantize the transform coefficients, and then generate the final bitstream through entropy coding. This process has been optimized through decades of engineering and performs well under medium to high bitrate conditions, achieving high compression efficiency while ensuring the visual quality of the reconstructed image.
[0054] In recent years, image compression methods based on deep learning neural networks have gradually emerged. The typical approach of this type of method is to use an encoding network to map the original image to a low-dimensional latent representation, and then compress the latent representation into a bitstream through entropy coding; during decoding, a decoding network is used to restore the latent representation to the reconstructed image.
[0055] This process is similar in concept to traditional encoding and decoding, but the difference lies in the fact that the transformations used for feature extraction and reconstruction are learned by neural networks, rather than manually designed mathematical transformations. For example, learning-based image compression methods, represented by the ELIC model, have been able to compete with traditional standards in the medium bitrate range, and even surpass them in some metrics, by jointly optimizing bitrate loss and distortion loss.
[0056] However, both traditional encoding / decoding and learning-based compression methods share a common optimization goal: pixel-level distortion, represented by mean squared error (MSE). This optimization approach works well when the bitrate is sufficient, resulting in a sufficiently small pixel difference between the reconstructed image and the original image, which is close to reality in human perception.
[0057] However, when the bitrate is compressed to an extremely low level, the amount of information contained in the latent representation becomes severely insufficient, and the decoding network cannot reconstruct rich image details with only limited information. At this point, models that optimize MSE tend to generate an average of all possible reconstruction results. In this case, the general outline and color of the image are still present, but the texture details are smoothed out, resulting in an overall blurry visual effect.
[0058] This phenomenon has a theoretical basis in the field of signal processing: when information loss is severe, the optimal solution to minimize the mean square error is the output conditional expectation, and the conditional expectation itself is smooth.
[0059] In other words, the blurring problem of traditional methods at low bit rates is not a defect in engineering implementation, but an inevitable result determined by the optimization objective itself.
[0060] To improve visual quality at low bit rates, researchers have begun to introduce generative models into image compression tasks, especially the diffusion model, which has made breakthroughs in recent years.
[0061] The core capability of the diffusion model lies in its ability to fill in missing details and generate a visually realistic and natural image, given a noisy or incomplete input, through an iterative denoising process. The basic idea behind using the diffusion model for image compression is that the encoder transmits only a small amount of compressed information, while the decoder utilizes the diffusion model's generative capabilities to complete the details, thereby achieving a high-quality reconstruction result at an extremely low bitrate.
[0062] This approach is theoretically very attractive, but in practice it exposes two unavoidable problems.
[0063] The first problem is reasoning efficiency.
[0064] The standard diffusion model's denoising process is iterative, typically requiring 20 to 50 iterations to generate an image of acceptable quality. Each denoising step calls a massive neural network (such as U-Net), resulting in extremely high computational overhead. In image compression decoding scenarios, this means decoding a single image requires dozens of network inferences, often taking several seconds, far from meeting the speed requirements of real-time communication, video conferencing, and live streaming. Theoretically, the number of denoising steps could be reduced to one, significantly improving speed, but the standard diffusion model experiences a significant drop in output quality during single-step denoising—because single-step denoising lacks sufficient constraints; without explicit guidance, the model can only rely on its prior knowledge to randomly generate results, often deviating from the original image content.
[0065] The second issue is rebuilding consistency.
[0066] The diffusion model is essentially a generative model, and its denoising process includes a random sampling step, meaning the same input may produce different outputs each time it runs. In image generation tasks, this diversity is advantageous; however, in image compression and reconstruction tasks, the goal is to recreate the original image, not create a new one. This randomness can lead to significant deviations in content between the reconstructed image and the original image—for example, facial features may be replaced, or text content may be altered. This is unacceptable in practical applications requiring faithful reconstruction.
[0067] The root cause of the two problems mentioned above can be attributed to the same point: existing image compression methods based on diffusion models lack a guiding information mechanism that is both lightweight and accurately describes the differences between the original and reconstructed images. If the guiding information is absent or inaccurate, single-step denoising cannot directionally recover the missing details; if the guiding information exists in the form of pixel-level residual images, its data volume is already close to that of a complete image. Introducing such auxiliary information almost negates the purpose of compression, and under the constraint of extremely low bitrates, this additional bitrate overhead is simply unbearable.
[0068] In view of this, this application provides an image compression method based on neural networks. At the encoding end, the difference between the reconstructed image and the original image is extracted in the form of semantic text description, and the text information and image feature representation are written together into the bitstream. At the decoding end, the missing semantic details can be filled in directionally during the image reconstruction process using the text information as a guiding condition, thereby solving the problems of lack of guidance in image reconstruction and deviation of reconstructed content from the original image without significantly increasing the bitrate.
[0069] The exemplary embodiments of this specification will now be described in detail with reference to the accompanying drawings.
[0070] Please see Figure 1 , Figure 1 A schematic diagram of the overall system architecture of the neural network involved in this application is shown.
[0071] like Figure 1 As shown, the neural network (10) involved in this application consists of two core parts: An encoding network (100) is deployed on the transmitting side and a decoding network (200) is deployed on the receiving side.
[0072] The encoding network (100) and the decoding network (200) can be deployed on different devices that are interconnected by the network, and the two communicate with each other by transmitting code streams (300).
[0073] The encoding network (100) is responsible for receiving the target image (101) to be compressed and compressing the target image (101) to generate the corresponding bitstream (300). The bitstream (300) contains two types of information: one is the image feature representation (301) obtained after compressing and encoding the target image (101); the other is text information (302) used to describe the semantic differences between the reconstructed image based on the image feature representation and the target image (101).
[0074] The decoding network (200) is responsible for receiving the bitstream (300) and, based on the text information (302) and image feature representation (301) contained in the bitstream (300), completing image reconstruction and finally outputting the reconstructed target image (201).
[0075] The above-mentioned encoding network (100) and decoding network (200) can be trained centrally and deployed separately: during the training phase, the encoding network (100) and decoding network (200) are trained end-to-end as a whole; during the deployment phase, the trained encoding network (100) and decoding network (200) are split and deployed on the transmitting side device and the receiving side device respectively, and the bit stream is transmitted and received through network connection.
[0076] Please see Figure 2 , Figure 2 This is a flowchart illustrating an image compression method based on a neural network, as shown in this specification. The neural network can employ... Figure 1 The network structure shown may specifically include an encoding network for compressing and encoding the image and a decoding network for reconstructing the compressed image; the method includes the following execution process: Step 202: Obtain the target image to be compressed; The execution subject of the above method can be any computing device that has the encoding end network (100) installed and running, such as a smartphone, personal computer, server, edge computing node, etc. In practical applications, the encoding end network (100) can be integrated into the image acquisition and transmission application as a software module, or it can be deployed on the server as an independent inference service to compress the uploaded images before transmission or storage.
[0077] The target image refers to the original image that needs to be compressed. The source of the target image can be diverse, such as real-time images captured by the user through a camera, image files read from local storage, image data received through a network interface, etc.
[0078] The target image can be in formats including but not limited to common image formats such as PNG, JPEG, BMP, and TIFF. Before being input to the encoding network (100), it can be decoded into the form of a raw pixel matrix (i.e., an array of pixel values in color spaces such as RGB or YUV).
[0079] Before acquiring the target image, necessary preprocessing operations can be performed on the target image.
[0080] For example, the target image can be cropped or scaled to the fixed input resolution required by the neural network (such as 256×256 pixels); pixel values can be normalized, mapping them from the integer range of [0, 255] to the floating-point range of [0, 1] or [-1, 1] to adapt to the input requirements of the neural network; and image data can be converted into tensor format for processing by subsequent network layers.
[0081] Step 204: Input the target image into the encoding end network so that the encoding end network can compress and encode the target image to generate a bitstream corresponding to the target image; wherein, the bitstream includes image feature representation obtained by compressing and encoding the target image; and text information used to describe the semantic differences between the reconstructed image reconstructed based on the image feature representation and the target image. After the target image is acquired and preprocessed, it can be input into the encoding end network (100) and compressed and encoded by the encoding end network (100) to generate the corresponding bit stream (300).
[0082] It should be noted that the image feature representation mentioned above refers to the representation of the original image mapped into a certain feature space after being encoded by a neural network. Compared with the original pixel representation, the image feature representation is usually lower in dimensionality and more compact, but retains the main structural information of the image. After further compression (such as sampling and quantization), the image feature representation is encoded into the main part of the bitstream, which is the basis for the decoding end to recover the image structure.
[0083] The aforementioned text information refers to textual content described in natural language about the semantic differences between the reconstructed image and the original image.
[0084] For example, suppose the original image is a photograph of a face. The compressed image feature representation can only roughly restore the facial contours and skin tone, but details of the eyes (such as double eyelid lines and the layering of eyelashes) are lost during compression. The corresponding text information might be: "The person's eyes have clear double eyelid lines and thick eyelashes." This text describes the semantic gap between the reconstructed image and the original image, providing directional guidance for the decoding end to complete the details.
[0085] It can be seen that text information is essentially a semantic residual. Compared to the pixel-level residual images (i.e., the difference image obtained by subtracting the original image from the reconstructed image pixel by pixel) that may be used in traditional solutions, the data volume of text information is much smaller than that of an image. A piece of natural language text typically only requires tens to hundreds of bytes after encoding, while a pixel-level residual image is itself an image, and its encoding overhead is unbearable in low bitrate scenarios. Therefore, using text information as a semantic residual can transmit key semantic completion instructions to the decoding end with extremely low additional bitrate overhead.
[0086] In order to compress the target image into a bitstream containing image feature representation and text information, the encoding end network (100) adopts a staged processing method internally.
[0087] In some embodiments, the encoding end network includes a first encoding network and a deep compression network; wherein the first encoding network is used to encode the target image into an image feature representation; the deep compression network is used to compress the image feature representation into a latent feature representation, and to perform compressed encoding on the latent feature representation; inputting the target image into the encoding end network to compress and encode the target image to generate a bitstream corresponding to the target image includes: inputting the target image into the first encoding network to encode the target image into an image feature representation; obtaining the image feature representation output by the first encoding network, and further inputting the image feature representation into the deep compression network to compress the image feature representation into a latent feature representation, and to perform compressed encoding on the latent feature representation to obtain a bitstream corresponding to the target image.
[0088] Specifically, please see Figure 3A and Figure 3B , Figure 3A The encoding end network (100) is shown. Figure 3B A detailed network architecture diagram of the decoding network (200) is shown, in which the encoding network (100) is structurally divided into two processing stages: The first stage is undertaken by the first encoding network (110). The first encoding network (110) receives the original target image (101) as input, maps it from the pixel space to the feature space, and outputs the image feature representation (111). The core function of this step is to extract the semantic features of the image, so that the subsequent processing no longer deals with the original pixels, but with feature vectors that are more semantically expressive.
[0089] The second stage is undertaken by the deep compression network (120). The deep compression network (120) receives the image feature representation (111) output by the first coding network (110), and further compresses it into a lower-dimensional latent feature representation (121) through downsampling and other means. Then, the latent feature representation (121) is compressed and encoded (such as entropy encoding) to finally generate the part of the bitstream (300) that contains the image feature representation.
[0090] The two stages work in series, and the target image (101) goes through a complete compression path of "feature extraction → deep compression → entropy coding". The advantage of this staged design is that the first encoding network (110) can focus on transforming pixel information into semantically rich feature representations, while the deep compression network (120) focuses on extreme compression of the feature representations. The two are trained for their own optimization goals, with clear division of labor, which helps to balance feature semantic quality and compression efficiency at extremely low bit rates.
[0091] In practical applications, when extracting image features, a single coding branch often struggles to simultaneously consider both detailed information (such as high-frequency information like texture and edges) and structural information (such as low-frequency information like object contours and spatial layout). To address this issue, the embodiments in this specification have specifically designed the internal structure of the first coding network (110).
[0092] In some embodiments, the first encoding network includes multiple network branches; wherein the multiple network branches include a main encoder for encoding the target image into a first detail feature representation; and an auxiliary encoder for encoding the target image into a first structural feature representation; the target image is input to the first encoding network to encode the target image into an image feature representation; the target image is input to the main encoder and the auxiliary encoder respectively, so that the main encoder encodes the target image into a first detail feature representation and the auxiliary encoder encodes the target image into a first structural feature representation; the first detail feature representation output by the main encoder and the first structural feature representation output by the auxiliary encoder are obtained, and feature fusion is performed on the first detail feature representation and the first structural feature representation to obtain an image feature representation corresponding to the target image.
[0093] like Figure 3A As shown, the first encoding network (110) adopts a dual-branch parallel processing structure internally, including a main encoder (112) and an auxiliary encoder (113). The target image (101) is simultaneously input to the main encoder (112) and the auxiliary encoder (113), and the two encoders process the target image independently to extract their respective focused feature types.
[0094] The main encoder (112) is responsible for extracting the detail features of the target image and outputting the first detail feature representation (112a). The so-called "detail features" refer to information such as texture, edges, and high-frequency changes in the image, such as the weave of fabric, the direction of the veins in leaves, and the skin texture of a person's face. This type of information is crucial for reconstructing the perceptual realism of the image, but it is also the part that is most easily lost during the compression process.
[0095] The auxiliary encoder (113) is responsible for extracting the structural features of the target image and outputting the first structural feature representation (113a). "Structural features" refer to information such as the contours, boundaries, and spatial layout of objects in the image, such as the positional relationships of facial features, the lines of buildings, and the relative positions of objects in a scene. This type of information determines whether the overall semantic content of the reconstructed image is consistent with the original. Figure 1 Accuracy is the foundation of image reconstruction fidelity.
[0096] After obtaining the outputs of the two encoders, the first detail feature representation (112a) and the first structural feature representation (113a) are fused to obtain the image feature representation (111). This image feature representation (111) contains complementary information from both branches, providing more comprehensive input for the subsequent deep compression network (120).
[0097] In the dual-branch structure, the network structures of the main encoder (112) and the auxiliary encoder (113) are different, and the feature representations output by the two usually differ in spatial resolution and number of channels. If they are directly fused without processing, the correspondence between the features will be misaligned, resulting in the fusion result failing to effectively integrate the information of the two types of features. To solve this problem, the embodiments of this specification introduce convolutional layers in the main encoder and the auxiliary encoder respectively for feature alignment.
[0098] In some embodiments, at least one convolutional layer is deployed in the main encoder and the auxiliary encoder respectively; wherein the at least one convolutional layer is used to adjust the resolution of the first detail feature representation or the first structural feature representation to a first target value; and to adjust the number of channels of the first detail feature representation or the first structural feature representation to a second target value; At this point, the first detail feature representation output by the main encoder and the first structural feature representation output by the auxiliary encoder are obtained, and feature fusion is performed on the first detail feature representation and the first structural feature representation. Specifically, this may include: obtaining the first detail feature representation output by the main encoder via the at least one convolutional layer with a resolution of the first target value and a channel number of the second target value, and obtaining the first structural feature representation output by the auxiliary encoder via the at least one convolutional layer with a resolution of the first target value and a channel number of the second target value; and superimposing the obtained first detail feature representation and the first structural feature representation in the channel dimension with pixel values to obtain an image feature representation corresponding to the target image.
[0099] like Figure 3A As shown, the main encoder (112) deploys at least one convolutional layer (114a) at its output, and the auxiliary encoder (113) also deploys at least one convolutional layer (114b) at its output. The convolutional layer here can be the last layer of the main encoder or the auxiliary encoder, or it can be one of the last few layers. Its core functions are twofold: First, the spatial resolution of the feature representation is adjusted to the first target value. For example, if the spatial resolution of the first detail feature representation (112a) output by the main encoder (112) is 32×32, and the spatial resolution of the first structural feature representation (113a) output by the auxiliary encoder (113) is 16×16, then through their respective convolutional layers (114a, 114b), the resolution of both is uniformly adjusted to the same target value (such as 32×32) to ensure that they correspond position by position in the spatial dimension.
[0100] Secondly, the number of channels in the feature representation is adjusted to the second target value. In a neural network, the number of channels in the feature representation determines the dimension of the feature vector at each spatial location. Due to different network designs, the main encoder (112) and the auxiliary encoder (113) may have different numbers of channels in their output features. By using convolutional layers, the number of channels in both is adjusted to a unified second target value, ensuring that they can be directly superimposed in terms of channel dimension.
[0101] After alignment, the first detail feature representation (112a) and the first structural feature representation (113a), whose resolution and number of channels are now unified, are superimposed on pixel values along the channel dimension. "Superimposing pixel values along the channel dimension" means that for each corresponding position in space, the feature vectors from the two branches are directly concatenated together to form a new feature vector with the number of channels equal to the sum of the two.
[0102] For example, if both aligned branches have 128 channels, the resulting image feature representation (111) will have 256 channels. This operation is simple and efficient, and can retain all feature information from both branches without loss, providing informative input for subsequent processing.
[0103] It should be noted that in practical applications, feature fusion can also be performed without channel-wise concatenation. Instead, element-wise addition can be used, directly adding the feature values of the two branches at corresponding positions and channels. The advantage of this method is that the number of feature channels remains unchanged after fusion, and the computational cost is relatively low. The disadvantage is that the feature information of the two branches may interfere with each other during the addition process. The specific fusion method used can be selected based on the actual network design requirements and target bitrate.
[0104] In order to further compress the image feature representation into a compact latent representation suitable for entropy coding, the internal structure of the deep compression network (120) in this embodiment is designed as follows.
[0105] In some embodiments, the deep compression network includes a compression network and a second encoding network; wherein the compression network is used to compress the image feature representation into a latent feature representation in a downsampling manner; and the second encoding network is used to compress and encode the latent feature representation to obtain a bitstream corresponding to the target image. In this case, the image feature representation is further input into the deep compression network, so that the deep compression network compresses the image feature representation into a latent feature representation, and the latent feature representation is compressed and encoded to obtain a bitstream corresponding to the target image. Specifically, this may include: The image feature representation is further input into the compression network, so that the compression network compresses the image feature representation into a latent feature representation in a downsampling manner; the latent feature representation output by the compression network is obtained, and the obtained latent feature representation is further input into the second encoding network, so that the second encoding network compresses and encodes the latent feature representation to obtain a bitstream corresponding to the target image.
[0106] like Figure 3A As shown, the deep compression network (120) is internally composed of two modules, a compression network (122) and a second encoding network (123), connected in series.
[0107] The compression network (122) receives the image feature representation (111) as input and performs spatial dimension compression on it by downsampling, transforming it into a lower-dimensional latent feature representation (121).
[0108] Downsampling means that by using a convolution operation with a stride greater than 1 (i.e., strided convolution), the spatial resolution of the input features is reduced by a certain proportion, while the number of channels is increased. This reduces spatial redundancy and preserves the main information of the image in a more compact form.
[0109] For example, the spatial resolution of the input image feature representation (111) is 32×32 and the number of channels is 256. After downsampling processing by the compression network (122), the spatial resolution of the output latent feature representation (121) can be reduced to 16×16 or 8×8, and the number of channels is adjusted accordingly. The overall data volume is greatly reduced, creating favorable conditions for subsequent entropy coding.
[0110] The second encoding network (123) receives the latent feature representation (121) output by the compression network (122), compresses and encodes it, and generates compressed data that is finally written into the bitstream (300).
[0111] The role of compression coding is to convert the numerical information in the latent feature representation (121) into binary codewords and to minimize the total length of the codewords. This process is usually achieved using entropy coding techniques, the specific methods of which will be described in detail later.
[0112] After completing the compression encoding of the target image, the encoding end network (100) also needs to complete a key task: generate text information describing the semantic differences between the reconstructed image and the target image, and write it into the bitstream.
[0113] To achieve this goal, the encoder network (100) is configured with a local decoding branch to simulate part of the reconstruction process at the encoder without transmitting the complete reconstructed image, thereby obtaining a local reconstruction result for comparison with the original image.
[0114] In some embodiments, the encoding network further includes a first decoding network corresponding to the first encoding network; the first decoding network includes multiple network branches; wherein the multiple network branches include a main decoder corresponding to the main encoder; and an auxiliary decoder corresponding to the auxiliary encoder; In this case, the method further includes: The acquired latent feature representation is input into the auxiliary decoder, so that the auxiliary decoder decodes the latent feature representation into a second structural feature representation; The second structural feature representation output by the auxiliary decoder is obtained, and the obtained second structural feature representation is further input into the main decoder so that the main decoder decodes the second structural feature representation into a second detail feature representation; Then, the second detail feature representation output by the main decoder is obtained, and the obtained second detail feature representation and the target image are input to the LLM accessed by the encoding end network, so that the LLM performs inference calculation on the second detail feature representation and the target image to generate text information describing the semantic differences between the second detail feature representation and the target image; the text information output by the LLM is obtained, the text information is encoded, and the encoded text information is written into the bitstream corresponding to the target image.
[0115] Please see Figure 4 , Figure 4 A schematic diagram of the data flow of the local decoding branch and semantic residual extraction process inside the encoding end network (100) is shown.
[0116] like Figure 4As shown, the first decoding network (130) also adopts a dual-branch structure internally, including an auxiliary decoder (131) and a main decoder (132), which are structurally corresponding to the auxiliary encoder (113) and the main encoder (112) at the encoding end, respectively.
[0117] At this point, the workflow of the local decoding branch on the encoding end is as follows: First, the latent feature representation (121) output by the deep compression network (120) is input to the auxiliary decoder (131). The auxiliary decoder (131) is responsible for decoding the latent feature representation (121) into a second structural feature representation (131a).
[0118] The second structural feature representation mentioned above is a structural feature obtained by decoding the latent feature representation (121) locally at the encoding end. It reflects the restoration of image structural information under the current compression level. Compared with the first structural feature representation (113a), the second structural feature representation (131a) has undergone a round trip of compression and decompression, and inevitably suffers some information loss.
[0119] Next, the second structural feature representation (131a) output by the auxiliary decoder (131) is input to the main decoder (132). The main decoder (132) decodes the second structural feature representation (131a) into a second detail feature representation (132a). The second detail feature representation (132a) represents the image features containing detailed information reconstructed locally at the encoding end.
[0120] After obtaining the second detail feature representation (132a) output by the main decoder (132), the second detail feature representation (132a) and the original target image (101) are input together into the LLM (140) accessed by the encoding end network (100). The LLM (140) performs inference calculations on the second detail feature representation (132a) and the target image (101), identifies the differences between the two at the semantic level, and outputs a description of the differences in the form of natural language text, i.e., text information (141).
[0121] It should be noted that in the input received by LLM (140), the target image (101) represents "what the original image should look like", and the second detail feature representation (132a) represents "what the reconstructed image should look like at the current compression level". The task of LLM (140) is to find the semantic gap between the two and describe this gap in natural language.
[0122] For example, LLM (140) might output a text description such as "The brick texture of the building is clearly visible in the original image, but this detail is blurred in the reconstructed image". In a specific embodiment, LLM (140) can use a large language model with multimodal understanding capabilities, such as GPT-4o, which can process image and text inputs simultaneously, directly compare and analyze image features and the original image, and generate accurate semantic difference descriptions.
[0123] After obtaining the text information (141) output by LLM (140), the text information (141) is encoded and the encoded text information is written into the bitstream (300).
[0124] The specific methods for encoding text information can include: first, tokenizing the text to convert it into a token sequence; then, converting the token sequence into a corresponding embedding index; and finally, entropy encoding (such as arithmetic encoding) is applied to the embedding index, compressing it into binary codewords and writing it into the bitstream. After the above encoding, the number of bits occupied by the text information in the bitstream is far less than that of a pixel-level residual image, and the additional bitrate overhead is minimal.
[0125] Of course, in practical applications, instead of using LLM to generate semantic difference descriptions, other visual-language models (VLM) with image-text comparison capabilities can be used. As long as the model can receive two images or image features as input and output natural language text describing the semantic differences between the two, it can meet the requirements of this application.
[0126] In order to achieve the optimal division of labor between detail feature extraction and structural feature extraction, the embodiments of this specification have designed the specific network implementations of the main encoder (112), auxiliary encoder (113), main decoder (132) and auxiliary decoder (131) as follows.
[0127] In some embodiments, the main encoder includes an encoder contained in the VAE portion of a diffusion model fine-tuned using LoRA (Low-Rank Adaptation) technology; the auxiliary encoder includes a compression network contained in the encoder end of a pre-trained and parameter-frozen deep compression codec; wherein the compression network supports downsampling to compress the input feature representation into a latent feature representation; the main decoder includes a decoder contained in the VAE portion of the diffusion model fine-tuned using LoRA technology; the auxiliary decoder includes a decompression network contained in the decoder end of a pre-trained and parameter-frozen deep compression codec; wherein the decompression network supports upsampling to decompress the input latent feature representation into an image feature representation corresponding to the target image.
[0128] First, we will introduce the selection of the main encoder (112) and the main decoder (132).
[0129] Diffusion models commonly employ a submodule called VAE (Variational Autoencoder) for transformation between pixel space and latent feature space. A VAE consists of an encoder and a decoder: the encoder compresses and maps the input image from pixel space to a low-dimensional latent space, extracting high-level semantic features; the decoder restores the feature representation from the latent space back to pixel space, generating the reconstructed image. VAE encoders, pre-trained on large-scale image datasets, possess excellent image detail feature extraction capabilities, encoding detailed information such as texture, edges, and color variations of the input image into a compact latent representation.
[0130] In this embodiment, the VAE encoder is used as the main encoder (112), and the VAE decoder is used as the main decoder (132). To better adapt the VAE to image compression tasks, this embodiment uses LoRA technology to fine-tune the VAE.
[0131] It's important to note that LoRA is a lightweight neural network fine-tuning technique. Its basic idea is to add a bypass structure consisting of two low-rank matrices in parallel next to the key layers (usually the weight matrices of the attention layer) of the model without modifying the original parameters of the pre-trained model. During training, only the parameters of this bypass structure are updated, while the original parameters remain frozen. Because the number of parameters in the low-rank matrix is much smaller than the original weight matrix, LoRA fine-tuning introduces very few additional parameters, resulting in a much lower training cost than full-scale fine-tuning, while simultaneously aligning the model's output behavior towards the target task.
[0132] In this specification, the fine-tuning of VAE based on LoRA technology can better adapt the output feature representation to the downstream processing requirements of image compression tasks while retaining the original detailed feature extraction capabilities of VAE, without the need for complete retraining of the large pre-trained model.
[0133] The selection of auxiliary encoder (113) and auxiliary decoder (131) will be introduced next.
[0134] Deep compression codecs are neural network models specifically designed and trained for image compression tasks. The compression network at the encoder end is specifically responsible for compressing image feature representations into latent feature representations through downsampling, and has a strong ability to perceive and preserve the structural information of images.
[0135] This specification allows the compression network contained in the encoder end of the deep compression codec to be used as the auxiliary encoder (113), and the decompression network contained in the decoder end to be used as the auxiliary decoder (131). In order to avoid the gradient signals of other modules from interfering with the converged parameters of the auxiliary encoder and auxiliary decoder during joint training, the parameters of the auxiliary encoder (113) and auxiliary decoder (131) can be frozen, that is, their parameters are not updated during the training phase.
[0136] The advantage of this combination is that the main encoder (112) is good at extracting detail features, and the auxiliary encoder (113) is good at extracting structural features. The two complement each other, enabling the dual-branch structure to obtain high-quality detail information and structural information at the same time. The main decoder (132) and the auxiliary decoder (131) also correspond to detail recovery and structural restoration respectively, forming symmetry in the structure at both ends of the encoding and decoding, which helps the training process to converge stably.
[0137] In some embodiments, the diffusion model may include SD-Turbo; the deep compression codec may include the ELIC model; the compression network may include the analysis transform network in the ELIC model; and the decompression network may include the synthesis transform network in the ELIC model.
[0138] It's worth noting that SD-Turbo (Stable Diffusion Turbo) is a text-to-image generation model released by Stability AI that utilizes distillation acceleration. SD-Turbo employs the Adversarial Diffusion Distillation (ADD) training method to compress the generation process, which originally required 20 to 50 iterative denoising steps, into 1 to 4 steps, enabling it to output high-quality images with only one or a very small number of network forward propagations.
[0139] The VAE module inside SD-Turbo is optimized while retaining the structure of the original Stable Diffusion model VAE. Both its encoder and decoder have good image feature processing capabilities and fast inference speed, making it suitable for applications with real-time requirements. By using SD-Turbo's VAE encoder as the main encoder (112) and VAE decoder as the main decoder (132), the image detail understanding capabilities accumulated by SD-Turbo during large-scale training can be utilized to provide strong support for the extraction of detailed features in image compression tasks.
[0140] ELIC (Efficient Learned Image Compression with Unevenly Grouped Space-Channel Contextual Adaptive Coding) is an efficient learning-based image compression model.
[0141] The ELIC model internally includes an Analysis Transform Network (hereinafter referred to as...). ) and Synthesis Transform Network (hereinafter referred to as These two components are responsible for feature compression at the encoding end and feature restoration at the decoding end, respectively.
[0142] Analysis of transform networks ( As the compression network in the auxiliary encoder (113), it is a layer-by-layer downsampled volume. Integral neural networks are responsible for compressing the input image feature representation into a potential feature representation with lower spatial resolution and more compact semantics through stride convolution.
[0143] In this specification, the transformation network is analyzed ( The number of downsampled convolutional layers was adjusted to 2 to accommodate the overall network's requirements for resolution transformation.
[0144] Synthetic Transformation Network ( As the decompression network in the auxiliary decoder (131), it is a layer-by-layer upsampling convolutional neural network responsible for gradually restoring the latent feature representation to a higher spatial resolution image feature representation through upsampling operations such as transposed convolution or sub-pixel convolution.
[0145] In this specification, the synthetic transform network ( The number of upsampling convolutional layers was adjusted to 3 to match the downsampling depth of the analysis transform network, ensuring symmetrical matching of the feature space resolution at both ends of the encoding and decoding.
[0146] Of course, in practical applications, deep compression codecs may not use the ELIC model. Instead, they may use other learning-based image compression models with similar analytical and synthetic transformation structures, such as the ScaleHyperprior model, the Joint Autoregressive and Hierarchical Priors model, etc. As long as the encoder of the model contains a network module that can compress features in a downsampling manner and the decoder contains a network module that can restore features in an upsampling manner, it can be used to replace the ELIC model as an auxiliary encoder and auxiliary decoder.
[0147] Step 206: Transmit the bitstream to the decoding network so that the decoding network can perform image reconstruction based on the text information and image feature representation contained in the bitstream to obtain the target image.
[0148] After the encoding network (100) completes the compression encoding, it transmits the bitstream (300) containing image feature representation (301) and text information (302) to the decoding network (200) through the network. After receiving the bitstream (300), the decoding network (200) parses out the image feature representation (301) and text information (302) from it. Using the text information (302) as a guiding condition, it combines the image feature representation (301) to complete the image reconstruction and finally outputs the reconstructed target image (201).
[0149] In some embodiments, the decoding network includes a deep decompression network, a generation network, and a second decoding network; The deep decompression network is used to perform compression decoding on the latent feature representation contained in the bitstream in accordance with the compression encoding, and to decompress the latent feature representation after compression decoding to obtain the image feature representation corresponding to the target image. The generating network is used to take the text information contained in the bitstream as a guiding condition and process the image feature representation output by the deep decompression network into a third detail feature representation for reconstructing the target image; The second decoding network is used to decode the latent feature representation after compression and decoding output by the deep decompression network into a third structural feature representation for reconstructing the target image, and to fuse the third structural feature representation and the third detail feature representation to obtain a reconstructed feature representation for reconstructing the target image, and to reconstruct the target image based on the reconstructed feature representation.
[0150] like Figure 3B As shown, the decoding network (200) is composed of three functional modules connected in series: a deep decompression network (210), a generation network (220), and a second decoding network (230).
[0151] The deep decompression network (210) is the first processing module at the decoding end. Its responsibility is to perform compression decoding on the latent feature representation contained in the bitstream (300) and restore it to the image feature representation through upsampling. The processing of the deep decompression network (210) corresponds to the inverse process of the deep compression network (120) at the encoding end. The two are symmetrical in structure and parameters.
[0152] The generator network (220) is the core module of the decoder. It is responsible for processing the image feature representation output by the deep decompression network (210) into a third detail feature representation (222) containing rich details, under the semantic guidance of the text information (302). The text information (302) plays the role of guiding conditions here, providing the generator network (220) with a clear semantic direction, enabling it to selectively supplement the image details lost due to compression, rather than randomly generating them without constraints.
[0153] The second decoding network (230) is the last processing module of the decoding end. It is responsible for decoding the latent feature representation into the structural feature representation, and then fusing the structural feature representation with the detail feature representation output by the generator network (220) to obtain the reconstructed feature representation. Finally, the reconstructed feature representation is decoded into the reconstructed target image (201).
[0154] This division of labor among the three modules reflects the core idea of the decoding end design in this application: the restoration of structural information and the generation of detail information are handled independently by different modules, avoiding mutual interference between the two; after each module has been processed independently, the structural information and detail information are integrated into a complete reconstructed representation through a fusion operation. The resulting reconstructed image is both structurally faithful to the original image and has a high degree of perceptual realism in terms of detail.
[0155] In some embodiments, the decoding network performs image reconstruction based on the text information and image feature representation contained in the bitstream to obtain the target image, which may specifically include: The latent feature representation contained in the bitstream is input into the deep decompression network, so that the deep decompression network performs compression decoding on the latent feature representation corresponding to the compression encoding, and decompresses the latent feature representation after compression decoding to obtain the image feature representation corresponding to the target image; The latent feature representation and the image feature representation after compression and decoding are obtained from the deep decompression network. The text information contained in the bitstream and the obtained image feature representation are then input into the generator network. The generator network uses the text information as a guiding condition to process the image feature representation into a third detail feature representation for reconstructing the target image. The third detail feature representation output by the generating network is obtained. The latent feature representation after compression and decoding and the obtained third detail feature representation are input into the second decoding network. The second decoding network decodes the latent feature representation after compression and decoding into a third structural feature representation for reconstructing the target image. The third structural feature representation and the third detail feature representation are fused to obtain a reconstructed feature representation for reconstructing the target image. The target image is then reconstructed based on the reconstructed feature representation.
[0156] Please see Figure 5 , Figure 5 A schematic diagram of the data flow between the modules within the decoding network (200) is shown.
[0157] like Figure 5 As shown, the complete execution flow of the decoding end network (200) is as follows: In the first step, the deep decompression network (210) reads the latent feature representation (301) from the bitstream (300), performs compression decoding operation on it (i.e., the inverse process corresponding to the compression encoding at the encoding end), and obtains the latent feature representation (211) before quantization; then, it decompresses (i.e. upsamples) the latent feature representation (211) and restores it to the image feature representation (212). The deep decompression network (210) outputs two results at the same time: the latent feature representation (211) after compression decoding and the image feature representation (212), which are used by the subsequent second decoding network (230) and the generation network (220), respectively.
[0158] In the second step, the generating network (220) receives two inputs: image feature representation (212) from the deep decompression network (210) and text information (302) parsed from the bitstream (300).
[0159] The generator network (220) processes the image feature representation (212) with text information (302) as a guiding condition and outputs the third detail feature representation (222). The third detail feature representation (222) is a feature representation that recovers the detail information after semantic guidance. Compared with the image feature representation (212), it contains richer high-frequency detail information.
[0160] In the third step, the second decoding network (230) receives two inputs: the latent feature representation (211) after compression and decoding from the deep decompression network (210), and the third detail feature representation (222) from the generator network (220).
[0161] The second decoding network (230) first decodes the latent feature representation (211) into the third structural feature representation (231), then fuses the third structural feature representation (231) with the third detail feature representation (222) to obtain the reconstructed feature representation (232), and finally decodes the reconstructed feature representation (232) into the reconstructed target image (201).
[0162] In some embodiments, the deep decompression network includes a third decoding network and a decompression network; The third decoding network is used to perform compression decoding on the latent feature representation corresponding to the compression encoding; The decompression network is used to decompress the latent feature representation into an image feature representation corresponding to the target image using the above sampling method; In this case, the latent feature representation contained in the bitstream is input to the deep decompression network, so that the deep decompression network performs compression and decoding on the latent feature representation corresponding to the compression encoding, and decompresses the latent feature representation after compression and decoding to obtain the image feature representation corresponding to the target image, which may specifically include: The latent feature representation contained in the bitstream is input to the third decoding network, so that the third decoding network performs compression decoding on the latent feature representation corresponding to the compression encoding; the latent feature representation after compression decoding output by the third decoding network is obtained, and the obtained latent feature representation after compression decoding is further input to the decompression network, so that the decompression network decompresses the latent feature representation into an image feature representation corresponding to the target image in an upsampling manner.
[0163] like Figure 3B As shown, the deep decompression network (210) is internally composed of a third decoding network (213) and a decompression network (214) connected in series.
[0164] The third decoding network (213) is responsible for performing entropy decoding, which restores the binary codewords in the bitstream (300) to a numerical sequence of latent feature representations. This process corresponds to the entropy encoding process performed by the second encoding network (123) at the encoding end. Both use the same probability model and encoding / decoding rules to ensure that the decoding result is numerically consistent with the latent feature representation before encoding (within the quantization precision range). The third decoding network (213) outputs the latent feature representation (211) after compressed decoding.
[0165] The decompression network (214) receives the latent feature representation (211) output by the third decoding network (213), and gradually enlarges its spatial resolution by upsampling to restore it to the image feature representation (212) corresponding to the target image resolution.
[0166] It should be noted that upsampling is the inverse of downsampling and is usually achieved through transposed convolution. Each layer of transposed convolution increases the spatial resolution of the feature map by a certain factor while adjusting the number of channels, gradually restoring the compact latent feature representation to a higher-dimensional image feature representation.
[0167] In some embodiments, the generative network includes a learning network and an image generation network; The learning network is used to learn the image semantic features described by the text information contained in the bitstream; the image generation network is used to use the image semantic features as guiding conditions to generate a third detail feature representation for reconstructing the target image based on the image feature representation output by the deep decompression network. In this case, the text information contained in the bitstream and the obtained image feature representation are further input into the generator network, so that the generator network uses the text information as a guiding condition to process the image feature representation into a third detail feature representation for reconstructing the target image. Specifically, this may include: The text information contained in the bitstream is input into the learning network so that the learning network learns the image semantic features described by the text information; the image semantic features output by the learning network are obtained, and the image semantic features and the obtained image feature representation are further input into the image generation network so that the image generation network uses the image semantic features as a guiding condition to generate a third detail feature representation for reconstructing the target image based on the image feature representation.
[0168] like Figure 3BAs shown, the generator network (220) is internally composed of two sub-modules: the learning network (221) and the image generation network (223).
[0169] The learning network (221) is responsible for extracting image semantic features (221a) from text information (302).
[0170] The aforementioned image semantic features refer to the transformation of natural language text into a high-dimensional vector representation that can be directly processed by neural networks. This vector corresponds to image features in the semantic space, thereby providing operable guiding signals for image generation networks.
[0171] For example, in one case, if the text information (302) is “the brick texture on the surface of the building is clearly visible and the brick joints are dark gray”, the learning network (221) will transform this text into a feature vector that corresponds to the visual concept of “brick texture and dark gray brick joints” in the image semantic space, namely the image semantic feature (221a).
[0172] The image generation network (223) receives two inputs: image semantic features (221a) from the learning network (221) and image feature representations (212) from the deep decompression network (210).
[0173] Among them, the image generation network (223) uses the image semantic features (221a) as the guiding condition to process the image feature representation (212) and output the third detail feature representation (222).
[0174] The core function of the image generation network (223) is to restore the high-frequency detail information lost during compression to the image feature representation (212) under explicit semantic guidance, so that the output third detail feature representation (222) is significantly better than the input image feature representation (212) in terms of detail richness.
[0175] In some embodiments, the learning network includes a CLIP network; the image generation network includes a denoising network in a diffusion model fine-tuned based on LoRA technology. In this case, the image semantic features and the obtained image feature representation are further input into the image generation network, so that the image generation network uses the image semantic features as a guiding condition to generate a third detail feature representation for reconstructing the target image based on the image feature representation. Specifically, this may include: The image semantic features and the obtained image feature representation are further input into the denoising network, so that the denoising network uses the image semantic features as a guiding condition to denoise the image feature representation, thereby generating a third detail feature representation for reconstructing the target image.
[0176] The learning network (221) adopts CLIP (Contrastive Language-Image Pre-Training) network.
[0177] CLIP network is a multimodal pre-trained model proposed by OpenAI. Its core capability is to align the outputs of the text encoder and the image encoder in the same semantic space by contrastive learning training on large-scale image-text pairs.
[0178] Specifically, the CLIP network's text encoder can transform the input natural language text into a high-dimensional feature vector that is semantically very close to the feature vector of an image describing the same content. This image-text semantic alignment characteristic makes the CLIP network very suitable as a tool for extracting semantic features from images from textual information.
[0179] In this specification, the CLIP network receives text information (302) parsed from the bitstream (300), performs semantic encoding on the text content, and outputs image semantic features (221a). The image semantic features (221a) accurately correspond to the visual concepts described by the text in the semantic space, providing a clear guiding direction for the subsequent image generation network (223).
[0180] The image generation network (223) employs a denoising network (224) in a diffusion model fine-tuned based on LoRA technology. In the diffusion model, the core function of the denoising network is to predict and remove the noise components from a given input with noise or incomplete information, under the constraints of guiding conditions, and output a feature representation that is closer to the distribution of the real image. In the image compression scenario of this application, the denoising network (224) does not face random Gaussian noise, but rather an image feature representation (212) with severely lost detail information due to high compression. Guided by the image semantic features (221a), the denoising network (224) performs denoising processing on the image feature representation (212), identifies and recovers the missing high-frequency detail information, and outputs a third detail feature representation (222).
[0181] It is worth noting that, since the image semantic features (221a) provide explicit semantic guidance conditions, the denoising network (224) operates under conditions with explicit directional constraints. Its denoising process only needs to perform one step (i.e., single-step denoising), without the need for dozens of iterations like the standard diffusion model. This characteristic is the key to achieving real-time decoding in this application.
[0182] Of course, in practical applications, the denoising network (224) may not use the LoRA-based diffusion model denoising network, but instead use other generative networks with conditional image generation capabilities, such as Conditional Generative Adversarial Networks (GANs) or Flow-based Generative Models. These alternatives, given sufficient guidance, can also achieve directional restoration of image details in fewer inference steps.
[0183] In some embodiments, the diffusion model includes SD-Turbo; the denoising network includes the U-Net network in the SD-Turbo model.
[0184] The U-Net network (224) in SD-Turbo is a denoising network accelerated by adversarial diffusion distillation training. Its network structure resembles the letter U and consists of a contraction path (encoder part) and an expansion path (decoder part), with multi-scale feature information transmitted between the two sides through skip connections. The contraction path reduces the spatial resolution of the feature map layer by layer while extracting high-level semantic information; the expansion path restores the spatial resolution layer by layer while using skip connections to introduce low-level detail information from the contraction path. Text guidance conditions (i.e., image semantic features (221a)) are injected into each layer of the U-Net network through a cross-attention mechanism, applying semantic constraints during the feature processing of each layer, so that the denoising process of the network always proceeds in the semantic direction of the text description.
[0185] The U-Net network in SD-Turbo, trained with distillation acceleration, can complete the full denoising mapping in a single forward propagation step without multiple iterations. Under the explicit guidance of image semantic features (221a), the U-Net network (224) only needs one forward propagation to process the compressed image feature representation (212) into a third detail feature representation (222) containing rich details, with fast decoding speed, meeting the needs of real-time application scenarios. The U-Net network (224) is also fine-tuned using LoRA technology, so that it can better adapt to the distribution characteristics of potential feature representations in image compression tasks while retaining the original detail recovery capability.
[0186] In some embodiments, the second decoding network includes multiple network branches; The plurality of network branches include an auxiliary decoder for decoding the compressed and decoded latent feature representation into a third structural feature representation for reconstructing the target image; and a main decoder for decoding the reconstructed feature representation into the target image. In this scenario, the second decoding network decodes the compressed latent feature representation into a third structural feature representation for reconstructing the target image, and fuses the third structural feature representation with the third detail feature representation to obtain a reconstructed feature representation for reconstructing the target image. The reconstruction of the target image based on this reconstructed feature representation may specifically include: The latent feature representation after compression and decoding is input to the auxiliary decoder, which decodes it into a third structural feature representation for reconstructing the target image. The third structural feature representation output by the auxiliary decoder is obtained, and the obtained third structural feature representation and the third detail feature representation are fused to obtain a reconstructed feature representation for reconstructing the target image. The reconstructed feature representation is further input to the main decoder, which decodes it into the target image to complete the reconstruction of the target image.
[0187] like Figure 3B As shown, the second decoding network (230) adopts a dual-branch structure with the auxiliary decoder (231) and the main decoder (232) connected in series.
[0188] The auxiliary decoder (231) receives the compressed and decoded latent feature representation (211) output by the deep decompression network (210) and decodes it into a third structural feature representation (231a). The third structural feature representation (231a) carries structural information such as the contour, object boundaries, and overall layout of the image. This information comes from the main components that are compressed and preserved in the latent feature representation (211) and plays the role of "skeleton" in image reconstruction.
[0189] After obtaining the third structural feature representation (231a) output by the auxiliary decoder (231), it is fused with the third detail feature representation (222) output by the generator network (220) to obtain the reconstructed feature representation (232).
[0190] It should be noted that the specific operation of feature fusion can be to concatenate the two along the channel dimension or to add them element by element. The specific method can be selected according to the overall network design. The fused reconstructed feature representation (232) contains both structural and detailed information, and is the most complete feature-level description of the target image.
[0191] The main decoder (232) receives the reconstructed feature representation (232) as input, maps it from the feature space back to the pixel space, and outputs the final reconstructed target image (201).
[0192] Among them, the master decoder (232) is essentially an image rendering network responsible for transforming abstract feature vectors into a matrix of pixel values visible to the human eye, completing the final step of image reconstruction.
[0193] In some embodiments, the deep compression network includes an encoder-side network of a pre-trained and parameter-frozen deep compression codec; the deep decompression network includes at least a decoder-side network of a pre-trained and parameter-frozen deep compression codec.
[0194] In this specification, both the deep compression network (120) and the deep decompression network (210) are the corresponding end networks of the deep compression codec that have been pre-trained and whose parameters have been frozen.
[0195] It should be explained that "pre-training completed" means that the deep compression codec has completed a complete training process on an independent image compression task before the image compression neural network training of this application begins, and its parameters have converged to a state that can achieve good compression performance.
[0196] The so-called parameter freezing refers to the fact that during the overall neural network training process of this application, the parameters of the deep compression network (120) and the deep decompression network (210) do not participate in gradient updates and always remain at the values when pre-training is completed.
[0197] The reason for adopting the pre-training freezing strategy is that the deep compression codec has already established an accurate probability model matching relationship between the encoder and decoder during the pre-training stage. This matching relationship is the basis for entropy coding to approach the lower bound of information theory. If the parameters of the deep compression network (120) or the deep decompression network (210) are updated during joint training, it may destroy the probability model matching established between the encoder and decoder, leading to a decrease in entropy coding efficiency and even causing instability in the training process.
[0198] After freezing the parameters, the deep compression network (120) and the deep decompression network (210) serve as fixed information compression and decompression channels, providing a stable feature transfer basis for the joint training of the entire network. Other trainable modules (such as the main encoder, main decoder, generator network, etc.) are optimized on this basis, making the training process more stable and the convergence speed faster.
[0199] In some embodiments, the deep compression codec includes the ELIC model; The encoder-side network is a network composed of the analytical transformation network, the advanced prior analytical transformation network, the SCCTX network, and the arithmetic encoder in the ELIC model. The decoder-side network is a network composed of the synthesis transform network, the super-prior synthesis transform network, the SCCTX network, and the decoder in the ELIC model. Accordingly, at this time, the compression network includes the analysis transform network in the ELIC model; the second encoding network includes a network composed of the super-prior analysis transform network, the SCCTX network and the encoder in the ELIC model; and the third decoding network includes a network composed of the super-prior synthesis transform network, the SCCTX network and the arithmetic decoder in the ELIC model; and the decompression network includes the synthesis transform network in the ELIC model.
[0200] Please see Figure 3A and Figure 3B The complete architecture of the ELIC model can be divided into encoder-side network and decoder-side network according to the encoding and decoding functional boundaries, and deployed in the encoder-side network (100) and decoder-side network (200) respectively. The functions of each component of the ELIC model and their specific roles in this specification are described in detail below.
[0201] Among them, the analysis transform network ( The transform network (121) is the backbone downsampling network of the ELIC model encoder, consisting of multiple strided convolution layers connected in series. Its function is to compress the input image feature representation (111) layer by layer, increasing the number of channels while reducing spatial resolution, and outputting a compact latent feature representation (121). In this specification, the transform network (121) is analyzed. As the compression network (122) in the deep compression network (120), it is responsible for compressing the image feature representation (111) output by the first encoding network (110) into a latent feature representation (121). As mentioned above, the transform network (122) is analyzed in this specification. The number of downsampled convolutional layers was adjusted to 2 to accommodate the resolution changes required by the overall network.
[0202] Transformation network of prior analysis ( ), is the encoding component of the hyperprior mechanism in the ELIC model. Its function is to extract information from the quantized latent feature representation. Extracting "side information" (denoted as z) from the latent features. Side information is a representation of the latent features. The description of statistical properties, specifically including those used for estimation The mean of the probability distribution of each element in the middle and variance Information. Transformation network of prior analysis ( The extracted edge information (z) is quantized and encoded separately, and written into the bitstream as auxiliary information transmitted to the decoding end. In this specification, the advanced prior analysis transform network (… Together with the SCCTX network and the arithmetic encoder, it forms the second coding network (123) in the deep compression network (120).
[0203] The SCCTX (Space-Channel Context) network is a key contextual modeling component in the ELIC model used for entropy coding. Its function is to utilize the contextual information of the elements already encoded around the current element to make a more accurate estimate of the conditional probability distribution of the current element.
[0204] Specifically, the SCCTX network integrates the mean and variance information provided by the prior network with the local statistical information provided by the spatial context model to estimate an accurate Gaussian distribution for each latent feature representation element. This serves as a probabilistic model for arithmetic coding. The more accurate the probabilistic model, the closer the bitrate of the arithmetic code is to the theoretical lower bound of information entropy, and the higher the compression efficiency.
[0205] In practical applications, the SCCTX network is typically deployed separately at the encoder and decoder ends. The encoder-side SCCTX network is coupled with the advanced prior analysis transform network (APT). The SCCTX network and the arithmetic encoder together constitute the second coding network (123); the SCCTX network at the decoding end and the super-prior synthesis transform network ( The third decoding network (213) is formed together with the arithmetic decoder.
[0206] The arithmetic encoder represents latent features based on the probabilistic model provided by the SCCTX network. Each element is arithmetically encoded and converted into a binary codeword, which is then written into the bitstream (300).
[0207] The arithmetic decoder performs the corresponding inverse operation at the decoding end, and reconstructs the latent feature representation from the bitstream (300) according to the same probability model. A numerical sequence.
[0208] It should be explained that arithmetic coding is an efficient entropy coding method that is close to the lower bound of information theory. Compared with traditional entropy coding methods such as Huffman coding, arithmetic coding can encode the probability distribution of non-integer bits, and theoretically can achieve a lower code rate.
[0209] Super-prior synthesis transformation network ( (), is the decoding component of the super-prior mechanism, whose function is to decode the side information received by the decoding end. Reduction to latent feature representation Mean of each element and variance The estimate is provided to the SCCTX network and the Gaussian entropy model to guide the arithmetic decoding process.
[0210] In this specification, the advanced prior synthesis transform network ( Together with the SCCTX network and the arithmetic decoder, it forms the third decoding network (213) in the deep decompression network (210).
[0211] Synthetic Transformation Network ( ), is the backbone upsampling network at the ELIC model decoder end, and its function is similar to the analysis transform network ( Conversely, it is responsible for representing the quantized latent features. By upsampling layer by layer through transposed convolution, spatial resolution is restored, and image feature representation is output.
[0212] In this specification, the synthetic transform network ( As the decompression network (214) in the deep decompression network (210), it is responsible for restoring the latent feature representation (211) after compression and decoding into the image feature representation (212). As mentioned above, the synthesis transform network ( The number of upsampled convolutional layers has been adjusted to 3 in this specification.
[0213] Through the above-mentioned splitting and reconfiguration of the components of the ELIC model, the encoder network (100) only carries the encoder network of the ELIC model, and the decoder network (200) only carries the decoder network of the ELIC model. The two exchange information through the bitstream (300).
[0214] This split deployment approach not only meets the requirement of separate deployment of the encoding and decoding ends in practical application scenarios, but also makes full use of the efficient compression capabilities accumulated by the ELIC model in the pre-training stage, avoiding the large amount of computing resources required to train the deep compression module from scratch.
[0215] The following provides a detailed explanation of the training data preparation, training loss design, and training strategy for the neural network (10).
[0216] Before training begins, a sufficient number of two-dimensional image data needs to be collected to form a training dataset. The images in the training dataset should cover a variety of scene categories, including natural landscapes, human faces, urban buildings, and text images, to ensure that the trained neural network (10) has good generalization ability for various image contents. The collected images can be uniformly cropped to a fixed resolution (e.g., 256×256 pixels) to meet the neural network (10)'s requirement for consistent input size. During training, data augmentation operations such as random horizontal flipping and random cropping can be performed on the training images to increase the diversity of training data and further improve the generalization performance of the model.
[0217] In some embodiments, the neural network further includes a discriminant network; wherein the discriminant network is used to determine the image type of the input target image; the image type includes the original target image and the reconstructed target image; In this case, the training loss used to train the neural network can specifically include the following three parts: Rate-distortion loss associated with compression and image reconstruction of the target image; wherein the rate-distortion loss includes bitrate loss and distortion loss; Perceptual loss is used to describe the semantic loss of the reconstructed target image compared to the original target image; The discriminant network calculates the discriminant loss for the input target image.
[0218] like Figure 3B As shown, the neural network (10) is also configured with a discriminant network (240) during the training phase. The discriminant network (240) is an image classification network whose input is an image and whose output is the probability that the image belongs to the "original target image" or the "reconstructed target image".
[0219] Under the adversarial training framework, the discriminant network (240) and the image reconstruction network (i.e., the overall generative network consisting of the encoder network (100) and the decoder network (200)) form a game relationship: On the one hand, the discriminant network (240) strives to learn to distinguish between the original image and the reconstructed image; On the other hand, the generative network strives to make the reconstructed image sufficient to "deceive" the discriminative network, making it unable to distinguish the reconstructed image from the original image.
[0220] This adversarial training mechanism can drive generative networks to produce visually more realistic reconstruction results.
[0221] As mentioned earlier, the training loss of the neural network (10) can be composed of the following three parts: Rate distortion loss: Rate distortion loss includes two components: bit rate loss and distortion loss.
[0222] The bitrate loss is used to constrain the size of the encoded bitstream.
[0223] In this specification, the bitrate is estimated using the cross-entropy formula: in, This represents the quantized latent feature representation. The Gaussian entropy model representing the ELIC model. Estimation of the probability distribution of each element. The intuitive meaning of this formula is: the higher the probability of a latent feature representing an element, the fewer bits are needed to encode it; conversely, the lower the probability, the more bits are required. Calculating the expected number of bits required to encode all elements yields an estimate of the overall code rate. The model learns to make the distribution of latent feature representations more concentrated and predictable by minimizing this code rate loss, thereby reducing the total number of bits required for encoding.
[0224] Distortion loss is used to measure the pixel-level difference between the reconstructed image and the original image. In this specification, distortion loss is expressed as mean squared error (MSE) loss: in, This represents the value of the i-th pixel in the original image. This represents the pixel value at the corresponding location in the reconstructed image, where N is the total number of pixels in the image. The mean squared error loss measures the pixel-by-pixel numerical difference between the reconstructed image and the original image. Minimizing this loss can prevent large pixel shifts in the reconstructed image and ensure the basic fidelity of the reconstructed image at the pixel level.
[0225] Perceptual loss, which relies solely on mean squared error loss for optimization, can easily lead to overly smooth reconstructed images and loss of texture details. Therefore, this application introduces perceptual loss into the training loss to compensate for the limitations of mean squared error loss at the perceptual level.
[0226] In some embodiments, the perceptual loss can be represented by LPIPS (Learned Perceptual Image PatchSimilarity) loss. LPIPS extracts multi-scale feature maps of the original and reconstructed images using pre-trained deep neural networks (such as VGG, AlexNet, or SqueezeNet), and calculates the difference between the two in the feature space. The calculation process is as follows: First, feature representations at each of the L scales of the original and reconstructed images are extracted using a pre-trained network. Then, L2 normalization is performed on the features along the channel dimension to obtain normalized feature maps at each scale. and .
[0227] Next, for each scale Calculate the element-wise mean squared error between normalized feature maps, and then apply the learned weights to the mean squared error of each channel. We then perform weighting to obtain the perceived loss components at this scale.
[0228] Finally, the perceptual loss components at all scales are summed to obtain the final LPIPS perceptual loss value.
[0229] Compared to mean squared error loss, LPIPS loss measures the difference in image perception (i.e., whether it looks like the human eye) rather than simply the difference in pixel values.
[0230] By minimizing the LPIPS loss, the model is guided to preserve high-frequency details such as texture and edges that are sensitive to the human eye in the reconstructed image, thereby improving the reconstruction quality in subjective visual evaluation.
[0231] Discriminative loss is an adversarial loss defined based on the Generative Adversarial Network (GAN) framework.
[0232] In some embodiments, the discriminative network (240) can use a DINOv2 pre-trained model as its backbone network. DINOv2 is a visual Transformer model trained based on self-supervised learning, which performs well in various image feature extraction tasks. On this basis, the discriminative network (240) can also introduce a registers mechanism to enhance its feature discrimination ability.
[0233] It should be noted that the registers mechanism adds several learnable register tokens to the input sequence of the Transformer, enabling the model to store global information in these registers when processing image features. This avoids the pollution of local feature maps by global information and improves the model's ability to perceive differences in local regions. This is particularly beneficial for the discriminative network (240) to distinguish subtle texture differences between the original image and the reconstructed image.
[0234] In some embodiments, the loss of the discriminant network (240) can be a binary cross-entropy loss, the goal of which is to make the discriminant network (240) output a high confidence (close to 1) for the original target image and a low confidence (close to 0) for the reconstructed target image. The adversarial loss of the generator (i.e., the overall image compression and reconstruction network) is derived in reverse, the goal of which is to make the reconstructed image able to "fool" the discriminant network (240) so that it also outputs a high confidence for the reconstructed image.
[0235] By minimizing the adversarial loss of the generator, the generator network (220) can be driven to produce highly realistic reconstructed images that are difficult for the discriminator network (240) to distinguish, thereby maintaining high visual quality even under extremely low bitrate conditions.
[0236] It should be noted that in practical applications, the discriminant network (240) and the generator network can be optimized alternately through backpropagation: in each training step, the parameters of the generator network are first fixed and the parameters of the discriminant network (240) are updated; then the parameters of the discriminant network (240) are fixed and the parameters of the generator network are updated. The two networks interact and improve together during training, ultimately enabling the generator network to generate visually realistic reconstructed images.
[0237] In some embodiments, the discriminant network may include a classification network for classifying the input target image; the image type of the target image output by the classification network includes the original target image and the reconstructed target image; correspondingly, the discriminant loss is represented by the cross-entropy loss for classifying the target image.
[0238] As a classification network, the discriminative network (240) outputs a binary classification head, which outputs the probability value of the input image belonging to the "original target image" category. Cross-entropy loss is typically used as the discriminative loss, and its penalty for the confidence level of the classification result is monotonic: the higher the confidence level of the misclassification, the greater the loss. This characteristic gives the discriminative network (240) a clear directional constraint during training, enabling it to continuously improve its ability to distinguish between the original and reconstructed images, thus providing a more effective adversarial training signal for the generative network.
[0239] In order to achieve stable convergence at extremely low bitrate targets while ensuring the visual quality of the reconstructed images, the embodiments in this specification adopt a phased, loose-to-tight training strategy.
[0240] In some embodiments, training the neural network includes: The first stage of training involves training the neural network using the weighted sum of the bitrate loss, the distortion loss, and the perceptual loss as the target training loss; wherein, in the first stage of training, the weight value of the bitrate loss is set to a first weight value. The second stage of training involves using the weighted sum of the bitrate loss, the distortion loss, the perceptual loss, and the discriminative loss as the target training loss to train the neural network. In this second stage, the weight value of the bitrate loss is set as a second weight value, which is greater than the first weight value.
[0241] Please see Figure 6 , Figure 6A schematic diagram of the loss function for a two-stage training strategy is shown.
[0242] The target training loss in the first stage of training can be a weighted sum of bitrate loss, distortion loss, and perceptual loss, and its loss function can be expressed as: in, This is the bitrate loss weight for the first stage (i.e., the first weight value). The weights for the mean squared error loss are... Weights for perceived loss. In the first stage, It is set to a relatively small value, and the bitrate constraint is relatively loose.
[0243] The core objective of the first stage of training is to warm up the model and establish effective information exchange and collaboration among the network modules. Under relaxed bitrate constraints, the model prioritizes ensuring the quality of the reconstructed image (minimizing mean squared error loss and perceptual loss), while gradually adapting the output distributions of modules such as the first encoding network (110), deep compression network (120), first decoding network (130), generator network (220), and second decoding network (230), laying a stable foundation for subsequent fine-tuning of extreme compression. In this stage, the text information generated by LLM (140) and the image semantic features extracted by CLIP network (221) begin to play a guiding role, enabling the model to learn how to use semantic difference information to assist in image reconstruction.
[0244] The target training loss in the second stage of training can include a weighted sum of bitrate loss, distortion loss, perceptual loss, and discriminative loss, and its loss function can be expressed as: in, This is the bitrate loss weight for the second stage (i.e., the second weight value). To determine the weight of the loss. In the second stage, The bitrate constraint has been significantly strengthened.
[0245] The core objective of the second phase of training is to compress the bitrate to an ultra-low target value while maintaining reconstruction quality. This involves significantly increasing the bitrate loss weights. This forces the model to learn to represent image information with fewer bits, actively discarding redundant information that does not contribute much to the final visual quality, thus achieving radical bit rate compression.
[0246] Meanwhile, the newly introduced discriminant loss As a quality gatekeeper mechanism, when the bitrate is strongly compressed, the adversarial training incentivizes the generator network (220) to more actively utilize textual semantic guidance information to recover the details lost due to compression, thus preventing the reconstruction quality from significantly degrading due to over-compression.
[0247] This two-stage training strategy, which progresses from loose to tight, can essentially be understood as an "implicit bitrate pruning" method. In other words, the model does not directly delete network parameters, but instead forces the network to learn to discard unimportant information by adjusting the weights of the loss function, achieving a compression effect similar to network pruning, but avoiding the risk that the model may have difficulty converging when directly applying high-intensity bitrate constraints.
[0248] It should be noted that the weights of each loss ( These are all hyperparameters and can be flexibly adjusted according to the actual optimization goals and target bitrate.
[0249] For example, if you want to achieve higher perceived quality at a lower bit rate, you can appropriately increase the bit rate. and The value can be increased appropriately if you want to reduce pixel-level distortion while maintaining perceived quality. The value of .
[0250] By flexibly adjusting these weights, the image compression neural network (10) in this specification can be adapted to different application scenarios and bit rate targets, and has high practical flexibility.
[0251] In some embodiments, the encoding method used to compress the target image may include entropy coding.
[0252] Entropy coding is a type of lossless coding technique based on the principles of information theory. Its core idea is to assign shorter codewords to high-probability symbols and longer codewords to low-probability symbols according to the probability distribution of the symbols to be encoded, so that the average code length after encoding is as close as possible to the theoretical lower bound of information entropy.
[0253] In this specification, entropy coding is used to represent the quantized latent features. The embedded index of the text information is encoded to generate the binary codewords that are finally written into the code stream (300).
[0254] In some embodiments, entropy coding can be implemented using arithmetic coding.
[0255] Arithmetic coding achieves encoding by mapping the entire sequence of symbols to be encoded to a sub-interval within the interval [0, 1), where the length of the sub-interval is determined by the joint probability of the symbol sequence. Compared with traditional entropy coding methods such as Huffman coding, arithmetic coding can encode fractional bits, theoretically achieving a compression ratio closer to the lower bound of information entropy. Its compression advantage is particularly significant when the probability distribution is skewed (i.e., some symbols appear with a much higher probability than others).
[0256] In this specification, the probabilistic model for arithmetic coding is jointly provided by the super-prior mechanism of the ELIC model and the SCCTX network. As previously described, this probabilistic model is capable of representing latent features. Estimate the exact Gaussian distribution for each element Supported by precise probabilistic models, arithmetic coding can compress the bit rate to near the theoretical limit of information entropy, providing a strong guarantee for achieving ultra-low bit rate goals.
[0257] Of course, in practical applications, entropy coding can also use other entropy coding methods such as Huffman coding or Canonical Huffman coding.
[0258] For example, Huffman coding has lower implementation complexity than arithmetic coding and has certain practical value on devices with limited computing resources; however, its coding efficiency is slightly lower than that of arithmetic coding, especially for symbol sets with a high degree of probability skewness, where the efficiency difference is more obvious.
[0259] The specific entropy coding method to be used can be selected based on the trade-off between coding efficiency and computational complexity required for the actual application scenario.
[0260] Figure 7 This is a schematic structural diagram of an electronic device provided in an exemplary embodiment.
[0261] like Figure 7As shown, device 700 mainly consists of a communication interface 702, a user interface 704, a processor 706, and a data storage 708. These components are interconnected and communicate with each other via a system bus, network, or other connection mechanism 710. The communication interface 702 enables device 700 to communicate with other devices, access networks, and transmission networks via analog or digital modulation. For example, the communication interface 702 may include a chipset and antenna for wireless communication with a radio access network or access point. Furthermore, the communication interface 702 can be a wired interface such as Ethernet, Token Ring, or a USB port, or a wireless interface such as Wi-Fi, Bluetooth, Global Positioning System (GPS), or a wide-area wireless interface (e.g., WiMAX or LTE). Of course, the communication interface 702 can also support other forms of physical layer interfaces and standard or proprietary communication protocols. The communication interface 702 may also include multiple physical communication interfaces, such as Wi-Fi, Bluetooth, and wide-area wireless interfaces.
[0262] User interface 704 includes receiving user input and providing output to the user. Therefore, user interface 704 may include input components such as a keypad, keyboard, touch-sensitive or presence-sensitive panel, computer mouse, trackball, joystick, microphone, still camera, and video camera, and output components such as a display screen (which may be combined with a touch-sensitive panel), CRT, LCD, LED, display using DLP technology, printer, and other similar devices known or developed in the future. User interface 704 may also generate auditory output via speakers, speaker jacks, audio output ports, audio output devices, headphones, and other similar devices known or developed in the future. In some embodiments, user interface 704 may include software, circuitry, or other forms of logic capable of transmitting and receiving data from external user input / output devices. Additionally or alternatively, device 700 may support remote access from other devices via communication interface 702 or another physical interface (not shown). User interface 704 may be configured to receive user input, the position and movement of which may be indicated by indicators or cursors described herein. User interface 704 may also be configured as a display device for rendering or displaying text fragments.
[0263] Processor 706 may contain one or more general-purpose processors and / or special-purpose processors.
[0264] Data storage 708 may include one or more volatile and / or non-volatile storage components and may be integrated wholly or partially with processor 706. Data storage 708 may include removable and non-removable components.
[0265] Processor 706 is capable of executing program instructions 718 (e.g., compiled or uncompiled program logic and / or machine code) stored in data storage 708 to perform the various functions described herein. Data storage 708 may contain a non-transitory computer-readable medium on which program instructions are stored, which, when executed by device 700, enable device 700 to perform any methods, processes, or functions disclosed in this specification and / or the accompanying drawings. Processor 706 executing program instructions 718 may result in processor 706 using data 712.
[0266] For example, program instructions 718 may include an operating system 722 (e.g., an operating system kernel, device drivers, and / or other modules) installed on device 700 and one or more applications 720 (e.g., a browser, social application, or game application). Similarly, data 712 may include operating system data 716 and application data 714. Operating system data 716 is primarily accessible to the operating system 722, while application data 714 is primarily accessible to one or more applications 720. Application data 714 may reside in a file system visible or hidden from the user of device 700.
[0267] Application 720 can communicate with operating system 722 through one or more application programming interfaces (APIs). These APIs help application 720 read and / or write application data 714, transmit or receive information via communication interface 702, receive or display information on user interface 704, etc.
[0268] In some terminology, application 720 may be simply referred to as an app. Furthermore, application 720 can be downloaded to device 700 through one or more online app stores or app markets. However, applications can also be installed on device 700 in other ways, such as through a web browser or a physical interface on device 700 (e.g., a USB port).
[0269] Please refer to Figure 8 This specification also proposes an image compression device based on a neural network, which can be applied to applications such as... Figure 7 The device shown is used to implement the technical solution of this specification. The device may include: The acquisition module 801 acquires the original target image to be compressed; The generation module 802 inputs the target image into the encoding end network, so that the encoding end network compresses and encodes the original target image at the encoding end to generate a bitstream corresponding to the target image; wherein, the bitstream includes image feature representation obtained by compressing and encoding the target image; and text information used to describe the semantic differences between the reconstructed image reconstructed based on the image feature representation and the target image. The reconstruction module 803 transmits the bitstream to the decoding network so that the decoding network can perform image reconstruction based on the text information and image feature representation contained in the bitstream to obtain the target image.
[0270] For ease of description, the above devices are described by dividing them into various modules or units based on their functions. Of course, when implementing one or more of these specifications, the functions of each module or unit can be implemented in the same or different software and / or hardware, or a module that performs the same function can be implemented by a combination of multiple sub-modules or sub-units, etc. The device embodiments described above are merely illustrative. For example, the division of units is only a logical functional division; in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed.
[0271] Based on the same concept as the methods described above, this specification also provides an electronic device, including: a processor; a memory for storing processor-executable instructions; wherein the processor performs the steps of the method as described in any of the above embodiments by executing the executable instructions.
[0272] Based on the same concept as the methods described above, this specification also provides a computer-readable storage medium having computer instructions stored thereon that, when executed by a processor, implement the steps of the methods as described in any of the above embodiments.
[0273] Based on the same concept as the methods described above, this specification also provides a computer program product, including a computer program / instructions that, when executed by a processor, implement the steps of the methods as described in any of the above embodiments.
[0274] What those skilled in the art will understand is: In this specification, the terms include, encompass, or any other variations thereof are intended to cover a non-exclusive inclusion, such that a process, method, product, or apparatus that includes a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, product, or apparatus. Without further limitation, the presence of additional identical or equivalent elements in a process, method, product, or apparatus that includes said elements is not excluded.
[0275] In this specification, "one," "a," and "the" do not specifically refer to the singular, but may also include the plural.
[0276] In this specification, ordinal numbers such as "first," "second," etc., do not necessarily indicate order; they are often used to distinguish between objects. For example, "first server" and "second server" usually refer to two servers. To differentiate between these two servers, they are described as "first server" and "second server." Of course, sometimes these two servers may be the same server.
[0277] In this specification, unless explicitly stated otherwise, the receipt and transmission of data are not necessarily direct; they can be indirect. For example, A receiving data sent by B can be understood as A directly receiving the data sent by B, or it can be understood as A indirectly receiving the data sent by B through other entities such as C. Similarly, B sending data to A can be understood as B sending the data directly to A, or it can be understood as B indirectly sending the data to A through other entities such as C. Here, C can be one entity, or it can be two or more entities.
[0278] In this specification, unless explicitly stated otherwise, the relationships between structures can be direct or indirect. For example, when describing a connection between A and B, unless explicitly stated that A and B are directly connected, it should be understood that A can be directly connected to B or indirectly connected to B. Similarly, when describing A above B, unless explicitly stated that A is directly above B (AB is adjacent and A is above B), it should be understood that A can be directly above B or indirectly above B (AB is separated by other elements, and A is above B). And so on.
[0279] This specification uses specific terms to describe embodiments thereof. For example, "an embodiment," "an embodiment," and / or "a number of embodiments" refer to a particular feature, structure, or characteristic associated with at least one embodiment of this specification. Therefore, it should be emphasized and noted that an embodiment, an embodiment, or an alternative embodiment mentioned two or more times in different locations in this specification does not necessarily refer to the same embodiment. Furthermore, those skilled in the art can combine and integrate the different embodiments or examples described in this specification, as well as the features of those different embodiments or examples, without contradiction.
[0280] Although one or more embodiments of this specification provide method steps as described in the embodiments or flowcharts, it is understood that the order of steps listed in the embodiments or flowcharts is only one of many possible execution orders and does not represent the only execution order. Therefore, when the claims involve method steps, any changes or adjustments to the order of such steps, or the parallelism between steps, are also within the scope of protection of the claims.
Claims
1. A neural network-based image compression method, wherein the neural network includes an encoding network for compressing and encoding images and a decoding network for reconstructing the compressed images; the method includes: Obtain the target image to be compressed; The target image is input to the encoding network, which compresses and encodes the target image to generate a bitstream corresponding to the target image; wherein the bitstream contains image feature representations obtained by compressing and encoding the target image. And textual information used to describe the semantic differences between the reconstructed image reconstructed based on the image feature representation and the target image; The bitstream is transmitted to the decoding network, whereby the decoding network performs image reconstruction based on the text information and image feature representation contained in the bitstream to obtain the target image.
2. The method as described in claim 1, wherein the encoding end network comprises a first encoding network and a deep compression network; wherein, The first encoding network is used to encode the target image into an image feature representation; the deep compression network is used to compress the image feature representation into a latent feature representation, and to perform compressed encoding on the latent feature representation; The target image is input to the encoding network, whereby the encoding network compresses and encodes the target image to generate a bitstream corresponding to the target image, including: The target image is input into the first encoding network so that the first encoding network encodes the target image into an image feature representation; The image feature representation output by the first encoding network is obtained, and the image feature representation is further input into the deep compression network so that the deep compression network compresses the image feature representation into a latent feature representation, and the latent feature representation is compressed and encoded to obtain a bitstream corresponding to the target image.
3. The method of claim 2, wherein the first encoding network comprises multiple network branches; wherein, The plurality of network branches include a main encoder for encoding the target image into a first detail feature representation; And an auxiliary encoder for encoding the target image into a first structural feature representation; The target image is input into the first encoding network so that the first encoding network encodes the target image into an image feature representation; The target image is input to the main encoder and the auxiliary encoder respectively, so that the main encoder encodes the target image into a first detail feature representation, and the auxiliary encoder encodes the target image into a first structural feature representation; The first detail feature representation output by the main encoder and the first structural feature representation output by the auxiliary encoder are obtained, and feature fusion is performed on the first detail feature representation and the first structural feature representation to obtain an image feature representation corresponding to the target image.
4. The method of claim 3, wherein at least one convolutional layer is deployed in both the main encoder and the auxiliary encoder; wherein, The at least one convolutional layer is used to adjust the resolution of the first detail feature representation or the first structural feature representation to a first target value; and to adjust the number of channels of the first detail feature representation or the first structural feature representation to a second target value; Obtain the first detail feature representation output by the main encoder and the first structural feature representation output by the auxiliary encoder, and perform feature fusion on the first detail feature representation and the first structural feature representation, including: Obtain the first detail feature representation of the main encoder outputting through the at least one convolutional layer with a resolution of the first target value and a channel number of the second target value, and obtain the first structural feature representation of the auxiliary encoder outputting through the at least one convolutional layer with a resolution of the first target value and a channel number of the second target value; The obtained first detail feature representation and first structural feature representation are superimposed on pixel values along the channel dimension to obtain an image feature representation corresponding to the target image.
5. The method of claim 4, wherein the deep compression network comprises a compression network and a second encoding network; wherein, The compression network is used to compress the image feature representation into a latent feature representation using a downsampling method; the second encoding network is used to compress and encode the latent feature representation to obtain a bitstream corresponding to the target image; The image feature representation is further input into the deep compression network, which compresses the image feature representation into a latent feature representation, and then performs compression encoding on the latent feature representation to obtain a bitstream corresponding to the target image, including: The image feature representation is further input into the compression network, so that the compression network compresses the image feature representation into a latent feature representation in a downsampling manner; The latent feature representation output by the compression network is obtained, and the obtained latent feature representation is further input into the second encoding network so that the latent feature representation is compressed and encoded by the second encoding network to obtain a bitstream corresponding to the target image.
6. The method of claim 5, wherein the encoding network further includes a first decoding network corresponding to the first encoding network; the first decoding network includes multiple network branches; wherein, The plurality of network branches include a master decoder corresponding to the master encoder; And, an auxiliary decoder corresponding to the auxiliary encoder; The method further includes: The acquired latent feature representation is input into the auxiliary decoder, so that the auxiliary decoder decodes the latent feature representation into a second structural feature representation; The second structural feature representation output by the auxiliary decoder is obtained, and the obtained second structural feature representation is further input into the main decoder so that the main decoder decodes the second structural feature representation into a second detail feature representation; The second detail feature representation output by the main decoder is obtained, and the obtained second detail feature representation and the target image are input to the LLM accessed by the encoding end network, so that the LLM can perform inference calculation on the second detail feature representation and the target image to generate text information describing the semantic differences between the second detail feature representation and the target image; The text information output by the LLM is obtained, the text information is encoded, and the encoded text information is written into the bitstream corresponding to the target image.
7. The method of claim 6, wherein the main encoder comprises the encoder included in the VAE portion of a diffusion model fine-tuned based on LoRA technology; the auxiliary encoder comprises a compression network included in the encoder end of a pre-trained and parameter-frozen deep compression codec; wherein, The compressed network supports the following sampling method to compress the input feature representation into a latent feature representation; The main decoder includes the decoder contained in the VAE part of the diffusion model, which is fine-tuned based on LoRA technology; the auxiliary decoder includes a decompression network contained in the decoder end of a pre-trained and parameter-frozen deep compression codec; wherein, the decompression network supports decompressing the input latent feature representation into an image feature representation corresponding to the target image by oversampling.
8. The method of claim 7, wherein the diffusion model includes SD-Turbo; the deep compression codec includes the ELIC model; the compression network includes the analysis transform network in the ELIC model; and the decompression network includes the synthesis transform network in the ELIC model.
9. The method of claim 6, wherein the decoding network comprises a deep decompression network, a generation network, and a second decoding network; in, The deep decompression network is used to perform compression decoding on the latent feature representation contained in the bitstream in accordance with the compression encoding, and to decompress the latent feature representation after compression decoding to obtain the image feature representation corresponding to the target image; The generating network is used to take the text information contained in the bitstream as a guiding condition and process the image feature representation output by the deep decompression network into a third detail feature representation for reconstructing the target image; The second decoding network is used to decode the latent feature representation after compression and decoding output by the deep decompression network into a third structural feature representation for reconstructing the target image, and to fuse the third structural feature representation and the third detail feature representation to obtain a reconstructed feature representation for reconstructing the target image, and to reconstruct the target image based on the reconstructed feature representation.
10. The method of claim 9, wherein the decoding network performs image reconstruction based on the text information and image feature representation contained in the bitstream to obtain the target image, comprising: The latent feature representation contained in the bitstream is input into the deep decompression network, so that the deep decompression network performs compression decoding on the latent feature representation corresponding to the compression encoding, and decompresses the latent feature representation after compression decoding to obtain the image feature representation corresponding to the target image; The latent feature representation and the image feature representation after compression and decoding are obtained from the deep decompression network. The text information contained in the bitstream and the obtained image feature representation are then input into the generator network. The generator network uses the text information as a guiding condition to process the image feature representation into a third detail feature representation for reconstructing the target image. The third detail feature representation output by the generating network is obtained. The latent feature representation after compression and decoding and the obtained third detail feature representation are input into the second decoding network. The second decoding network decodes the latent feature representation after compression and decoding into a third structural feature representation for reconstructing the target image. The third structural feature representation and the third detail feature representation are fused to obtain a reconstructed feature representation for reconstructing the target image. The target image is then reconstructed based on the reconstructed feature representation.
11. The method of claim 10, wherein the deep decompression network comprises a third decoding network and a decompression network; wherein, The third decoding network is used to perform compression decoding on the latent feature representation corresponding to the compression encoding; the decompression network is used to decompress the latent feature representation into an image feature representation corresponding to the target image by means of upsampling; The latent feature representation contained in the bitstream is input into the deep decompression network, which performs compression and decoding on the latent feature representation corresponding to the compression encoding, and then decompresses the latent feature representation after compression and decoding to obtain the image feature representation corresponding to the target image, including: The latent feature representation contained in the bitstream is input to the third decoding network, so that the third decoding network performs compression decoding on the latent feature representation corresponding to the compression encoding; The latent feature representation after compression and decoding output by the third decoding network is obtained, and the obtained latent feature representation after compression and decoding is further input into the decompression network so that the latent feature representation is decompressed into an image feature representation corresponding to the target image by the third decoding network in an upsampling manner.
12. The method of claim 11, wherein the generative network comprises a learning network and an image generation network; wherein, The learning network is used to learn the image semantic features described by the text information contained in the bitstream; the image generation network is used to use the image semantic features as guiding conditions to generate a third detail feature representation for reconstructing the target image based on the image feature representation output by the deep decompression network; The text information contained in the bitstream and the obtained image feature representation are further input into the generator network, so that the generator network uses the text information as a guiding condition to process the image feature representation into a third detail feature representation for reconstructing the target image, including: The text information contained in the bitstream is input into the learning network so that the learning network can learn the image semantic features described by the text information from the text information; The image semantic features output by the learning network are obtained, and the image semantic features and the obtained image feature representation are further input into the image generation network, so that the image generation network uses the image semantic features as a guiding condition to generate a third detail feature representation for reconstructing the target image based on the image feature representation.
13. The method of claim 12, wherein the learning network comprises a CLIP network; and the image generation network comprises a denoising network in a diffusion model fine-tuned based on LoRA technology; The image semantic features and the acquired image feature representation are further input into the image generation network, so that the image generation network uses the image semantic features as a guiding condition to generate a third detail feature representation for reconstructing the target image based on the image feature representation, including: The image semantic features and the obtained image feature representation are further input into the denoising network, so that the denoising network uses the image semantic features as a guiding condition to denoise the image feature representation, thereby generating a third detail feature representation for reconstructing the target image.
14. The method of claim 13, wherein the diffusion model includes SD-Turbo; and the denoising network includes the U-net network in the SD-Turbo model.
15. The method of claim 12, wherein the second decoding network comprises a plurality of network branches; wherein, The plurality of network branches include an auxiliary decoder for decoding the compressed and decoded latent feature representation into a third structural feature representation for reconstructing the target image; and a main decoder for decoding the reconstructed feature representation into the target image; The second decoding network decodes the compressed latent feature representation into a third structural feature representation for reconstructing the target image, and fuses the third structural feature representation and the third detail feature representation to obtain a reconstructed feature representation for reconstructing the target image, and reconstructs the target image based on the reconstructed feature representation, including: The latent feature representation after compression and decoding is input into the auxiliary decoder, so that the auxiliary decoder decodes the latent feature representation after compression and decoding into a third structural feature representation for reconstructing the target image; The third structural feature representation output by the auxiliary decoder is obtained, and the obtained third structural feature representation and the third detail feature representation are fused to obtain a reconstructed feature representation for reconstructing the target image; The reconstructed feature representation is further input into the main decoder, which decodes the reconstructed feature representation into the target image to complete the reconstruction of the target image.
16. The method of claim 15, wherein the deep compression network includes an encoder-side network of a pre-trained and parameter-frozen deep compression codec; and the deep decompression network includes at least a decoder-side network of a pre-trained and parameter-frozen deep compression codec.
17. The method of claim 16, wherein the deep compression codec includes an ELIC model; in, The encoder-side network is a network composed of the analytical transformation network, the advanced prior analytical transformation network, the SCCTX network, and the encoder in the ELIC model. The decoder-side network is a network composed of the synthesis transform network, the super-prior synthesis transform network, the SCCTX network, and the decoder in the ELIC model. Accordingly, the compression network includes the analysis transformation network in the ELIC model; the second encoding network includes a network composed of the super-prior analysis transformation network, the SCCTX network, and the encoder in the ELIC model; The third decoding network includes a network composed of the super-prior synthesis transform network, the SCCTX network, and the arithmetic decoder in the ELIC model; the decompression network includes the synthesis transform network in the ELIC model.
18. The method of claim 17, wherein the neural network further comprises a discriminative network; wherein, The discriminant network is used to determine the image type of the input target image; the image type includes the original target image and the reconstructed target image; The training loss used to train the neural network includes: Rate-distortion loss associated with compression and image reconstruction of the target image; wherein the rate-distortion loss includes bitrate loss and distortion loss; Perceptual loss is used to describe the semantic loss of the reconstructed target image compared to the original target image; The discriminant network calculates the discriminant loss for the input target image.
19. The method of claim 18, wherein the perceptual loss is represented by LPIPS loss; The discriminant network includes a classification network for classifying the input target image; the image type of the target image output by the classification network includes the original target image and the reconstructed target image; correspondingly, the discriminant loss is represented by the cross-entropy loss for classifying the target image.
20. The method of claim 1, wherein the encoding method used to compress and encode the target image includes entropy encoding.
21. The method of claim 18, wherein training the neural network comprises: The first stage of training involves training the neural network using the weighted sum of the bitrate loss, the distortion loss, and the perceptual loss as the target training loss; wherein, in the first stage of training, the weight value of the bitrate loss is set to a first weight value. The second stage of training involves using the weighted sum of the bitrate loss, the distortion loss, the perceptual loss, and the discriminative loss as the target training loss to train the neural network. In this second stage, the weight value of the bitrate loss is set as a second weight value, which is greater than the first weight value.
22. A neural network, comprising: The encoding network compresses and encodes the input target image to generate a bitstream corresponding to the target image; wherein the bitstream includes an image feature representation obtained by compressing and encoding the target image; and text information describing the semantic differences between the reconstructed image based on the image feature representation and the target image. A decoding network is used to perform image reconstruction based on the text information and image feature representation contained in the bitstream transmitted to the local network, so as to obtain the target image.
23. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor, when executing the program, implements the method as claimed in any one of claims 1 to 21.
24. A computer program product comprising a computer program / instructions that, when executed by a processor, implement the method as described in any one of claims 1-21.
