Semantic communication-oriented high texture fidelity image compression method and system independent of sending end model

Abstract

The invention discloses a semantic communication-oriented high texture fidelity image compression method and system independent of a sending end model, and the method comprises the steps: carrying out the extraction of semantic information, structural information, an entity region and texture information for a given original image, and obtaining a semantic representation, a structural representation, an entity mask set and a texture representation; based on the semantic representation, the structure representation, the entity mask set and the texture representation, performing rate-distortion optimization on the structure representation to obtain an optimal structure residual error, and then encoding the semantic representation, the texture representation and the optimal structure residual error to obtain a code stream; and at a decoding end, decoding the code stream to obtain a semantic representation, a lossy structure residual error and a lossy texture representation, and then reconstructing an image through a multi-modal image generation model. According to the method, the dependence of encoding and decoding ends on model synchronization can be effectively reduced, the texture fidelity of a reconstructed image is improved, and the bit rate is reduced.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of semantic communication, and relates to an image compression technology in semantic communication, in particular to a high-texture-fidelity image compression method and system not dependent on a sending end model for semantic communication. BACKGROUND

[0002] Semantic communication represents a new communication paradigm, the core of which is to break through the dependence of traditional communication on accurate and error-free transmission of bit-level data, and instead focus on the effective communication of the inherent meaning of information. In today's era of information explosion and increasingly prominent context awareness needs, this change is particularly critical. In the face of massive data emerging in scenarios such as video conferencing, Internet of Vehicles and autonomous driving, Internet of Things, and immersive XR applications, a fundamental challenge is that not all transmitted information has equal value to the target task of the receiving end. Semantic communication, through its intelligent understanding and refinement capabilities, can identify and prioritize the transmission of the most relevant and valuable semantic content. This not only significantly improves communication efficiency (such as reducing latency and improving robustness), but also effectively alleviates network bandwidth pressure and promotes efficient use of communication and computing resources. Therefore, semantic communication has shown great application potential and transformative power in solving the efficiency bottleneck and information redundancy problems faced by traditional communication in the processing of massive data.

[0003] In the research of image semantic communication, especially in the direction of image compression, the current technology mostly adopts an end-to-end deep learning architecture. This approach relies on neural networks to complete the encoding and decoding of image data, but its normal operation depends on the high coordination of the models of both communication parties. That is, the receiving party must pre-save a model configuration matching the sending party to adapt to the transmission needs in different application scenarios. However, this method is extremely sensitive to changes in model parameters, and even a small change can cause a large deviation in the reconstruction result. Therefore, how to maintain the consistency of the models at both ends of the communication has become an important challenge affecting system performance.

[0004] In the literature "R. Zheng, X. Song, X. Zuo, M. Yang, D. Gao and X. Xie, "A Transmitter-Model Unaware Generative Image Compression Framework for Semantic Communication," ICASSP 2025, pp. 1-5", a human understandable modality is introduced as an intermediate representation, which alleviates the need for strict synchronization between the receiver and transmitter models in end-to-end image compression methods to some extent. However, although this method has made some progress in reducing model dependency, the fidelity of the reconstructed image in texture details is still significantly insufficient, affecting the overall visual quality and semantic expression ability.

[0005] In summary, the existing image compression technology in semantic communication still faces two core problems while achieving efficient semantic transmission: first, the high dependence on the transmitter model limits the flexibility and adaptability of the system; second, the insufficient fidelity of the reconstructed image in texture details affects the visual quality and accurate restoration of high-level semantic information. SUMMARY

[0006] To overcome the above-mentioned shortcomings of the prior art, the purpose of the present application is to provide a high-texture-fidelity image compression method and system for semantic communication independent of the transmitter model, which uses a multi-modal representation containing semantic information, structural information, and texture information of the original image as the intermediate layer transmission content, so that the decoding end does not need to know the specific model used by the encoding end in advance, thereby effectively reducing the dependence of the encoding and decoding ends on model synchronization. In addition, the present application introduces a small resolution image as an expression modality of texture information, and combines the rate-distortion optimization (RDO) mechanism to further optimize the structure modality using mask information and texture modality, in order to improve the texture fidelity of the reconstructed image and reduce the bit rate.

[0007] To achieve the above-mentioned purpose, the technical solution adopted by the present application is:

[0008] A high-texture-fidelity image compression method for semantic communication independent of the transmitter model, characterized by the following steps:

[0009] Step 1, for a given original image, extract semantic information, structural information, entity region, and texture information respectively to obtain semantic representation, structural representation, entity mask set, and texture representation;

[0010] Step 2, based on the semantic representation, structural representation, entity mask set, and texture representation, perform rate-distortion optimization on the structural representation to obtain optimal structural residual, and then encode the semantic representation, texture representation, and optimal structural residual to obtain a code stream;

[0011] Step 3: At the decoding end, the bitstream is decoded to obtain semantic representation, lossy structural residual and lossy texture representation, and then the image is reconstructed through a multimodal image generation model.

[0012] This invention also provides a high-texture-fidelity image compression system for semantic communication that is independent of the sender model, comprising an encoder and a decoder, wherein:

[0013] The encoding end is used to extract semantic information, structural information, entity regions and texture information from the original image to obtain semantic representation, structural representation, entity mask set and texture representation. Based on the extraction results, rate-distortion optimization is performed on the structural representation to obtain the optimal structural residual. Then the semantic representation, texture representation and optimal structural residual are encoded to obtain the bitstream.

[0014] The decoding end is used to decode the bitstream to obtain semantic representation, lossy structural residuals and lossy texture representation, and then reconstruct the image through a multimodal image generation model.

[0015] Compared to existing technologies, this invention introduces low-resolution images as a representation of texture information and combines structural and semantic information for collaborative rate-distortion optimization during the encoding process. This achieves high texture fidelity while ensuring that the decoder does not rely on the encoder model. The multimodal joint optimization design of this invention not only guarantees the semantic accuracy of the image content but also significantly enhances the realism of the reconstructed image in terms of detailed textures, further reducing the bitrate. Attached Figure Description

[0016] Figure 1 This is a schematic diagram of the image compression system structure proposed in this invention.

[0017] Figure 2 This is a schematic diagram of the rate-distortion optimization module proposed in this invention.

[0018] Figure 3 This is a schematic diagram of an image compression system according to a specific embodiment of the present invention. Detailed Implementation

[0019] The embodiments of the present invention will now be described in detail with reference to the accompanying drawings and examples.

[0020] like Figure 1 As shown, the present invention is a high-texture-fidelity image compression system for semantic communication that does not depend on the sending end model. Its framework mainly includes an encoding end and a decoding end.

[0021] The encoding end is used to extract semantic information, structural information, entity regions, and texture information from natural images to obtain semantic representation, structural representation, mask set, and texture representation. Based on the extracted representations, rate-distortion optimization is performed on the structural representation to obtain structural residuals. Subsequently, the semantic representation, texture representation, and structural residuals are encoded.

[0022] The decoding end is used to restore the structural residuals transmitted from the encoding end to the optimal structural representation, and then reconstruct the image through a multimodal generative model.

[0023] Furthermore, the encoding end of the present invention includes a semantic information extraction module, a structural information extraction module, an entity region extraction module, a texture information extraction module, a rate-distortion optimization (RDO) module, and an intermediate representation encoding module.

[0024] The semantic information extraction module extracts modalities containing semantic information from the original image, such as text, as the semantic representation. Therefore, the semantic information extraction module can use an image-to-text model for semantic information extraction.

[0025] The structural information extraction module extracts modalities containing structural information from the original image, such as binary edge maps, as the structural representation. Therefore, the structural information extraction module can employ an edge detection model for structural information extraction.

[0026] The entity region extraction module extracts the region masks of each entity in the original image, obtaining a mask set containing all entities in the original image, i.e., the entity mask set. The entity masks are used to help the subsequent rate-distortion optimization module focus on the parts of the structural representation that are most helpful to the relevant task metrics. Therefore, the entity region extraction module can use an entity segmentation model for entity region extraction.

[0027] The texture information extraction module extracts modalities containing texture information from the original image, such as a low-resolution image after downsampling the original image, as the texture representation. Therefore, the texture information extraction module can use a downsampling operation to extract texture information.

[0028] The rate-distortion optimization module optimizes the data redundancy of the structural representation based on the semantic representation, structural representation, texture representation and entity mask set to obtain the optimal structural residual.

[0029] The intermediate representation encoding module is used to encode the extracted semantic representation, texture representation, and the obtained optimal structural residual into a bitstream, which is then sent into the channel.

[0030] Furthermore, the decoding end of the present invention mainly includes an intermediate representation decoding module and a multimodal image generation model.

[0031] The intermediate representation decoding module decodes the transmitted bitstream into various representation modalities, specifically semantic representation, lossy texture representation, and lossy structural residual.

[0032] Multimodal generative models are used to reconstruct the original image from various intermediate representations, including semantic representation, lossy texture representation, and optimal structural representation. The optimal structural representation is obtained by fusing the texture structure of the lossy texture representation with the lossy structural residual.

[0033] Corresponding to the above system, the present invention provides a generative image compression method for semantic communication that does not depend on the sending end model, which mainly includes the following steps;

[0034] Step 1: Given the original image (natural image or artistic creation), firstly, in the encoding end, the semantic information extraction module is used to extract the semantic representation, the structural information extraction module is used to extract the structural representation, the entity region extraction module is used to extract the entity mask set, and the texture information extraction module is used to extract the texture representation.

[0035] The extraction behavior in this step can be further described as follows:

[0036] Step 11: In the semantic information extraction module, given the input image x, the image-to-text model G... text (·), to obtain the semantic representation IR of the original image. semantic =G text (x). For example, based on MiniCPM-V2.6, the corresponding descriptive text is obtained from the original image, i.e., semantic representation (IR). semantic .

[0037] Step 12: In the structural information extraction module, given the input image x, the edge detection model G... edge (·), to obtain the structural representation IR containing the structural information of the original image. struct =G edge (x). For example, based on PidiNet, the corresponding binary edge map, i.e., the structural representation IR, is obtained from the original image. struct .

[0038] Step 13: In the entity region extraction module, given the input image x, the entity segmentation model G... mask (·), thus obtaining the entity mask set M = G of the original image. mask (x). For example, the corresponding entity mask set M is obtained from the original image based on the Segment Anything Model.

[0039] Step 14: In the texture information extraction module, given the input image x, perform the downsampling operation downsample(·) to obtain the texture representation IR containing the texture information of the original image. texture=downsample(x).

[0040] Step 2: Then, based on semantic representation, structural representation, entity mask set, and texture representation, rate-distortion optimization is performed on the structural representation to remove details irrelevant to the task and optimize to obtain the optimal structural residual. After that, the semantic representation, texture representation, and optimal structural residual are encoded to obtain the bitstream.

[0041] In this step, rate-distortion optimization is performed on the structural representation based on semantic representation, structural representation, entity mask set, and texture representation to obtain structural residuals. These residuals are used at the decoding end to reconstruct the optimal structural representation. Specifically, referring to the figure, the following input is given:

[0042] Semantic representation IR semantic IR structure representation struct Texture representation IR texture Entity mask set M, area threshold A min Original Image I original (i.e., the input image x), weight λ.

[0043] Step 21: Initialization.

[0044] RD opt =∞,mask opt =zeros,Diff opt =zerosM filtered ={}

[0045] Among them, RD opt For the optimal rate-distortion value, mask opt For the corresponding optimal mask, Diff opt M represents the corresponding optimal structural residual. filtered Let be the set of masks filtered by area, and zeros represent a matrix with all values ​​being 0.

[0046] IR representation of texture texture Encoding and decoding are performed to obtain a lossy texture representation (IR). texture-loss , IR texture-los The lossy texture structure representation S is obtained by feeding it into the edge detection model. texture-loss The calculation formula is as follows:

[0047] IR texture-lossy =Decode(Encode(IR) texture ))

[0048] S texture-loss =G edge (IR texture-los )

[0049] Where Decode(·) is the decoding operation (specifically implemented as BPG decoding), Encode(·) is the encoding operation (specifically implemented as BPG encoding), and G edge (·) represents the edge detection model (specifically implemented as PidiNet).

[0050] Step 22: Mask filtering, update M filtered The formula is as follows:

[0051] M filtered ={m∈M|A m ≥A min}

[0052] Among them, A m Let m be the area of ​​the mask.

[0053] Step 23: Sort.

[0054] For M filtered Sort the elements in ascending order to get M sorted .

[0055] Step 24: Calculate the current structural representation IR sturct-mask ,as follows:

[0056]

[0057] m current =mask opt ∪m

[0058] IR sturct-mask =IR struct ·m current

[0059] Diff sturct =IR sturct-mas -S texture-los

[0060] Diff sturct-lossy =Decode(Encode(Diff) sturct ))

[0061] IR struct-curren =Diff sturct-loss +S texture-los

[0062] Where Decode(·) is the decoding operation (specifically implemented as BPG decoding), Encode(·) is the encoding operation (specifically implemented as BPG encoding), and m is M. sorted The mask in m current For the mask of the current iteration, IR sturct-maDiff represents the structure after masking (i.e., the binary edge map after masking). stur The structural residual, i.e., the difference between the masked structural representation and the lossy compressed texture structural representation, is called Diff. sturct-lossy For lossy structural residuals, IR struct-curren This is the final structural representation used for image generation. The purpose of this step is to calculate the structural representation after the current mask, and to ensure that the current structural representation is as consistent as possible with the structural representation at the decoding end, especially in the case of subsequent lossy compression.

[0063] Step 25: Utilize semantic representation for IR semantic IR struct-current and IR texture-los Rate-distortion optimization is performed using the following formula:

[0064] I generated =G E (IR semantic IR struct-curr IR texture-lossy )

[0065] RD = BitRate(C current )+λ·Distortion(I generated ,I original )

[0066] Among them, I generated Let G be the generated image of the current iteration, RD be the rate-distortion value of the current iteration, and G be the value of the generated image of the current iteration. E (·) represents the multimodal image generation model at the encoding end (specifically implemented as multiple ControlNet and StableDiffusion models), BitRate(·) is the bit rate for calculating the input conditions (specifically implemented as calculating the BPP under the sum of the sizes of semantic representation, structural representation and texture representation), and Distortion(·) is the distortion value between the generated image and the original image (specifically implemented as calculating the DISTS value between the two images).

[0067] Update RD opt mask opt and Diff opt The rule is: if RD <RD opt ,but:

[0068] RD opt =RD, mask opt =m current Diff opt =Diff sturct

[0069] After completing step 25, return to step 24 until all masks have been calculated.

[0070] Step 26: Once all masks m have been calculated, obtain and output the optimal structural residual Diff. opt .

[0071] Steps 21 to 26 are the specific process of rate-distortion optimization. For the rate-distortion optimization module, the inputs are semantic representation, structural representation, entity mask set and texture representation, and the output is only the optimal structural residual.

[0072] Subsequently, the intermediate representations that need to be encoded are transmitted to the intermediate representation encoding module. Specifically, the semantic representation (i.e., text) is passed to the text encoding algorithm, namely the Lampel-Ziv algorithm, for encoding to obtain the semantic representation bitstream. The optimal structural residual (residual of the binary edge map) obtained from the rate-distortion optimization module and the texture representation (low-resolution image) are passed to the same image encoding algorithm, namely the BPG algorithm, for encoding to obtain the texture representation bitstream and the structural residual bitstream. The resulting bitstreams are then transmitted to the channel.

[0073] Step 3: At the decoding end, the bitstream is decoded to obtain semantic representation, lossy structural residual and lossy texture representation, and then the image is reconstructed through a multimodal image generation model.

[0074] The decoding in step 3 specifically involves:

[0075] The obtained bitstream is transmitted to the intermediate representation decoding module. Specifically, the semantic representation bitstream is decoded using a text decoding algorithm, namely the Lampel-Ziv algorithm, to obtain the semantic representation. The structural residual bitstream and the texture representation bitstream are decoded using an image decoding algorithm, namely the BPG algorithm, to obtain the lossy structural residual and lossy texture representation.

[0076] This step reconstructs the image using a multimodal image generation model, and the method is as follows:

[0077] Given the decoded semantic representation IR semantic Defective structural residuals Diff opt-lossy Lossy texture representation IR texture-lossy To obtain the reconstructed image

[0078] S texture-los =G edge (IR texture-lossy )

[0079] IR struct-opt =Diff opt-lossy +S texture-lo

[0080]

[0081] Among them, Stexture-lossy For the structural representation of lossy textures, G edge (·) represents the edge detection model (specifically implemented as PidiNet), IR struct-opt For optimal structure representation, IR texture-loss For lossy texture representation, Diff opt-loss For the lossy structural residual after decoding, G D A multimodal image generation model for the decoding end (specifically implemented as multiple ControlNet and StableDiffusion models).

[0082] One specific embodiment of the present invention is as follows: Figure 3 As shown, at the encoding end, the semantic representation, structural representation, and texture representation of the image are extracted. The semantic representation is the image's descriptive text, the structural representation is the edge map, and the texture representation is the downsampled original image. The structural residual between the structural representation and the texture representation is calculated, and the bitstream is obtained through encoding. At the decoding end, the above information is sufficient to decode and generate an image with a very high similarity to the original image. It is worth noting that the structural representation represents a data modality containing the structural information of the original image; the binary edge map is just one type. Based on this scheme, at the encoding end of this invention, various representations only extract key information from the original image (only this key information is sufficient for a multi-modal image generation model to reconstruct a high-fidelity image), and the representations require only a small amount of bit storage. For example, for an original image with a resolution of 512*512, the semantic representation + structural residual + texture representation occupy a total of 0.96KB to 1.28KB, while a completely uncompressed original image is 768KB. By transmitting "semantic representation + structural residual + texture representation" and utilizing the powerful generation capabilities of the image generation model, compression is achieved.

[0083] In summary, this invention transmits only intermediate representations, rather than tensors in end-to-end image compression (these tensors can only be processed by specific models). This allows the decoding end to directly process these common modalities without relying on the sending end's model, using various image generation models. Furthermore, this invention improves texture fidelity by transmitting texture representations and further reduces the bitrate by using texture representations to process and optimize structural representations.

Claims

1. A high-texture-fidelity image compression method for semantic communication that does not depend on the sender model, characterized in that, Includes the following steps: Step 1: For a given original image, extract semantic information, structural information, entity regions and texture information respectively to obtain semantic representation, structural representation, entity mask set and texture representation; Step 2: Based on the semantic representation, structural representation, entity mask set and texture representation, rate-distortion optimization is performed on the structural representation to obtain the optimal structural residual. Then, the semantic representation, texture representation and optimal structural residual are encoded to obtain the bitstream. Step 3: At the decoding end, the bitstream is decoded to obtain semantic representation, lossy structural residual and lossy texture representation, and then the image is reconstructed through a multimodal image generation model.

2. The high-texture-fidelity image compression method for semantic communication independent of the sender model according to claim 1, characterized in that, Step 1: Semantic information is extracted using an image-to-text model to obtain text containing the semantic information of the original image, which serves as the semantic representation. Structural information is extracted using an edge detection model to obtain a binary edge map containing structural information of the original image, which serves as the structural representation. Entity regions are extracted using an entity segmentation model to obtain a mask set containing all entities in the original image, namely the entity mask set. The entity mask is used to help the subsequent rate-distortion optimization module focus on the part of the structural representation that is most helpful to the relevant task indicators. Texture information is extracted using a downsampling operation to obtain an image containing the texture information of the original image, which serves as the texture representation.

3. The high-texture-fidelity image compression method for semantic communication independent of the sender model according to claim 1, characterized in that, Step 2, the method for rate-distortion optimization, is as follows: Step 21, initialize the optimal rate-distortion value RD opt =∞, the corresponding optimal mask opt =zeros, corresponding to the optimal structural residual Diff opt =zeros, the set of area-filtered masks M filtered ={}; Represents IR for textures texture Encoding and decoding are performed to obtain a lossy texture representation (IR). texture-lossy , IR texture-lossy The lossy texture structure representation S is obtained by feeding it into the edge detection model. texture-lossy ;zeros represents a matrix where all values ​​are 0; Step 22: Perform mask filtering and update M. filtered ; Step 23, for M filtered Sort the elements in ascending order to get M sorted ; Step 24, calculate the current structure representation as follows: m current =mask opt ∪m GO sturct-mask =IR struct ·m current Diff sturct =IR sturct-mask -S texture-lossy Diff sturct-lossy =Decode(Encode(Diff sturct )) IR struct-current =Diff sturct-lossy +S texture-lossy Where m is M sorted The mask in m current For the mask of the current iteration, IR sturct-mask For the masked structure representation, IR struct For structural representation, Diff sturct For structural residuals, Diff sturct-lossy For lossy structural residuals, Decode(·) is the decoding operation, Encode(·) is the encoding operation, and IR... struct-current This is the structural representation ultimately used for image generation; Step 25, utilize semantic representation IR semantic IR struct-current and IR texture-lossy Perform rate-distortion optimization and update RD. opt mask opt and Diff opt ; Step 26: After all masks m have been calculated, the optimal structural residual Diff is obtained. opt .

4. The high-texture-fidelity image compression method for semantic communication independent of the sender model according to claim 3, characterized in that, The lossy texture representation and the lossy texture structure representation are calculated as follows: IR texture-lossy =Decode(Encode(IR texture )) S texture-lossy =G edge (GO texture-lossy ) Among them, G edge (·) represents the edge detection model; M filtered The update formula is as follows: M filtered ={m∈M|A m ≥A min } Where M is the set of entity masks, A m Let A be the area of ​​mask m. min Area threshold; The formula for rate distortion optimization is as follows: I generated =G E (AND semantic ,AND struct-current ,AND texture-lossy ) RD=BitRate(C current )+λ·Distortion(I generated ,I original ) If RD <RD opt Then update: RD opt =RD,mask opt =m current ,Diff opt =Diff sturct Among them, I generated For the generated image of the current iteration, I original The image is the original image, RD is the rate-distortion value of the current iteration, and G is the original image. E (·) represents the multimodal image generation model at the encoding end, BitRate(·) represents the bit rate for calculating the input conditions, Distortion(·) represents the distortion value between the generated image and the original image, and λ is the weight.

5. The high-texture-fidelity image compression method for semantic communication independent of the sender model according to claim 4, characterized in that, The bitrate of the input conditions is calculated as the BPP value of the sum of the sizes of the semantic representation, structural representation, and texture representation; the distortion value between the generated image and the original image is calculated as the DISTS value between the generated image and the original image.

6. The high-texture-fidelity image compression method for semantic communication independent of the sender model according to claim 1, characterized in that, In step 2, the semantic representation is encoded using a text encoding algorithm to obtain a semantic representation bitstream, and the texture representation and optimal structural residual are encoded using the same image encoding algorithm to obtain a texture representation bitstream and a structural residual bitstream.

7. The high-texture-fidelity image compression method for semantic communication independent of the sender model according to claim 1, characterized in that, In step 3, the semantic representation bitstream is processed by a text decoding algorithm to obtain the semantic representation, and the structural residual bitstream and texture representation bitstream are processed by an image decoding algorithm to obtain the lossy structural residual and lossy texture representation.

8. The high-texture-fidelity image compression method for semantic communication independent of the sender model, as described in any one of claims 1 to 7, is characterized in that... Step 3, which involves reconstructing the image using a multimodal image generation model, is implemented as follows: S texture-lossy =G edge (GO texture-lossy ) IR struct-opt =Diff opt-lossy +S texture-lossy Among them, S texture-lossy For the structural representation of lossy textures, G edge (·) represents the edge detection model, IR texture-lossy For lossy texture representation, IR struct-opt For the optimal structure representation, Diff opt-lossy For the lossy structural residual after decoding, G D This is a multimodal image generation model for the decoding end.

9. A system for implementing the high-texture-fidelity image compression method for semantic communication independent of the sender model as described in any one of claims 1 to 8, comprising an encoding end and a decoding end, characterized in that: The encoding end is used to extract semantic information, structural information, entity regions and texture information from the original image to obtain semantic representation, structural representation, entity mask set and texture representation. Based on the extraction results, rate-distortion optimization is performed on the structural representation to obtain the optimal structural residual. Then the semantic representation, texture representation and optimal structural residual are encoded to obtain the bitstream. The decoding end is used to decode the bitstream to obtain semantic representation, lossy structural residuals and lossy texture representation, and then reconstruct the image through a multimodal image generation model.

10. The system according to claim 9, characterized in that: The encoding end includes a semantic information extraction module, a structural information extraction module, an entity region extraction module, a texture information extraction module, a rate-distortion optimization module, and an intermediate representation encoding module; The semantic information extraction module extracts modalities containing semantic information of the original image, i.e., semantic representations; The structural information extraction module extracts modalities containing structural information of the original image, i.e., structural representations; The entity region extraction module extracts the region mask of each entity in the original image and forms the entity mask set. The texture information extraction module extracts the modal containing the texture information of the original image, i.e., the texture representation; The rate-distortion optimization module optimizes the data redundancy of the structural representation based on semantic representation, structural representation, texture representation, and entity mask set. The intermediate representation encoding module is used to encode the semantic representation, texture representation, and optimal structure residual into a bitstream; The decoding end includes an intermediate representation decoding module and a multimodal image generation model; The intermediate representation module decodes the transmitted bitstream into semantic representation, lossy structural residuals, and lossy texture representation; The multimodal generative model is used to reconstruct the original image from semantic representation, lossy structural residuals, and lossy texture representation.

Images