Semantic communication oriented system and method for generative image compression independent of sender model

Abstract

The application discloses a generative image compression system and method for semantic communication independent of a sending end model, comprising an encoder and a decoder; the encoder is used for extracting semantic information of a natural image, and further optimizing the extracted semantic information data by rate-distortion optimization; the decoder is used for restoring the optimized semantic information transmitted by the encoder through a channel, and reconstructing an image through a multi-modal generative model. The image compression system provided by the application takes a mode understandable by human beings as an intermediate representation, and the decoder does not need to know a specific model used by the encoder in advance; a rate-distortion optimization module based on a mask is contained in the system, which can effectively remove information of the intermediate mode with low semantic correlation, maintain the perceptual quality of the reconstructed image, and reduce the bit rate.

Description

TECHNICAL FIELD

[0001] The present application belongs to the field of image compression technology in the field of semantic communication, and specifically relates to a generative image compression system and method for semantic communication not dependent on a sending end model. BACKGROUND

[0002] Semantic communication is a new communication paradigm that focuses on the transmission of the meaning behind data rather than the precise transmission of bit-level data, as in traditional communication. This communication method is particularly important today when data volume is exploding and context-aware communication is becoming increasingly important. In applications such as video conferencing, intelligent transportation systems, and the Internet of Things, the amount of data transmitted is huge, but not all transmitted information is crucial to the decision-making process of the receiver. Semantic communication improves communication efficiency, reduces network load, and ensures optimal use of resources by intelligently filtering and transmitting the most relevant content. This approach has great potential to address the limitations of traditional communication methods in handling large amounts of data while maintaining efficiency and relevance.

[0003] In the field of semantic communication, especially in image compression, existing technical solutions mainly rely on end-to-end learning frameworks. These frameworks optimize neural network models to achieve image compression and reconstruction, but they usually require synchronization of deep models between the transmitter and the receiver. This design requires the receiver to know and store the corresponding model of the sender in advance to adapt to different transmission scenarios. In addition, these frameworks are very sensitive to changes in model parameters, and even minor adjustments can cause significant changes in output results, which increases the difficulty of model synchronization. Therefore, it is crucial to ensure that the models on both ends are synchronized.

[0004] Comparative document 1, Lei E, Uslu Y B, Hassani H, et al. Text+sketch: Image compression at ultra low rates [J]. arXiv preprint arXiv:2307.01944, 2023, discloses a technical solution for image compression based on text and sketches at ultra-low rates, but only considers text and sketches in the extraction of semantic information. Since it uses an end-to-end compression framework for the compression of sketches, the decoding end still needs to know the encoding model used by the encoding end in advance. This solution requires high model synchronization between the encoding end and the decoding end.

[0005] In summary, the current semantic communication technology based on deep learning is often realized in an end-to-end manner, which makes it highly required to synchronize the models between the sending end and the receiving end during communication, and even a slight parameter change may cause serious impact on the decoding process. Since manufacturers usually need to customize the used technology according to different tasks and requirements, the end-to-end communication manner limits the flexibility to some extent. SUMMARY

[0006] In order to overcome the deficiencies of the prior art, the purpose of the present application is to provide a generative image compression system and method for semantic communication not dependent on the model of the sending end, which takes the modality understandable by human beings as the intermediate representation, and the decoder does not need to know the specific model used by the encoder in advance; the system contains a rate-distortion optimization module based on mask, which can effectively remove the information of the intermediate modality that is lowly related to semantics, maintain the perceptual quality of the reconstructed image and reduce the bit rate.

[0007] In order to achieve the above purpose, the technical scheme adopted by the present application is:

[0008] The generative image compression system for semantic communication not dependent on the model of the sending end comprises an encoder and a decoder.

[0009] The encoder is used for extracting semantic information from a natural image, optimizing the extracted semantic information data in terms of rate and distortion, and then encoding.

[0010] The decoder is used for restoring the optimized semantic information transmitted by the encoder, and reconstructing an image through a multi-modal generative model.

[0011] The encoder comprises a semantic information extraction module, a rate-distortion optimization module and a semantic information encoding module.

[0012] The semantic information extraction module is used for converting an image into a pre-defined representation, which is usually a modality understandable by human beings, i.e. semantic information data (a data form containing semantic information, such as a sketch, an edge map, a depth map, a segmentation map, a subject reference image, text, human body posture key points and a series of data abstractly expressing specific features in an input image, which is a modality that can be directly understood by human beings).

[0013] The rate-distortion optimization module is used for optimizing the data redundancy in the semantic information data in terms of rate and distortion.

[0014] The semantic information encoding module is used for encoding the optimized semantic information data into a code stream.

[0015] The decoder comprises a semantic information decoding module and a multi-modal generative model.

[0016] The semantic information decoding module decodes the code stream corresponding to the optimized semantic information data into a human understandable modality;

[0017] The multi-modal generative model is used to reconstruct the semantic information data (in the form of a human understandable modality) into the original image.

[0018] The method for generative image compression facing semantic communication does not depend on the generative model of the sending end, and comprises the following steps:

[0019] Step 1: Given an image (natural image or artistic creation), first extract semantic information from the image at the encoder, usually in the form of a human understandable modality, i.e. semantic information data; the form of human understandable modality is specifically implemented as text, sketch, entity mask;

[0020] Step 2: Then, rate-distortion optimization is performed on the semantic information data to remove details irrelevant to the task, and then encoding is performed;

[0021] Step 3: At the decoding end, the decoded semantic information is reconstructed into the original image through a multi-modal image generation model.

[0022] In step 1, the extraction of semantic information from the image is based on an image-to-text model, an image-to-condition model, and an image-to-mask model;

[0023] Image-to-text model: through this model, the input image is converted into descriptive text containing semantic content; the generated descriptive text provides a high-level content summary of the image for reconstructing the image content;

[0024] Image-to-condition model: through this model, various human understandable modalities are generated for tasks related to the semantic content of the input image;

[0025] Image-to-mask model: through this model, masks are generated for different regions or instances within the input image, which help the encoder and subsequent image generation process focus on the most relevant parts of the image, thereby preserving as much semantic information as possible.

[0026] The step 1 is specifically:

[0027] Step (1): Given an input image x, an image-to-text model G text (·) is used to obtain the descriptive text G text = G text (x) of the input image; based on MiniCPM-V2.6, the corresponding descriptive text of the input image is obtained;

[0028] Step (2): Given an input image x, an image-to-condition model G condition(·), to obtain the specific condition C containing the semantic information of the input image condition = G condition (x); obtain the corresponding sketch from the input image based on the HED detector in ControlNet;

[0029] Step (3): Given the input image x, the image-to-mask model G mask (·), to obtain the entity mask set M = G mask (x) of the input image; obtain the corresponding mask set from the input image based on the Segment Anything Model.

[0030] The step 2 is specifically:

[0031] The image-to-condition model outputs condition data, in the specific implementation of the scheme, the image-to-condition model is a model for extracting the sketch of the input image, and the output condition data is the sketch. In the framework, the sketch is only an implementation manner, and can be any data type acted on by a mask, such as an edge map, a depth map, or a segmentation map;

[0032] Rate-distortion optimization is performed on the sketch based on the mask to obtain an optimized sketch;

[0033] The specific process is:

[0034] The following inputs are given:

[0035] Condition data C condition , which is the output of the image-to-condition model, the mask set M, the area threshold A min , the original image I original , that is, the input image x, and the weight λ;

[0036] Step (1): initialization:

[0037] RD opt = ∞, mask op t = zeros, M filtered = {}

[0038] Wherein, RD opt is the optimal rate-distortion value, mask opt is the corresponding optimal mask, and M filtered is the mask set filtered by area;

[0039] Step (2): mask filtering:

[0040] M filtered = {m ∈ M | A m ≥ A min}

[0041] Wherein, Am is the area of the mask m;

[0042] Step (3): Ordering:

[0043] M filtered is sorted in ascending order to get M sorted

[0044] Step (4): Rate-distortion optimization:

[0045]

[0046] m current = mask opt ∪ m

[0047] C current = Decode(Encode(C condition · m current ))

[0048] I generated = MultiModalImageGeneration(C current )

[0049] RD = BitRate(C current ) + λ · Distortion(I generated , I original )

[0050] where Decode(·) is the decoding operation (specifically implemented as BPG decoding), Encode(·) is the encoding operation (specifically implemented as BPG encoding), m current is the mask of the current iteration, C current is the condition data of the current iteration, I generated is the generated image of the current iteration, RD is the rate-distortion value of the current iteration, MultiModalImageGeneration(·) is the multi-modal image generation (specifically implemented as in the multi-modal image generation chapter), BitRate(·) is the calculation of the code rate of the input condition (specifically implemented as the calculation of the BPP of the text and sketch), Distortion(·) is the calculation of the distortion value between the generated image and the original image (specifically implemented as the calculation of the DISTS value between the two images);

[0051] If RD < RD opt , then update:

[0052] RD opt = RD, mask opt = m current

[0053] Step (5): Output:

[0054] output the optimized condition data C opt = C condition ·mask opt .

[0055] Steps (1)-(5) are the specific process of rate-distortion optimization. For the rate-distortion optimization module, the input is the output condition data and the mask set of the condition model and the mask model, and the output is only the optimized condition data, i.e., the optimized sketch.

[0056] The descriptive text obtained from the text model is directly transmitted to the text encoder, i.e., the Lampel-Ziv algorithm, and the optimized condition data obtained from the rate-distortion optimization module, i.e., the optimized sketch, is transmitted to the condition encoder, i.e., the BPG algorithm. Both encoders are used to compress data into a code stream and transmit it to the channel. Step 3 decoding is specifically:

[0057] The text is decompressed using the Lempel-Ziv algorithm, and the optimized sketch is decompressed using the BPG algorithm.

[0058] In step 3, at the decoding end, there are a text decoder and a condition decoder, which are used to decode the code stream transmitted by the channel into their respective data, i.e., the text decoder decodes the text code stream into text, and the condition decoder decodes the condition code stream into condition data, i.e., the decoded optimized sketch, which is then transmitted to the multi-modal generation model.

[0059] The output of the text decoder, i.e., the Lampel-Ziv algorithm, is the descriptive text, which is also the input of the multi-modal generation model. The output of the condition encoder, i.e., the BPG algorithm, is the optimized condition data, i.e., the optimized sketch in this scheme, which is also the input of the multi-modal generative model.

[0060] The descriptive text and the optimized condition data together form the decoded semantic information data.

[0061] In step 3, the multi-modal image generation is specifically:

[0062] Given the decoded semantic information data Multi-modal generative model G M (·), to obtain the reconstructed image Based on ControlNet, the input semantic information data is reconstructed into an image.

[0063] The semantic information data contains the decoded descriptive text and the optimized decoded sketch

[0064] Advantages of the present application:

[0065] The image compression framework for semantic communication proposed by the present application is different from the end-to-end implementation adopted in the common semantic communication framework, and the decoding end in the framework does not need to know the neural network model used by the encoding end in advance. Meanwhile, in the encoding end of the present framework, the extracted semantic information (expressed as human understandable modalities) is further optimized to eliminate the semantic irrelevant information therein through the mask-based rate-distortion optimization, thereby improving the encoding efficiency. BRIEF DESCRIPTION OF DRAWINGS

[0066] Figure 1 The schematic diagram of the generative image compression framework proposed by the present application.

[0067] Figure 2 The schematic diagram of a specific implementation of the framework proposed by the present application. DETAILED DESCRIPTION

[0068] The present application will be further described in detail below with reference to the accompanying drawings.

[0069] Figure 1 The generative image compression framework proposed by the present application includes an encoder and a decoder;

[0070] The encoder is used for extracting semantic information from a natural image and further optimizing the extracted semantic information data by using rate-distortion optimization;

[0071] The decoder is used for restoring the optimized semantic information transmitted by the encoder through a channel and reconstructing an image through a multi-modal generative model.

[0072] The encoder includes a semantic information extraction module, a rate-distortion optimization module and a semantic information encoding module;

[0073] The semantic information extraction module is used for converting an image into a pre-defined representation, which is hereinafter uniformly referred to as semantic information data (a data form containing semantic information, such as a sketch, an edge map, a depth map, a segmentation map, a subject reference image, text, human body posture key points and a series of data abstractly expressing specific features in an input image, which is expressed as a human understandable modality);

[0074] The rate-distortion optimization module is used for further optimizing the data redundancy in the semantic information data;

[0075] The semantic information encoding module is used for encoding the optimized semantic information data into a code stream.

[0076] The decoder includes a semantic information decoding module and a multi-modal generative model;

[0077] The semantic information decoding module decodes the code stream corresponding to the optimized semantic information data into a human understandable modality.

[0078] The multi-modal generative model is used to reconstruct the semantic information data (in the form of a human understandable modality) into the original image.

[0079] The specific method of generative image compression for semantic communication not relying on the sending end model is as follows:

[0080] Step 1: In this framework, given an image (natural image or artistic creation), first extract semantic information from the image at the encoder, usually in the form of a human understandable modality; the specific implementation of the human understandable modality form is text, sketch, entity mask;

[0081] Step 2: Then, further optimize these semantic information through rate-distortion optimization to remove details irrelevant to the task, and then encode;

[0082] Step 3: At the decoding end, the decoded semantic information is reconstructed into the original image through a multi-modal image generation model.

[0083] The step 1 is specifically as follows:

[0084] Step (1): Given an input image x, an image-to-text model G text (·) is used to obtain the descriptive text C text of the input image = G text (x); where the specific implementation of G text (·) is "MiniCPM-V 2.6"

[0085] Step (2): Given an input image x, an image-to-condition model G condition (·) is used to obtain a specific condition C condition containing the semantic information of the input image = G condition (x); in the specific implementation scheme, the condition is a sketch, where G condition (·) is a sketch mode of the edge detection model "HED" in "ControlNet"

[0086] Step (3): Given an input image x, an image-to-mask model G mask (·) is used to obtain the entity mask set M = G mask (x) of the input image, where the specific implementation of G mask (·) is the entity segmentation model "Segment Anything Model".

[0087] The step 2 is specifically as follows:

[0088] Based on the mask, the sketch is rate-distortion optimized to obtain an optimized sketch;

[0089] The specific process is:

[0090] Given the following input:

[0091] Conditional data C condition , i.e. the output of the image to the conditional model, the mask set M, the area threshold A min , the original image I original (i.e. the input image x), the weight λ

[0092] Step (1): initialization:

[0093] RD opt = ∞, mask opt = zeros, M filtered = {}

[0094] Wherein, RD opt is the optimal rate-distortion value, mask opt is the corresponding optimal mask, M filtered is the mask set filtered by area;

[0095] Step (2): mask filtering:

[0096] M filtered = {m ∈ M | A m ≥ A min}

[0097] Wherein, A m is the area of the mask m;

[0098] Step (3): sorting:

[0099] Sort M filtered in ascending order to obtain M sorted

[0100] Step (4): rate-distortion optimization:

[0101]

[0102] m current = mask opt ∪m

[0103] C current = Decode(Encode(C condition ·m current ))

[0104] I generated = MultiModalImageGeneration(C current )

[0105] RD = BitRate(C current )+ λ · Distortion(I generated ,I original )

[0106] Where Decode(·) is the decoding operation (implemented as BPG decoding), Encode(·) is the encoding operation (implemented as BPG encoding), m current is the mask of current iteration, C current is the condition data of current iteration, I generated is the generated image of current iteration, RD is the rate-distortion value of current iteration, MultiModalImageGeneration(·) is the multi-modal image generation (implemented as the multi-modal image generation chapter), BitRate(·) is the calculation of the code rate of input conditions (implemented as the calculation of the BPP of text and sketch), Distortion(·) is the calculation of the distortion value between the generated image and the original image (implemented as the calculation of the DISTS value between the two images);

[0107] If RD < RD opt , then update:

[0108] RD opt = RD, mask opt = m current

[0109] Step (5): output:

[0110] Output the optimized condition data C opt = C condition · mask opt .

[0111] Steps (1)-(5) are the specific process of rate-distortion optimization. For the rate-distortion optimization module, the input is the output condition data and the mask set of the condition model and the mask model, and the output is only the optimized condition data, i.e. the optimized sketch.

[0112] The descriptive text obtained from the text model is directly passed to the text encoder, i.e. the Lampel-Ziv algorithm, and the optimized condition data obtained from the rate-distortion optimization module, i.e. the optimized sketch, is passed to the condition encoder, i.e. the BPG algorithm. The two encoders are used to compress the data into a code stream and pass it to the channel.

[0113] The text is losslessly compressed using the Lempel-Ziv algorithm, and the optimized sketch is lossily compressed using the BPG algorithm with a quantization parameter QP = 12.

[0114] The step 3 decoding is specifically:

[0115] The text is decompressed using the Lempel-Ziv algorithm, and the optimized sketch is decompressed using the BPG algorithm.

[0116] In step 3, the multi-modal image generation is specifically:

[0117] Given the decoded semantic information data Multi-modal generative model G M (·), to obtain the reconstructed image

[0118] The semantic information data Contains decoded descriptive text And the optimized decoded sketch The specific implementation of the multi-modal generative model is "ControlNet".

[0119] This process aims to enhance image transmission and generation using multi-modal data processing. By integrating various models, this framework can effectively encode and decode semantic information in images and generate high-quality images in a multi-modal environment. The overall process mainly consists of two components: an encoder and a decoder.

[0120] The semantic information extraction module includes an image-to-text model, an image-to-condition model, and an image-to-mask model.

[0121] Image-to-text model: Through this model, the input image is converted into descriptive text containing semantic content; the generated descriptive text provides a high-level summary of the content of the image, which can then be used to reconstruct the image content.

[0122] In the specific implementation of the present application, MiniCPM-V 2.6 is selected as the image-to-text model, which accepts images and text as input and generates high-quality image content descriptions using prompts.

[0123] Image-to-condition model: The input image is simultaneously sent to the image-to-condition model, which generates various human-understandable modal representations for tasks related to semantics, such as edges, styles, and reference images of main objects, etc. These representations are crucial for accurately reconstructing semantic information related to tasks in the decoding phase.

[0124] In the present invention, HED detectors in ControlNet are used to obtain sketches corresponding to the input image. Compared to edge maps with rich details and natural structures, more abstract sketches usually only capture the spatial structure of the input image. However, when dealing with more complex scenes, sketches often contain visually confusing and semantically irrelevant information, which provides potential for further compression of image data.

[0125] Image-to-mask model: Image-to-mask models generate masks for different regions or instances within an image. These masks help the encoder and subsequent image generation process focus on the most relevant parts of the image, thereby preserving as much semantic information as possible.

[0126] In the present invention, the "Segment Anything Model" is employed to obtain these masks.

[0127] Algorithm 1

[0128] The specific process is as follows:

[0129] Given the following inputs:

[0130] Conditional data C, mask set M, area threshold A min , original image I original , weight λ

[0131] Step (1): Initialization:

[0132] RD opt = ∞, mask opt = zeros, M filtered = {}

[0133] where RD opt is the optimal rate-distortion value, mask opt is the corresponding optimal mask, and M filtered is the area-filtered mask set;

[0134] Step 2: Mask filtering:

[0135] M filtered = {m ∈ M | A m ≥ A min}

[0136] where A m is the area of mask m;

[0137] Step 3: Sorting:

[0138] Sort M filtered in ascending order to obtain M sorted

[0139] Step 4: Rate-distortion optimization:

[0140]

[0141] m current =mask opt ∪m

[0142] C current =Decode(Encode(C·m current ))

[0143] I generated =MultiModalImageGeneration(C current )

[0144] RD=BitRate(C current )+λ·Distortion(I generated ,I original )

[0145] Where Decode(·) is the decoding operation (implemented as BPG decoding), Encode(·) is the encoding operation (implemented as BPG encoding), m current is the mask of current iteration, C current is the condition data of current iteration, I generated is the generated image of current iteration, RD is the rate-distortion value of current iteration, MultiModalImageGeneration(·) is the multi-modal image generation (implemented as the multi-modal image generation section), BitRate(·) is the code rate of input condition (implemented as the BPP of text and sketch), Distortion(·) is the distortion value between the generated image and the original image (implemented as the DISTS value between two images);

[0146] If RD < RD opt , then update:

[0147] RD opt =RD, mask opt =m current

[0148] Step 5: Output:

[0149] Output the optimized condition data C opt =C·mask opt

[0150] Lossless compression of text with Lempel-Ziv algorithm, lossy compression of the optimized sketch with BPG algorithm at quantization parameter QP = 12.

[0151] To achieve a better balance between compression and representation capability, the rate-distortion optimization strategy fuses the semantic information extracted based on the semantic mask.

[0152] The distortion metric of the rate-distortion optimization depends on the specific task. In the proposed method, to minimize the irrelevant information while maintaining the perceptual quality and semantic consistency of the reconstructed image;

[0153] The present application proposes a mask-based rate-distortion optimization, as shown in Algorithm 1.

[0154] Before rate-distortion optimization, first, mask filtering is established to filter out masks with an area less than 1%, because these small areas are usually considered to be semantically unimportant, and excluding these areas also improves the coding efficiency of the encoder. Next, using the DISTS metric, the rate-distortion optimization further refines the semantic information in the conditional data and finds the optimal representation under a given rate constraint.

[0155] In the method in the present application, after obtaining the descriptive text and the mask sketch, the Lempel-Ziv algorithm is applied to losslessly compress the text. For the mask sketch processed by image processing, the BPG algorithm with a quantization parameter QP=12 is used for lossy compression.

[0156] In the decoder, the text and sketch code streams are decompressed using the Lampel-Ziv algorithm and the BPG algorithm, respectively, to obtain the text and the optimized mask sketch, which are used as input to reconstruct the image using the multi-modal generative model "ControlNet" (for details, see step 3 above)

[0157] The present application extracts human understandable modalities as intermediate representations in the communication process, thereby eliminating the need to rely on the model of the sender;

[0158] The present application uses a mask-based rate-distortion optimization module to further optimize human understandable modalities containing semantic information.

Claims

1. A generative image compression method for semantic communication independent of the sender model, characterized in that, Comprising the following steps; Step 1: Given an image, first extract semantic information from the image at the encoder, obtain semantic information data; for text, sketch, entity mask; Step 2: Then, rate-distortion optimization is performed on the semantic information data to remove details irrelevant to the task, and then encoding is performed; Step 3: At the decoding end, the decoded semantic information is reconstructed into the original image through a multi-modal image generation model; The step 1 is specifically: Step (1): Given an input image x, an image-to-text model G text (·) is applied to obtain the descriptive text C text = G text (x) for the input image; the corresponding descriptive text is obtained from the input image based on MiniCPM-V2.6; Step (2): Given an input image x, image-to-condition model G condition (·) to get a specific condition G condition condition (x); get the corresponding sketch from the input image based on the HED detector in ControlNet;​ Step (3): Given an input image x, an image-to-mask model G mask (·) is applied to obtain a set of entity masks M = G mask (x) for the input image based on the Segment Anything Model; The step 2 is specifically, rate-distortion optimization is performed on the sketch based on the mask, and the optimized sketch is obtained, The specific process is: Given the following input: Sketch C condition i.e. the output of the image-to-condition model, the mask set M, the area threshold A min , the original image I original for the input image x, the weight l; Step (1): Initialization: RD opt = ∞, mask opt = zeros, M filtered = {} wherein RD opt is the optimal rate-distortion value, mask opt is the corresponding optimal mask, M filtered is the set of masks filtered by area; Step (2): Mask filtering: M filtered = {m e M | A m ≥ A min} where A m is the area of the mask m; Step (3): Sorting: M filtered M sorted Step (4): Rate-distortion optimization: For each mask m in M_sorted, the following steps are performed; m current = mask opt ∪, C current = Decode(Encode(C condition ·m current )) I generated = MultiModalImageGeneration(C current ) RD = BitRate(C current ) + λ · Distortion(I generated ,I original ) where Decode(·) is a decoding operation, Encode(·) is an encoding operation, m current is the mask of the current iteration, C current is the sketch of the current iteration, I generated is the generated image of the current iteration, RD is the rate-distortion value of the current iteration, MultiModalImageGeneration(·) is a multi-modal image generation, BitRate(·) is a code rate calculation of an input condition, Distortion(·) is a distortion value calculation between a generated image and an original image. If RD < RD opt then update: RD opt = RD, mask opt = m current Step (5): Output: Output optimized sketch C opt = C condition • mask opt .

2. The generative image compression method for semantic communication sender-agnostic model according to claim 1, wherein, In the step 1, the extraction of semantic information from the image is based on the image-to-text model, the image-to-condition model, and the image-to-mask model; Image-to-text model: Through this model, the input image is converted into descriptive text containing semantic content; The generated descriptive text provides a high-level content summary of the image, which is used to reconstruct the image content; Image-to-condition model: Through this model, a sketch is generated for a task related to the semantic content of the input image; Image-to-mask model: Through this model, masks are generated for different regions or instances within the input image, which help the encoder and subsequent image generation process focus on the most relevant parts of the image, thereby preserving as much semantic information as possible.

3. The generative image compression method for semantic communication sender-agnostic model according to claim 1, characterized in that, Steps (1) to (5) are the specific process of rate-distortion optimization. For the rate-distortion optimization module, the input is the output sketch and mask set of the condition model and the mask model, and the output is only the optimized sketch; The descriptive text obtained from the text model is directly passed to the text encoder, and the text encoder uses the Lampel-Ziv algorithm. The optimized sketch obtained from the rate-distortion optimization module is passed to the condition encoder, and the condition encoder uses the BPG algorithm. Two encoders are used to compress data into a code stream and pass it to the channel.

4. The generative image compression method for semantic communication sender-agnostic model according to claim 3, characterized in that, In the step 3, at the decoding end, there are a text decoder and a condition decoder, which are used to decode the code stream transmitted by the channel into their respective data, i.e. the text decoder decodes the text code stream into text, and the condition decoder decodes the condition code stream into a sketch, i.e. the decoded and optimized sketch, which is then passed to the multi-modal generation model; The text decoder uses the Lampel-Ziv algorithm, and the output is the descriptive text, which is also the input of the multi-modal generation model; The condition encoder uses the BPG algorithm, and the output is the optimized sketch, which is also the input of the multi-modal generation model; The descriptive text and the optimized sketch together constitute the decoded semantic information data.

5. The generative image compression method for semantic communication sender-agnostic model according to claim 4, characterized in that, In the step 3, the multi-modal image generation is specifically: Given decoded semantic information data Multimodal generative model G M (·), to obtain a reconstructed image reconstruct the input semantic information data into an image based on the ControlNet. The semantic information data comprising decoded descriptive text and optimized decoded sketches of 6. A generative image compression system for implementing the method of any one of claims 1 to 5, characterized in that, Including an encoder and a decoder; The encoder is used to extract semantic information from a natural image, and to perform rate-distortion optimization on the extracted semantic information data, and then to encode; The decoder is used for restoring the optimized semantic information delivered by the encoder, and reconstructing the image through the multi-modal generative model.

7. The generative image compression system for semantic communication sender-agnostic model according to claim 6, wherein, The encoder comprises a semantic information extraction module, a rate-distortion optimization module and a semantic information encoding module. The semantic information extraction module is used for converting the image into a pre-defined representation, i.e., semantic information data. The rate-distortion optimization module is used for performing rate-distortion optimization on the data redundancy in the semantic information data. The semantic information encoding module is used for encoding the optimized semantic information data into a code stream.

8. The generative image compression system for semantic communication sender-agnostic model according to claim 6, wherein, The decoder comprises a semantic information decoding module and a multi-modal generative model. The semantic information decoding module decodes the code stream corresponding to the optimized semantic information data into a human-understandable modality. The multi-modal generative model is used for reconstructing the semantic information data into the original image.

Images