Semantic communication-oriented generative image compression system and method independent of sending end model

Abstract

The invention discloses a semantic communication-oriented generative image compression system and method independent of a sending end model. The semantic communication-oriented generative image compression system comprises an encoder and a decoder, the encoder is used for performing semantic information extraction on a natural image and further optimizing the extracted semantic information data by using distortion optimization; and the decoder is used for restoring the optimized semantic information transmitted by the encoder through the channel, and reconstructing an image through a multi-modal generative model. According to the image compression system provided by the invention, modals which can be understood by human beings are used as intermediate representation, and a decoder does not need to know a specific model used by an encoder in advance; the system comprises a mask-based rate-distortion optimization module which can effectively remove information, which is low in semantic correlation, of an intermediate mode, maintain the perception quality of a reconstructed image and reduce the bit rate.

Description

Technical Field

[0001] The present invention belongs to the technical field of image compression in the field of semantic communication, and in particular relates to a generative image compression system and method for semantic communication that is independent of a sending end model. Background Art

[0002] Semantic communication is an emerging communication paradigm that focuses more on the transmission of the meaning behind the data than traditional communication, which focuses on the precise transmission of bit-level data. This communication method is particularly important today as data volumes surge and context-aware communication becomes increasingly important. In applications such as video conferencing, intelligent transportation systems, and the Internet of Things, the amount of data transmitted is huge, but not all transmitted information is critical to the recipient's decision-making process. Semantic communication improves communication efficiency, reduces network load, and ensures optimal use of resources by intelligently filtering and transmitting the most relevant content. This approach has great potential to address the limitations of traditional communication methods that have difficulty maintaining efficiency and relevance when processing large amounts of data.

[0003] In the field of semantic communication, especially image compression, existing technical solutions mainly rely on end-to-end learning frameworks. These frameworks achieve image compression and reconstruction by optimizing neural network models, but they usually require synchronization of deep models between transmitters and receivers. This design requires the receiver to know and store the corresponding model of the transmitter in advance to adapt to different transmission scenarios. In addition, these frameworks are very sensitive to changes in model parameters. Even minor adjustments may lead to significant changes in output results, which increases the difficulty of model synchronization. Therefore, how to ensure the synchronization of models at both ends is crucial.

[0004] Comparative document 1, Lei E, Uslu YB, Hassani H, et al. Text+sketch: Image compression at ultra low rates [J]. arXiv preprint arXiv: 2307.01944, 2023, discloses a technical solution for image compression based on text and sketches at ultra-low bit rates, but only considers text and sketches in the extraction of semantic information. Since the compression of sketches adopts an end-to-end compression framework, the decoder still needs to know the encoding model used by the encoder in advance. This solution requires a high degree of model synchronization between the encoder and the decoder.

[0005] In summary, current semantic communication technologies based on deep learning are often implemented in an end-to-end manner, which places high demands on model synchronization between the sender and the receiver during the communication process. Even a small parameter change may cause serious impact on the decoding process. Since manufacturers usually need to customize the technology used according to different tasks and requirements, the end-to-end communication method limits this flexibility to a certain extent. Summary of the invention

[0006] In order to overcome the shortcomings of the above-mentioned prior art, the purpose of the present invention is to provide a generative image compression system and method for semantic communication that is independent of the sender model. The image compression system uses a modality that can be understood by humans as an intermediate representation, and the decoder does not need to know the specific model used by the encoder in advance; the system includes a mask-based rate-distortion optimization module that can effectively remove information of the intermediate modality that has low semantic relevance, maintain the perceptual quality of the reconstructed image and reduce the bit rate.

[0007] In order to achieve the above object, the technical solution adopted by the present invention is:

[0008] A generative image compression system for semantic communication that does not rely on the sender model, including an encoder and a decoder;

[0009] The encoder is used to extract semantic information from natural images, perform rate-distortion optimization on the extracted semantic information data, and then encode;

[0010] The decoder is used to restore the optimized semantic information transmitted by the encoder, and then reconstruct the image through the multimodal generative model.

[0011] The encoder includes a semantic information extraction module, a rate-distortion optimization module and a semantic information encoding module;

[0012] The semantic information extraction module is used to convert the image into a predefined representation, which is usually expressed in a human-understandable modality, i.e., semantic information data (data in the form of semantic information, such as sketches, edge maps, depth maps, segmentation maps, subject reference images, texts, human posture key points, etc., which abstractly express specific features in the input image and are expressed in a human-intuitively understandable modality);

[0013] The rate-distortion optimization module is used to perform rate-distortion optimization on data redundancy in the semantic information data;

[0014] The semantic information encoding module is used to encode the optimized semantic information data into a code stream.

[0015] The decoder includes a semantic information decoding module and a multimodal generative model;

[0016] The semantic information decoding module decodes the code stream corresponding to the optimized semantic information data into a human-understandable modality;

[0017] The multimodal generative model is used to reconstruct semantic information data (expressed in human-understandable modalities) into the original image.

[0018] A generative image compression method for semantic communication that does not rely on a sender model comprises the following steps:

[0019] Step 1: Given an image (natural image or artistic creation), first extract semantic information from the image at the encoder, usually in a human-understandable modality, i.e., semantic information data; the human-understandable modality is specifically manifested as text, sketches, entity masks;

[0020] Step 2: Then, the semantic information data is rate-distortion optimized to remove details irrelevant to the task, and then encoded;

[0021] Step 3: At the decoding end, the decoded semantic information is used to reconstruct the original image through a multimodal image generation model.

[0022] In the step 1, extracting semantic information from the image is implemented based on an image-to-text model, an image-to-conditional model, and an image-to-mask model;

[0023] Image-to-text model: Through this model, the input image is converted into descriptive text containing semantic content; the generated descriptive text provides a high-level content summary of the image and is used to reconstruct the image content;

[0024] Image-to-conditional model: This model generates various human-understandable modality representations for tasks that are semantically related to the input image;

[0025] Image-to-mask model: Through this model, masks are generated for different regions or instances within the input image. The masks help the encoder and subsequent image generation process focus on the most relevant parts of the image, thereby preserving semantic information as much as possible.

[0026] The step 1 is specifically as follows:

[0027] Step (1): Given an input image x, the image-to-text model G text (·), get the descriptive text G of the input image text =G text (x); Obtain the corresponding descriptive text from the input image based on MiniCPM-V2.6;

[0028] Step (2): Given an input image x, image to conditional model G condition(·), we get the specific condition C that contains the semantic information of the input image. condition =G condition (x); Get the corresponding sketch from the input image based on the HED detector in ControlNet;

[0029] Step (3): Given an input image x, the image-to-mask model G mask (·), we get the entity mask set M = G of the input image mask (x); Based on the Segment Anything Model, obtain the corresponding mask set from the input image.

[0030] The step 2 is specifically as follows:

[0031] The image-to-conditional model outputs conditional data. In the specific implementation of this solution, the image-to-conditional model is a model that extracts a sketch of the input image, and the output conditional data is the sketch. In this framework, the sketch is just an implementation method and can be any data type that is masked, such as edge maps, depth maps, and segmentation maps;

[0032] Perform rate-distortion optimization on the sketch based on the mask to obtain an optimized sketch;

[0033] The specific process is:

[0034] Given the following input:

[0035] Conditional Data C condition , that is, the output of the image to the conditional model, the mask set M, and the area threshold A min , original image I original (i.e. input image x), weight λ;

[0036] Step (1): Initialization:

[0037] RD opt =∞,mask op t=zeros,M filtered ={}

[0038] Among them, RD opt is the optimal rate-distortion value, mask opt is the corresponding optimal mask, M filtered is the mask set after area filtering;

[0039] Step (2): Mask filtering:

[0040] M filtered ={m∈M|A m ≥A min}

[0041] Among them, Am is to calculate the area of ​​mask m;

[0042] Step (3): Sorting:

[0043] To M filtered Sort in ascending order to get M sorted

[0044] Step (4): Rate-distortion optimization:

[0045]

[0046] m current =mask opt ∪m

[0047] C current =Decode(Encode(C condition ·m current ))

[0048] I generated =MultiModalImageGeneration(C current )

[0049] RD=BitRate(C current )+λ·Distortion(I generated ,I original )

[0050] Among them, Decode(·) is a decoding operation (specifically implemented as BPG decoding), Encode(·) is an encoding operation (specifically implemented as BPG encoding), m current is the mask of the current iteration, C current is the conditional data of the current iteration, I generated is the generated image of the current iteration, RD is the rate-distortion value of the current iteration, MultiModalImageGeneration(·) is multimodal image generation (the specific implementation is the same as the multimodal image generation section), BitRate(·) is the bit rate of the calculated input condition (the specific implementation is to calculate the BPP of text and sketch), Distortion(·) is the distortion value between the generated image and the original image (the specific implementation is to calculate the DISTS value between two images);

[0051] If RD <RD opt , then update:

[0052] RD opt =RD,mask opt =m current

[0053] Step (5): Output:

[0054] Output optimized condition data C opt =C condition ·Mask opt .

[0055] Steps (1) to (5) are the specific process of rate-distortion optimization. For the rate-distortion optimization module, the input is the output conditional data and mask set of the conditional model and the mask model, and the output is only the optimized conditional data, that is, the optimized sketch;

[0056] The descriptive text obtained from the text model is directly passed to the text encoder, namely the Lampel-Ziv algorithm, and the optimized conditional data obtained from the rate-distortion optimization module, namely the optimized sketch, is passed to the conditional encoder, namely the BPG algorithm. The two encoders are used to compress the data into a code stream and pass it to the channel. The decoding of step 3 is specifically as follows:

[0057] The text is decompressed using the Lempel-Ziv algorithm; the optimized sketch is decompressed using the BPG algorithm.

[0058] In the step 3, at the decoding end, a text decoder and a conditional decoder are included, and the text decoder and the conditional decoder are used to decode the code stream transmitted by the channel into respective data, that is, the text decoder decodes the text code stream into text, and the conditional decoder decodes the conditional code stream into conditional data, that is, the decoded optimized sketch, and then the two are jointly passed to the multimodal generation model;

[0059] The output of the text decoder, i.e. the Lampel-Ziv algorithm, is the descriptive text, which is also the input of the multimodal generative model; the output of the conditional encoder, i.e. the BPG algorithm, is the optimized conditional data, i.e. the optimized sketch in this scheme, which is also the input of the multimodal generative model;

[0060] The descriptive text and the optimized conditional data together constitute the decoded semantic information data.

[0061] In step 3, the multimodal image generation is specifically as follows:

[0062] Given the decoded semantic information data Multimodal generative model G M (·), and the reconstructed image is obtained Reconstruct the input semantic information data into an image based on ControlNet;

[0063] The semantic information data Contains decoded descriptive text The optimized decoding sketch of and

[0064] Beneficial effects of the present invention:

[0065] The image compression framework for semantic communication proposed in the present invention is different from the end-to-end implementation used in common semantic communication frameworks. The decoding end in the framework does not need to know the neural network model used by the encoding end in advance. At the same time, in the encoding end of this framework, the semantic information extracted (expressed as a human-understandable modality) is further eliminated through mask-based rate-distortion optimization, thereby improving the coding efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0066] Figure 1 Schematic diagram of the generative image compression framework proposed in the present invention.

[0067] Figure 2 The figure is a schematic diagram of a specific implementation of the framework proposed in the present invention. DETAILED DESCRIPTION

[0068] The present invention will be further described in detail below in conjunction with the accompanying drawings.

[0069] Figure 1 The proposed generative image compression framework includes an encoder and a decoder;

[0070] The encoder is used to extract semantic information from natural images, and further optimize the extracted semantic information data using rate distortion optimization;

[0071] The decoder is used to restore the optimized semantic information transmitted by the encoder through the channel, and then reconstruct the image through the multimodal generative model.

[0072] The encoder includes a semantic information extraction module, a rate-distortion optimization module and a semantic information encoding module;

[0073] The semantic information extraction module is used to convert the image into a predefined representation, which is hereinafter referred to as semantic information data (a data form containing semantic information, such as a series of data that abstractly expresses specific features in the input image, such as sketches, edge maps, depth maps, segmentation maps, subject reference images, texts, human posture key points, etc., which are expressed in a modality that can be intuitively understood by humans);

[0074] The rate-distortion optimization module is used to further optimize data redundancy in the semantic information data;

[0075] The semantic information encoding module is used to encode the optimized semantic information data into a code stream.

[0076] The decoder includes a semantic information decoding module and a multimodal generative model;

[0077] The semantic information decoding module decodes the code stream corresponding to the optimized semantic information data into a human-understandable modality;

[0078] The multimodal generative model is used to reconstruct semantic information data (expressed in human-understandable modalities) into the original image.

[0079] The specific method of generative image compression for semantic communication that does not rely on the sender model is as follows:

[0080] Step 1: In this framework, given an image (natural or artistic), semantic information is first extracted from the image at the encoder, usually in the form of a human-understandable modality; the human-understandable modality is specifically manifested as text, sketches, entity masks;

[0081] Step 2: Then, the semantic information is further optimized through rate-distortion optimization to remove details irrelevant to the task before encoding;

[0082] Step 3: At the decoding end, the decoded semantic information is used to reconstruct the original image through a multimodal image generation model.

[0083] The step 1 is specifically as follows:

[0084] Step (1): Given an input image x, the image-to-text model G text (·), get the descriptive text C of the input image text =G text (x); where G text (·) is implemented as "MiniCPM-V 2.6"

[0085] Step (2): Given an input image x, image to conditional model G condition (·), we get the specific condition C that contains the semantic information of the input image. condition =G condition (x); In the specific implementation, the condition is a sketch, where G condition (·) is the sketch mode of the edge detection model "HED" in "ControlNet"

[0086] Step (3): Given an input image x, the image-to-mask model G mask (·), we get the entity mask set M = G of the input image mask (x), where G mask The specific implementation of (·) is the entity segmentation model “Segment Anything Model”.

[0087] The step 2 is specifically as follows:

[0088] Perform rate-distortion optimization on the sketch based on the mask to obtain an optimized sketch;

[0089] The specific process is:

[0090] Given the following input:

[0091] Conditional Data C condition , that is, the output of the image to the conditional model, the mask set M, and the area threshold A min , original image I original (i.e., input image x), weight λ

[0092] Step (1): Initialization:

[0093] RD opt =∞,mask opt =zeros,M filtered ={}

[0094] Among them, RD opt is the optimal rate-distortion value, mask opt is the corresponding optimal mask, M filtered is the mask set after area filtering;

[0095] Step (2): Mask filtering:

[0096] M filtered ={m∈M|A m ≥A min}

[0097] Among them, A m is to calculate the area of ​​mask m;

[0098] Step (3): Sorting:

[0099] To M filtered Sort in ascending order to get M sorted

[0100] Step (4): Rate-distortion optimization:

[0101]

[0102] m current =mask opt ∪m

[0103] C current =Decode(Encode(C condition ·m current ))

[0104] I generated =MultiModalImageGeneration(C current )

[0105] RD=BitRate(C current )+λ·Distortion(I generated ,I original )

[0106] Among them, Decode(·) is a decoding operation (specifically implemented as BPG decoding), Encode(·) is an encoding operation (specifically implemented as BPG encoding), m current is the mask of the current iteration, C current is the conditional data of the current iteration, I generated is the generated image of the current iteration, RD is the rate-distortion value of the current iteration, MultiModalImageGeneration(·) is multimodal image generation (the specific implementation is the same as the multimodal image generation section), BitRate(·) is the bit rate of the calculated input condition (the specific implementation is to calculate the BPP of text and sketch), Distortion(·) is the distortion value between the generated image and the original image (the specific implementation is to calculate the DISTS value between two images);

[0107] If RD <RD opt , then update:

[0108] RD opt =RD,mask opt =m current

[0109] Step (5): Output:

[0110] Output optimized condition data C opt =C condition ·Mask opt .

[0111] Steps (1) to (5) are the specific process of rate-distortion optimization. For the rate-distortion optimization module, the input is the output conditional data and mask set of the conditional model and the mask model, and the output is only the optimized conditional data, that is, the optimized sketch;

[0112] The descriptive text obtained from the text model is directly passed to the text encoder, namely the Lampel-Ziv algorithm, and the optimized conditional data obtained from the rate-distortion optimization module, namely the optimized sketch, is passed to the conditional encoder, namely the BPG algorithm. The two encoders are used to compress the data into a code stream and pass it to the channel.

[0113] The text is losslessly compressed using the Lempel-Ziv algorithm, and the optimized sketch is losslessly compressed using the BPG algorithm with a quantization parameter QP=12.

[0114] The decoding of step 3 is specifically as follows:

[0115] The text is decompressed using the Lempel-Ziv algorithm; the optimized sketch is decompressed using the BPG algorithm.

[0116] In step 3, the multimodal image generation is specifically as follows:

[0117] Given the decoded semantic information data Multimodal generative model G M (·), and the reconstructed image is obtained

[0118] The semantic information data Contains decoded descriptive text The optimized decoding sketch of and The specific implementation of the multimodal generation model is "ControlNet".

[0119] The pipeline aims to enhance image transmission and generation by leveraging multimodal data processing. By integrating various models, the framework can effectively encode and decode semantic information in images and generate high-quality images in multimodal environments. Its overall pipeline mainly consists of two components: encoder and decoder.

[0120] The semantic information extraction module includes an image-to-text model, an image-to-conditional model, and an image-to-mask model;

[0121] Image-to-text model: With this model, an input image is converted into descriptive text containing semantic content; the generated descriptive text provides a high-level content summary of the image, which can then be used to reconstruct the image content.

[0122] In the specific implementation of the present invention, MiniCPM-V 2.6 is selected as the image-to-text model, which accepts images and text as inputs and uses prompt words to generate high-quality image content descriptions.

[0123] Image-to-conditional model: The input image is fed into an image-to-conditional model, which generates various human-understandable modal representations for semantically related tasks, such as edges, styles, and reference images of subject objects. These representations are critical for accurately reconstructing task-related semantic information in the decoding stage.

[0124] In this paper, the HED detector in ControlNet is used to obtain a sketch corresponding to the input image. Compared with the detailed and natural edge map, the more abstract sketch usually only captures the spatial structure of the input image. However, when dealing with more complex scenes, the sketch often contains visually confusing and semantically irrelevant information, which provides potential for further compressing image data.

[0125] Image-to-mask model: The image-to-mask model generates masks for different regions or instances within the image. These masks help the encoder and subsequent image generation process focus on the most relevant parts of the image, thereby preserving semantic information as much as possible.

[0126] In the present invention, the "Segment Anything Model" is adopted to obtain these masks.

[0127] Algorithm 1

[0128] The specific process is:

[0129] Given the following input:

[0130] Conditional data C, mask set M, area threshold A min , original image I original , weight λ

[0131] Step (1): Initialization:

[0132] RD opt =∞,mask opt =zeros,M filtered ={}

[0133] Among them, RD opt is the optimal rate-distortion value, mask opt is the corresponding optimal mask, M filtered is the mask set after area filtering;

[0134] Step 2: Mask filtering:

[0135] M filtered ={m∈M|A m ≥A min}

[0136] Among them, A m is to calculate the area of ​​mask m;

[0137] Step 3: Sorting:

[0138] To M filtered Sort in ascending order to get M sorted

[0139] Step 4: Rate-distortion optimization:

[0140]

[0141] m current =mask opt ∪m

[0142] C current =Decode(Encode(C·m current ))

[0143] I generated =MultiModalImageGeneration(C current )

[0144] RD=BitRate(C current )+λ·Distortion(I generated ,I original )

[0145] Among them, Decode(·) is a decoding operation (specifically implemented as BPG decoding), Encode(·) is an encoding operation (specifically implemented as BPG encoding), m current is the mask of the current iteration, C current is the conditional data of the current iteration, I generated is the generated image of the current iteration, RD is the rate-distortion value of the current iteration, MultiModalImageGeneration(·) is multimodal image generation (the specific implementation is the same as the multimodal image generation section), BitRate(·) is the bit rate of the calculated input condition (the specific implementation is to calculate the BPP of text and sketch), Distortion(·) is the distortion value between the generated image and the original image (the specific implementation is to calculate the DISTS value between two images);

[0146] If RD <RD opt , then update:

[0147] RD opt =RD,mask opt =m current

[0148] Step 5: Output:

[0149] Output optimized condition data C opt =C mask opt

[0150] The text is losslessly compressed using the Lempel-Ziv algorithm, and the optimized sketch is losslessly compressed using the BPG algorithm with a quantization parameter QP=12.

[0151] To achieve a better balance between compression and representation ability, a rate-distortion optimization strategy is proposed to fuse the extracted semantic information based on semantic masks.

[0152] The distortion metric for rate-distortion optimization depends on the specific task. In the proposed method, in order to minimize irrelevant information while maintaining the perceptual quality and semantic consistency of the reconstructed image;

[0153] The present invention proposes a mask-based rate-distortion optimization, as shown in Algorithm 1.

[0154] Before rate-distortion optimization, mask filtering is first established to filter out masks with a coverage area less than 1%, because these small areas are usually considered semantically unimportant, and excluding these areas also improves the encoding efficiency of the encoder. Next, using the DISTS metric, rate-distortion optimization further refines the semantic information in the conditional data and finds the optimal representation under a given rate constraint.

[0155] In the method of the present invention, after obtaining the descriptive text and the mask sketch, the Lempel-Ziv algorithm is applied to perform lossless compression on the text. For the mask sketch after image processing, the BPG algorithm with a quantization parameter QP=12 is used for lossy compression.

[0156] In the decoder, the Lampel-Ziv algorithm and the BPG algorithm are used to decompress the text and sketch code streams respectively to obtain the text and the optimized mask sketch, which are used as input to reconstruct the image using the multimodal generative model "ControlNet" (see step 3 above for the specific process)

[0157] The present invention extracts human-understandable modalities as intermediate representations in the communication process, thereby eliminating the need to rely on a model at the sender end;

[0158] The present invention adopts a mask-based rate-distortion optimization module to further optimize the human-understandable modality containing semantic information.

Claims

1. A generative image compression system for semantic communication that does not rely on the sender model, characterized by: Includes encoder and decoder; The encoder is used to extract semantic information from natural images, perform rate-distortion optimization on the extracted semantic information data, and then encode; The decoder is used to restore the optimized semantic information transmitted by the encoder, and then reconstruct the image through the multimodal generative model.

2. The generative image compression system for semantic communication independent of the sender model according to claim 1, characterized in that: The encoder includes a semantic information extraction module, a rate-distortion optimization module and a semantic information encoding module; The semantic information extraction module is used to convert the image into a predefined representation, which is usually expressed in a human-understandable modality, i.e., semantic information data; The rate-distortion optimization module is used to perform rate-distortion optimization on data redundancy in the semantic information data; The semantic information encoding module is used to encode the optimized semantic information data into a code stream.

3. A generative image compression system for semantic communication that does not rely on the sender model, characterized in that: The decoder includes a semantic information decoding module and a multimodal generative model; The semantic information decoding module decodes the code stream corresponding to the optimized semantic information data into a human-understandable modality; The multimodal generative model is used to reconstruct the semantic information data into the original image.

4. A generative image compression method for semantic communication independent of a transmitter model based on the system according to any one of claims 1 to 3, characterized in that: The steps include: Step 1: Given an image, first extract semantic information from the image at the encoder, i.e., semantic information data; the human-understandable modal form is specifically realized as text, sketch, entity mask; Step 2: Then, the semantic information data is rate-distortion optimized to remove details irrelevant to the task, and then encoded; Step 3: At the decoding end, the decoded semantic information is used to reconstruct the original image through a multimodal image generation model.

5. The generative image compression method for semantic communication independent of the sender model according to claim 4, characterized in that: In the step 1, extracting semantic information from the image is implemented based on an image-to-text model, an image-to-conditional model, and an image-to-mask model; Image-to-text model: Through this model, the input image is converted into descriptive text containing semantic content; the generated descriptive text provides a high-level content summary of the image and is used to reconstruct the image content; Image-to-conditional model: This model generates various human-understandable modality representations for tasks that are semantically related to the input image; Image-to-mask model: Through this model, masks are generated for different regions or instances within the input image. The masks help the encoder and subsequent image generation process focus on the most relevant parts of the image, thereby preserving semantic information as much as possible.

6. The generative image compression method for semantic communication independent of the sender model according to claim 4, characterized in that: The step 1 is specifically as follows: Step (1): Given an input image x, the image-to-text model G text (·), get the descriptive text C of the input image text =G text (x); Obtain the corresponding descriptive text from the input image based on MiniCPM-V2.6; Step (2): Given an input image x, image to conditional model G condition (·), we get the specific condition C that contains the semantic information of the input image. condition =G condition (x); Get the corresponding sketch from the input image based on the HED detector in ControlNet; Step (3): Given an input image x, the image-to-mask model G mask (·), we get the entity mask set M = G of the input image mask (x), obtain the corresponding mask set from the input image based on the Segment Anything Model.

7. The generative image compression method for semantic communication independent of the sending end model according to claim 4, characterized in that: Specifically, step 2 is to perform rate-distortion optimization on the sketch based on the mask to obtain an optimized sketch. The specific process is: Given the following input: Conditional Data C condition , that is, the output of the image to the conditional model, the mask set M, and the area threshold A min , original image I original (i.e. input image x), weight λ; Step (1): Initialization: RD opt =∞,mask opt =zeros,M filtered ={} Among them, RD opt is the optimal rate-distortion value, mask opt is the corresponding optimal mask, M filtered is the mask set after area filtering; Step (2): Mask filtering: M filtered ={m∈M|A m ≥A min } Among them, A m is to calculate the area of ​​mask m; Step (3): Sorting: To M filtered Sort in ascending order to get M sorted Step (4): Rate-distortion optimization: m current =mask opt ∪m C current =Decode(Encode(C condition ·m current )) I generated =MultiModalImageGeneration(C current ) RD=BitRate(C current )+λ·Distortion(I generated ,I original ) Among them, Decode(·) is the decoding operation, Encode(·) is the encoding operation, and m current is the mask of the current iteration, C current is the conditional data of the current iteration, I generated is the generated image of the current iteration, RD is the rate-distortion value of the current iteration, MultiModalImageGeneration(·) is multimodal image generation, BitRate(·) is the bit rate of the calculated input condition, Distortion(·) is the distortion value between the calculated generated image and the original image; If RD <RD opt , then update: RD opt =RD,mask opt =m current Step (5): Output: Output optimized condition data C opt =C condition ·Mask opt .

8. The generative image compression method for semantic communication independent of the sender model according to claim 7, characterized in that: Steps (1) to (5) are the specific process of rate-distortion optimization. For the rate-distortion optimization module, the input is the output conditional data and mask set of the conditional model and the mask model, and the output is only the optimized conditional data, that is, the optimized sketch; The descriptive text obtained from the text model is directly passed to the text encoder, namely the Lampel-Ziv algorithm, and the optimized conditional data obtained from the rate-distortion optimization module, namely the optimized sketch, is passed to the conditional encoder, namely the BPG algorithm. The two encoders are used to compress the data into a code stream and pass it to the channel.

9. The generative image compression method for semantic communication independent of the sender model according to claim 8, characterized in that: In the step 3, at the decoding end, a text decoder and a conditional decoder are included, and the text decoder and the conditional decoder are used to decode the code stream transmitted by the channel into respective data, that is, the text decoder decodes the text code stream into text, and the conditional decoder decodes the conditional code stream into conditional data, that is, the decoded optimized sketch, and then the two are jointly passed to the multimodal generation model; The output of the text decoder, i.e. the Lampel-Ziv algorithm, is the descriptive text, which is also the input of the multimodal generative model; the output of the conditional encoder, i.e. the BPG algorithm, is the optimized conditional data, i.e. the optimized sketch, which is also the input of the multimodal generative model; The descriptive text and the optimized conditional data together constitute the decoded semantic information data.

10. The generative image compression method for semantic communication independent of the sender model according to claim 9, characterized in that: In step 3, the multimodal image generation is specifically as follows: Given the decoded semantic information data Multimodal generative model G M (·), and the reconstructed image is obtained Reconstruct the input semantic information data into an image based on ControlNet; The semantic information data Contains decoded descriptive text The optimized decoding sketch of and

Images