An image semantic communication system based on large models

Abstract

The present invention discloses an image semantic communication system based on a large model, comprising the following steps: S1‑1: text semantic extraction, using a semantic extractor to convert an original image with low semantic density into a corresponding text description with high semantic density; S1‑2: semantic coding and channel coding, ensuring effective semantic information transmission on a physical channel; S1‑3: channel decoding and semantic decoding; S1‑4: image reconstruction: using a text encoder, a diffusion U‑Net and an image decoder in a semantic reconstructor to reconstruct the image; S1‑5: progressive learning assisted by experience replay, the encoder and decoder assisted by experience replay continuously sample from the experience replay during the training phase; the beneficial technical effects of the present invention include: improving the efficiency of transmitting semantics, improving the stability of semantics and channel coding, and enhancing the generalization ability of the system through experience replay.

Description

Technical Field

[0001] The invention belongs to the field of semantic communication, and in particular relates to an image semantic communication system based on a large model. Background Art

[0002] With the rapid development of deep learning, researchers have begun to explore end-to-end image semantic communication systems based on deep neural networks. For example, image semantic communication systems built using deep learning methods such as convolutional neural networks and Vision Transformers (ViT) have surpassed traditional solutions.

[0003] Although significant progress has been made in the study of image semantic communication based on deep learning, some problems still exist: (1) Traditional image semantic communication systems directly encode the entire image and extract low-level semantic information in pixels, which has a large amount of natural signals with spatial redundancy. (2) Image semantic communication systems often operate in dynamic environments, causing the feature distribution and channel state of the transmitted image data to drift over time, so the performance of the semantic communication system will degrade. (3) Traditional semantic communication systems based on deep learning are trained under specific signal-to-noise ratio conditions, while the signal-to-noise ratio in real communication environments changes dynamically. Summary of the invention

[0004] The present invention aims to solve at least one of the technical problems existing in the prior art; to this end, the present invention proposes an image semantic communication system based on a large model, which is used to solve the technical problems of spatial redundancy in image semantics, poor generalization performance of the image semantic communication system and uncertain signal-to-noise ratio during wireless communication.

[0005] To achieve the above object, the present invention provides an image semantic communication system based on a large model, comprising the following steps:

[0006] S1-1: Text semantic extraction: Image semantic extraction is performed using the image encoder and text decoder in the semantic extractor, thereby converting the original image with low semantic density into a corresponding text description with high semantic density;

[0007] S1-2: Semantic Coding and Channel Coding: The text information generated by the semantic extractor is then fed into the semantic encoder; the semantic encoder analyzes the text information and converts it into a compact semantic representation; the semantic encoder consists of a semantic coding layer and a signal-to-noise ratio weighting module; the semantic features generated after encoding are channel coded and modulated by the channel encoder to ensure efficient transmission of semantic information over the physical channel;

[0008] S1-3: Channel decoding and semantic decoding: At the receiving end, the information transmitted through the physical channel is received and decoded using a channel decoder; the channel decoder adopts the opposite structure of the channel encoder; the semantic decoder consists of a semantic decoding layer and a signal-to-noise ratio weighting module, which aims to reverse the semantic encoding process to restore the original text information;

[0009] S1-4: Image reconstruction: The text encoder, diffuse U-Net and image decoder of the semantic reconstructor are used to reconstruct the image. The specific steps are as follows: the text information is first converted into a conditional vector by the text encoder; then, the diffuse U-Net converts the noisy image into a latent image feature vector aligned with the conditional vector; finally, the latent image feature vector is processed by the image decoder to generate the final reconstructed image;

[0010] S1-5: Progressive learning assisted by experience replay: During the training phase of this system, the latest samples are stored in a temporary experience replay; when the temporary experience replay becomes full, a clustering method is used to select representative temporary samples to transfer to the fixed experience replay; then, the temporary experience replay is cleared; the experience replay-assisted encoder and decoder continue to sample from the temporary experience replay and the fixed experience replay during the training phase.

[0011] Further: in S1-1, the semantic extractor includes: a guided language image pre-training model is adopted on the sending end, and the guided language image pre-training model includes: an image encoder, a text encoder, a text decoder and an image decoder.

[0012] Furthermore: the image encoder includes a feature extraction module based on Vision Transformer, which divides the input image x into a patch sequence , and for the patch sequence Each patch in is encoded, and each patch represents a fixed-size image region; the vector of each patch is processed through multiple encoder layers with multi-head self-attention sublayers and feed-forward sublayers to generate a feature vector representation of the image.

[0013] Furthermore: the text decoder adopts the BERT structure, which is composed of multiple stacked decoder layers, each decoder layer includes three sublayers: a causal self-attention sublayer, a cross attention sublayer and a feedforward sublayer; the image features are decoded into specific text feature vectors through these three layers; the last layer of the text decoder converts the text feature vector into the next tag in the prediction sequence through linear projection and softmax function; then, in the generation process, each output text will be used as the input of the next time step until the corresponding text description of the image with high semantic density is generated.

[0014] Further: the semantic coding, channel coding, semantic decoding and channel decoding in steps S1-2 and S1-3 all use a signal-to-noise ratio weighting module, and the working process of the signal-to-noise ratio weighting module is as follows:

[0015] S5-1: SNR Scaling: First, the SNR projection module scales the SNR value to the same dimension as the feature vector in the encoder and decoder; the SNR projection module is a fully connected layer feedforward network, as shown in the following formula:

[0016] ;

[0017] in, represents the activation function, is the signal-to-noise ratio value;

[0018] and They represent the weights and biases of the fully connected layers in the signal-to-noise ratio scaling module respectively;

[0019] S5-2: Feature weighting: weighting the input features with the signal-to-noise ratio vector Combined, a weighting factor V is obtained, which records the importance of each intermediate feature vector to the semantic / channel encoder and semantic / channel decoder, as shown in the following formula:

[0020] ;

[0021] Among them, Tanh is the activation function; f is the feature map of the input feature F after passing through a fully connected layer, as shown in the following formula:

[0022] ;

[0023] and Respectively represent the weight and bias of the fully connected layer in the feature weighting module;

[0024] Finally, the input feature F is multiplied by the weighting factor V to obtain the adjustment vector C, as shown in the following formula:

[0025] ;

[0026] in, represents the i-th element in the adjustment vector C, represents the i-th element in the input feature vector F, represents the i-th element in the weighting factor V;

[0027] The signal-to-noise ratio weighting module is embedded into the feature vectors of the semantic / channel encoder and decoder to enhance the robustness of the system.

[0028] Further: in the step S1-4, the semantic reconstructor also includes a stable diffusion model for image reconstruction, and the specific steps are as follows:

[0029] S6-1: Given a semantic text , the text encoder takes as input the semantic text Converted into a fixed-dimensional semantic vector, the specific process is: first, the semantic text Each word in is mapped to a fixed-length vector through word embedding; these word embeddings are used as input to the text encoder; the encoder iteratively performs multi-head self-attention sublayer and feedforward sublayer operations, and finally produces a sequence of text feature vectors;

[0030] S6-2: Feature generator: The initial image feature vector consisting of pure noise is input into the image feature generator to generate a noise feature vector; the sequence consisting of text feature vectors is injected into the noise feature vector to guide noise removal; through multiple iterations, the noise is gradually removed and an image feature vector containing text information is obtained; the noise removal step adopts the U-Net structure, which adopts an encoder-decoder structure based on convolutional neural networks to retain spatial information while generating image semantic information;

[0031] S6-3: Image decoder: Due to the low computational efficiency of the diffusion operation, the image denoising process is performed in a compressed semantic / feature space, i.e., multiple denoising iterations are performed in the reduced semantic / feature space, which improves the image computational efficiency;

[0032] Finally, the decoder of the variational autoencoder is used to map the feature data in the semantic / feature space back to the pixel space to reconstruct an image that is semantically consistent. Since the variational autoencoder has learned the potential structure of the distribution of a large amount of image data, the decoder can provide more detailed information through upsampling and interpolation during the decoding process, which is consistent with the key semantics in the image, thereby enhancing the image quality in the pixel space.

[0033] Further: in the step S1-5, an encoder and decoder assisted by experience replay are introduced, and the learning strategy based on experience replay solves the problem of selective forgetting in incremental learning by diversifying the content of experience replay; an experience replay assisted method with temporary experience replay and fixed experience replay is designed for the encoder and decoder assisted by experience replay; the workflow of the encoder and decoder assisted by experience replay is as follows:

[0034] Will and Defined as a set representing the dynamic samples stored in the temporary experience replay and the fixed experience replay;

[0035] in, represents the i-th sample in the temporary experience replay;

[0036] represents the jth sample in the fixed experience replay;

[0037] When the temporary experience replay pool becomes full, representative samples need to be selected from it and transferred to the fixed experience replay.

[0038] Further: the process of selecting representative samples is as follows:

[0039] S8-1: Short-term typical sample evaluation: During the inference phase, new samples being processed are continuously added to the temporary experience replay; when the number of samples in the temporary experience replay exceeds the specified maximum value, an evaluation operation is performed; the main goal is to evaluate the typicality of temporary samples and use the K-Medoids clustering algorithm to select the most typical k temporary samples ;

[0040] S8-2: Long-term sample selection: The main goal is to select samples from temporary experience replay that are significantly different from samples in fixed experience replay to ensure diversity in experience replay; use Gaussian function to calculate samples and each sample in the fixed experience replay The average similarity score M between them is shown in the following formula:

[0041] ;

[0042] in, is the Gaussian parameter;

[0043] When the calculated similarity score is greater than a given threshold When , the sample is transferred from the temporary experience replay to the fixed experience replay, as shown in the following formula:

[0044] ;

[0045] After the selection is completed, It is cleared to buffer new samples for the next round; then the semantic encoder and semantic decoder are trained through incremental learning using both temporary experience replay and fixed experience replay.

[0046] The beneficial technical effects of the present invention include:

[0047] 1. The present invention is an image semantic communication system based on a large model, which introduces a semantic extractor based on a guided language image pre-training model on the sending end and a semantic reconstructor based on a stable diffusion model on the receiving end, which are used to generate high-quality text descriptions consistent with the image and reconstruct images matching the text descriptions. High-level semantics are extracted from images with low-level pixels to enhance the semantic density of the transmitted information.

[0048] 2. The present invention is an image semantic communication system based on a large model, which introduces a signal-to-noise ratio weighting module to dynamically adjust semantic coding and channel coding according to different signal-to-noise ratio conditions, thereby improving channel coding to combat strong channel noise.

[0049] 3. The present invention is a large-model-based image semantic communication system that introduces an experience replay-assisted codec to track changes in dynamic environments, including two types of experience replay storage pools: temporary experience replay and fixed experience replay. This enables the semantic encoder and decoder to view all knowledge with different distributions in previously trained data while learning from new data, thereby avoiding catastrophic forgetting. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] In order to more clearly illustrate the embodiments of the present invention and the technical solutions in the prior art, the drawings required for use in the embodiments and the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some implementation methods recorded in the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0051] Figure 1 It is a schematic diagram of the process of the present invention. DETAILED DESCRIPTION

[0052] In order to make the purpose, technical solutions and advantages of the present application clearer, the present application is described and illustrated below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not intended to limit the present application. Based on the embodiments provided in the present application, all other embodiments obtained by ordinary technicians in the field without making creative work are within the scope of protection of the present application.

[0053] Obviously, the drawings described below are only some examples or embodiments of the present application. For ordinary technicians in this field, the present application can also be applied to other similar scenarios based on these drawings without creative work. In addition, it can also be understood that although the efforts made in this development process may be complicated and lengthy, for ordinary technicians in this field related to the content disclosed in this application, some changes in design, manufacturing or production based on the technical content disclosed in this application are just conventional technical means, and should not be understood as insufficient content disclosed in this application.

[0054] Unless otherwise specified, all embodiments and optional embodiments of the present application can be combined with each other to form a new technical solution.

[0055] See also Figure 1 , is a schematic diagram of the process of the present invention, comprising the following steps:

[0056] S1-1: Text Semantic Extraction: Semantic extraction is performed using the image encoder and text decoder in the semantic extractor, thereby converting the original image with low semantic density into a corresponding text description with high semantic density;

[0057] S1-2: Semantic Coding and Channel Coding: The text information generated by the semantic extractor is then fed into the semantic encoder; the semantic encoder analyzes the text information and converts it into a compact semantic representation; the semantic encoder consists of a semantic coding layer and a signal-to-noise ratio weighting module; the semantic features generated after encoding are channel coded and modulated by the channel encoder to ensure efficient transmission of semantic information over the physical channel;

[0058] S1-3: Channel decoding and semantic decoding: At the receiving end, the information transmitted through the physical channel is received and decoded using a channel decoder; the channel decoder adopts the opposite structure of the channel encoder; the semantic decoder consists of alternating semantic decoding layers and signal-to-noise ratio weighting modules, aiming to reverse the semantic encoding process to restore the original text information;

[0059] S1-4: Image reconstruction: The text encoder, diffuse U-Net and image decoder of the semantic reconstructor are used to reconstruct the image. The specific steps are as follows: the text information is first converted into a conditional vector by the text encoder; then, the diffuse U-Net converts the noisy image into a latent image feature vector aligned with the conditional vector; finally, the latent image feature vector is processed by the image decoder to generate the final reconstructed image;

[0060] S1-5: Progressive learning assisted by experience replay: During the training phase of this system, the latest samples are stored in a temporary experience replay; when the temporary experience replay becomes full, a clustering method is used to select representative temporary samples to transfer to the fixed experience replay; then, the temporary experience replay is cleared; the experience replay-assisted encoder and decoder continue to sample from the temporary experience replay and the fixed experience replay during the training phase.

[0061] Specifically:

[0062] In order to enhance the semantic density and interpretability of semantic communication, a large model called guided language image pre-training model is used on the sending end to build a semantic extractor; the semantic extractor includes a series of vision and language related knowledge components, including: image encoder, text encoder, text decoder and image decoder; semantic extraction is performed using the image encoder and text decoder in this semantic extractor, thereby converting the original image with low semantic density into the corresponding text description with high semantic density;

[0063] The image encoder contains a feature extraction module based on Vision Transformer (ViT), which splits the input image x into a sequence of patches , and for the patch sequence Encode each patch in , where each patch represents a fixed-size image region; process the vector of each patch through multiple encoder layers with multi-head self-attention sublayers and feed-forward sublayers to generate a textual representation of the image, which corresponds to the input image x feature;

[0064] Then, the patch sequence Input into the image encoder to extract visual features from the image;

[0065] S1-1: Text semantic extraction: Use the image encoder and text decoder in the semantic extractor to perform image semantic extraction;

[0066] The specific workflow of the image encoder is as follows:

[0067] S1-1-1: The multi-head self-attention sublayer captures global and local information in the image. The output of the multi-head self-attention layer is calculated by the following formula:

[0068] ;

[0069] in, is the patch sequence The ith patch in ;

[0070] MSA is a multi-head self-attention operator;

[0071] LN is the layer normalization operator in ViT;

[0072] j is the jth coding layer, , L is the total number of coding layers;

[0073] S1-1-2: In the feedforward sublayer, the vector of each patch is nonlinearly transformed, that is, the global and local information in the image captured in the multi-head self-attention sublayer is nonlinearly transformed as shown in the following formula:

[0074] ;

[0075] in, and are the weights and biases of the feedforward sublayer of the image encoder in the guided language image pre-training model (BLIP), and GeLU represents the activation function;

[0076] S1-1-3: The output of the image encoder is shown in the following formula:

[0077] ;

[0078] in, represents the output of the jth encoder layer;

[0079] The text decoder adopts the BERT structure, which consists of multiple stacked decoder layers. Each decoder layer includes three sublayers: causal self-attention sublayer, cross attention sublayer, and feedforward sublayer.

[0080] The specific workflow of the text decoder is as follows:

[0081] S1-1-4: The causal self-attention sublayer uses a mask matrix to prevent the current tag from accessing the information of future tags. The output of the causal self-attention sublayer is shown in the following formula:

[0082] ;

[0083] Among them, CSA is the causal self-attention operator;

[0084] is the initial mark;

[0085] S1-1-5: Cross-attention sublayer: Cross-attention allows each labeled vector to interact with the feature vector of the input image. The output of the cross-attention sublayer in the first text decoder layer is shown in the following formula:

[0086] ;

[0087] Among them, CA is the cross attention operator;

[0088] S1-1-6: The feedforward layer includes a linear layer and an activation function. The output of the feedforward sublayer of the first text decoder layer is shown in the following formula:

[0089] ;

[0090] in, and They are the weights and biases of the feedforward sublayer in the text decoder of the Guided Language Image Pre-training model (BLIP), and ReLU represents the activation function;

[0091] The last layer of the text decoder transforms the output into the next token in the prediction sequence through linear projection and softmax function; then, in the generation process, each output text will be used as input for the next time step until a corresponding text description with high semantic density is generated for the image.

[0092] S1-2: Semantic Coding and Channel Coding, the semantic encoder consists of a semantic coding layer and a signal-to-noise ratio weighting module; the encoder layer analyzes the text information and converts it into a compact semantic representation; the signal-to-noise ratio weighting module allows the semantic encoder to optimize the encoding process and maintain reliable semantic transmission even in the presence of different channel conditions; the semantic features generated after encoding are channel coded and modulated by the channel encoder to ensure effective semantic information transmission on the physical channel; similarly, the channel encoder also consists of alternating feedforward layers and signal-to-noise ratio weighting modules;

[0093] The signal-to-noise ratio weighting module works as follows:

[0094] S1-2-1: SNR Scaling: First, the SNR projection module scales the SNR value to the same dimension as the feature vector in the encoder and decoder; the SNR projection module is a feed-forward network with a fully connected layer, as shown in the formula:

[0095] ;

[0096] Among them, GeLU represents the activation function, is the signal-to-noise ratio value;

[0097] and They represent the weights and biases of the fully connected layers in the signal-to-noise ratio scaling module respectively;

[0098] S1-2-2: Feature weighting: The input feature is combined with the signal-to-noise ratio vector p to obtain a weighting factor V, which records the importance of each intermediate feature vector to the semantic / channel encoder and semantic / channel decoder, as shown in the following formula:

[0099] ;

[0100] Among them, Tanh is the activation function; f is the feature map of the input feature F after passing through a fully connected layer, as shown in the following formula:

[0101] ;

[0102] and Respectively represent the weight and bias of the fully connected layer in the feature weighting module;

[0103] Finally, the input feature F is multiplied by the weighting factor V to obtain the adjustment vector C, as shown in the following formula:

[0104] ;

[0105] in, represents the i-th element in the adjustment vector C, represents the i-th element in the input feature vector F, represents the i-th element in the weighting factor V;

[0106] The signal-to-noise ratio weighting module is embedded into the feature vectors of the semantic / channel encoder and decoder to enhance the robustness of the system.

[0107] S1-4: In order to better understand the received text information, a semantic reconstructor including a stable diffusion model is designed for image reconstruction. The text encoder, diffusion U-Net and image decoder of the semantic reconstructor are used for image reconstruction. The specific steps performed by the stable diffusion model are as follows:

[0108] S1-4-1: Given a semantic text , the text encoder takes as input the semantic text Converted into a fixed-dimensional semantic vector, the specific process is: first, the semantic text Each word in is mapped to a fixed-length vector through word embedding; these word embeddings are used as input to the text encoder; the encoder iteratively performs multi-head self-attention sublayer and feedforward sublayer operations, and finally produces a sequence of text feature vectors;

[0109] S1-4-2: Feature generator: The initial image feature vector consisting of pure noise is input into the image feature generator to generate a noise feature vector; the sequence consisting of text feature vectors is injected into the noise feature vector to guide noise removal; through multiple iterations, the noise is gradually removed, and an image feature vector containing text information is obtained; the noise removal step adopts the U-Net structure, which adopts an encoder-decoder structure based on convolutional neural networks to retain spatial information while generating image semantic information;

[0110] S1-4-3: Image decoder: Due to the low computational efficiency of the diffusion operation, the image denoising process is performed in a compressed semantic / feature space, i.e., multiple denoising iterations are performed in the reduced semantic / feature space, which improves the image computational efficiency;

[0111] Finally, the decoder of the variational autoencoder is used to map the feature data in the semantic / feature space back to the pixel space to reconstruct an image that is semantically consistent. Since the variational autoencoder has learned the potential structure of the distribution of a large amount of image data, the decoder can provide more detailed information through upsampling and interpolation during the decoding process, which is consistent with the key semantics in the image, thereby enhancing the image quality in the pixel space.

[0112] S1-5: Experience replay-assisted encoders and decoders are introduced for the purpose of experience replay-assisted progressive learning: During the training phase of this system, the latest samples are stored in a temporary experience replay; when the temporary experience replay becomes full, a clustering method is used to select representative temporary samples to transfer to the fixed experience replay; then, the temporary experience replay is cleared; the experience replay-assisted encoders and decoders continue to sample from the temporary experience replay and the fixed experience replay during the training phase;

[0113] The workflow of the experience playback-assisted encoder and decoder is as follows:

[0114] Will and Defined as a set representing the dynamic samples stored in the temporary experience replay and the fixed experience replay;

[0115] in, represents the i-th sample in the temporary experience replay;

[0116] represents the jth sample in the fixed experience replay;

[0117] When the temporary experience replay pool becomes full, representative samples need to be selected from it and transferred to the fixed experience replay;

[0118] The process of selecting a representative sample is as follows:

[0119] Step 1: Short-term typical sample evaluation: During the inference phase, new samples being processed are continuously added to the temporary experience replay; when the number of samples in the temporary experience replay exceeds the specified maximum value, an evaluation operation is performed; the main goal is to evaluate the typicality of temporary samples and use the K-Medoids clustering algorithm to select the most typical k temporary samples ;

[0120] Step 2: Long-term sample selection: The main goal is to select samples from temporary experience replay that are significantly different from samples in fixed experience replay to ensure diversity in experience replay; use Gaussian function to calculate samples and each sample in the fixed experience replay The average similarity score M between them is shown in the following formula:

[0121] ;

[0122] in, is the Gaussian parameter;

[0123] When the calculated similarity score is greater than a given threshold When , the sample is transferred from the temporary experience replay to the fixed experience replay, as shown in the following formula:

[0124] ;

[0125] After the selection is completed, It is cleared to buffer new samples for the next round; then the semantic encoder and semantic decoder are trained through incremental learning using both temporary experience replay and fixed experience replay.

[0126] The following is an embodiment of the present invention:

[0127] The configuration of the experiment in this embodiment is described in detail as follows:

[0128] The experimental training and testing environment involves Windows 2016 servers with Python 3.8, PyTorch 1.8.0, and CUDA 11.6. The computing resources are provided by Intel(R) Xeon(R) Silver 4210R CPU @ 2.40GHz and NVIDIA Tesla T4.

[0129] In order to evaluate the performance of the large model-based image semantic communication system (LAM-ISC) in image classification tasks, it is compared with the convolutional neural network-based image semantic communication system (CSC) and the ViT-based image semantic communication system (VSC); in addition, the bandwidth compression capability of the present system is evaluated by considering the compression ratio between the transmitted data and the original image as the evaluation metric.

[0130] The experimental results are shown in Table 1 below;

[0131]

[0132] Table 1 shows that the image classification accuracy of the present invention is significantly better than that of the CSC image classification task based on the convolutional neural network and the VSC image classification task based on ViT. Secondly, the LAM-ISC transmission data compression rate proposed by the present invention is the highest, which can effectively save the transmission bandwidth. In addition, LAM-ISC exhibits the least number of trainable parameters, thereby achieving the lowest training complexity.

[0133] The above is a very detailed description of one or more embodiments of the present invention, but the content is only a specific example of the present invention and cannot be considered to limit the scope of application of the present invention. Any other methods and changes proposed based on the content of the present invention should fall within the scope of patent protection of the present invention.

Claims

1. An image semantic communication system based on a large model, characterized in that: The following steps are involved: S1-1: Text semantic extraction: Image semantic extraction is performed using the image encoder and text decoder in the semantic extractor, thereby converting the original image with low semantic density into a corresponding text description with high semantic density; S1-2: Semantic Coding and Channel Coding: The text information generated by the semantic extractor is then fed into the semantic encoder; the semantic encoder analyzes the text information and converts it into a compact semantic representation; the semantic encoder consists of a semantic coding layer and a signal-to-noise ratio weighting module; the semantic features generated after encoding are channel coded and modulated by the channel encoder to ensure efficient transmission of semantic information over the physical channel; S1-3: Channel decoding and semantic decoding: At the receiving end, the information transmitted through the physical channel is received and decoded using a channel decoder; the channel decoder adopts the opposite structure of the channel encoder; the semantic decoder consists of a semantic decoding layer and a signal-to-noise ratio weighting module, which aims to reverse the semantic encoding process to restore the original text information; S1-4: Image reconstruction: The text encoder, diffuse U-Net and image decoder of the semantic reconstructor are used to reconstruct the image. The specific steps are as follows: the text information is first converted into a conditional vector by the text encoder; then, the diffuse U-Net converts the noisy image into a latent image feature vector aligned with the conditional vector; finally, the latent image feature vector is processed by the image decoder to generate the final reconstructed image; S1-5: Incremental learning assisted by experience replay: During the training phase of this system, the latest samples are stored in a temporary experience replay; When the temporary experience replay becomes full, a clustering method is used to select representative temporary samples to transfer to the fixed experience replay; then, the temporary experience replay is cleared; the experience replay-assisted encoder and decoder continue to sample from the temporary experience replay and the fixed experience replay during the training phase; The semantic coding, channel coding, semantic decoding and channel decoding in steps S1-2 and S1-3 all use a signal-to-noise ratio weighting module, and the working process of the signal-to-noise ratio weighting module is as follows: S5-1: SNR Scaling: First, the SNR projection module scales the SNR value to the same dimension as the feature vector in the encoder and decoder; the SNR projection module is a fully connected layer feedforward network, as shown in the following formula: ； in, represents the activation function, is the signal-to-noise ratio; and They represent the weights and biases of the fully connected layers in the signal-to-noise ratio scaling module respectively; S5-2: Feature weighting: weighting the input features with the signal-to-noise ratio vector Combined, a weighting factor V is obtained, which records the importance of each intermediate feature vector to the semantic / channel encoder and semantic / channel decoder, as shown in the following formula: ； Among them, Tanh is the activation function; f is the feature map of the input feature F after passing through a fully connected layer, as shown in the following formula: ； and Respectively represent the weight and bias of the fully connected layer in the feature weighting module; Finally, the input feature F is multiplied by the weighting factor V to obtain the adjustment vector C, as shown in the following formula: ； in, represents the i-th element in the adjustment vector C, represents the i-th element in the input feature vector F, represents the i-th element in the weighting factor V; The signal-to-noise ratio weighting module is embedded into the feature vectors of the semantic / channel encoder and decoder to enhance the robustness of the system; In step S1-4, the semantic reconstructor further includes a stable diffusion model for image reconstruction, and the specific steps are as follows: S6-1: Given a semantic text , the text encoder takes as input the semantic text Converted into a fixed-dimensional semantic vector, the specific process is: first, the semantic text Each word in is mapped to a fixed-length vector through word embedding; these word embeddings are used as input to the text encoder; the encoder iteratively performs multi-head self-attention sublayer and feedforward sublayer operations, and finally produces a sequence of text feature vectors; S6-2: Feature generator: The initial image feature vector consisting of pure noise is input into the image feature generator to generate a noise feature vector; the sequence consisting of text feature vectors is injected into the noise feature vector to guide noise removal; through multiple iterations, the noise is gradually removed and an image feature vector containing text information is obtained; the noise removal step adopts the U-Net structure, which adopts an encoder-decoder structure based on convolutional neural networks to retain spatial information while generating image semantic information; S6-3: Image decoder: Due to the low computational efficiency of the diffusion operation, the image denoising process is performed in a compressed semantic / feature space, i.e., multiple denoising iterations are performed in the reduced semantic / feature space, which improves the image computational efficiency; Finally, the decoder of the variational autoencoder is used to map the feature data in the semantic / feature space back to the pixel space to reconstruct an image that is semantically consistent. Since the variational autoencoder has learned the potential structure of the distribution of a large amount of image data, the decoder can provide more detailed information through upsampling and interpolation during the decoding process, which is consistent with the key semantics in the image, thereby enhancing the image quality in the pixel space.

2. The image semantic communication system based on a large model according to claim 1, characterized in that: In S1-1, the semantic extractor includes: a guided language image pre-training model is adopted on the sending end, and the guided language image pre-training model includes: an image encoder, a text encoder, a text decoder and an image decoder.

3. The image semantic communication system based on a large model according to claim 2, characterized in that: The image encoder includes a feature extraction module based on Vision Transformer, which extracts the input image Split into a sequence of patches , and for the patch sequence Each patch in is encoded, and each patch represents a fixed-size image region; the vector of each patch is processed through multiple encoder layers with multi-head self-attention sublayers and feed-forward sublayers to generate a feature vector representation of the image.

4. The image semantic communication system based on a large model according to claim 2, characterized in that: The text decoder adopts the BERT structure, which consists of multiple stacked decoder layers, each of which includes three sublayers: causal self-attention sublayer, cross attention sublayer and feedforward sublayer; the image features are decoded into specific text feature vectors through these three layers; the last layer of the text decoder converts the text feature vector into the next tag in the prediction sequence through linear projection and softmax function; then, in the generation process, each output text will be used as the input of the next time step until the corresponding text description of the image with high semantic density is generated.

5. The image semantic communication system based on a large model according to claim 1, characterized in that: In step S1-5, an encoder and decoder assisted by experience replay are introduced. The learning strategy based on experience replay solves the problem of selective forgetting in incremental learning by diversifying the content of experience replay. An experience replay assisted method with temporary experience replay and fixed experience replay is designed for the encoder and decoder assisted by experience replay. The workflow of the encoder and decoder assisted by experience replay is as follows: Will and Defined as a set representing the dynamic samples stored in the temporary experience replay and the fixed experience replay; in, represents the i-th sample in the temporary experience replay; represents the jth sample in the fixed experience replay; When the temporary experience replay pool becomes full, representative samples need to be selected from it and transferred to the fixed experience replay.

6. The image semantic communication system based on a large model according to claim 5, characterized in that: The process of selecting a representative sample is as follows: S8-1: Short-term typical sample evaluation: During the inference phase, new samples being processed are continuously added to the temporary experience replay; when the number of samples in the temporary experience replay exceeds the specified maximum value, an evaluation operation is performed; the main goal is to evaluate the typicality of temporary samples and use the K-Medoids clustering algorithm to select the most typical k temporary samples ; S8-2: Long-term sample selection: The main goal is to select samples from temporary experience replay that are significantly different from samples in fixed experience replay to ensure diversity in experience replay; use Gaussian function to calculate samples and each sample in the fixed experience replay The average similarity score M between them is shown in the following formula: ； in, is the Gaussian parameter; When the calculated similarity score is greater than a given threshold When , the sample is transferred from the temporary experience replay to the fixed experience replay, as shown in the following formula: ； After the selection is completed, It is cleared to buffer new samples for the next round; then the semantic encoder and semantic decoder are trained through incremental learning using both temporary experience replay and fixed experience replay.

Images