Lightweight generative image semantic communication method and system for packet loss channel

Abstract

The application provides a lightweight generative image semantic communication method and system for a packet loss channel, comprising: using a lightweight encoder at a sending end to encode an image into a semantic bit sequence and packet transmission; at a receiving end, using a preset mark to fill in lost bits, and using a generative decoder to reconstruct the image according to incomplete bit information. The method adopts a lightweight network containing an efficient long-range attention module, and designs a three-stage training process containing erasure-aware weighted generative adversarial training, so that the decoder can complete the missing information with high quality. The application does not need to retransmit, and can obtain a reconstructed image with better perceptual quality in a packet loss scene, and the model is lightweight and efficient, and is suitable for time delay sensitive and resource limited scenes.

Description

Technical Field

[0001] This invention relates to the field of digital communication technology, and more specifically, to a lightweight generative image semantic communication method and system for packet loss channels. Background Technology

[0002] In recent years, with the development of services such as satellite communication, emergency image transmission, remote monitoring, and real-time video, communication systems face higher demands for reliable transmission under conditions of limited bandwidth, limited terminal computing power, and channel fluctuations. Traditional communication systems typically aim to accurately transmit all source bits, but in scenarios of wireless fading, network congestion, or unstable links, complete bit transmission often requires a large amount of redundant coding or retransmission mechanisms, resulting in significant latency and resource overhead, and thus having certain limitations in resource-constrained scenarios.

[0003] Semantic communication, a novel communication paradigm centered on transmitting the meaning of information, leverages neural networks to extract task-related or semantically relevant information, thereby reducing transmission redundancy and improving transmission efficiency in complex channel environments. Existing deep learning joint source-channel coding schemes typically map images directly to continuous semantic features and transmit them end-to-end in the physical channel. However, these schemes often require significant modifications to existing communication protocols and modulation transmission links, making practical deployment challenging.

[0004] To enhance compatibility with existing digital communication systems, bit-level semantic communication methods are gaining increasing attention. These methods further map semantic features into discrete bit sequences, enabling them to be packaged, transmitted, and decoded according to conventional digital communication links. In real-world networks, bit sequences are typically transmitted as data packets, thus inevitably susceptible to packet loss due to network congestion, channel abrupt changes, or sudden interference. Traditional retransmission mechanisms can recover lost data packets, but they significantly increase end-to-end latency, making them unsuitable for latency-sensitive applications such as emergency image transmission and real-time video transmission.

[0005] Existing learning-based image transmission methods for packet loss typically improve robustness by stacking complex neural networks or heavy context models. While these methods can recover lost content to some extent, they generally suffer from long inference times, high model complexity, and high deployment costs, limiting their application on resource-constrained edge devices.

[0006] Therefore, it is necessary to propose an image semantic communication method that takes into account packet loss robustness, perception reconstruction quality, low inference latency, and digital communication compatibility. Summary of the Invention

[0007] To address the shortcomings of existing technologies, the present invention aims to provide a lightweight generative image semantic communication method and system for packet loss channels. It seeks to solve the technical problems in existing packet-level semantic communication systems, which struggle to recover high-quality images from partially received bits while ensuring lightweight models and low inference latency in packet loss scenarios, and are prone to artifacts when supplementing image details.

[0008] According to the present invention, a lightweight generative image semantic communication system for packet loss channels includes: a lightweight BitSem encoder, a packet segmentation module, a packet loss channel, a depacketization module, and a lightweight BitSem decoder at the receiving end. The lightweight BitSem encoder at the transmitting end receives the input raw image and encodes the raw image into a semantic bit sequence; The packet segmentation module adopts an interleaved grouping strategy to evenly and alternately distribute bits at different spatial locations in the semantic bit sequence to multiple different data packets; The data packet is transmitted to the unpacking module through the packet loss channel for unpacking, and the bit gaps caused by the data packet loss are filled with a preset missing marker to form a complete bit sequence; Finally, the lightweight BitSem decoder at the receiving end reconstructs the image based on the complete bit sequence.

[0009] Preferably, it also includes a conditional discriminator used only during the model training phase, which does not participate in the image inference and reconstruction process after training is completed.

[0010] Preferably, the lightweight BitSem encoder is composed of a multi-level convolutional downsampling layer and a high-efficiency long-range attention ELAB module cascaded together; The lightweight BitSem decoder and encoder have a symmetrical structure, including multi-level transposed convolutional upsampling layers and an ELAB module; The ELAB module achieves low-complexity modeling of global semantic dependencies of images through local feature extraction and grouped multi-scale self-attention mechanism.

[0011] Preferably, the conditional discriminator adopts a PatchGAN structure, using the original image / reconstructed image and semantic bit sequence as input; First, the bit sequence is upsampled to the image scale and concatenated with image features. Then, the ELAB module is fused to extract block-level features, and the local authenticity probability map of the image is output.

[0012] According to the present invention, a lightweight generative image semantic communication method for packet loss channels is provided, employing the aforementioned lightweight generative image semantic communication system for packet loss channels. The communication method includes: Step S1: The sending end acquires the original image, inputs it into the trained lightweight BitSem encoder, extracts the semantic features of the image, and converts the continuous semantic features into discrete semantic bit sequences through Gumbel-Softmax differentiable mapping; Step S2: The packet segmentation module adopts an interleaved grouping strategy to evenly distribute bits at different spatial locations in the semantic bit sequence to multiple different data packets; Step S3: The data packet is transmitted through the packet loss channel. The receiving end identifies the index of the lost data packet and records the bit gap position. The retransmission mechanism is not enabled throughout the process. Step S4: The receiving end unpacks and recovers the valid bits, fills all the lost bit positions with learnable missing tags, and generates a complete bit representation that combines valid bits and missing tags; Step S5: Input the complete bit representation into the trained generative lightweight BitSem decoder to reconstruct the complete image based on the valid bits and missing marker semantics.

[0013] Preferably, the generative lightweight BitSem decoder is obtained through a training process that includes generative adversarial training, enabling it to generate semantically consistent image content in bit-gap regions.

[0014] Preferably, the generative adversarial training employs an erasure-aware weighted strategy, which includes: Generate an erase mask based on the bit gap position; A weight matrix is ​​constructed based on the erase mask, and the weight matrix applies a higher weight to the generative adversarial loss corresponding to the bit gap region than to the correct reception region. The generative decoder and the corresponding conditional discriminator are trained using a weighted loss function.

[0015] Preferably, the training process further includes multi-layer feature matching constraints, which improve the structural consistency and semantic authenticity of the reconstructed image by minimizing the feature map differences between the reconstructed image and the original image in multiple intermediate layers of the conditional discriminator.

[0016] Preferably, the training process of the method includes the following three stages: In the first stage, the backbone network of the encoder and decoder is pre-trained in a packet loss-free environment. In the second stage, based on the pre-trained model, a packet loss channel model is introduced to jointly optimize packet loss robustness. In the third stage, the encoder is frozen, and a conditional discriminator is introduced to perform generative adversarial training on the decoder.

[0017] Preferably, the lightweight BitSem encoder is composed of a multi-level convolutional downsampling layer and a high-efficiency long-range attention ELAB module cascaded together; The lightweight BitSem decoder and encoder have a symmetrical structure, including multi-level transposed convolutional upsampling layers and an ELAB module; The ELAB module achieves low-complexity modeling of global semantic dependencies of images through local feature extraction and grouped multi-scale self-attention mechanism.

[0018] Compared with the prior art, the present invention has the following beneficial effects: 1. This invention does not aim to fully recover all source bits, but rather uses semantic decoding and generative reconstruction capabilities to recover image content from incomplete received bits. It can recover images from incomplete bits without retransmission or a large amount of forward error correction redundancy, achieving high packet loss robustness and low latency, and is particularly suitable for image transmission scenarios that are sensitive to latency.

[0019] 2. This invention distributes feature bits evenly across multiple data packets using a packet interleaving method, mitigating the concentrated damage to local semantic content caused by sudden packet loss. The lightweight BitSem encoding / decoding structure, combined with the ELAB module, can capture global semantic dependencies of images with low parameter count and computational complexity, enabling the system to possess both strong semantic representation capabilities and low-latency inference capabilities.

[0020] 3. Regarding reconstruction quality, this invention utilizes a generative model to enhance the texture completion capability of lost regions and distinguishes the different reconstruction needs of correct receiving regions and erased regions through an erasure-aware weighted strategy. This ensures that the correct receiving regions maintain pixel fidelity, while the erased regions prioritize texture generation. Furthermore, multi-layer feature matching constraints can suppress pseudo-details that may be generated during generative adversarial training, thereby making the texture details in the lost regions of the reconstructed image more realistic, with fewer artifacts, and resulting in higher overall visual quality.

[0021] 4. Experimental results show that the present invention exhibits superior perception quality compared to traditional FEC schemes and existing learning-based schemes under high packet loss rate and sudden packet loss scenarios. It also has an advantage of approximately 20 times the inference speed and is suitable for application scenarios that are sensitive to both reliability and latency, such as emergency image transmission, satellite communication, remote monitoring, real-time video transmission, and resource-constrained edge devices. Attached Figure Description

[0022] Other features, objects, and advantages of the present invention will become more apparent from the following detailed description of non-limiting embodiments with reference to the accompanying drawings: Figure 1 This is a block diagram of a lightweight generative image semantic communication system for packet loss channels provided in an embodiment of the present invention; Figure 2 This is a schematic diagram comparing the PSNR performance of the method of this embodiment and the baseline scheme under different packet loss rates; Figure 3 This is a schematic diagram comparing the LPIPS performance of the method of this embodiment and the baseline scheme under different packet loss rates. Figure 4 This is a schematic diagram comparing the FID performance of the method of this embodiment and the baseline scheme under different packet loss rates. Detailed Implementation

[0023] The present invention will now be described in detail with reference to specific embodiments. These embodiments will help those skilled in the art to further understand the present invention, but do not limit the invention in any way. It should be noted that those skilled in the art can make several changes and improvements without departing from the concept of the present invention. These all fall within the protection scope of the present invention.

[0024] Example 1 According to the present invention, a lightweight generative image semantic communication method and system for packet loss channels are provided, such as... Figure 1 As shown, the system includes: a lightweight BitSem encoder, a packet segmentation module, a packet loss channel, a depacketization module, a lightweight BitSem decoder at the receiving end, and a conditional discriminator used during the training phase.

[0025] Specifically, the workflow of this communication method and system includes two parts: an inference process and a training process. The inference process describes the end-to-end image transmission process after training is completed, while the training process describes how the model parameters are obtained through three-stage training.

[0026] The first part, the reasoning process, can be divided into the sending process, the channel transmission process, and the receiving reconstruction process, including the following: Step S1: The sending end acquires the image to be transmitted and inputs it into the trained lightweight BitSem encoder. This encoder extracts the image features, maps the original image from pixel space to semantic feature space, and uses learnable bitmapper blocks to convert continuous semantic features into discrete semantic bit sequences. This bit sequence can be transmitted according to the packet-level transmission method of existing digital communication systems, thereby improving the compatibility of the semantic communication system with existing communication links.

[0027] Specifically, the encoder is a deep neural network-based module used to map high-dimensional pixel data to a compact low-dimensional representation containing rich semantic information. In this embodiment, the lightweight BitSem encoder consists of multiple cascaded convolutional downsampling layers and an efficient long-range attention module (ELAB). The convolutional downsampling layers are responsible for progressively reducing the spatial size of the feature maps while increasing the number of channels, thereby extracting hierarchical image features; while the efficient long-range attention module is used to efficiently capture the global semantic dependencies of the image. The encoder first encodes the input image into a set of semantic feature maps with continuous values, and further includes a learnable bit mapper for compatibility with digital communication systems. This mapper employs a Gumbel-Softmax reparameterization technique to convert the continuous semantic feature maps into a binary sequence of semantic bits. It is understood that this process is differentiable during training, allowing end-to-end optimization of the entire encoder via gradient descent.

[0028] Step S2: The generated semantic bit sequence is passed to the packet segmentation module, which segments and encapsulates the long bit stream into multiple data packets. To effectively combat the common burst packet loss in networks, the packet segmentation module employs an interleaved grouping strategy. Specifically, this strategy does not simply segment the bit stream sequentially, but rather distributes feature bits from different spatial locations in the semantic bit sequence evenly and interleavedly into N different data packets. The advantage of this strategy is that when a burst packet loss occurs in the network, resulting in the loss of several consecutive data packets, the lost bits are dispersed in the original semantic feature map, rather than concentrated in one or a few spatially adjacent regions. This reduces the destructive impact of burst packet loss on key structural regions of the image, providing a better foundation for reconstruction at the receiving end.

[0029] Step S3: Multiple data packets are transmitted through a packet loss channel. A packet loss channel is a simulation of a real-world network environment, such as a wireless communication channel or a congested internet. In this channel, some data packets may be lost due to various unpredictable factors. The method proposed in this invention aims to avoid the additional latency caused by retransmissions by not relying on traditional retransmission mechanisms such as Automatic Repeat Request (ARQ). During transmission, some data packets may be randomly or continuously lost due to factors such as network congestion, link fluctuations, wireless fading, or sudden interference. The receiving end detects the location of the lost data packets and records it accordingly.

[0030] Step S4: The receiving end unpacks the successfully received data packets and replaces the bit positions corresponding to lost data packets with learnable missing data markers, forming a bit representation containing received bits and missing data markers. Specifically, the unpacking module identifies successfully arrived data packets and lost data packets based on information such as the sequence number of the data packets. It unpacks the successfully received data packets and restores the bit information carried within them to their corresponding positions in the original semantic bit sequence according to predetermined interleaving rules.

[0031] For bit gaps caused by packet loss, the system does not simply fill them with 0s or 1s, but uses a pre-defined, learnable missing bit marker. This missing bit marker is an optimizable parameter whose value is learned along with the codec during training. By using a specific marker to fill all lost bits, the system can explicitly indicate to subsequent decoders which information is reliable (received bits) and which information is completely missing (bits filled with the missing bit marker). In this way, all received bits and the filled missing bit marker together constitute a complete sequence of received bits.

[0032] Step S5: The receiving end inputs the aforementioned bit representations into the trained lightweight BitSem decoder. The decoder performs semantic-aware reconstruction based on the received bits and missing markers, directly recovering the image. This decoder is the core reconstruction unit of the entire system, and it is also a generative module based on a deep neural network. Its network structure is designed to be symmetrical with the lightweight BitSem encoder, mainly consisting of multiple transposed convolutional upsampling layers and an efficient long-range attention module (ELAB). The transposed convolutional layers are responsible for progressively enlarging the spatial size of the feature maps to recover the details and resolution of the image, while the ELAB module continues to utilize the long-range dependencies between features during the decoding process to ensure the global consistency of the generated content. The unique feature of the lightweight BitSem decoder is that it is trained as a powerful generative model, capable not only of recovering the corresponding image region based on the received valid bits, but more importantly, of generating semantically reasonable and visually natural image content on the regions marked by missing markers based on the surrounding received semantic information. Finally, the decoder outputs a complete, high-quality reconstructed image.

[0033] To overcome the degradation in reconstruction quality caused by packet loss, this invention employs a generative adversarial training method during the training phase. Specifically, a conditional discriminator is introduced during training to constrain the semantic similarity between the original and reconstructed images, guiding the decoder to synthesize semantically consistent texture details in the erased regions. Since the conditional discriminator is only used to optimize decoder parameters during the training phase and does not participate in the aforementioned inference and deployment process, this invention can improve the perceptual reconstruction quality in packet loss scenarios without increasing the computational burden during the actual deployment phase.

[0034] The second part, the training process, is as follows: Training Step 1: Backbone Network Pre-training. First, a lightweight BitSem codec is trained in a packet loss-free environment. This enables the encoder to extract effective image semantic features and gives the decoder stable image reconstruction capabilities. This stage primarily uses pixel-level reconstruction error as the optimization objective, providing an initial model for subsequent packet loss-robust training.

[0035] Training Step 2: Packet Loss Robustness Optimization. Based on the pre-trained model, a packet loss channel model is introduced for end-to-end joint optimization. Since the packet segmentation process in this invention acts as an interleaver, it can evenly distribute feature bits across multiple data packets. Therefore, during the training phase, the erasure of a data packet can be statistically equivalent to random bit erasure on the feature map. Based on this equivalence, the erasure ratio is randomly sampled in each training iteration, and a portion of the bits in the bit sequence is randomly masked according to this ratio. The receiver replaces the masked bit positions with learnable missing data markers, and the decoder then reconstructs the image. Through this training phase, the codec and learnable missing data markers are jointly optimized, enabling the model to recover the image from a portion of the received semantic bits.

[0036] Training Step 3: Generative Adversarial Training. Building upon the joint optimization of packet loss robustness, a conditional discriminator is introduced to distinguish between real and reconstructed images. This is combined with an erasure-aware weighted strategy, ensuring the model prioritizes semantically consistent texture generation in lost regions and pixel fidelity in correctly received regions. Simultaneously, multi-layer feature matching constraints are introduced to suppress pseudo-details that may arise during adversarial training. After this training phase, the encoder, learnable missing data markers, and decoder used for inference deployment are retained. The conditional discriminator serves only as a training aid and does not participate in actual transmission or receiver reconstruction.

[0037] Through the aforementioned reasoning process and training algorithm, this invention establishes packet loss robustness and generative perceptual reconstruction capabilities during the training phase. During the inference phase, it can recover semantically consistent, visually high-quality images from partially received data packets without retransmission or additional complex error correction decoding. The following section uses an image transmission task as an example to detail the network structure, variable definitions, training objectives, and experimental verification process of this invention.

[0038] Example 2 This embodiment takes image transmission as an example to specifically describe the lightweight generative image semantic communication method and system for packet loss channels of the present invention as follows: The input image is denoted as ,in, , and These represent the image height, width, and number of channels, respectively. The sending end first utilizes the lightweight BitSem encoder. Extract semantic features from the image and map them into a binary feature map. The corresponding bit rate is .

[0039] Subsequently, the packet segmentation module evenly distributes the feature bits to In a data packet. This packet splitting process can be regarded as an interleaving operation, which evenly distributes bits from different spatial locations into different data packets, thereby reducing the concentrated damage to local semantic regions caused by the loss of consecutive packets.

[0040] Data packets are transmitted through a packet loss channel. Due to network congestion, channel fluctuations, or sudden fading, some data packets may be randomly or suddenly lost. The receiver can detect the index of the lost data packets. After unpacking, the receiver replaces the bit positions corresponding to the lost data packets with learnable markers. The received bit sequence is then input into the BitSem decoder. Directly reconstruct the image :

[0041] in, This represents the received bit sequence after packet erasure and missing marker completion.

[0042] This invention employs the efficient long-range attention module ELAB to construct a lightweight BitSem encoder and decoder. ELAB includes a local feature extraction module and a grouped multi-scale self-attention module, which can capture the global semantic dependencies of an image with low parameter and computational cost by utilizing shifted convolution and accelerated self-attention mechanisms. The encoder consists of multiple convolutional downsampling layers and the ELAB module cascaded together, compressing the input image into a compact semantic representation step by step; the decoder adopts a symmetrical structure, restoring the image resolution step by step through transposed convolution and the ELAB module. To adapt to end-to-end digital transmission, the continuous features output by the encoder are converted into discrete bit sequences through a learnable bit mapper. Specifically, the bit mapping module uses a three-dimensional convolutional layer to generate a probability distribution of bit values ​​of 0 or 1 for each position, and uses the Gumbel-Softmax method for differentiability sampling, thereby maintaining backpropagation feasibility during the training phase and generating discrete bit sequences during the inference phase.

[0043] During the training phase, this invention introduces a conditional generative adversarial framework. (Conditional discriminator) To reconstruct images and corresponding bit sequence The input is used to determine whether the reconstructed image has semantic consistency with the real image at the local image patch level. The discriminator adopts the PatchGAN structure, which first aligns the bit sequence to the image scale through convolution and upsampling, then concatenates it with image features, and outputs a patch-level true / false probability map after processing by convolutional layers, normalization layers and ELAB modules.

[0044] The training process of this invention includes three stages: The first stage involves pre-training the backbone network. The BitSem encoder and decoder are jointly trained in a packet loss-free environment, using pixel mean square error as the objective function, enabling the lightweight backbone network to possess stable image compression and reconstruction capabilities.

[0045] The second stage involves joint optimization of packet loss robustness. A packet loss channel model is introduced based on the pre-trained model. Since the packet segmentation process of this invention distributes bits evenly across different data packets, packet loss can be equivalent to random bit erasure on the feature map. In each round of training, the packet loss ratio is randomly sampled. Then, corresponding erasure operations are performed on the bit sequence to optimize the robustness of the codec against packet loss. The objective function of this stage can be expressed as:

[0046] in, These represent the parameters of the BitSem codec. This represents the erase operator determined by the packet loss mode. for Norm.

[0047] The third stage is generative adversarial training. In this stage, the encoder is frozen, and the decoder and conditional discriminator are jointly trained, allowing the decoder to further shift from pixel-level regression to perceptual quality optimization. The discriminator learns to distinguish between real images and reconstructed images recovered from damaged bit sequences, while the decoder generates semantically consistent texture details while maintaining pixel reconstruction accuracy.

[0048] Specifically, in each training iteration, the conditional discriminator semantic bit sequence As conditional inputs, the original images are received respectively. and reconstructed images And determine the probability that it is a real sample. In standard conditional adversarial training, the discriminator loss... It can be represented as:

[0049] When updating the generative decoder, the decoder needs to generate a reconstructed image that can fool the discriminator, with an adversarial loss term. It can be represented as:

[0050] Meanwhile, to avoid the generated image deviating from the original image, this invention retains pixel-level reconstruction constraints in the standard generator target. Therefore, the generator decoder loss... It can be represented as:

[0051] in, This represents the trade-off coefficient between pixel fidelity and perceptual generation quality. The standard adversarial loss described above applies a uniform weight to the entire image, failing to differentiate between the different reconstruction needs of correctly received regions and regions with packet loss. Since correctly received regions require more attention to maintaining pixel consistency, while erased regions require more attention to restoring texture details, this invention further proposes the following erase-perceptual weighted strategy and introduces multi-layer feature matching constraints.

[0052] (1) Erasure perception weighting strategy To address the issue that lost and correctly received regions have different reconstruction requirements, this invention proposes an erasure-aware weighted strategy. For correctly received regions, pixel fidelity should be prioritized; for erased regions, due to more severe information loss, semantically consistent texture generation should be given greater emphasis. A binary erasure mask is defined as follows:

[0053] in, This indicates that a bit has been lost at the corresponding position in the feature map. This indicates that the signal was received correctly at that location. Therefore, the weight matrix is ​​constructed as follows:

[0054] in, This represents the enhancement strength of the loss generation for the lost region. Based on this weight matrix, the erasure-aware generation loss of this invention... It can be represented as:

[0055] in, To generate adversarial loss terms, This indicates element-wise multiplication.

[0056] Through the above design, the system can enhance the generative texture recovery capability in the lost region and maintain strong pixel consistency in the correct receiving region, thereby avoiding the problems of over-generation in the receiving region or over-smoothing in the lost region caused by the uniform weight strategy.

[0057] (2) Multi-layer feature matching constraints Relying solely on generative adversarial loss may introduce spurious textures. To further improve semantic consistency and suppress generative artifacts, this invention introduces multi-layer feature matching constraints. Specifically, feature maps of the real and reconstructed images are extracted from multiple intermediate layers of the discriminator, and both are constrained to maintain consistency in the feature space.

[0058] in, Indicates the discriminator's first... Intermediate feature maps output by the layer. This indicates the number of feature elements in this layer. Indicates the total number of floors. for Norm.

[0059] Finally, the training objective of the generator decoder can be expressed as:

[0060] in, The feature matching loss weights are used. The discriminator loss also incorporates erase-aware weights, causing the discriminator to focus on the generation quality of the lost regions, which can be expressed as:

[0061] Example 3 This embodiment provides experimental verification results of the method proposed in this application to demonstrate its beneficial effects compared to existing technologies. To verify the effectiveness of the invention, this embodiment performs training and testing on the DIV2K image dataset. The dataset is divided into training, validation, and test sets in an 8:1:1 ratio. Random cropping is performed during the training phase. Image patches were cropped from the center during the testing phase. The number of data packets was set to N=16, and the bit rate was 0.5 bpp. All experiments were conducted on an Intel Xeon Silver 4214R CPU and an Nvidia GeForce RTX 4090Ti graphics card with 48 GB of video memory.

[0062] This invention is compared with two baseline methods: the first is the Lightweight ResiComp method, which enhances the packet loss robustness of neural network codecs through Masked Visual Markup Modeling (MTMV); the second is a method that combines image compression algorithms with ideal packet-level forward error correction codes, including combinations of BPG, JPEG2000, and JPEG AI with different redundancy ratios of FEC.

[0063] The experiment used PSNR, LPIPS, and FID as evaluation metrics. PSNR measures pixel-level fidelity, LPIPS measures perceptual similarity in feature space, and FID measures the distance between the generated image distribution and the real image distribution.

[0064] Figures 2 to 4 The illustration schematically demonstrates the performance comparison between the method of this invention and the baseline scheme at different packet loss rates in an independent packet loss scenario.

[0065] In scenarios with independent and identically distributed packet loss, this invention demonstrates stronger robustness under high packet loss rates. When the packet loss rate reaches 0.875, this invention improves PSNR by approximately 1.8 dB compared to the method combining BPG and JPEG AI with ideal FEC. In perceptual quality metrics such as LPIPS and FID, this invention outperforms the comparison scheme at all packet loss rates. Introducing generative adversarial training significantly improves image perceptual quality; further introducing a perceptually weighted erasure strategy reduces LPIPS and FID by approximately 0.03 and 4, respectively, with minimal impact on PSNR.

[0066] In burst packet loss scenarios, this invention employs a three-state Markov model to simulate relevant packet loss patterns in real-world networks. Experimental results show that this invention achieves superior perception reconstruction quality in both moderate and severe burst packet loss scenarios. For example, in a scenario with an average packet loss rate of 13.8%, this invention achieves an LPIPS of 0.1702 and an FID of 129.37, outperforming BPG+FEC and lightweight ResiComp. In a severe burst packet loss scenario with an average packet loss rate of 32.3%, this invention simultaneously achieves higher PSNR and superior perception quality.

[0067] In terms of computational complexity, the model of this invention has 6.15M parameters, 18.06G FLOPs, and encoding and decoding inference times of approximately 11.2 ms and 11.3 ms, respectively. Compared to the lightweight ResiComp, this invention has lower parameter count and computational complexity, and its inference speed can be improved by approximately 20 times, making it suitable for deployment in latency-sensitive and computationally limited edge communication scenarios.

[0068] Those skilled in the art will understand that, besides implementing the system and its various devices, modules, and units provided by this invention in the form of purely computer-readable program code, the same functions can be achieved entirely through logical programming of the method steps, making the system and its various devices, modules, and units of this invention function in the form of logic gates, switches, application-specific integrated circuits, programmable logic controllers, and embedded microcontrollers. Therefore, the system and its various devices, modules, and units provided by this invention can be considered as a hardware component, and the devices, modules, and units included therein for implementing various functions can also be considered as structures within the hardware component; alternatively, the devices, modules, and units for implementing various functions can be considered as both software modules implementing the method and structures within the hardware component.

[0069] Specific embodiments of the present invention have been described above. It should be understood that the present invention is not limited to the specific embodiments described above, and those skilled in the art can make various changes or modifications within the scope of the claims, which do not affect the essence of the present invention. Unless otherwise specified, the embodiments and features of the present invention can be arbitrarily combined with each other.

Claims

1. A lightweight generative image semantic communication system for packet loss channels, characterized in that, include: The transmitter includes a lightweight BitSem encoder, packet segmentation module, packet loss channel, and depacketization module, while the receiver includes a lightweight BitSem decoder. The lightweight BitSem encoder at the transmitting end receives the input raw image and encodes the raw image into a semantic bit sequence; The packet segmentation module adopts an interleaved grouping strategy to evenly and alternately distribute bits at different spatial locations in the semantic bit sequence to multiple different data packets; The data packet is transmitted to the unpacking module through the packet loss channel for unpacking, and the bit gaps caused by the data packet loss are filled with a preset missing marker to form a complete bit sequence; Finally, the lightweight BitSem decoder at the receiving end reconstructs the image based on the complete bit sequence.

2. The lightweight generative image semantic communication system for packet loss channels according to claim 1, characterized in that, It also includes a conditional discriminator used only during the model training phase, which does not participate in the image inference and reconstruction process after training.

3. The lightweight generative image semantic communication system for packet loss channels according to claim 1, characterized in that, The lightweight BitSem encoder consists of a cascaded multi-level convolutional downsampling layer and a high-efficiency long-range attention ELAB module; The lightweight BitSem decoder and encoder have a symmetrical structure, including multi-level transposed convolutional upsampling layers and an ELAB module; The ELAB module achieves low-complexity modeling of global semantic dependencies of images through local feature extraction and grouped multi-scale self-attention mechanism.

4. The lightweight generative image semantic communication system for packet loss channels according to claim 2, characterized in that, The conditional discriminator adopts the PatchGAN structure, taking the original image / reconstructed image and semantic bit sequence as input; First, the bit sequence is upsampled to the image scale and concatenated with image features. Then, the ELAB module is fused to extract block-level features, and the local authenticity probability map of the image is output.

5. A lightweight generative image semantic communication method for packet loss channels, characterized in that, The lightweight generative image semantic communication system for packet loss channels as described in any one of claims 1 to 4 includes a communication method comprising: Step S1: The sending end acquires the original image, inputs it into the trained lightweight BitSem encoder, extracts the semantic features of the image, and converts the continuous semantic features into discrete semantic bit sequences through Gumbel-Softmax differentiable mapping; Step S2: The packet segmentation module adopts an interleaved grouping strategy to evenly distribute bits at different spatial locations in the semantic bit sequence to multiple different data packets; Step S3: The data packet is transmitted through the packet loss channel. The receiving end identifies the index of the lost data packet and records the bit gap position. The retransmission mechanism is not enabled throughout the process. Step S4: The receiving end unpacks and recovers the valid bits, fills all the lost bit positions with learnable missing tags, and generates a complete bit representation that combines valid bits and missing tags; Step S5: Input the complete bit representation into the trained generative lightweight BitSem decoder to reconstruct the complete image based on the valid bits and missing marker semantics.

6. The lightweight generative image semantic communication method for packet loss channels according to claim 5, characterized in that, The generative lightweight BitSem decoder is obtained through a training process that includes generative adversarial training, enabling it to generate semantically consistent image content in bit-gap regions.

7. The lightweight generative image semantic communication method for packet loss channels according to claim 6, characterized in that, The generative adversarial training employs an erasure-aware weighted strategy, which includes: Generate an erase mask based on the bit gap position; A weight matrix is ​​constructed based on the erase mask, and the weight matrix applies a higher weight to the generative adversarial loss corresponding to the bit gap region than to the correct reception region. The generative decoder and the corresponding conditional discriminator are trained using a weighted loss function.

8. The lightweight generative image semantic communication method for packet loss channels according to claim 6, characterized in that, The training process also includes multi-layer feature matching constraints, which improve the structural consistency and semantic authenticity of the reconstructed image by minimizing the difference between the feature maps of the reconstructed image and the original image in multiple intermediate layers of the conditional discriminator.

9. The lightweight generative image semantic communication method for packet loss channels according to claim 5, characterized in that, The training process of the method includes the following three stages: In the first stage, the backbone network of the encoder and decoder is pre-trained in a packet loss-free environment. In the second stage, based on the pre-trained model, a packet loss channel model is introduced to jointly optimize packet loss robustness. In the third stage, the encoder is frozen, and a conditional discriminator is introduced to perform generative adversarial training on the decoder.

10. The lightweight generative image semantic communication method for packet loss channels according to claim 5, characterized in that, The lightweight BitSem encoder consists of a cascaded multi-level convolutional downsampling layer and a high-efficiency long-range attention ELAB module; The lightweight BitSem decoder and encoder have a symmetrical structure, including multi-level transposed convolutional upsampling layers and an ELAB module; The ELAB module achieves low-complexity modeling of global semantic dependencies of images through local feature extraction and grouped multi-scale self-attention mechanism.

Images