Image semantic communication method based on residual error and adaptive coding

Abstract

The invention belongs to the technical field of semantic communication, and particularly relates to an image semantic communication method based on residual errors and adaptive coding. The method comprises the steps of collaborative design of a sending end and a receiving end, differential processing of residual errors according to semantic features of images, dynamic adjustment of coding priorities of different parts of the residual errors, and optimization of a decoding process to resist channel noise interference; the method specifically comprises the following steps: extracting a residual error at a sending end, screening the residual error, adaptively encoding the residual error into a bit stream, and reconstructing an image at a receiving end; through innovative means such as a residual error screening algorithm, a semantic-based residual error self-adaptive coding and decoding algorithm and an image generation network integrated with a Transform architecture discriminator, the robustness of the system is effectively enhanced, the anti-noise interference capability and the semantic information retention capability are improved, and the method has an efficient compression characteristic, is suitable for various image data sets, and has a wide application prospect. And the compatibility with the existing communication system is good.

Description

Technical Field

[0001] This invention belongs to the field of semantic communication technology, specifically relating to an image semantic communication method based on residual and adaptive coding, which is used to improve the noise resistance and semantic information preservation capabilities of image semantic communication systems. Background Technology

[0002] In today's information-saturated era, image data is experiencing explosive growth. Traditional image communication systems have long relied on conventional image compression coding methods such as BPG and JPEG XL. These methods focus on achieving perfect reconstruction at the pixel level, but neglect the understanding and adaptation of the image's semantic content. To meet users' needs for image semantic content, researchers have proposed image semantic communication systems based on deep semantic segmentation and those based on entropy models, providing solutions for preserving the semantic information of images. However, most current simulation experiments of image semantic communication systems are conducted under ideal noise-free conditions, failing to consider the impact of the channel during actual communication, resulting in poor robustness. For example, under low signal-to-noise ratio conditions, these image semantic communication systems may experience a series of problems such as the "cliff effect" or poor reconstruction quality at the receiving end.

[0003] To address the problems existing in the aforementioned semantic communication systems, this invention designs an image semantic communication method based on residual and adaptive coding. This method introduces a series of innovative components, including a residual filtering algorithm, a semantic-based residual adaptive coding and decoding algorithm, a receiver-side image restoration algorithm, and an image generation network integrating a Transformer architecture discriminator. This not only improves the semantic information preservation capability of the image semantic communication system but also endows it with excellent noise resistance, better meeting current needs for semantic communication systems. Summary of the Invention

[0004] To address the problems of existing image semantic communication methods, such as the "cliff effect" and poor image quality at the receiving end in low signal-to-noise ratio environments, this invention aims to provide an image semantic communication method based on residual and adaptive coding. By using semantically guided residual differentiation processing and an autoregressive adaptive entropy encoding and decoding mechanism, the method improves the semantic information preservation capability and noise resistance performance in low signal-to-noise ratio scenarios, while achieving efficient bandwidth utilization.

[0005] The proposed image semantic communication method based on residual and adaptive coding includes a collaborative design between the transmitter and receiver. Its core is to differentiate the residual based on the semantic features of the image, dynamically adjust the coding priority of different parts of the residual, and optimize the decoding process to resist channel noise interference. Specifically, it includes: residual extraction at the transmitter, residual filtering processing, adaptive encoding of the residual into a bitstream, and image reconstruction at the receiver. This involves a semantic segmentation network, a residual filtering algorithm, an image generation network with an integrated Transformer architecture discriminator trained by a loss function, a semantic-based adaptive residual coding algorithm at the transmitter trained by a loss function, a semantic-based adaptive residual decoding algorithm at the receiver, and an image restoration algorithm. The specific steps are as follows:

[0006] Step 1: Extracting residuals from the sending end

[0007] First, a semantic segmentation network is used to perform semantic segmentation on the input image to obtain a semantic map of the input image. Then, the semantic map is input into an image generation network with a Transformer architecture discriminator to generate a coarse reconstruction map. Finally, the residual is obtained by subtracting the input image from the coarse reconstruction map. At the same time, the semantic map is encoded into a bitstream for transmission.

[0008] Step 2: Residual screening process

[0009] Using the residual screening algorithm proposed in this invention (see...) Figure 2 (As shown) The residuals obtained in step 1 are processed; the residual filtering algorithm includes a category range selection module and a selective zeroing module based on semantics and block ideas; the category range selection module filters out the residuals corresponding to the categories of objects with large proportions respectively; the selective zeroing module performs zeroing processing on the residuals corresponding to the large proportion categories, increasing the proportion of zero pixel values ​​in the entire residual, thereby helping to achieve a higher compression rate;

[0010] Step 3: Adaptively encode the residual into a bitstream

[0011] First, an encoder consisting of convolution and normalization operations processes the residuals after step 2, transforming them into continuous vectors (i.e., latent representations). This mapping transforms the residuals from pixel space to latent space, preserving key information while reducing redundancy. Second, discrete latent representations of the residuals are generated through discrete quantization, providing a compact data form for subsequent probability distribution estimation and entropy coding. Next, the semantic graph undergoes importance processing to generate an importance matrix. Similarly, an encoder consisting of convolution and normalization operations transforms the importance matrix and the coarsely reconstructed graph into continuous vectors (i.e., local importance semantic features and global semantic features), which are then concatenated to form local-global semantic features. Finally, guided by these semantic features, the semantic-based adaptive residual coding algorithm proposed in this invention (see [link to original text]) is applied. Figure 3 The local-global entropy model (shown) adaptively estimates the probability distribution of the discrete latent representation of the residual; finally, based on the estimated probability distribution, the discrete latent representation of the residual and its shape parameters are encoded into a bit stream by arithmetic coding.

[0012] Step 4: Image reconstruction at the receiving end

[0013] The receiver first recovers the semantic map and shape parameters of the residual discrete latent representation from the bitstream, generates a coarse reconstructed map using an image generation network with parameters shared with the transmitter, and extracts local-global semantic features from the semantic map and the coarse reconstructed map. Then, it utilizes the semantic-based residual adaptive decoding algorithm proposed in this invention (see...). Figure 4 The autoregressive properties of the local-global entropy model (shown) are used to reconstruct the discrete latent representation of the residuals; then, the discrete latent representation of the residuals is decoded by a decoder consisting of upsampling, convolution and normalization operations to recover the residuals; finally, the residuals are combined with the coarse reconstruction map and restored to the original size by an image restoration algorithm [1].

[0014] Furthermore, in step 1:

[0015] The semantic segmentation network is a pre-trained PSPNet[2].

[0016] The image generation network is based on LGGAN[3], with the addition of a newly designed image generation network discriminator based on the Transformer architecture (see [link]). Figure 5 (As shown). This network discriminator divides the image into several small blocks, and then uses multiple layers of Transformers to handle the dependencies between each block, thereby effectively capturing global information of the image. This method not only improves the efficiency of feature extraction, but also enhances the discriminator's ability to understand the global structure of the image, thus improving the quality of images generated by the image generation network.

[0017] There are two types of semantic graph transmission methods, both of which use LDPC for channel transmission. Under low signal-to-noise ratio conditions, a retrained VQ-VAE model [4] is used to perform encoding at the sending end and decoding at the receiving end; while under high signal-to-noise ratio conditions, the semantic graph is first preprocessed with run-length encoding, and then the data is compressed by using the Zlib method [5], thereby achieving high-quality transmission of the semantic graph.

[0018] Furthermore, in step 2:

[0019] The residual filtering algorithm consists of a category range selection module based on semantics and block division, and a selective zeroing module. The category range selection module dynamically divides objects into large-proportion and small-proportion categories based on semantic information and block division strategies, and filters out the corresponding residuals. Specifically, it: (1) sets a category threshold β and statistically analyzes the proportion of different categories in the semantic graph; (2) accumulates the proportions of these categories in ascending order, stopping the accumulation when the accumulated value exceeds the preset category threshold; (3) filters the categories that participated in the accumulation process as small-proportion categories, and classifies the remaining categories as large-proportion categories. Since objects in an image are closely adjacent, processing the residuals corresponding to large-proportion categories may cause significant changes in the pixel values ​​around small-proportion categories. These pixel value changes may lead to a significant deviation in the semantic performance of the finally recovered image when measuring small-proportion category objects. Therefore, this rigid method of determining the range of small-proportion category objects should not be used; instead, the definition of the range of small-proportion category objects should be appropriately broadened. The semantic map is divided into several small blocks, and blocks containing categories with small percentages are all classified as such. The selective zeroing module then selectively ignores some information from the residuals corresponding to categories with large percentages. This involves setting a zeroing threshold and setting values ​​in the residuals whose absolute values ​​are less than the threshold to zero, thereby increasing the proportion of zero-pixel values ​​in the residuals. This residual filtering algorithm significantly improves the compression efficiency of the residuals while maintaining the same compression ratio, better preserves residual information from categories with small percentages, and ultimately enhances the semantic performance of the image reconstructed by the receiving end.

[0020] Further:

[0021] The semantic graph importance processing algorithm proceeds as follows: First, the same-category range selection module filters out objects of small percentage categories from the semantic graph. Then, it initializes a zero-based matrix of the same size as the semantic graph and divides the semantic graph into 16×16 blocks. Iterating through all blocks, if a block contains pixels of objects of a small percentage category, the corresponding element in the matrix is ​​changed to the category number of that small percentage category (the category number does not contain 0). If a block contains pixels of multiple small percentage categories, it is uniformly labeled as the category with the smallest percentage. Next, the pixel percentage of each small percentage category in the matrix is ​​calculated, sorted in ascending order of percentage, and then weighted inverted (the smaller the percentage, the higher the weight coefficient). Finally, the category numbers in the matrix are replaced with the weight coefficients to generate the importance matrix.

[0022] Furthermore, in step 3:

[0023] The aforementioned semantic-based residual adaptive coding algorithm at the transmitting end takes a semantic map, a coarsely reconstructed map, and residuals processed by a residual filtering algorithm as input, and outputs a bitstream. The core of this algorithm lies in extracting local-global semantic features from the semantic map and the coarsely reconstructed map, using these features to guide entropy coding to map data into a bitstream. Without the guidance of local-global semantic features, the bit consumption of entropy coding is directly related to the probability distribution of the data: data values ​​with high frequency are allocated fewer bits, while data values ​​with low frequency are allocated more bits. When local-global semantic features are introduced as guidance, the goal of the entropy encoder changes from "allocating bits based on data statistics" to "allocating bits based on local-global semantic features." Its core logic is that the Local-Global Entropy Model (LGEM) in the algorithm dynamically adjusts the estimation of the residual probability distribution based on local-global semantic features, thereby allocating more bits to residuals corresponding to small percentage categories and fewer bits to residuals corresponding to large percentage categories, prioritizing the semantic information of objects in small percentage categories.

[0024] Furthermore, in step 4:

[0025] The receiver-side semantic-based adaptive residual decoding algorithm takes a semantic map, a coarse reconstruction map, and a bitstream of the residual discrete latent representation as input, and outputs the reconstructed residual. The core of this algorithm lies in decoding the discrete latent representation of the residual from the bitstream using an autoregressive approach. Specifically: First, the receiver decodes the semantic map from the bitstream and performs a coarse reconstruction using an image generation network; then, it extracts local-global semantic features from the semantic map and the coarse reconstruction map; next, guided by these semantic features, the Local-Global Entropy Model (LGEM) reconstructs the probability distribution of the residual discrete latent representation position-by-position using an autoregressive method, while simultaneously using arithmetic decoding to obtain the residual discrete latent representation at the corresponding position, avoiding noise sensitivity issues caused by directly transmitting probability distribution parameters and enhancing anti-interference capabilities; finally, a decoder composed of upsampling, convolution, and normalization operations decodes the discrete latent representation of the residual, achieving residual reconstruction.

[0026] Furthermore:

[0027] The Local-Global Entropy Model (LGEM) consists of a context model based on a masked attention mechanism and a distribution estimation module (see [link]). Figure 8 (As shown). The modules in the context model adopt the Transformer architecture, which captures the long-distance dependencies of the discrete latent representations of the residuals through masked self-attention; the distribution estimation module combines local-global semantic features to dynamically adjust the probability distribution estimation, allocating more bits to the residuals of small-proportion classes.

[0028] Furthermore:

[0029] The loss function employs rate-distortion joint optimization; the rate term represents the encoding cost of the residual latent representation, and the distortion term includes pixel-level error and a small percentage of class-weighted error. Compression efficiency and semantic fidelity are balanced by coefficients α and λ. The rate term is shown in Equation (1), the distortion term is shown in Equation (2), and the hybrid loss function is shown in Equation (3).

[0030]

[0031] L=R+λD (3)

[0032] Among them, y hat Let x be the input image, representing the residual latent representation. To reconstruct the image at the receiving end, p represents the expectation of all possible input images x. x It is the image distribution; It is an entropy model for y hat Prediction of probability distributions; It is based on the definition of entropy in information theory, representing the encoding y. hat Theoretically minimum number of bits required; M is the square of the Euclidean distance between the original image and the reconstructed image; low This is a small percentage category mask, where the small percentage category part is 1 and the rest is 0, used to filter out small percentage categories; the coefficient α determines the priority of the small percentage category region. The larger α is, the more the model focuses on protecting the reconstruction accuracy of the residuals corresponding to the small percentage category; the coefficient λ is used to control the trade-off between the rate part and the distortion part; when λ increases, the compression ratio is sacrificed for higher pixel-level accuracy; when λ decreases, pixel-level accuracy is sacrificed for higher compression ratio.

[0033] The main technical features and functional advantages of this invention are as follows:

[0034] (1) Strong robustness: With the help of entropy coding and entropy decoding technology, it can stably decode under low signal-to-noise ratio conditions, effectively mitigating the "cliff effect".

[0035] (2) Strong anti-noise interference capability: The residual is decoded by autoregression at the receiving end, which effectively avoids the influence of noise on the probability distribution during transmission.

[0036] (3) Strong semantic information preservation capability: By guiding residual coding with local-global semantic features, the semantic information of vulnerable small-proportion categories is prioritized and more bits are allocated to them, thereby improving the semantic preservation capability.

[0037] (4) High-efficiency compression: By selective zeroing and dynamic bit allocation, the compression efficiency of residuals is improved and bandwidth consumption is reduced.

[0038] (5) Strong compatibility: It adopts the method of transmitting residuals and is suitable for various datasets such as Cityscapes and Ade20K. Attached Figure Description

[0039] Figure 1 This is a flowchart illustrating the image semantic communication method based on residual and adaptive coding of the present invention.

[0040] Figure 2 This is a flowchart of the residual screening algorithm in this invention.

[0041] Figure 3 This is a flowchart of the semantic-based residual adaptive coding algorithm in this invention.

[0042] Figure 4 This is a flowchart of the semantic-based residual adaptive decoding algorithm in this invention.

[0043] Figure 5 This is a flowchart of the image generation network discriminator based on Transformer in this invention.

[0044] Figure 6This is a flowchart illustrating the sending end processing in this invention.

[0045] Figure 7 This is a flowchart illustrating the receiving end processing in this invention.

[0046] Figure 8 This is the local-global entropy model in the adaptive encoding / decoding algorithm of this invention.

[0047] Figure 9 This is a comparison of the MIoU performance of various methods on the Cityscapes dataset under different signal-to-noise ratio conditions.

[0048] Figure 10 This is a comparison of the FID performance of various methods on the Cityscapes dataset under different signal-to-noise ratio conditions.

[0049] Figure 11 This is a comparison of the MIoU performance of various methods on the Ade20K dataset under different signal-to-noise ratio conditions.

[0050] Figure 12 This is a comparison of the FID performance of various methods on the Ade20K dataset under different signal-to-noise ratio conditions. Detailed Implementation

[0051] The present invention will be further described below with reference to the embodiments and accompanying drawings.

[0052] It should be noted that the present invention is not limited to the specific embodiments described, but should include improvements, equivalent conversions and modifications based on the scope of the claims.

[0053] Figure 2 This is a schematic diagram of the residual screening algorithm in this invention. The algorithm includes a category range selection module based on semantics and block-based thinking, and a selective zeroing module. The specific workflow is as follows:

[0054] Step 1: First, process the semantic graph through a category range selection module based on semantics and block segmentation to dynamically divide it into categories with large and small proportions of objects. The specific steps are as follows:

[0055] (1) Set a category threshold value β and count the proportion of different categories in the semantic graph;

[0056] (2) Accumulate the proportions of these categories in ascending order, and stop accumulating when the accumulated value exceeds the preset category threshold.

[0057] (3) Select the categories that participated in the accumulation process as small percentage categories, and classify the remaining categories as large percentage categories. Since objects in an image are closely adjacent, the pixel values ​​around small percentage objects may change significantly when processing the residuals corresponding to large percentage category objects. These changes in pixel values ​​may cause a large deviation in the semantic performance of the final restored image when measuring the small percentage category objects. Therefore, this method of rigidly determining the range of small percentage category objects should not be used. Instead, the definition of the range of small percentage category objects should be appropriately relaxed. That is, divide the semantic map into several 16×16 blocks, classify the blocks containing small percentage categories as small percentage categories, and classify the rest as large percentage categories, thereby realizing the division of the range of large percentage categories and small percentage categories;

[0058] Step 2: In the selective zeroing module, based on the size proportion categories defined in the previous step, the residual regions corresponding to the large proportion categories are selected. Then, a zeroing threshold is set, setting values ​​in the residuals whose absolute values ​​are less than the zeroing threshold to zero, thereby significantly increasing the proportion of zero pixel values ​​in the residuals. Through this optimized residual selection algorithm, the compression efficiency of the residuals can be effectively improved while maintaining the compression ratio, and the residual information of the small proportion categories can be fully preserved, ultimately enhancing the semantic fidelity of the reconstructed image at the receiving end. In this invention, the category threshold value is 0.2 (representing that the total proportion of small proportion categories accounts for 20% of the entire image), and the zeroing threshold is 30 (representing that residual values ​​with an absolute value less than 30 are set to zero).

[0059] Figure 3 This is a schematic diagram of the semantic-based residual adaptive coding algorithm in this invention, including the importance processing algorithm, the local-global entropy model, and arithmetic coding. The specific workflow is as follows:

[0060] First, the semantic graph is processed to generate an importance matrix. Specifically, similar to the category range selection module based on semantics and block division, objects of small percentages are selected from the semantic graph. Then, a zero-based matrix of the same size as the semantic graph is initialized, and the semantic graph is evenly divided into several 16×16 blocks. All blocks are iterated over; if a block contains pixels of objects of small percentages, the corresponding element in the matrix is ​​changed to the category number of that small percentage (the category number does not contain 0); if a block contains pixels of multiple small percentages, it is uniformly labeled as the category with the smallest percentage. Next, the pixel percentage of each small percentage category in the matrix is ​​calculated, sorted in ascending order of percentage, and then weighted inverted (the smaller the percentage, the higher the weight coefficient). Finally, the category numbers in the matrix are replaced with the weight coefficients to generate the importance matrix.

[0061] Secondly, an encoder composed of convolution and normalization operations processes the importance matrix, coarse reconstruction image, and residuals respectively to obtain local importance semantic features, global semantic features, and latent representations of the residuals. These two features are then concatenated to obtain local-global semantic features. Next, the latent representations of the residuals are discretized. This is because discrete values ​​are more suitable for entropy coding, which helps achieve efficient compression and transmission of residual data. Then, the local-global semantic features and the discrete latent representations of the residuals are input into a local-global entropy model. During this process, the local-global entropy model adaptively estimates the probability distribution of the discrete latent representations of the residuals, guided by the local-global semantic features. Finally, arithmetic coding, based on the adaptively estimated probability distribution, allocates more bits to residuals corresponding to smaller percentage categories and fewer bits to residuals corresponding to larger percentage categories, thus prioritizing the semantic information of smaller percentage categories.

[0062] Figure 4 This is a schematic diagram of the semantic-based residual adaptive decoding algorithm in this invention, including an importance processing algorithm, a local-global entropy model that shares parameters with the encoding algorithm, and arithmetic decoding. The local-global entropy model comprises two parts: a context model based on a mask attention mechanism and a distribution estimation module. The specific workflow is as follows: First, similar to the encoding algorithm, local-global semantic features are obtained. Then, guided by these semantic features, the local-global entropy model (LGEM) reconstructs the probability distribution of the discrete latent representation of the residual position position-by-position using an autoregressive method. Simultaneously, arithmetic decoding is used to obtain the discrete latent representation of the residual at the corresponding position, avoiding the noise sensitivity problem caused by directly transmitting probability distribution parameters and enhancing anti-interference capability. Finally, a decoder consisting of upsampling, convolution operations, and normalization operations is used to decode the discrete latent representation of the residual, achieving residual reconstruction.

[0063] The stepwise decoding operation in the above autoregressive mode is as follows:

[0064] Step 1: Initialize the discrete latent representation of the residual as a tensor y with all zeros, and decode it point by point in order from left to right and from top to bottom;

[0065] Step 2: Use a context model based on mask attention mechanism to extract context features from the discrete latent representation y;

[0066] Step 3: Combining contextual features and local-global semantic features, the distribution estimation module is used to predict the probability distribution parameters μ[i][j] and σ[i][j] of the current position, and arithmetic decoding is used to restore the residual discrete latent representation y[i][j] of the position from the bit stream;

[0067] Step 4 updates y[i][j] into y, and repeats Steps 2-4 until all discrete latent representations of the residuals are decoded.

[0068] Figure 5 This is a flowchart illustrating the discriminator of the Transformer-based image generation network in this invention, including a block-shaping and linear flattening module, a Transformer encoder, and an average pooling operation. The specific workflow is as follows:

[0069] First, the image is divided into several 16×16 sub-blocks, and then each sub-block is flattened into a vector of length 768. A fully connected operation is used to map each sub-block vector to a high-dimensional space to obtain an embedding vector, and then relative position encoding information is added to the embedding vector.

[0070] Then, the embedding vector with added positional encoding information is input into a 6-layer Transformer encoder, which consists of multi-head attention, residual and normalization layers, and a feedforward neural network. Next, the embedding vector after passing through the 6-layer encoder is subjected to global average pooling to obtain a global feature vector. Global average pooling is chosen because it is suitable for capturing overall information and does not overemphasize extreme values ​​in a particular dimension, which helps improve the model's generalization ability and robustness.

[0071] Finally, the global feature vector is input into a fully connected layer, which outputs a scalar value. The Sigmoid function is then used to map the output to obtain the probability that the input image is the real image.

[0072] Figure 6 This is a schematic diagram of the sending end processing flow in this invention, including a semantic segmentation network, an image generation network, a residual filtering algorithm, and a semantic-based residual adaptive coding algorithm to encode the semantic map and residual into a bitstream. The specific workflow is as follows: First, semantic segmentation is performed on the input image to obtain its semantic map, which is then encoded into a bitstream. Next, the semantic map is input into the image generation network for semantic image generation, producing a coarse reconstructed image of the input image. Then, the input image and the coarse reconstructed image are subtracted to obtain the residual. Finally, the residual is first optimized using the residual filtering algorithm, and then encoded into a bitstream using the semantic-based residual adaptive coding algorithm.

[0073] Figure 7This is a schematic diagram of the receiving end processing flow in this invention, including an image generation network, a semantic-based residual adaptive decoding algorithm, and an image restoration algorithm to reconstruct the input image. The specific workflow is as follows: First, a semantic map is decoded from the bitstream, and a semantic image is generated using an image generation network that shares parameters with the sending end to obtain a coarse reconstructed image of the input image; second, the residual is decoded from the bitstream using the semantic-based residual adaptive decoding algorithm and combined with the coarse reconstructed image to obtain the reconstructed image; finally, the reconstructed image is restored to its original size using the image restoration algorithm.

[0074] Figure 8 This invention presents the local-global entropy model architecture. The model consists of two core components: a context model based on a masked attention mechanism and a distribution estimation module. The specific workflow is as follows: First, the latent representation is input into the context model, which extracts spatial context features through a masked attention mechanism. Then, these context features are concatenated with local-global semantic features to form a fused feature. Finally, the fused feature is input into the distribution estimation module, which outputs the probability distribution parameters of the latent representation. The entire process achieves probability distribution modeling through end-to-end learning, eliminating the need for explicit transmission of probability distribution parameters. This effectively suppresses the interference of channel noise on the probability distribution parameters and significantly improves the robustness of the system.

[0075] Experimental Example 1:

[0076] This experiment was simulated on different signal-to-noise ratios (SNR = 0, 5, 10, 15, 20 dB) and different datasets (Cityscapes, Ade20K). During communication, a low-density parity-check (LDPC) coding scheme was used for channel coding, with a fixed coding length of 648 and a code rate of 2 / 3. During transmission, binary phase-shift keying (BPSK) was used to modulate and demodulate the channel-coded data, and the bit rate after LDPC coding was 0.6 bpp. The average intersection-to-union ratio (MIoU) and Fréchet initial distance (FID) were used as evaluation metrics. To comprehensively evaluate the performance of this invention, the experiment was compared not only with the image semantic communication system based on deep semantic segmentation (DSSLIC) and the image semantic communication system based on the multi-reference entropy model (MLIC), but also with traditional image communication systems based on BPG and JPEG XL. In addition, the MIoU value was set to 0 and the FID value was set to 600 in the case of decoding failure due to noise interference. The results are as follows: Figure 9 , Figure 10 , Figure 11 and Figure 12 As shown.

[0077] Throughout the entire SNR range, this invention outperforms other comparative systems, especially under low SNR conditions, where its advantages are even more significant. In low SNR environments, traditional image communication systems based on BPG and JPEG XL are prone to decoding failures due to noise interference, resulting in a "cliff effect." In this invention, when SNR = 0dB and 5dB, the VQ-VAE method is used to transmit the semantic map, ensuring successful decoding of the semantic map at the receiving end. Then, an image generation network is used for coarse reconstruction, laying the foundation for the quality of the reconstructed image at the receiving end. Simultaneously, through a semantic-based residual adaptive encoding and decoding algorithm, more bits are allocated to the residuals corresponding to important categories in the image, i.e., categories with a small proportion, prioritizing the semantic performance of key regions in the reconstructed image at the receiving end. When SNR = 10dB, 15dB, and 20dB, the VQ-VAE method is no longer used to transmit the semantic graph. Instead, the semantic graph is first preprocessed using RLE coding, then lossless compression is performed using the Zlib method, and finally channel coding is performed using LDPC to achieve lossless transmission of the semantic graph. This allows the MIoU and FID indicators of the present invention to achieve a qualitative leap from SNR = 5dB to SNR = 10dB.

[0078] from Figure 9 and Figure 11 It can be seen that on both datasets, the MIoU values ​​of all methods improve with increasing SNR. However, the two traditional image communication systems based on BPG and JPEG XL, as well as DSSLIC, exhibit a significant "cliff effect" at SNR=0dB and SNR=5dB, indicating poor robustness. In contrast, the MIoU curve of this invention is the smoothest and performs best, especially at low SNR, where the advantages of this method are extremely obvious. This demonstrates that the method of this invention has strong resistance to channel interference and visual quality preservation capabilities. Figure 10 and Figure 12 As shown, on both datasets, the FID metric of the method of this invention has a significant advantage under all signal-to-noise ratio conditions; the lower the signal-to-noise ratio, the more pronounced the advantage. For example, in... Figure 10 In the experiment, when SNR=5dB, both traditional image communication systems based on BPG and JPEG XL, as well as DSSLIC, failed to decode. Although the MLIC method successfully decoded at the receiving end, the decoded image had a large error compared to the original image, with an FID value as high as 457. In contrast, the FID of the method of the present invention was only 374, indicating that the method of the present invention has a better semantic information preservation capability.

[0079] References

[0080] [1]Chen Z,Qin H,Guo Y,et al.Binarized Diffusion Model for ImageSuper-Resolution[J].2024.

[0081] [2]H.Zhao,J.Shi,X.Qi,X.Wang and J.Jia,"Pyramid Scene ParsingNetwork,"2017 IEEE Conference on Computer Vision and Pattern Recognition(CVPR),Honolulu,HI,USA,2017,pp.6230-6239,doi:10.1109 / CVPR.2017.660.

[0082] [3]H.Tang,D.Xu,Y.Yan,P.H.S.Torr and N.Sebe,"Local Class-Specific andGlobal Image-Level Generative Adversarial Networks for Semantic-Guided SceneGeneration,"2020 IEEE / CVF Conference on Computer Vision and PatternRecognition(CVPR),Seattle,WA,USA,2020,pp.7867-7876,doi:10.1109 / CVPR42600.2020.00789.

[0083] [4]Oord A V D,Vinyals O,Kavukcuoglu K.Neural Discrete RepresentationLearning[J].2017.DOI:10.48550 / arXiv.1711.00937.

[0084] [5]Deutsch P,Gailly J L.ZLIB Compressed Data Format Specificationversion 3.3[J].RFC Editor,1996.DOI:urn:ietf:rfc:1950。

Claims

1. An image semantic communication method based on residual and adaptive coding, characterized in that, The design includes collaborative design between the transmitter and receiver. The core of this design is to differentiate the residual based on the semantic features of the image, dynamically adjust the encoding priority of different parts of the residual, and optimize the decoding process to resist channel noise interference. Specifically, this includes: residual extraction at the transmitting end, residual filtering and processing, adaptive encoding of the residual into a bit stream, and image reconstruction at the receiving end; This involves a semantic segmentation network, a residual filtering algorithm, an image generation network with an integrated Transformer architecture discriminator trained with a loss function, a semantic-based residual adaptive coding algorithm at the transmitter and a semantic-based residual adaptive decoding algorithm at the receiver trained with a loss function, and an image restoration algorithm. The specific steps are as follows: Step 1: Extracting residuals from the sending end First, a semantic segmentation network is used to perform semantic segmentation on the input image to obtain a semantic map of the input image. Then, the semantic map is input into an image generation network with a Transformer architecture discriminator to generate a coarse reconstruction map. Finally, the residual is obtained by subtracting the input image from the coarse reconstruction map. At the same time, the semantic map is encoded into a bitstream for transmission. Step 2: Residual screening process The residuals obtained in step 1 are processed using a residual filtering algorithm. This residual filtering algorithm includes a category range selection module based on semantics and block-based thinking and a selective zeroing module. The category range selection module filters out the residuals corresponding to small-proportion category objects and large-proportion category objects respectively. The selective zeroing module performs selective zeroing processing on the residuals corresponding to large-proportion category objects, increasing the proportion of zero pixel values ​​in the entire residual, thereby helping to achieve a higher compression rate. Step 3: Adaptively encode the residual into a bitstream First, an encoder consisting of convolution and normalization operations processes the residuals after step 2, transforming them into continuous vectors, i.e., latent representations. This maps the residuals from the pixel space to the latent space, reducing redundancy while preserving key information. Second, discrete latent representations of the residuals are generated through discrete quantization, providing a compact data form for subsequent probability distribution estimation and entropy coding. Next, the semantic graph is processed to generate an importance matrix. Similarly, an encoder consisting of convolution and normalization operations transforms the importance matrix and the coarsely reconstructed graph into continuous vectors, i.e., local importance semantic features and global semantic features, respectively. These are then concatenated to form local-global semantic features. Then, guided by these semantic features, the probability distribution of the discrete latent representations of the residuals is adaptively estimated using the local-global entropy model in the semantic-based adaptive residual coding algorithm. Finally, based on the estimated probability distribution, the discrete latent representations of the residuals and their shape parameters are encoded into a bitstream using arithmetic coding. Step 4: Image reconstruction at the receiving end The receiver first recovers the shape parameters of the semantic map and the discrete latent representation of the residual from the bitstream. It then generates a coarse reconstruction map using an image generation network that shares parameters with the transmitter, and extracts local-global semantic features from the semantic map and the coarse reconstruction map. Next, leveraging the autoregressive properties of the local-global entropy model in the semantic-based residual adaptive decoding algorithm proposed in this invention, the discrete latent representation of the residual is reconstructed. Subsequently, a decoder consisting of upsampling, convolution, and normalization operations is used to decode the discrete latent representation and recover the residual. Finally, the residual is combined with the coarse reconstruction map, and an image restoration algorithm is used to restore it to its original size.

2. The image semantic communication method according to claim 1, characterized in that, In step 1: The semantic segmentation network is a pre-trained PSPNet; The image generation network described above is based on LGGAN, with the addition of an image generation network discriminator based on the Transformer architecture. This discriminator divides the image into several small blocks and then uses multiple layers of Transformers to process the dependencies between each block, thereby effectively capturing global information of the image, improving the efficiency of feature extraction, enhancing the discriminator's ability to understand the global structure of the image, and improving the quality of the images generated by the image generation network. There are two types of semantic graph transmission methods, both of which use LDPC for channel transmission. Under low signal-to-noise ratio (SNR) conditions, a retrained VQ-VAE model is used to perform encoding at the transmitter and decoding at the receiver. Under high SNR conditions, the semantic graph is first preprocessed with run-length encoding, and then the data is compressed using the Zlib method to achieve high-quality transmission of the semantic graph.

3. The image semantic communication method according to claim 2, characterized in that, In step 2: The category range selection module dynamically divides the residuals into residuals corresponding to objects of a large proportion and residuals corresponding to objects of a small proportion based on semantic information and block strategy. The specific steps are as follows: (1) Set a category threshold value β and count the proportions of different categories in the semantic graph; (2) Accumulate the proportions of these categories in ascending order, and stop accumulating when the accumulated value is greater than the preset category threshold value; (3) Filter the categories that participated in the accumulation process into small proportion categories. The selective zeroing module selectively ignores some information for the residuals corresponding to categories with a large proportion. That is, it sets a zeroing threshold and sets the values ​​in the residuals whose absolute values ​​are less than the zeroing threshold to zero, thereby increasing the proportion of zero pixel values ​​in the residuals.

4. The image semantic communication method according to claim 3, characterized in that, The semantic-based residual adaptive coding algorithm described in step 3 takes the semantic map, the coarse reconstruction map, and the residual processed by the residual filtering algorithm as inputs, and outputs a bit stream. The algorithm extracts local-global semantic features from the semantic map and the coarse reconstruction map, and uses these features to guide entropy coding to map data into a bit stream. When local-global semantic features are introduced as guidance, the goal of the entropy encoder changes from "allocating bits based on data statistics" to "allocating bits based on local-global semantic features". The local-global entropy model in the algorithm dynamically adjusts the estimation of the residual probability distribution according to the local-global semantic features, thereby allocating more bits to the residuals corresponding to the small percentage of categories and less bits to the residuals corresponding to the large percentage of categories, thus prioritizing the semantic information of objects in the small percentage of categories.

5. The image semantic communication method according to claim 4, characterized in that, The semantic-based adaptive residual decoding algorithm described in step 4 takes a bitstream of semantic graph, coarse reconstruction graph, and discrete latent representation of residual as input, and outputs the reconstructed residual. This algorithm uses an autoregressive approach to decode the discrete latent representation of residual from the bitstream. Specifically, the receiving end decodes the semantic graph from the bitstream and performs coarse reconstruction using an image generation network. Then, it extracts local-global semantic features from the semantic graph and coarse reconstruction graph. Next, guided by these semantic features, the local-global entropy model reconstructs the discrete latent representation of residual position-by-position using an autoregressive method. Finally, a decoder consisting of upsampling, convolution, and normalization operations decodes the discrete latent representation of residual, thus reconstructing the residual.

6. The image semantic communication method according to claim 5, characterized in that, The local-global entropy model consists of a context model based on a mask attention mechanism and a distribution estimation module. The module in the context model adopts a Transformer architecture and captures the long-distance dependencies of the discrete latent representation of the residuals through mask self-attention. The distribution estimation module combines local-global semantic features to dynamically adjust the probability distribution estimation and allocate more bits to the residuals corresponding to the small percentage of classes.

7. The image semantic communication method according to claim 6, characterized in that, The loss function employs rate-distortion joint optimization; the rate term represents the encoding cost of the residual latent representation, and the distortion term includes pixel-level error and a small percentage of class-weighted error. Compression efficiency and semantic fidelity are balanced by coefficients α and λ. The rate term is shown in Equation (1), the distortion term is shown in Equation (2), and the hybrid loss function is shown in Equation (3). L=R+λD (3) Among them, y hat Let x be the discrete latent representation of the residual, and let x be the input image. To reconstruct the image at the receiving end, p represents the expectation of all possible input images x. x It is the image distribution; It is an entropy model for y hat Prediction of probability distributions; It is based on the definition of entropy in information theory, representing the encoding y. hat Theoretically minimum number of bits required; M is the square of the Euclidean distance between the original image and the reconstructed image; low This is a small percentage category mask, where the small percentage category part is 1 and the rest is 0, used to filter out small percentage categories; the coefficient α determines the priority of the small percentage category region. The larger α is, the more the model focuses on protecting the reconstruction accuracy of the residuals corresponding to the small percentage category; the coefficient λ is used to control the trade-off between the rate part and the distortion part; when λ increases, the compression ratio is sacrificed for higher pixel-level accuracy; when λ decreases, pixel-level accuracy is sacrificed for higher compression ratio.

Images