Model algorithm hardware collaborative optimization semantic communication model lightweight deployment method

Abstract

A model algorithm hardware collaborative optimization semantic communication model lightweight deployment method belongs to the field of communication, and comprises the following steps: designing a structured pruning method based on feature map information entropy and reconstruction sensitivity joint evaluation to prune a coding and decoding structure of a semantic communication model; carrying out quantitative perception training by adopting TensorRT INT8, and carrying out dynamic range calibration through a calibration set; an end-to-end acceleration architecture from PyTorch to an FPGA is realized based on a Xilinx Vitis tool chain, end-to-end deployment reasoning acceleration on a VU13PFPGA is realized, and a high-energy-efficiency semantic communication prototype is provided for resource-constrained IoT equipment. According to the method, the complexity of the model is remarkably reduced through the FERS structured pruning method; the compression ratio and the reconstruction quality are effectively balanced by combining a precision perception quantification technology; through end-to-end hardware collaborative design, multi-dimensional optimization of calculation, storage and energy consumption is realized, and finally a high-performance terminal deployment target is achieved.

Description

Technical Field

[0001] This invention belongs to the field of communication technology, specifically relating to a lightweight deployment method for semantic communication models that utilizes model-algorithm-hardware co-optimization. Background Technology

[0002] With the deep integration of artificial intelligence and communication technology, the Internet of Things (IoT) is evolving from "interconnection of everything" to "intelligent interconnection of everything". Semantic communication, as a key technology, significantly improves communication efficiency by extracting and transmitting task-related semantic features.

[0003] The core of semantic communication models lies in their encoding and decoding structures, which must simultaneously handle high-dimensional image features and complex domain channel transmission. The classic Joint Source-channel Coding (JSCC) technique, through end-to-end joint optimization, maintains high-fidelity reconstruction capabilities even in low signal-to-noise ratio (SINR) environments, making it an ideal paradigm for image semantic communication. In particular, deep learning-based JSCC frameworks (such as DeepJSCC) demonstrate significant advantages in bandwidth-constrained IoT scenarios due to the powerful feature extraction and noise resistance capabilities of neural networks. JSCC-based semantic communication transmission frameworks involve unique operations such as complex numerical mapping, power normalization, and IQ modulation. The encoder must map the input image into a complex numerical signal that meets power constraints, while the decoder must reconstruct the image from the noisy signal. This process is far more sensitive to the distribution of model weights and activation values ​​than traditional semantic transmission tasks, and complex number operations lack efficient support on resource-constrained terminal hardware. However, deep learning-based semantic communication models typically have a huge number of parameters (ranging from millions to tens of millions) and high computational complexity (inference on a single 224×224 image can reach GFLOPs level). Directly deploying them on IoT terminals (such as edge sensors and embedded cameras) that are limited by computing power, memory, and power consumption will lead to problems such as high latency, high energy consumption, and inability to guarantee real-time performance.

[0004] The closest existing technical solutions to this invention mainly focus on two dimensions: model lightweighting and hardware acceleration. Regarding model lightweighting, MobileNet employs a depthwise separable convolutional architecture, achieving an 8-9x speedup by decomposing standard convolutions into depthwise convolutions and pointwise convolutions; ShuffleNet, through channel shuffling operations and modular design of grouped convolutions, significantly reduces the number of parameters while maintaining feature transfer efficiency. Furthermore, the classic model pruning method L1-norm uses the L1 norm of the convolutional kernel weights as an importance indicator to independently optimize the channel pruning ratio of each layer. At the hardware deployment level, Xilinx's Vitis AI and NVIDIA's TensorRT provide FPGA deployment toolchains. Existing research based on these toolchains to accelerate ResNet-50 classification on the Zynq platform has achieved 142 FPS, focusing on improving computational throughput and memory access efficiency.

[0005] For lightweight deployment of complex semantic communication models, existing technologies suffer from the following main shortcomings: First, while general lightweight models (such as depthwise separable convolution in MobileNet and channel shuffling in ShuffleNet) can reduce the number of parameters, their architecture is primarily optimized for image classification tasks and fails to meet the complex complex numerical feature reconstruction requirements in semantic communication, resulting in significant degradation in reconstruction quality during JSCC encoding and decoding. Second, traditional pruning methods (such as L1-norm) rely solely on weight magnitudes for channel pruning, ignoring the correlation between the information entropy distribution of feature maps and the reconstruction sensitivity of the decoder, leading to the loss of key semantic features. Finally, existing FPGA deployment toolchains (such as VitisAI) only support real number operations, making it difficult to handle complex numerical data streams in semantic communication, and lacking pipeline optimization for encoder-decoder collaboration, thus failing to fully leverage the parallel architecture advantages of FPGAs. Summary of the Invention

[0006] Existing general-purpose lightweight model methods are difficult to adapt to the complex numerical data stream processing architecture and feature sensitivity requirements unique to semantic communication, resulting in a significant decrease in the quality of model reconstruction after compression. At the same time, existing hardware deployment schemes lack optimization support for the complex domain data collaborative pipeline between encoder and decoder, and cannot efficiently utilize hardware resources to achieve end-to-end acceleration.

[0007] Therefore, in order to address the core contradiction in deploying semantic communication models on resource-constrained IoT terminal devices—namely, how to significantly reduce the computational complexity, storage requirements, and energy consumption of the model while minimizing the sacrifice of semantic information reconstruction quality and task accuracy—this invention provides a lightweight deployment method for semantic communication models that features model-algorithm-hardware co-optimization. This method enables the efficient deployment of complex semantic communication models under conditions of severely limited IoT terminal device resources, achieving low-latency, low-power, and high-precision semantic information transmission and reconstruction.

[0008] This invention provides a lightweight deployment method for semantic communication models with hardware-based model algorithm co-optimization, which mainly includes the following three key technical points:

[0009] 1. Algorithm level: For the semantic communication codec structure, a structured pruning method based on joint evaluation of feature map information entropy and reconstruction sensitivity (FERS) is designed to minimize reconstruction loss while compressing the number of model parameters; at the same time, TensorRT INT8 quantization-aware training is adopted, and dynamic range calibration of the calibration set is used to reduce the quantization error of JSCC complex features.

[0010] 2. Hardware deployment level: Based on the Xilinx Vitis toolchain, an end-to-end acceleration architecture from PyTorch to FPGA is implemented, realizing end-to-end deployment inference acceleration on VU13P FPGA.

[0011] 3. System level: Build a full-link solution of "model compression-hardware acceleration" to provide a high-efficiency semantic communication prototype for resource-constrained IoT devices.

[0012] The technical solution adopted by this invention to solve the technical problem is as follows:

[0013] This invention provides a lightweight deployment method for semantic communication models that utilizes model-algorithm-hardware co-optimization, which mainly includes the following steps:

[0014] (1) Design a structured pruning method based on Feature map information entropy and Reconstruction Sensitivity (FERS) to prune the encoding and decoding structure of the semantic communication model;

[0015] (2) TensorRT INT8 quantization perception training was used, and dynamic range calibration was performed using a calibration set;

[0016] (3) Based on the Xilinx Vitis toolchain, an end-to-end acceleration architecture from PyTorch to FPGA is implemented, enabling end-to-end deployment and inference acceleration on VU13P FPGA, providing a high-efficiency semantic communication prototype for resource-constrained IoT devices.

[0017] Furthermore, the JSCC encoder in the semantic communication model adopts an encoder architecture based on a convolutional neural network, consisting of a series of convolutional layers, ReLU activation functions, and normalization layers. The convolutional layers are used to extract image features and combine them layer by layer to form channel input samples. The ReLU activation function enables the model to learn the nonlinear mapping from the source signal space to the encoded signal space. The output of the last convolutional layer is normalized to obtain the channel input samples.

[0018] Furthermore, the JSCC encoder in the semantic communication model uses a function Map an n-dimensional input image x to a channel input sample of length k. in Let n be a real vector space. Let z be a k-dimensional complex vector space, and z be a complex field sample, satisfying the average transmit power constraint. P is the average transmit power constraint. For mathematical expectation, z * It is the conjugate transpose of z. For the encoding function f θ The direct output vector, yes The conjugate transpose of .

[0019] Furthermore, the JSCC decoder in the semantic communication model transforms the operations performed by the JSCC encoder through a series of transposed convolutions with nonlinear activation functions, mapping image features to estimates of the original transmitted image.

[0020] Furthermore, the JSCC encoder in the semantic communication model maps the input image to channel input samples z and transmits them through a physical channel η. This physical channel η introduces random corruption into the transmitted symbols. This means that the physical channel η is modeled as a series of untrainable layers, defined by the transfer function. express, This is a corrupted, complex signal.

[0021] Furthermore, in the structured pruning method based on the joint evaluation of feature map information entropy and reconstruction sensitivity, the output feature map F of the c-th channel of the JSCC encoder in the semantic communication model is... c ∈R H×W Calculate information entropy:

[0022]

[0023] Where H is the feature map F c The height of feature map F is W. c width, The normalized energy probability distribution is defined as follows: i and j are the row and column indices of the pixel positions in the feature map, respectively, and the entropy value ε is... c The larger the value, the more dispersed the feature value distribution, the higher the information richness, and the higher the channel importance.

[0024] Furthermore, in the structured pruning method based on the joint evaluation of feature map information entropy and reconstruction sensitivity, the quantification of the reconstruction sensitivity of the JSCC decoder in the semantic communication model is as follows:

[0025]

[0026] Where, x n For the nth original input signal / image, For the original model, x n The reconstruction results For the pruned model on x n The reconstruction results are given, where PSNR is the peak signal-to-noise ratio and N is the total number of test samples.

[0027] Furthermore, in the structured pruning method based on the joint evaluation of feature map information entropy and reconstruction sensitivity, the channel adaptive fusion is expressed as:

[0028] Score c =α·Norm(ε c )+β·β(SNR)·Norm(ΔPSNR c )

[0029] Wherein, the channel factor β(SNR) = 1 - e -0.1·SNR Normalization function Score c The higher the final comprehensive importance score of the c-th channel, the more important the channel is and the more it should be retained during pruning; α and β are manually set weight coefficients, which are the weight coefficients of the information entropy term and the reconstruction sensitivity term, respectively.

[0030] Furthermore, the specific implementation process of step (2) is as follows:

[0031] 1) Load the trained PyTorch framework model and set it to inference mode, and prepare tensors consistent with the model input; convert the structure and weights of the PyTorch framework model to ONNX format using the ONNX export function of the PyTorch framework model.

[0032] 2) Use TensorRT to calibrate the two sub-models after splitting and converting to ONNX format using the calibration set, generating calibration files p1.cache and p2.cache for the two sub-models respectively; manually rewrite the specific values ​​in the calibration files of the two sub-models generated in the first generation to the values ​​in the calibration file of the complete model.

[0033] 3) Read the calibration data in the calibration file, parse out the quantization parameters of each layer, reorganize these parameters in JSON format, store them as key-value pairs, and generate a new JSON file to meet the format requirements of the calibration file when generating a file that can be directly loaded and executed by the FPGA.

[0034] Furthermore, the specific implementation process of step (3) is as follows:

[0035] Load the PyTorch framework model and prepare the corresponding input data; use the conversion function of the PyTorch2caffe tool to export the structure and weights of the PyTorch framework model into two files: caffemodel and prototxt; use the JSON calibration table generated during quantization to compile the generated Caffe model into an NVDLA format file using the compiler tool nvdla_compiler, which is then used for inference verification on the FPGA.

[0036] The beneficial effects of this invention are:

[0037] The core advantage of this invention lies in achieving a breakthrough in the collaborative optimization of semantic communication model characteristics and hardware deployment constraints. Through an adaptive structured pruning method based on joint evaluation of Feature map information entropy and Reconstruction Sensitivity (FERS), model complexity is significantly reduced while preserving key semantic features. Combined with accuracy-aware quantization techniques, compression ratio and reconstruction quality are effectively balanced. End-to-end hardware co-design achieves multi-dimensional optimization of computation, storage, and energy consumption, ultimately achieving the goal of high-performance, low-resource-consumption terminal deployment. Attached Figure Description

[0038] Figure 1 The flowchart shows a lightweight deployment system for a semantic communication model that utilizes model algorithm and hardware co-optimization, as provided by this invention.

[0039] Figure 2 This is a schematic diagram illustrating the implementation principle of a lightweight deployment method for semantic communication models that utilizes model algorithm and hardware co-optimization, as provided by this invention.

[0040] Figure 3 This is a diagram of the JSCC decoder and its architecture.

[0041] Figure 4 This is a flowchart of a structured pruning method based on joint evaluation of Feature map information entropy and Reconstruction Sensitivity (FERS).

[0042] Figure 5 Flowchart for optimizing INT8 quantization in the PyTorch framework model.

[0043] Figure 6 This document outlines the complete process for deploying PyTorch framework models on an FPGA.

[0044] Figure 7 To assess the reconstruction performance of the Semantic Communication Model (SC) under different pruning strategies in AWGN channels with a signal-to-noise ratio of 0-20 dB.

[0045] Figure 8 The images show reconstructed under different pruning strategies in a 15dB-AWGN channel environment. In the figure, (a) is the original image, (b) is the unpruned image, (c) is the reconstructed image after L1-norm pruning, (d) is the FERS reconstructed image, and (e) is the version deployed on FPGA. Detailed Implementation

[0046] The present invention will be further described in detail below with reference to the accompanying drawings.

[0047] In a first aspect, the present invention provides a lightweight deployment system for semantic communication models that features hardware-model collaborative optimization.

[0048] like Figure 1As shown, this invention provides a lightweight deployment system for a semantic communication model with hardware-model co-optimization. This system is an end-to-end deployment system for image semantic communication models used in resource-constrained IoT terminals for classification tasks. It mainly includes the following modules: a transmitter, a wireless channel, and a receiver. The transmitter deploys an image semantic encoder model, and the receiver deploys an image semantic decoder model. The image semantic encoder model mainly includes a data preprocessing module and a Joint Source-Channel Encoder (JSCC encoder). The image semantic decoder model mainly includes a Joint Source-Channel Decoder (JSCC decoder) and an image classification module. The data preprocessing module is mainly used to preprocess the input image, including cropping, pixel value normalization, and data augmentation. The JSCC encoder is mainly used to extract and compress the preprocessed input image to generate a compressed semantic feature vector. The JSCC decoder is mainly used to reconstruct the semantic feature vector and send the reconstructed image to the image classification module. The image classification module is mainly used to classify the reconstructed image. The image classification module specifically uses the lightweight ResNet-50 classifier in the PyTorch framework model. Its biggest advantage lies in the introduction of residual connections, which effectively solves the gradient vanishing and network degradation problems in deep networks.

[0049] This invention provides a lightweight deployment system for semantic communication models with model-algorithm-hardware co-optimization. Taking a task-driven image semantic communication model as its core, the system first preprocesses the data. Then, the sending end extracts and compresses the input image through a JSCC encoder to generate a compressed semantic feature vector. The receiving end reconstructs the semantic feature vector through a JSCC decoder and sends the reconstructed image to the image classification module.

[0050] For both the transmitting and receiving ends, this invention employs lightweight pruning and quantization methods. Specifically, to address the distortion issue caused by the model lightweighting process, this invention introduces a FERS structured pruning module in the image semantic communication model, based on the structural characteristics of the semantic communication model. This module utilizes a structured pruning method jointly evaluated by Feature Map Information Entropy and Reconstruction Sensitivity (FERS) to lightweight key feature channels, minimizing reconstruction loss while compressing the number of model parameters. The optimized feature stream is then further processed by the JSCC decoder to complete image reconstruction. Finally, the reconstructed image is fed into an image classification module optimized by collaborative L1-norm pruning and INT8 quantization.

[0051] The present invention provides a lightweight deployment system for semantic communication models that uses model algorithm hardware co-optimization, which is ultimately implemented through FPGA, thereby efficiently completing the lightweight deployment and acceleration of image semantic communication models.

[0052] Secondly, the present invention provides a lightweight deployment method for semantic communication models with model-algorithm-hardware co-optimization, which is based on a lightweight deployment system for semantic communication models with model-algorithm-hardware co-optimization provided in the first aspect.

[0053] like Figure 2 As shown, this invention provides a lightweight deployment method for semantic communication models with model-algorithm-hardware co-optimization, which uses source-channel joint coding technology to achieve wireless image transmission. The specific implementation process is as follows:

[0054] Step S1: Data preprocessing;

[0055] The input image is preprocessed, including cropping, pixel value normalization, and data augmentation, in order to make it uniform in size and meet the input requirements of the JSCC encoder neural network.

[0056] Step S2: Encoding operation;

[0057] The sending end extracts and compresses the input image using the JSCC encoder to generate a compressed semantic feature vector.

[0058] Specifically, the JSCC encoder uses a function Map an n-dimensional input image x to a channel input sample z of length k, where Let n be a real vector space. Let z be a k-dimensional complex vector space, and z be a sample from the complex field.

[0059] like Figure 3 As shown, the JSCC encoder can specifically adopt an encoder architecture based on convolutional neural networks, mainly composed of a series of convolutional layers (conv), ReLU activation functions, and normalization layers. The convolutional layers are used to extract image features and combine them layer by layer to form channel input samples; while the ReLU activation function enables the model to learn a nonlinear mapping from the source signal space to the encoded signal space. The output of the last convolutional layer of the encoder is shown. After normalization, the final result is The channel input sample z satisfies the average transmit power constraint. Where P is the average transmit power constraint. For mathematical expectation, z * It is the conjugate transpose of z. For the encoding function f θ The direct output vector, yes The conjugate transpose of .

[0060] Step S3: Data transmission;

[0061] After the encoding operation, the real and imaginary parts of the input sample z are directly transmitted through the IQ components of the digital signal. z is transmitted through the physical channel η, which introduces random corruption to the transmitted symbols. In order to optimize the entire neural network system in an end-to-end manner, the physical channel η is modeled as a series of non-trainable layers, which are defined by transfer functions. express.

[0062] Step S4: Decoding operation;

[0063] The receiving end reconstructs the semantic feature vector using the JSCC decoder and then processes the reconstructed image. The image is fed into the image classification module.

[0064] Specifically, the JSCC decoder uses a function Damaged composite signal Mapping to the original input image The estimate.

[0065] like Figure 3 As shown, similar to the structure of the JSCC encoder, the JSCC decoder transforms the operations performed by the JSCC encoder through a series of transposed convolutions (with non-linear activation functions) to map image features to estimates of the original transmitted image.

[0066] Specifically, Figure 3 In this context, 224×224×3 and 224×224×64 represent the dimensional representation of tensors (data blocks) in deep learning. For example, the feature dimension of the original input image is 224×224×3, meaning the feature map's spatial dimensions are 224 in height and 224 in width. The "3" represents the three color channels of a standard RGB color image. After the first convolutional layer, the width and height of the input tensor remain unchanged, but the number of channels increases from 3 to 64. After the second convolutional layer, the width and height of the input tensor are compressed to half their original values, i.e., 112×112, and so on. These dimensional transformations collectively describe the morphological changes of data as it flows through the network, intuitively demonstrating how the JSCC encoder progressively compresses the input image and extracts features, and how the JSCC decoder progressively reconstructs it back to the original image size.

[0067] Step S5: Image classification;

[0068] The reconstructed images are fed into an image classification module that has undergone collaborative L1-norm pruning and INT8 quantization optimization.

[0069] S5.1: Pruning;

[0070] In image semantic communication systems, there are fundamental functional differences between image encoding, transmission, and decoding processes and image classification processes: the core task of the former is to achieve high-fidelity image compression and reconstruction, and its complex domain feature map contains structured spatial information; the latter focuses on high-level semantic feature extraction. If traditional L1-norm pruning is directly applied to the image encoding, transmission, and decoding processes, it will lead to a significant decrease in physical channel noise resistance and image reconstruction PSNR. To address this characteristic, this invention employs a pruning strategy jointly evaluated by L1-norm and FERS, using adapted pruning methods for different parts of the image semantic communication model.

[0071] Specifically, for the image encoding, transmission, and decoding processes, a structured pruning method based on joint evaluation of Feature Map Information Entropy and Reconstruction Sensitivity (FERS) is adopted. The specific process is as follows: Figure 4 As shown:

[0072] The output feature map F of the c-th channel of the JSCC encoder c ∈R H×W Calculate information entropy:

[0073]

[0074] Where H is the feature map F c The height of feature map F is W. c width, Let be the normalized energy probability distribution, where i and j are the row and column indices of the pixel positions in the feature map, respectively. The entropy value ε... c The larger the value, the more dispersed the feature value distribution (higher information richness), and the higher the channel importance.

[0075] The quantization of the reconstruction sensitivity of the JSCC decoder can be expressed as:

[0076]

[0077] Where, x n For the nth original input signal / image, For the original model, x n The reconstruction results For the pruned model on x n The reconstruction results are given, where PSNR is the peak signal-to-noise ratio and N is the total number of test samples.

[0078] Channel adaptive fusion can be expressed as:

[0079] Score c =α·Norm(ε c )+β·β(SNR)·Norm(ΔPSNR c )

[0080] Wherein, the channel factor β(SNR) = 1 - e -0.1·SNR Normalization function Score c The higher the final comprehensive importance score of the c-th channel, the more important the channel is and the more it should be retained during pruning; α and β are manually set weight coefficients, which are the weight coefficients of the information entropy term and the reconstruction sensitivity term, respectively.

[0081] Specifically, for the image classification process, the L1-norm pruning strategy is adopted.

[0082] S5.2: INT8 quantization optimization;

[0083] To overcome deployment bottlenecks on resource-constrained devices, this invention employs TensorRT's INT8 quantization-aware training and reduces quantization errors in JSCC complex features through dynamic range calibration of a calibration set. Specifically, this invention uses 8-bit quantization with the aid of TensorRT and a calibration set.

[0084] The 8-bit quantization process of the PyTorch framework model is as follows: Figure 5 (Engine refers to the inference engine; simply put, it's the "executable program" that runs at high speed in actual production after the model is "compiled" by TensorRT. It is the core of the entire model conversion and deployment process.) As shown:

[0085] S5.2.1: Load the trained PyTorch framework model and set it to inference mode, and prepare tensors consistent with the model input; convert the structure and weights of the PyTorch framework model to ONNX format through the ONNX export function of the PyTorch framework model, so as to accelerate inference on different inference engines or hardware.

[0086] S5.2.2: Using TensorRT, the two sub-models, after being split and converted to ONNX format, are calibrated separately using calibration sets, generating calibration files p1.cache (calibration table 1) and p2.cache (calibration table 2) for the two sub-models. However, since the calibration data for the sub-models only covers the quantization range of their respective parts, and the complete model requires unified calibration data to ensure global consistency during inference, the specific values ​​in the initially generated calibration files of the two sub-models need to be manually rewritten to the values ​​in the calibration file of the complete model.

[0087] S5.2.3: Read the calibration data in the calibration file, parse out the quantization parameters of each layer, then reorganize these parameters in JSON format, store them as key-value pairs, and generate a new JSON file to meet the format requirements of the calibration file when generating a file (NVDLA file) that can be directly loaded and executed by the FPGA.

[0088] In the above process, the calibration dataset plays a crucial role in generating quantization parameters during the INT8 precision conversion. During forward inference, by analyzing the dynamic range distribution characteristics of the activation tensors at each layer, the nonlinear mapping parameters between weights and activation values ​​(i.e., quantization scaling factor and zero-point offset) are accurately calculated. These parameters minimize the information loss between the original FP32 output and the INT8 quantized output, achieving a precision conversion from 32-bit floating-point numbers to 8-bit integers (-128 to 127). This significantly improves inference efficiency while maintaining the model's predictive power, and is the core mechanism for reducing quantization errors.

[0089] To achieve effective deployment of semantic communication models on IoT terminals, an end-to-end acceleration architecture from PyTorch to FPGA is implemented based on the Xilinx Vitis toolchain. The complete process of deploying the PyTorch framework model to the FPGA development board is as follows: Figure 6 As shown, the overall process starts with the model trained using the PyTorch framework. First, the PyTorch framework model is loaded and the corresponding input data is prepared. Then, using the conversion function of the PyTorch2caffe tool, the structure and weights of the PyTorch framework model are exported as two files: a caffemodel file describing the model's network structure and a prototxt file storing the model's weights. After the conversion, the generated Caffe model is compiled into an NVDLA format file using the nvdla_compiler tool, along with a JSON calibration table generated during quantization. This file is ultimately used for inference verification on the FPGA.

[0090] To verify the effectiveness of the present invention, the following experiments were conducted.

[0091] 1. Dataset and Environment;

[0092] This invention utilizes the Mini-ImageNet dataset in its experiments. This dataset contains 60,000 images, covering a wide variety of objects in IoT scenarios. Specifically, the dataset is divided into training, validation, and test sets in an 8:1:1 ratio. The semantic communication framework of this invention is implemented based on PyTorch and...

[0093] Training was performed on an NVIDIA RTX 4090 GPU. During training, the Adam optimizer was used with an initial learning rate of 1×10⁻⁶. -4 As training rounds progress, the learning rate gradually decreases to 1×102. -6 During model training, the input data is randomly cropped to 224×224 pixels, and data augmentation techniques such as flipping, rotation, and brightness adjustment are applied to improve the model's generalization ability. Furthermore, this invention uses Peak Signal-to-Noise Ratio (PSNR) and Structural Similarity Index (SSIM) to evaluate the model's reconstruction performance on the test dataset. Simultaneously, dynamic noise of 5-20 dB is dynamically injected during training to simulate common noise environments in IoT scenarios, thereby further enhancing the model's robustness.

[0094] The hardware platform used in the experiment of this invention is the VU13P, with a clock frequency of 50MHz. The VU13P FPGA development board is an advanced hardware development platform based on the Xilinx Virtex UltraScale series FPGA chip (specifically model: XCVU13P-FHGB2104). This chip integrates core resources such as large-capacity logic units, high-speed memory, and DSP slices, and connects to external devices through multiple high-speed interfaces (such as PCIe, QSFP28, and FMC), constructing a high-performance and highly flexible hardware development platform. In addition, the development board is typically equipped with large-capacity DDR4 SDRAM to support high-speed data processing and algorithm implementation. By providing multiple high-speed interfaces (such as PCIe Gen3x16, QSFP28, and FMC), it can meet the data transmission needs of diverse application scenarios.

[0095] 2. Results Analysis;

[0096] The performance of three pruning strategies—unpruned, L1-norm pruning, and FERS pruning—was compared in an additive white Gaussian noise (AWGN) channel (SNR = 0–20 dB), and was evaluated using PSNR and SSIM metrics, respectively.

[0097] The results are as follows Figure 7As shown, the unpruned model (Unpruned Model-SC) consistently delivers the best performance; at SNR = 15dB, its PSNR reaches 30.00dB and SSIM is 0.86, effectively preserving rich details with near-lossless performance. However, due to its high computational complexity, large number of parameters, and large model size, it is not suitable for direct deployment in IoT devices. In contrast, L1-norm pruning (L1-norm Pruning Model-SC) exhibits significant performance degradation across all SNR levels, with a PSNR approximately 3dB lower than the unpruned model, indicating that its single-threshold pruning strategy leads to the loss of key features. FERS pruning (FERS Pruning Model-SC (Proposed)) demonstrates balanced performance within the SNR range of 10–20dB, outperforming not only L1-norm pruning but also approaching the performance of the unpruned model (at SNR = 20dB, the difference is only approximately 1.5% PSNR / 1.2% SSIM). This result fully demonstrates that FERS, a sensitivity-based adaptive pruning method, can achieve a good balance between model complexity and reconstruction accuracy in medium SNR environments, while ensuring the feasibility of hardware deployment. However, in low...

[0098] In scenarios with SNR (<3dB), FERS pruning actually performs worse than L1-norm pruning. This may be because the preserved high-frequency channels amplify noise, thus masking key signals. Overall, these results demonstrate that sensitivity-based pruning techniques such as FERS can effectively balance model complexity and reconstruction performance in medium SNR environments while maintaining the practicality of hardware deployment.

[0099] Table 1 more intuitively illustrates the reconstruction metrics of different pruning strategies in a 15dB-AWGN channel environment. The image reconstructed after L1-norm pruning (PSNR 26.79dB / SSIM 0.77) shows serrated beak edges and blurred feather texture. The image reconstructed by FERS (PSNR 29.08dB / SSIM 0.84) accurately preserves the details of the duck's eye, feather boundaries, and the continuity of background leaf veins, showing high consistency with the unpruned model (PSNR 29.78dB / SSIM 0.86). While the FPGA-based version (PSNR 28.18dB / SSIM 0.81) exhibits slight artifacts at the feather tips, it still effectively maintains the structural integrity of the duck's comb, fully verifying the efficiency of this invention in hardware implementation.

[0100] Table 1 Reconstruction metrics of different pruning strategies in a 15dB-AWGN channel environment

[0101] Original image Unpruned L1-norm FEARS FPGA deployment image Figure 8 (a) Figure 8 (b) Figure 8 (c) Figure 8 (d) Figure 8 (e) PSNR - 29.78dB 26.79dB 29.08dB 28.18dB SSIM - 0.86 0.77 0.84 0.81

[0102] Table 2 shows the model metrics for different lightweight strategies when SNR = 15dB in an AWGN channel.

[0103] Unprocessed Pruning 8-bit quantization Pruning + 8-bit quantization FPGA deployment Parameters 24.12M 13.36MB 24.12M 13.36M 13.36M accuracy 73.25％ 72.95％ 73.05％ 72.2％ 71.5％ Reasoning time 4.47ms 4.44ms 3.97ms 3.64ms 1.45s Model size 92.35MB 51.26MB 24.23MB 13.83MB 13.83MB

[0104] As shown in Table 2, this invention compares the effects of pruning, 8-bit quantization, and their combinations across four dimensions: image quality, model efficiency, inference speed, and accuracy. FERS pruning, by analyzing feature map entropy and reconstruction sensitivity, reduces the model parameters from 24.12M to 13.36M (a 44.6% decrease), while simultaneously compressing the model size to 51.26MB (a 44.4% decrease). TensorRT-based 8-bit quantization, while maintaining the same number of parameters, further reduces the model size by 74.9% (to 24.23MB) with only a 0.2% loss in accuracy. Finally, combining the two methods (pruning + 8-bit quantization) achieves a significant compression of 85.2% (to 13.83MB).

[0105] This invention addresses the structural characteristics of semantic communication models and combines lightweight and accelerated collaborative optimization with hardware deployment constraints. While significantly reducing model parameters and computational load, it maintains high semantic task accuracy and reconstruction quality, thus solving the technical bottleneck of the feasibility and effectiveness of semantic communication models on resource-constrained IoT terminal devices.

[0106] The above description is only a preferred embodiment of the present invention. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the principle of the present invention, and these improvements and modifications should also be considered within the scope of protection of the present invention.

Claims

1. A semantic communication model lightweight deployment method for model algorithm hardware collaborative optimization, characterized in that, Includes the following steps: (1) Design a structured pruning method based on joint evaluation of feature map information entropy and reconstruction sensitivity to prune the encoding and decoding structure of semantic communication model; (2) TensorRT INT8 quantization perception training was used, and dynamic range calibration was performed using a calibration set; (3) Based on the Xilinx Vitis toolchain, an end-to-end acceleration architecture from PyTorch to FPGA is implemented, enabling end-to-end deployment and inference acceleration on VU13PFPGA, providing a high-efficiency semantic communication prototype for resource-constrained IoT devices.

2. The semantic communication model lightweight deployment method for model algorithm hardware co-optimization according to claim 1, characterized in that, The JSCC encoder in the semantic communication model adopts a convolutional neural network-based encoder architecture, consisting of a series of convolutional layers, ReLU activation functions, and normalization layers. The convolutional layers are used to extract image features and combine them layer by layer to form channel input samples. The ReLU activation function enables the model to learn the nonlinear mapping from the source signal space to the encoded signal space. The output of the last convolutional layer is normalized to obtain the channel input samples.

3. The lightweight deployment method for semantic communication models with hardware co-optimization of model algorithms according to claim 1, characterized in that, The JSCC encoder in the semantic communication model uses a function Map an n-dimensional input image x to a channel input sample of length k. in Let n be a real vector space. Let z be a k-dimensional complex vector space, and z be a complex field sample, satisfying the average transmit power constraint. P is the average transmit power constraint. For the mathematical expectation, z * It is the conjugate transpose of z. For the encoding function f θ The direct output vector, yes The conjugate transpose of .

4. The lightweight deployment method for semantic communication models with hardware co-optimization of model algorithms according to claim 1, characterized in that, The JSCC decoder in the semantic communication model transforms the operations performed by the JSCC encoder through a series of transposed convolutions with nonlinear activation functions, mapping image features to estimates of the original transmitted image.

5. The lightweight deployment method for semantic communication models with hardware co-optimization of model algorithms according to claim 1, characterized in that, The JSCC encoder in the semantic communication model maps the input image to channel input samples z and transmits them through a physical channel η. This physical channel η introduces random corruption into the transmitted symbols. This means that the physical channel η is modeled as a series of untrainable layers, defined by the transfer function. express, This is a corrupted, complex signal.

6. The lightweight deployment method for semantic communication models with hardware co-optimization of model algorithms according to claim 1, characterized in that, In the structured pruning method based on joint evaluation of feature map information entropy and reconstruction sensitivity, the output feature map F of the cth channel of the JSCC encoder in the semantic communication model is c ∈R H×W Calculate the information entropy: Where H is the feature map F c The height of feature map F is W. c width, The normalized energy probability distribution is defined as follows: i and j are the row and column indices of the pixel positions in the feature map, respectively, and the entropy value ε is... c The larger the value, the more dispersed the feature value distribution, the higher the information richness, and the higher the channel importance.

7. The lightweight deployment method for semantic communication models with hardware co-optimization of model algorithms according to claim 1, characterized in that, In the structured pruning method based on the joint evaluation of feature map information entropy and reconstruction sensitivity, the quantification of the reconstruction sensitivity of the JSCC decoder in the semantic communication model is as follows: Where, x n For the nth original input signal / image, For the original model, x n The reconstruction results For the pruned model on x n The reconstruction results are given, where PSNR is the peak signal-to-noise ratio and N is the total number of test samples.

8. The lightweight deployment method for semantic communication models with hardware co-optimization of model algorithms according to claim 1, characterized in that, In the structured pruning method based on joint evaluation of feature map information entropy and reconstruction sensitivity, the channel adaptive fusion is represented as: Score c = a - Norm (e c ) + b - b (SNR) - Norm (APSNR c ) Wherein, the channel factor β(SNR) = 1 - e -0.1·SNR Normalization function Score c The higher the final comprehensive importance score of the c-th channel, the more important the channel is and the more it should be retained during pruning; α and β are manually set weight coefficients, which are the weight coefficients of the information entropy term and the reconstruction sensitivity term, respectively.

9. The lightweight deployment method for semantic communication models with hardware co-optimization of model algorithms according to claim 1, characterized in that, The specific implementation process of step (2) is as follows: 1) Load the trained PyTorch framework model and set it to inference mode, and prepare tensors consistent with the model input; convert the structure and weights of the PyTorch framework model to ONNX format using the ONNX export function of the PyTorch framework model. 2) Use TensorRT to calibrate the two sub-models after splitting and converting to ONNX format using the calibration set, generating calibration files p1.cache and p2.cache for the two sub-models respectively; manually rewrite the specific values ​​in the calibration files of the two sub-models generated in the first generation to the values ​​in the calibration file of the complete model. 3) Read the calibration data in the calibration file, parse out the quantization parameters of each layer, reorganize these parameters in JSON format, store them as key-value pairs, and generate a new JSON file to meet the format requirements of the calibration file when generating a file that can be directly loaded and executed by the FPGA.

10. The lightweight deployment method for semantic communication models with hardware co-optimization of model algorithms according to claim 9, characterized in that, The specific implementation process of step (3) is as follows: Load the PyTorch framework model and prepare the corresponding input data; use the conversion function of the PyTorch2caffe tool to export the structure and weights of the PyTorch framework model into two files: caffemodel and prototxt; use the JSON calibration table generated during quantization to compile the generated Caffe model into an NVDLA format file using the compiler tool nvdla_compiler, which is then used for inference verification on the FPGA.

Images