Lightweight video semantic communication method

Abstract

The invention relates to a lightweight video semantic communication method, and belongs to the field of communication and artificial intelligence. The invention aims to solve the problem that the transmission efficiency and reliability of the existing video transmission method are reduced when bandwidth resources are limited or channel conditions are bad. According to the method, a deep neural network is adopted, motion vectors and residual errors between video frames are modeled, motion estimation and compensation are achieved, and stable transmission performance can be ensured under the condition of a low signal-to-noise ratio by using a joint source channel coder-decoder based on Swin Transformer and ECA-Net. According to the invention, a lightweight optical flow estimation network and an efficient signal-to-noise ratio adaptive module are introduced, so that the model complexity and the reasoning time are reduced, and the method is more suitable for edge equipment. According to the method, the cliff effect of traditional communication is overcome, the performance superior to that of an existing model and a traditional communication system is shown under the low to medium signal-to-noise ratio level, and the method has the advantages of being low in complexity and short in reasoning time.

Description

Technical Field

[0001] The invention belongs to the field of communication and artificial intelligence, and relates to a lightweight video semantic communication method. Background Art

[0002] With the advent of the Internet of Things era, IoT devices need to perceive video information in real time, resulting in the need to transmit huge traffic. However, the current bandwidth increase rate is far less than the traffic increase rate, so video compression is widely used to reduce the bandwidth requirements when transmitting video signals wirelessly. In modern wireless networks, video transmission tasks are usually divided into two main steps: source coding and channel coding. Source coding compresses the original video into a bit sequence, while channel coding converts these bit sequences into signals suitable for transmission through wireless channels (such as noise, fading, and interference). This separation-based approach is widely used in many fields because it can seamlessly transmit various source data on any wireless channel by changing the underlying channel coding. A significant advantage of this approach is that each component can be optimized independently, thereby improving the overall system performance. H.264 and H.265 are the current mainstream video source coding standards. When combined with the LDPC (Low-Density Parity-Check) channel coding scheme, they can significantly improve the efficiency and reliability of video transmission. These standards implemented using commercial software, with the support of highly parallel frameworks and optimized assembly code, can achieve faster encoding speeds. The H.265+LDPC scheme runs at a speed of 40ms to 600ms under different encoding settings (trading between encoding efficiency and encoding speed), while the encoding speed of H.264+LDPC is 6ms to 125ms. In contrast, the official reference software may run hundreds of times slower because it focuses more on the correct implementation of the standard without large-scale optimization.

[0003] In addition, with the widespread application and in-depth development of wireless communication technology, the traditional communication system architecture designed based on the separation principle has gradually revealed its inherent limitations. This separation design, that is, treating source coding and channel coding as two independent links, is convenient for optimization and innovation in their respective fields in theoretical architecture, but in actual wireless transmission environments, it is easy to cause an extreme situation called the "cliff effect". Specifically, when the wireless channel conditions deteriorate sharply and its performance drops below the preset threshold of the channel encoder, the source information that has been carefully compressed will become difficult to recover or even completely lost because it cannot effectively combat the noise and distortion in the channel, resulting in a sudden interruption of the communication link and a sharp decline in user experience.

[0004] Based on the powerful capabilities of joint source-channel coding and video compression technology using deep learning, some video semantic communication systems have been proposed, showing the potential to surpass traditional communication methods: DeepWiVe is the first end-to-end deep neural network joint source-channel coding video transmission scheme, which verifies that the joint source-channel coding method can avoid the cliff effect. DVST is a new end-to-end learnable wireless video transmission framework based on conditional coding of video frames. This coding method has powerful performance, but the neural network complexity is extremely high, resulting in DVST taking 280ms to transmit a video frame in a system with higher computing power.

[0005] At present, many edge devices (such as IoT devices) still do not have sufficient computing resources, which makes these devices unable to afford more complex neural network models. Therefore, lightweight video semantic communication models that can realize basic communication tasks are urgently needed by edge devices. However, there is currently no lightweight video semantic communication system to support it. Summary of the invention

[0006] In view of this, an object of the present invention is to provide a lightweight video semantic communication method.

[0007] In order to achieve the above object, the present invention provides the following technical solutions:

[0008] A lightweight video semantic communication method, the method comprising:

[0009] Input the current frame and the previous frame of the transmitter into the motion estimation network to generate motion vectors;

[0010] The generated motion vector and the previous frame are input into the motion compensation network to generate a predicted frame;

[0011] Subtract the current frame from the predicted frame to get the residual;

[0012] Concatenate the residual with the motion vector in the channel dimension;

[0013] Input the concatenated vector into a joint source-channel encoder and output an encoded vector;

[0014] transmitting the coded vector via a wireless channel;

[0015] Input the received encoding vector into a joint source-channel decoder and output a decoding vector;

[0016] Split the decoded vector into residual and motion vector;

[0017] The received residual enters the semantic refinement module to reduce the semantic error of the residual; the semantic refinement module is a U-Net style convolutional neural network based on deep separable convolution, which captures different semantic information of the image by gradually reducing and expanding the spatial resolution of the feature map to achieve image repair and reconstruction;

[0018] The received motion vector and the reconstructed previous frame are input into the motion compensation network to generate a reconstructed prediction frame;

[0019] The reconstructed prediction frame is added to the refined residual to obtain the reconstructed current frame.

[0020] Furthermore, the motion estimation network includes multiple convolutional neural networks, each of which calculates the residual optical flow of different levels of the pyramid.

[0021] Furthermore, the motion compensation network includes a warp operator and a convolutional neural network.

[0022] Furthermore, the joint source-channel encoder includes a semantic encoder and a channel encoder.

[0023] Furthermore, the semantic encoder is based on the Swin Transformer architecture.

[0024] Furthermore, the channel encoder includes a channel signal-to-noise ratio (SNR) adaptation module and a residual network (Residual Network, ResNet).

[0025] Furthermore, the SNR adaptation module is based on the Efficient Channel Attention Network (ECA-Net) architecture.

[0026] Furthermore, the joint source-channel decoder comprises a channel decoder and a semantic decoder.

[0027] Furthermore, the channel decoder includes a ResNet and an SNR adaptation module.

[0028] Furthermore, the semantic decoder is based on the Swin Transformer architecture.

[0029] The beneficial effects of the present invention are:

[0030] (1) This method can effectively avoid the "cliff effect" commonly seen in traditional communication systems and maintain stable transmission performance under low signal-to-noise ratio conditions, thus ensuring reliable transmission of video information in harsh channel environments.

[0031] (2) By introducing a lightweight optical flow estimation network and an efficient signal-to-noise ratio adaptation module, this method significantly reduces the model complexity and inference time, making it more suitable for edge devices with limited computing resources.

[0032] (3) This method can effectively compress video data, reduce bandwidth requirements, and use joint source channel coding technology to improve transmission efficiency and achieve faster video transmission.

[0033] (4) Through motion estimation and compensation technology, as well as a semantic codec based on Swin Transformer, this method can achieve high-quality reconstructed video and ensure the clarity and coherence of the video content.

[0034] (5) The lightweight design of this method enables faster inference time, which can meet the needs of real-time video transmission and is suitable for delay-sensitive application scenarios.

[0035] Other advantages, objectives and features of the present invention will be described in the following description to some extent, and to some extent, will be obvious to those skilled in the art based on the following examination and study, or can be taught from the practice of the present invention. The objectives and other advantages of the present invention can be realized and obtained through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] In order to make the purpose, technical solutions and advantages of the present invention more clear, the present invention will be described in detail below in conjunction with the accompanying drawings, wherein:

[0037] Figure 1 This is a system architecture diagram of the present invention;

[0038] Figure 2 is a structural diagram of a motion estimation network in the present invention;

[0039] Figure 3 is a structural diagram of a motion compensation network in the present invention;

[0040] Figure 4 This is a CNN network structure diagram in the motion compensation network of the present invention;

[0041] Figure 5 is a structural diagram of a joint source-channel codec in the present invention;

[0042] Figure 6 is a structural diagram of the semantic codec in the present invention; Figure 6 (a) is the semantic encoder; Figure 6 (b) is the semantic decoder;

[0043] Figure 7 It is a structural diagram of the SNR adaptation module in the present invention;

[0044] Figure 8 is a structural diagram of a channel codec in the present invention; Figure 8 (a) is a structural diagram of a channel encoder; Figure 8 (b) is a structural diagram of a channel decoder;

[0045] Fig. 9 It is a structural diagram of the semantic refining module in the present invention;

[0046] Fig.10 Performance curves of lightweight video semantic communication system with different compression rates on AWGN channel; Fig.10 (a) is the PSNR indicator; Fig.10 (b) is the MSSSIM indicator;

[0047] Fig.11 PSNR curves between the lightweight video semantic communication system and the traditional communication system with different SNR values ​​on the AWGN channel. DETAILED DESCRIPTION

[0048] The following describes the embodiments of the present invention by specific examples, and those skilled in the art can easily understand other advantages and effects of the present invention from the contents disclosed in this specification. The present invention can also be implemented or applied through other different specific embodiments, and the details in this specification can also be modified or changed in various ways based on different viewpoints and applications without departing from the spirit of the present invention. It should be noted that the illustrations provided in the following embodiments only illustrate the basic concept of the present invention in a schematic manner, and the following embodiments and features in the embodiments can be combined with each other without conflict.

[0049] Among them, the drawings are only used for illustrative explanations, and they only represent schematic diagrams rather than actual pictures, and should not be understood as limitations on the present invention. In order to better illustrate the embodiments of the present invention, some parts of the drawings may be omitted, enlarged or reduced, and do not represent the size of actual products. For those skilled in the art, it is understandable that some well-known structures and their descriptions in the drawings may be omitted.

[0050] The same or similar numbers in the drawings of the embodiments of the present invention correspond to the same or similar parts; in the description of the present invention, it should be understood that if the terms "upper", "lower", "left", "right", "front", "rear", etc. indicate the orientation or position relationship, they are based on the orientation or position relationship shown in the drawings, which is only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific orientation, be constructed and operate in a specific orientation. Therefore, the terms describing the position relationship in the drawings are only used for illustrative purposes and cannot be understood as limiting the present invention. For ordinary technicians in this field, the specific meanings of the above terms can be understood according to specific circumstances.

[0051] The present invention proposes a lightweight and efficient deep joint source channel coding system for end-to-end video transmission over wireless channels. The system uses deep neural networks to model motion vectors and residuals between video frames, thereby achieving effective motion estimation and compensation. It adopts a joint source channel codec based on Swin-transformer and ECA-Net, which can ensure robust transmission performance even under low signal-to-noise ratio (SNR) conditions. The innovation of this paper lies in the introduction of a lightweight optical flow estimation network and an efficient signal-to-noise ratio adaptation module, which significantly reduces the model complexity and reasoning time. The system was trained and tested using Vimeo-90k, UVG ​​and HEVC datasets. The results show that the system can not only overcome the cliff effect of traditional communications well, but also has very lightweight characteristics and extremely fast reasoning speed. In the specific implementation, the present invention will be described in detail from six parts: system architecture composition, motion estimation network structure, motion compensation network structure, joint source channel codec structure, semantic codec structure, SNR adaptation network structure, and channel codec structure.

[0052] 1. The architecture diagram of lightweight video semantic communication system is as follows Figure 1 shown.

[0053] S11: First, the current frame x t The previous frame x of the sender t-1 The motion estimation network is passed in for motion estimation to generate a motion vector m t , where θ me represents the parameters of the motion estimation network;

[0054] m t =f me (x t ,x t-1 ,θ me )

[0055] S12: Generated motion vector m t With the previous frame x t-1Enter the motion compensation network to generate predicted frames where θ mc-t Represents the parameters of the motion compensation network at the sending end;

[0056]

[0057] S13: Current frame x t With predicted frame Subtract the residual r t ;

[0058]

[0059] S14: residual r t With the motion vector m t Cascading in the channel dimension;

[0060] s t =concat(r t ,m t )

[0061] S15: cascaded vector s t Enter the joint source channel encoder, where θ se ,θ ce Represent the parameters of semantic encoder and channel encoder respectively, SNR represents signal-to-noise ratio;

[0062] z t =f ce (f se (s t ,θ se ),SNR,θ ce )

[0063] S16: The vector zt of the joint source channel encoder propagates in the channel and is affected by noise, where h represents the channel transmission matrix, N~CN(0,σ2) represents the wireless channel AWGN, and σ2 represents the noise variance;

[0064] z′ t =hz t +N

[0065] S17: At the receiving end, the vector z′ after passing through the physical channel t Enter the joint source channel decoder, where θ sd ,θ cd represents the parameters of semantic decoder and channel decoder, SNR represents signal-to-noise ratio;

[0066] s′ t =f sd (f cd (z′ t ,SNR,θcd ),θ sd )

[0067] S18: vector s′ after passing through the joint source channel decoder t is divided into the residual r′ t and motion vector m′ t ;

[0068] m′ t ,r′ t =divide(s′ t )

[0069] S19: Received motion vector m′ t and the reconstructed previous frame x′ t-1 Enter the motion compensation network at the receiving end to generate a predicted frame where θ mc-r Represents the parameters of the motion compensation network at the receiving end;

[0070]

[0071] S110: Received residual r′ t After the semantic refinement module, we get r″ t , further reducing its semantic error;

[0072] r″ t = refine(r′ t )

[0073] S111: Finally, the predicted frame generated by the receiving end and the refined residual r″ t Add to get the reconstructed current frame x′ t ;

[0074]

[0075] S112: Different from traditional communication systems, semantic communication systems consider the original video frame x in the semantic domain. t and reconstruct the video frame x′ t To determine whether there is a semantic gap between the two, this patent uses mean square error (MSE) to model the loss value:

[0076] L=L MSE (x′ t ,x t )

[0077] 2. The motion estimation network structure diagram is as follows Figure 2 shown.

[0078] The present invention uses a coarse-to-fine spatial pyramid structure to learn the residual flow of each pyramid level. Here, the reasoning process of the network is briefly described.

[0079] S21: Let {G0,...,G K} represents a set of trained convolutional neural network (ConvNet) models, each of which computes the residual optical flow at the kth level of the pyramid:

[0080]

[0081] S22: Convolutional Neural Network G k Use the value from the previous pyramid level m k-1 The upsampled stream and the frame of the kth layer To calculate the residual flow v k . In the second frame Sent to the convolutional neural network G k Before, use as The stream to distort the second frame The flow m at the kth pyramid level k :

[0082] m k =u(m k-1 )+v k

[0083] S23: Figure 2 As shown, from the downsampled image We start with an initial optical flow estimate with sum = zero to compute the residual flow m0 = v0 + 0 at the top of the pyramid. We upsample the resulting flow u(m0) and compare it with Pass them together to the convolutional neural network G1 to calculate the residual flow v1. At each pyramid level, use the formula m k =u(m k-1 )+v k Calculate flow m k . m k The same is propagated to the higher resolution layers of the pyramid until the full resolution stream is obtained. The working principle of the method using a 3-level pyramid is shown in the figure. In practice, a 5-level pyramid (K=4) is used.

[0084] 3. The motion compensation network structure diagram is as follows Figure 3 shown.

[0085] S31: Given a previously reconstructed frame x t-1 and motion vector v t , the motion compensation network obtains the expected and current frame x t The predicted frames are as close First, the previous frame x t-1Based on motion information t is warped to the current frame as follows:

[0086]

[0087] in, is the warping frame and w is the backward warping operation.

[0088] S32: To further improve the distortion frame The quality will and the reference frame x t-1 concatenate them as input and then feed them into another CNN to get a refined prediction frame The motion compensation network structure of the present invention is shown in 3, and the CNN network structure is as follows Figure 4 shown.

[0089] S33: In this CNN network, the data flow starts from the current frame and the previous frame x t-1 At the beginning, they are all input into the initial convolutional layer (Conv(3, 64, 1)) to extract basic features. These features are processed through a series of residual blocks, where skip connections allow information to bypass layers and prevent it from being lost during deeper processing. The network uses two pooling layers to reduce the feature map resolution, thereby capturing more global context. In the second half, the upsampling layer restores the resolution and additional residual blocks refine the features. Finally, a 3×3 convolutional layer (Conv(3, 3, 1)) outputs the reconstructed frame Throughout the process, skip connections ensure that both high-level and low-level information are effectively preserved and combined.

[0090] 4. The structure of the joint source-channel codec in the present invention is as follows: Figure 5 shown.

[0091] S41: residual r t With the motion vector m t After cascading, it will pass through the joint source channel codec system. The joint source channel codec of the present invention includes the semantic codec and channel codec mentioned above, which complete the function of feature extraction and compression of feature tensor and channel adaptation, and is the core component of the video semantic communication system, and is also the key to overcome the cliff effect and achieve elegant performance degradation.

[0092] 5. The semantic codec structure of the present invention is as follows Figure 6 shown.

[0093] The semantic codec of the present invention is built on the basis of the most advanced visual transformer Swin Transformer. Compared with ViT, Swin Transformer has some improvements. It introduces a windowed attention mechanism and multi-level feature representation when processing images. These improvements improve the model's ability to handle large-size images and complex visual patterns. Through more efficient information exchange and hierarchical processing, the performance on various visual tasks is significantly improved. In addition, the computational complexity of Swin Transformer is linear with the image size, while for the traditional ViT model, its computational complexity is quadratic.

[0094] S51: This structure uses three layers of Swin transformer layers for feature extraction. The number of Swin transformer blocks in each layer is 2, 4, and 2 respectively. The number of attention heads of the Swin transformer blocks in each layer is also different, which are 4, 8, and 4 respectively.

[0095] S52: Figure 6 As shown in (a), in the semantic encoder, a patch merging step is performed before each Swin transformer layer. This operation reduces the spatial dimension of the data by downsampling and merging adjacent feature blocks, thereby reducing the computational burden for further deep processing.

[0096] S53: Figure 6 As shown in (b), in the semantic decoder, each time after passing through the Swin transformer layer, it will go through a Patch segmentation step. This step is actually to refine and upsample the feature data. Through the segmentation operation, the features are distributed in a wider space, preparing for further detailed feature recovery.

[0097] 6. The SNR adaptation module structure of the present invention is as follows Figure 7 shown.

[0098] Most of the existing SNR adaptation modules are based on channel attention, such as the image semantic communication system ADJSCC, which originally proposed the SNR adaptation module, and the latest image semantic communication system WITT; such modules rely on a large number of fully connected layers to implement their functions. These fully connected layers are used to transform and modulate the input features so that according to specific conditions, this inevitably increases the complexity of the model.

[0099] Most existing methods are committed to developing more complex attention modules to obtain better performance, which inevitably increases the complexity of the model. In order to overcome the contradiction between performance and complexity, the SNR adaptation module of the present invention is based on an efficient channel attention (ECA) method, which adopts a local cross-channel interaction strategy without dimensionality reduction, which can be effectively implemented by one-dimensional convolution. The module requires only a few parameters and can bring performance improvement.

[0100] S61: The SNR adaptation module of the present invention abandons a large number of fully connected layers with high parameter counts and uses one-dimensional convolution instead, which greatly reduces the number of parameters while ensuring a slight improvement in performance.

[0101] S62: The data is first subjected to global average pooling to generate channel descriptors, which are then concatenated with the signal-to-noise ratio (SNR). Subsequently, local cross-channel interactions are performed through one-dimensional convolution (kernel size = 3), and finally, weights are generated using a sigmoid function and multiplied with the data. This enables a channel-wise attention mechanism, thereby enhancing the adaptability of the system under varying SNR conditions. The SNR adaptation module of the present invention is shown in FIG. Figure 7 shown.

[0102] 7. The channel codec structure of the present invention is as follows Figure 8 As shown, Figure 8 (a) is the structure diagram of the channel encoder. Figure 8 (b) is the channel decoder structure diagram:

[0103] The present invention cascades the SNR adaptation module and Resnet to form the channel codec module of the present invention, wherein Resnet can effectively avoid the gradient disappearance problem caused by the network becoming very deep.

[0104] 8. The structure diagram of the semantic refining module in the invention is as follows: Fig. 9 shown.

[0105] In the present invention, the residual transmitted through the channel and decoded at the receiving end will eventually be directly added to the predicted frame to generate the final frame. In this process, the information in the residual directly affects the final imaging quality, and the residual directly decoded through the channel will inevitably produce a small semantic error. Therefore, the residual decoded by the receiving end can pass through a semantic refinement module, which aims to further reduce the semantic error of the residual and ultimately improve the overall imaging quality.

[0106] The semantic refinement module is a U-Net style convolutional neural network based on deep separable convolution. Specifically, it captures different semantic information of the image by gradually reducing and expanding the spatial resolution of the feature map, thereby achieving image restoration and reconstruction.

[0107] In order to adapt to the lightweight and fast features of this model, this network uses Depthwise Separable Convolution to reduce the computational complexity while maintaining efficient feature extraction.

[0108] Residual Convolutional Block (RCB) is a convolutional neural network module that extracts information from input features through two convolutional layers and their subsequent batch normalization and ReLU activation functions, and combines residual connections to retain the original features, thereby effectively alleviating the gradient vanishing problem and improving the network training efficiency.

[0109] The skip connection structure of this network is similar to U-Net, which enables the network to retain and utilize low-level features by combining the features of the corresponding positions in the downsampling stage in the upsampling stage. This is particularly important for image restoration tasks because low-level features contain detailed information of the input image.

[0110] 9. The training details of the present invention are as follows:

[0111] Based on the above DNN and loss function, end-to-end semantic video transmission can be effectively realized. In order to improve the training efficiency of the lightweight video semantic communication system, the present invention uses a motion estimation module with pre-trained weights, and the parameters of the remaining modules are all generated by training. During the model training process, the present invention uses the data in the data set as input and jointly trains lightweight video semantic communication on different channels with different signal-to-noise ratios.

[0112] The present invention uses the Vimeo-90k dataset for training, which consists of multiple independent video clips, including various scenes and actions, and is used to evaluate different video processing tasks. Before training, the dataset is cropped to 256*256 pixels; in addition, in order to facilitate training and accelerate convergence, Gaussian white noise with a mean of 0 and a variance of 0.05 is added to the dataset during training; the Adam optimizer is used, and the learning rate is set to 2.5×10-5; the training SNR = {0, 2, 4, 6, 8, 10, 12, 14, 16}, each SNR is trained for five rounds, a total of 45 rounds, and the model is saved once for a certain number of steps; the Pytorch deep learning framework and NVIDIA40608G GPU are used.

[0113] 10. The model test details and some test results of the present invention are as follows:

[0114] Considering the generalization ability of the system and its universality at various resolutions, the present invention uses the UVG dataset different from the training dataset to evaluate the performance of the lightweight video semantic communication system. The UVG dataset is a widely used standard dataset for measuring the performance of video-related algorithms, including 7 high-frame-rate videos with a resolution of 1920×1080.

[0115] The present invention uses the Channel Bandwidth Ratio to measure the transmission rate of the communication system. The image dimension n is called the source bandwidth, and the channel dimension k is called the channel bandwidth. The ratio k / n is the channel bandwidth ratio (usually k < n). In this paper, the input image is an RGB image, so n = h×w×3, and in schemes other than the ideal channel capacity, k represents the number of analog symbols transmitted in the wireless channel.

[0116] The present invention uses two common distortion criteria, Peak Signal-to-Noise Ratio (PSNR) and Multi-Scale Structural Similarity (MS-SSIM), to measure the reconstruction quality of video frames. Usually, the Peak Signal-to-Noise Ratio is used to evaluate the difference between two video frames at the pixel level. The larger the Peak Signal-to-Noise Ratio, the higher the pixel similarity between the two video frames. The MS-SSIM measures the similarity between two video frames in the multi-scale structure, and the MS-SSIM produces a value between 0 (worst) and 1 (best).

[0117] Fig.10 Figure for the performance curves of the lightweight video semantic communication system with different compression ratios on the AWGN channel; on the AWGN channel, three system models with different channel bandwidth ratios of 0.01, 0.02, and 0.04 are prepared; the test video first uses UVG-Bosphorus, 1920×1080, GOP = 4 (one I-frame and three P-frames in each group of pictures), and the PSNR and MSSSIM test values are the average of the last three P-frames. It can be seen that the present invention overcomes the cliff effect of traditional communication and achieves an elegant performance degradation, as shown in Fig.10 (a) and Fig.10 (b).

[0118] The present invention is also compared with the classic video coding transmission scheme in the current mainstream wireless communication system. In particular, standard video codecs (H.264 and H.265) are used for source coding, and practical LDPC codes are used for channel coding. For simplicity, "+" is used to connect source coding, channel coding and modulation schemes, which are H.265+3 / 4LDPC+16QAM, H.264+3 / 4LDPC+16QAM, H.265+1 / 2LDPC+16QAM, H.264+1 / 2LDPC+16QAM; ffmpeg is used to perform H.265 encoding and H.264 encoding on UVG-Bosphorus video frames to generate bit files. To ensure fairness, GOP is also set to 4 during encoding, and the CBR of these traditional means is also controlled to 0.02.

[0119] It can be seen that as the signal-to-noise ratio decreases, the present invention achieves graceful performance degradation and overcomes the cliff effect of the traditional communication system. Since the present invention is implemented using a low-complexity neural network, the performance in the higher signal-to-noise ratio area is slightly lower than that of the traditional communication, such as Fig.11 shown.

[0120] In addition, the size of the model weights and the inference time are crucial, especially for edge devices with limited computing power. The main advantages of the present invention lie in these aspects. The total number of parameters of the present invention is 30.1M, which is very lightweight; in addition, the average inference time for transmitting a frame by the lightweight video semantic communication method was tested on a server equipped with an Intel(R) Core(TM) i5-12400F CPU and an RTX 4060GPU was 14ms, which is much faster than traditional communication methods such as the H.254+LDPC method and the H.265+LDPC method.

[0121] Due to the use of lightweight optical flow estimation network and joint source information using lightweight channel coding

[0122] The coding system of the present invention has low weight and much faster reasoning time than traditional communication means. It is the lightest and fastest semantic communication system at present, and is very suitable for use in edge devices.

[0123] This paper introduces a lightweight video semantic communication system, an innovative lightweight framework designed for the limited computing resources of edge devices. A key innovation of the system is the integration of a deep learning-based residual coding method, which has a simple structure and low complexity. In addition, the system also uses an advanced semantic codec and a channel codec. The semantic codec utilizes the state-of-the-art Swin Transformer architecture to enhance the system's ability to handle large images and complex visual patterns, while the channel codec integrates an ECA-Net-based SNR adaptation module, which is less complex and more effective than traditional SNR adaptation modules. The synergy of these technologies significantly reduces computational complexity and improves inference speed, making the system very suitable for deployment on resource-constrained devices.

[0124] Through extensive experimental evaluation, the superior performance of the lightweight video semantic communication system in mitigating the “cliff effect” commonly seen in traditional communication systems, which can cause a sudden drop in performance under certain SNR conditions, has been demonstrated. The lightweight video semantic communication system exhibits stable performance across a wide range of video resolutions and channel conditions, including AWGN and fading channels such as Rayleigh and Rice. Comparative analysis with existing models and traditional communication systems further highlights the superior performance of the lightweight video semantic communication system at low to moderate SNR levels. Although its performance may be slightly inferior to more complex systems at very high SNR levels, the low complexity and fast inference time of the lightweight video semantic communication system provide significant advantages in practical applications, making it an ideal choice for real-time video transmission in edge computing applications.

[0125] Finally, it should be noted that the above embodiments are only used to illustrate the technical solution of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solution of the present invention can be modified or replaced by equivalents without departing from the purpose and scope of the technical solution, which should be included in the scope of the claims of the present invention.

Claims

1. A lightweight video semantic communication method, characterized in that: The method includes: Input the current frame and the previous frame of the transmitter into the motion estimation network to generate motion vectors; The generated motion vector and the previous frame are input into the motion compensation network to generate a predicted frame; Subtract the current frame from the predicted frame to get the residual; Concatenate the residual with the motion vector in the channel dimension; Input the concatenated vector into a joint source-channel encoder and output an encoded vector; transmitting the coded vector via a wireless channel; Input the received encoding vector into a joint source-channel decoder and output a decoding vector; Split the decoded vector into residual and motion vector; The received residual enters the semantic refinement module to reduce the semantic error of the residual; the semantic refinement module is a U-Net style convolutional neural network based on deep separable convolution, which captures different semantic information of the image by gradually reducing and expanding the spatial resolution of the feature map to achieve image repair and reconstruction; The received motion vector and the reconstructed previous frame are input into the motion compensation network to generate a reconstructed prediction frame; The reconstructed prediction frame is added to the refined residual to obtain the reconstructed current frame.

2. A lightweight video semantic communication method according to claim 1, characterized in that: The motion estimation network includes multiple convolutional neural networks, each of which calculates the residual optical flow at a different level of the pyramid.

3. A lightweight video semantic communication method according to claim 1, characterized in that: The motion compensation network includes a warp operator and a convolutional neural network.

4. A lightweight video semantic communication method according to claim 1, characterized in that: The joint source-channel encoder comprises a semantic encoder and a channel encoder.

5. A lightweight video semantic communication method according to claim 4, characterized in that: The semantic encoder is based on the Swin Transformer architecture.

6. A lightweight video semantic communication method according to claim 4, characterized in that: The channel encoder includes a channel signal-to-noise ratio (SNR) adaptation module and a residual network (Residual Network, ResNet).

7. A lightweight video semantic communication method according to claim 6, characterized in that: The SNR adaptation module is based on the Efficient Channel Attention Network (ECA-Net) architecture.

8. A lightweight video semantic communication method according to claim 1, characterized in that: The joint source-channel decoder comprises a channel decoder and a semantic decoder.

9. A lightweight video semantic communication method according to claim 8, characterized in that: The channel decoder includes ResNet and SNR adaptation modules.

10. A lightweight video semantic communication method according to claim 8, characterized in that: The semantic decoder is based on the Swin Transformer architecture.

Images