Generative semantic communication system and method based on frame interpolation

Abstract

The invention relates to a generative semantic communication system and method based on frame interpolation, and belongs to the field of communication and artificial intelligence. The system comprises a semantic coding module and a channel coding module at a sending end, and a channel decoding module, a semantic decoding module and a frame interpolation module at a receiving end. The sending end extracts key semantic information of the dynamic image through semantic coding, and compresses and transmits at least two key frames; and a receiving end reconstructs the key frame through a decoding module, and generates an intermediate frame by means of a frame interpolation module, thereby restoring a complete dynamic image. In the aspect of system design, the semantic encoding and decoding module is based on a Swin Transform structure, and the frame interpolation module is based on an optical flow decoder, so that efficient semantic compression and decoding generation are realized. A training strategy adopts transfer learning, and an image transmission and frame generation module is gradually optimized, so that the model is quickly converged to an optimal state. The system structure and the implementation method are described in detail.

Description

Technical Field

[0001] The present invention belongs to the field of communication and artificial intelligence, and relates to a generative semantic communication system and method based on frame interpolation. Background Art

[0002] As an emerging communication paradigm, semantic communication has attracted widespread attention because of its great potential in compressing data redundancy, reducing bandwidth requirements, and improving transmission efficiency. In recent years, the rapid development of generative artificial intelligence (GAI) has also injected new vitality into semantic communication. Through pre-training of generative models and learning of large-scale data, GAI can generate high-quality voice, image, and video content, and is widely used in automatic image generation, text generation and other fields. These capabilities of generative AI naturally match the efficient transmission requirements of semantic communication: through semantic extraction, the communication system achieves effective data transmission under low bandwidth conditions, thereby ensuring the efficient dissemination of GAI content in 6G networks. In addition, by building a knowledge base and optimizing model parameters, generative AI further improves the intelligence and response efficiency of the semantic communication system.

[0003] For image transmission, especially dynamic image transmission scenarios, frame interpolation technology in generative AI has been widely studied and used to optimize dynamic image generation. Frame interpolation estimates the motion information between adjacent frames and automatically generates intermediate frames to construct a continuous video stream.

[0004] In the 6G network environment, application scenarios such as telemedicine, autonomous driving, AR / VR, and smart city monitoring have put forward higher requirements for the quality and efficiency of dynamic image transmission. Especially when bandwidth is limited, low latency, and transmission quality requirements coexist, how to achieve high-quality transmission under limited bandwidth conditions has become an important issue. To this end, this paper proposes a system architecture that combines the image semantic communication system with the frame interpolation generation technology. By efficiently transmitting key frames and restoring intermediate frames using generation technology, the amount of transmission data of dynamic images is greatly reduced, and the communication efficiency and image reconstruction quality are effectively improved. The Swin Transformer model is used in the system to encode and decode image semantics, ensuring the stability and efficiency of the input features of the generation module. The generation module is implemented based on the optical flow decoder, which can significantly reduce the transmission overhead and time delay while maintaining high-quality generation effects.

[0005] In addition, in order to achieve efficient training of the model, the system in this paper adopts a transfer learning strategy. First, the image transmission system without the frame interpolation module is trained end-to-end to ensure the robust transmission of semantic information on the communication link; then the frame interpolation module is added to achieve high-quality intermediate frame reconstruction by gradually optimizing the generated features. The final simulation results show that the system performs well in high-quality transmission and low latency, effectively responds to the communication challenges in the future 6G scenario, and achieves high-quality reconstruction of high-definition dynamic images under limited transmission bandwidth, which has good practicality and promotion value. Summary of the invention

[0006] In view of this, the purpose of the present invention is to provide a generative semantic communication system and method based on frame interpolation. The system includes a semantic coding module and a channel coding module deployed at the transmitting end, and a channel decoding module, a semantic decoding module and a frame interpolation module deployed at the receiving end. Compared with the traditional communication system, the semantic communication system in the present invention is implemented by deep learning methods at both the transmitting end and the receiving end to enhance the adaptability and transmission efficiency of the system.

[0007] In order to achieve the above object, the present invention provides the following technical solutions:

[0008] A generative semantic communication system based on frame interpolation, comprising:

[0009] Sending side:

[0010] A semantic encoding module, used for semantically encoding the input dynamic image, extracting key semantic information, and compressing at least two key frames into semantic features;

[0011] A channel coding module, used for performing channel coding on the semantic features to generate a transmission signal;

[0012] Receiver:

[0013] A channel decoding module, used to perform channel decoding on the received signal and restore semantic features;

[0014] A semantic decoding module is used to decode semantic features and reconstruct key frames;

[0015] The frame interpolation module is used to generate intermediate frames between key frames based on the reconstructed key frames, so as to restore the complete dynamic image.

[0016] Furthermore, the semantic encoding module and the semantic decoding module are both based on the Swin Transformer structure.

[0017] Furthermore, the frame interpolation module is based on an optical flow decoder.

[0018] Furthermore, the Swin Transformer module introduces a local window self-attention mechanism.

[0019] Furthermore, the frame interpolation module adopts a convolutional neural network (CNN).

[0020] A generative semantic communication method based on frame interpolation comprises the following steps:

[0021] Extract dynamic image key frames and concatenate the key frames as input;

[0022] The semantic encoding module based on the Swin Transformer structure is used to process the input layer by layer to obtain semantic features;

[0023] The semantic features are processed by a channel coding module to obtain a transmission signal;

[0024] Transmit the sending signal to the receiving end through the channel;

[0025] The received signal is processed using a channel decoding module to restore semantic features;

[0026] The semantic decoding module based on the Swin Transformer structure is used to restore the semantic features layer by layer until the key frame is reconstructed.

[0027] Extract features from the input of the Swin Transformer modules at each layer of the receiving end to obtain input features suitable for the frame interpolation module;

[0028] The input features are processed layer by layer using a frame interpolation module based on an optical flow decoder to obtain multiple frames of generated images.

[0029] Furthermore, the method adopts a training strategy of transfer learning, including:

[0030] In the first stage, the image transmission system without the frame interpolation module is trained end-to-end until convergence;

[0031] In the second stage, the frame interpolation module is trained separately to improve the image generation quality until the predetermined convergence criteria are reached.

[0032] Furthermore, the first stage uses a mean square error (MSE) loss function to optimize image transmission performance.

[0033] Furthermore, the second stage adopts the sum of Census loss based on photometric loss and Charbonnier loss as the loss function.

[0034] Furthermore, the feature extraction includes reconstructing the shape of the feature map tensor and reducing the number of channels.

[0035] The beneficial effect of the present invention is that the present invention can effectively reduce the amount of dynamic image transmission data, improve transmission efficiency, and maintain high-quality image reconstruction. The simulation results show that the system can generate dynamic images rich in motion information by only transmitting a limited number of key frames, which is suitable for high-definition transmission, telemedicine, autonomous driving, and intelligent monitoring in 6G network environments.

[0036] Other advantages, objectives and features of the present invention will be described in the following description to some extent, and to some extent, will be obvious to those skilled in the art based on the following examination and study, or can be taught from the practice of the present invention. The objectives and other advantages of the present invention can be realized and obtained through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] In order to make the purpose, technical solutions and advantages of the present invention more clear, the present invention will be described in detail below in conjunction with the accompanying drawings, wherein:

[0038] Figure 1 A system architecture diagram of a generative semantic communication system based on frame interpolation according to an embodiment of the present invention;

[0039] Figure 2 A schematic diagram of the logical structure of optical flow estimation of a frame interpolation module according to an embodiment of the present invention;

[0040] Figure 3 This is a network diagram of the Swin Transformer module of an embodiment of the present invention;

[0041] Figure 4 A network diagram of a frame interpolation module according to an embodiment of the present invention;

[0042] Figure 5 A flow chart of a generative semantic communication system method based on frame interpolation according to an embodiment of the present invention;

[0043] Figure 6 is a training flow chart of an embodiment of the present invention;

[0044] Figure 7 The PSNR comparison between the generated image and the reconstructed image through the AWGN channel in the embodiment of the present invention is shown;

[0045] Figure 8 The MS-SSIM comparison between the generated image and the reconstructed image through the AWGN channel of the embodiment of the present invention;

[0046] Fig. 9 Comparison of transmission time between an embodiment of the present invention and an image transmission system without generation function;

[0047] Fig.10The figure compares the number of channel resource frames occupied by the embodiment of the present invention and the image transmission system without the generation function. DETAILED DESCRIPTION

[0048] The following describes the embodiments of the present invention by specific examples, and those skilled in the art can easily understand other advantages and effects of the present invention from the contents disclosed in this specification. The present invention can also be implemented or applied through other different specific embodiments, and the details in this specification can also be modified or changed in various ways based on different viewpoints and applications without departing from the spirit of the present invention. It should be noted that the illustrations provided in the following embodiments only illustrate the basic concept of the present invention in a schematic manner, and the following embodiments and features in the embodiments can be combined with each other without conflict.

[0049] Among them, the drawings are only used for illustrative explanations, and they only represent schematic diagrams rather than actual pictures, and should not be understood as limitations on the present invention. In order to better illustrate the embodiments of the present invention, some parts of the drawings may be omitted, enlarged or reduced, and do not represent the size of actual products. For those skilled in the art, it is understandable that some well-known structures and their descriptions in the drawings may be omitted.

[0050] The same or similar numbers in the drawings of the embodiments of the present invention correspond to the same or similar parts; in the description of the present invention, it should be understood that if the terms "upper", "lower", "left", "right", "front", "rear", etc. indicate the orientation or position relationship, they are based on the orientation or position relationship shown in the drawings, which is only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific orientation, be constructed and operate in a specific orientation. Therefore, the terms describing the position relationship in the drawings are only used for illustrative purposes and cannot be understood as limiting the present invention. For ordinary technicians in this field, the specific meanings of the above terms can be understood according to specific circumstances.

[0051] The present invention provides a semantic communication system of frame interpolation, such as Figure 1 As shown, the system is composed of a semantic coding module and a channel coding module deployed at the transmitting end, and a channel decoding module, a semantic decoding module and a frame interpolation module deployed at the receiving end.

[0052] At the sending end, the image semantic communication system first performs semantic encoding and channel encoding on the input image x. Therefore, the sending signal at the sending end can be expressed as

[0053]

[0054] Where S(·) is the image semantic encoder; is the parameter of the image semantic encoder; C(·) is the channel encoder; α is the parameter of the channel encoder. The signal y from the transmitter reaches the receiver through the physical channel, and the signal obtained by the receiver can be expressed as

[0055]

[0056] Where h is the channel gain of the Rayleigh fading channel; n is additive white Gaussian noise (AWGN). The receiving end receives the signal After passing through the channel decoder and semantic decoder respectively, the image is reconstructed. This process is the reverse process of the transmitter. Therefore, the reconstructed image can be expressed as

[0057]

[0058] in, is the reconstructed image; C -1 (·) is the channel decoder; β is the parameter of the channel decoder; S -1 (·) is the image semantic decoder; δ is the parameter of the image semantic decoder.

[0059] The image quality evaluation indicator adopted in this paper is the Peak Signal to Noise Ratio (PSNR). Therefore, when training the image semantic communication system end-to-end, the mean square error (MSE) can be used as the loss function, that is,

[0060]

[0061] The frame interpolation module is based on optical flow estimation. The role of optical flow estimation is to find dense correspondences between adjacent frames. Given two input frames I0 and I1 at adjacent times, the purpose of frame interpolation is to synthesize an intermediate frame I t , where 0<t<1. Figure 2 As shown in Figure 1, to achieve this goal, we first need to extract the features of the input frame, that is, obtain a pyramid feature from each frame, and then gradually refine the bilateral intermediate flow field and the reconstructed intermediate features in a coarse-to-fine manner until the highest level of the pyramid is reached to obtain the final output. The pyramid features of frame I0 and frame I1 are generated at the kth (k∈{1,2,3,4}) layer by the pyramid feature encoder (PyramidEncoder). After extracting meaningful hierarchical representations, the pyramid features are processed through multiple decoders. Perform image distortion (warp), according to and generate In addition, the decoder also generates reconstructed intermediate features. To fill in the missing reference information and thus facilitate motion estimation

[0062] Due to the task-related nature, The input and The output of is different from other decoders. In summary, the features between decoders can be calculated as follows:

[0063]

[0064] in, represents the decoder in the pyramid, and [·] represents the connection operation. M is the single-channel merge mask derived from the sigmoid layer, whose elements range from 0 to 1, and R is the three-channel image residual, which can compensate for the details. Finally, we can synthesize the desired frame by the following formula

[0065]

[0066] Among them, w represents backward warping and ⊙ is element-wise multiplication. M adjusts the blending ratio according to bidirectional occlusion information, while R compensates for some details when flow-based generation is unreliable.

[0067] To train the decoder, this paper uses the sum of Census loss and Charbonnier loss based on photometric loss as the loss function, which can be expressed as

[0068]

[0069] in, and represents the generated intermediate frame and the real intermediate frame. The Charbonnier loss is ρ(x)=(x 2 +∈ 2 ) α Specifically, the parameters of the loss function are set to α = 0.5 and ∈ = 0.001, which is used as Replacement of loss. is the Census loss, which calculates the soft Hamming distance between image patches after the Census transformation.

[0070] Swin Transformer is an advanced visual model that uses a layered architecture design that enables it to effectively process image features at different resolutions. Unlike the global self-attention mechanism used by traditional Transformers, Swin Transformer introduces local window self-attention. Specifically, in each layer, the feature map is divided into non-overlapping windows, and self-attention calculations are performed within these windows. By moving windows between consecutive layers, the model is able to establish connections between different windows, significantly enhancing the flow of information. In addition, Swin Transformer significantly reduces the computational complexity by limiting the scope of self-attention calculations, allowing the model to process images of larger sizes without excessive computing resources.

[0071] In the generative semantic communication system based on frame interpolation proposed in this paper, the Swin Transformer module is used for semantic encoding and decoding. Figure 4 The specific process of a single Swin Transformer module is as follows

[0072]

[0073] in, and u (k) Represent the output features of W-MSA and MLP, respectively. k represents the number of layers. W-MSA and SW-MAS are multi-head self-attention modules, using regular and shifted window configurations, respectively. LN represents the layer normalization operation.

[0074] Each frame interpolation module can be regarded as a decoder. For a single module, the input features are stacked into overall features. In order to balance efficiency and maintain a large receptive field and number of channels in motion estimation and feature encoding, the convolution in the middle layer of the module only updates some channels of the output tensor of the previous layer. In addition, the use of residual connections and interleaved structures facilitates the propagation and joint refinement of information. In summary, Figure 3 The specific process of a single frame interpolation module is as follows

[0075]

[0076] in are input features. is the output feature. (k) and is an intermediate feature. Concat is a concatenation operation. Split is a segmentation operation. Conv and Deconv are deconvolution operations respectively.

[0077] In order to better illustrate the present invention, Figure 5As shown, an embodiment of the present invention further provides a semantic communication method applicable to a highly dynamic network, the method comprising:

[0078] 101. In semantic coding, two RGB images are input They are divided into non-overlapping blocks and arranged from left to right and from top to bottom.

[0079] 102. Next, these image patches will be processed by the Swin Transformer module. After the Swin Transformer module, the size of the feature will become Specifically, the number of channels increases layer by layer, while the spatial size of the image gradually decreases. This process is called a layer of the Swin Transformer module;

[0080] In the embodiment of the present invention, as the network level goes deeper, the number of channels of the feature map output by each layer of the Swin Transformer module at the transmitting end will increase layer by layer, and the size will decrease layer by layer, so as to gradually extract higher-level and more abstract semantic features. This layer-by-layer design enables the model to better process complex semantic information and effectively encode images.

[0081] 103. The output of the semantic coding is fed into the channel coding module to generate the transmission signal The transmission signal is transmitted through a wireless channel and is channel-decoded.

[0082] 104. The receiving end inputs the channel-decoded signal into the semantic decoding module, which includes the Swin Transformer module, to gradually restore the original information. In this process, the Swin Transformer module further enhances the accuracy of signal decoding and the quality of semantic reconstruction through its multi-level feature extraction capability, so that the entire transmission system can still effectively retain and restore high-quality semantic information in a complex channel environment.

[0083] In the embodiment of the present invention, semantic decoding and semantic encoding adopt a symmetrical structure. The input feature size of the SwinTransformer module at the receiving end corresponds to the output of the corresponding layer at the sending end. For the output of the receiving end, the number of channels C of the feature decreases layer by layer, while the size increases layer by layer, and finally a high-resolution image is reconstructed. Specifically, the output tensor size of the kth layer of the Swin Transformer module satisfies These hierarchical features constitute a pyramid feature structure with multi-scale information.

[0084] 105. At the receiving end, feature extraction is performed on the feature map input to each layer of the Swin Transformer module to construct the corresponding pyramid feature.

[0085] In the embodiment of the present invention, the feature extraction process includes the reconstruction of the shape of the feature map tensor and the deletion of the number of channels. The input features of the k-th layer Swin Transformer module at the receiving end are divided into two features after feature extraction. and The shape is The ordered blocks are reversed and restored to the feature size required for the convolutional neural network. The number of channels is reduced by sorting the information content, that is, the channels of each block are sorted according to the amount of information they contain, and the channels with more information are retained and the channels with less information are deleted. The amount of information contained in each block is determined by its mean. The higher the mean, the richer its information content. Therefore, only channels with high information content are retained to improve the effectiveness of feature expression.

[0086] 106. The features after feature extraction are applied to the frame interpolation module to finally generate the generated image.

[0087] In the embodiment of the present invention, the method adopts a training strategy based on transfer learning, combined with the architecture of the system, such as Figure 6 As shown in the figure. Specifically, it includes training an end-to-end image semantic communication system for image transmission and a frame interpolation module for image generation. The end-to-end image semantic communication system is trained based on the DIV2K dataset, and the frame interpolation module is trained using the Vimeo90K dataset. In the end-to-end system training, the learning rate is 1×10 -4 The Adam optimizer of the DIV2K dataset is used, and the batch size is set to 16. In the training of the frame interpolation module, the learning rate is still 1×10 -4 The Adam optimizer is used, but since the input becomes an image pair, the batch size of the processed Vimeo90K dataset is set to 6. The entire system is implemented based on PyTorch. The training and testing of all experiments are carried out in the Ubuntu 20.04+CUDA 11.0 environment, and a single RTX 3070GPU is used for calculations.

[0088] In the experiment, the input For example, the number of layers of the sending end and the corresponding receiving end is 4 in total. The setting values ​​of the input tensor shapes of the SwinTransformer module input and the frame interpolation module of each layer at the receiving end are shown in Table 1.

[0089] Table 1

[0090] Layer number k SwinTransformer Module Frame interpolation module 1 [2,3200,320] [1,80，40，96] 2 [2,12800,256] [1,160，80，72] 3 [2,51200,192] [1,320，160，48] 4 [2,204800，128] [1,640，320，32]

[0091] Figure 7 and Figure 8 The PSNR comparison between the generated image and the reconstructed image through the AWGN channel and the MS-SSIM comparison between the generated image and the reconstructed image through the AWGN channel are respectively shown. The simulation results show that the frame interpolation-based generative semantic image system provided by the present invention can transmit good dynamic images using only key frames.

[0092] Fig. 9 and Fig.10 The transmission time comparison and the number of frames of channel resources occupied by the image transmission system without the generation function are respectively shown. The simulation results show that the generative semantic image system based on frame interpolation provided by the present invention has lower transmission delay and higher bandwidth utilization compared with the image transmission system without the generation function.

[0093] The generative semantic communication system based on frame interpolation proposed in the present invention adopts an architecture with the Swin Transformer module as the core to realize the semantic encoding, decoding and transmission of images; at the receiving end, a frame interpolation module based on optical flow estimation is used to complete the task of generating images. The system can efficiently transmit dynamic images and significantly improve the transmission rate of high-definition dynamic images. Simulation results show that the system can achieve high-quality transmission of dynamic images under different channel conditions, and compared with the semantic communication system without generation function, the generative semantic communication system based on frame interpolation greatly reduces the amount of data and transmission time in dynamic image transmission.

[0094] Finally, it should be noted that the above embodiments are only used to illustrate the technical solution of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solution of the present invention can be modified or replaced by equivalents without departing from the purpose and scope of the technical solution, which should be included in the scope of the claims of the present invention.

Claims

1. A generative semantic communication system based on frame interpolation, characterized in that: include: Sending side: A semantic encoding module, used for semantically encoding the input dynamic image, extracting key semantic information, and compressing at least two key frames into semantic features; A channel coding module, used for performing channel coding on the semantic features to generate a transmission signal; Receiver: A channel decoding module, used to perform channel decoding on the received signal and restore semantic features; A semantic decoding module is used to decode semantic features and reconstruct key frames; The frame interpolation module is used to generate intermediate frames between key frames based on the reconstructed key frames, so as to restore the complete dynamic image.

2. The generative semantic communication system based on frame interpolation according to claim 1, characterized in that: The semantic encoding module and the semantic decoding module are both based on the Swin Transformer structure.

3. The generative semantic communication system based on frame interpolation according to claim 1, characterized in that: The frame interpolation module is based on an optical flow decoder.

4. The generative semantic communication system based on frame interpolation according to claim 1, characterized in that: The SwinTransformer module introduces a local window self-attention mechanism.

5. The generative semantic communication system based on frame interpolation according to claim 1, characterized in that: The frame interpolation module adopts a convolutional neural network (CNN).

6. A generative semantic communication method based on frame interpolation, characterized in that: The following steps are involved: Extract dynamic image key frames and concatenate the key frames as input; The semantic encoding module based on the Swin Transformer structure is used to process the input layer by layer to obtain semantic features; The semantic features are processed by a channel coding module to obtain a transmission signal; Transmit the sending signal to the receiving end through the channel; The received signal is processed using a channel decoding module to restore semantic features; The semantic decoding module based on the Swin Transformer structure is used to restore the semantic features layer by layer until the key frame is reconstructed. Extract features from the input of the Swin Transformer modules at each layer of the receiving end to obtain input features suitable for the frame interpolation module; The input features are processed layer by layer using a frame interpolation module based on an optical flow decoder to obtain multiple frames of generated images.

7. The generative semantic communication method based on frame interpolation according to claim 6, characterized in that: The method adopts a training strategy of transfer learning, including: In the first stage, the image transmission system without the frame interpolation module is trained end-to-end until convergence; In the second stage, the frame interpolation module is trained separately to improve the image generation quality until the predetermined convergence criteria are reached.

8. The generative semantic communication method based on frame interpolation according to claim 7, characterized in that: The first stage uses the mean square error (MSE) loss function to optimize image transmission performance.

9. The generative semantic communication method based on frame interpolation according to claim 7, characterized in that: The second stage uses the sum of Census loss based on photometric loss and Charbonnier loss as the loss function.

10. The generative semantic communication method based on frame interpolation according to claim 6, characterized in that: The feature extraction includes reconstructing the shape of the feature map tensor and reducing the number of channels.

Images