Training methods, devices, equipment, and storage media for multi-source semantic communication models
By training a multi-source semantic communication model in stages, the problems of low communication efficiency and resource waste in multi-source relay communication networks are solved, and efficient semantic information transmission and image restoration are achieved.
Patent Information
- Application Number
- CN202410645981.9
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-05-23
- Publication Date
- 2026-11-17
- Estimated Expiration
- 2044-05-23
AI Technical Summary
Existing technologies suffer from low communication efficiency and significant resource waste in multi-source relay communication networks, failing to effectively utilize the correlation between multiple signal sources.
A multi-source semantic communication model is adopted. The semantic sending module, semantic information fusion module and semantic recovery module are trained in stages and deployed at the source node, relay node and destination node respectively. The model is trained using a pre-built auxiliary receiver until the loss function converges, thereby realizing the extraction, fusion and recovery of semantic features.
It improves the training and communication efficiency of multi-source relay communication networks, reduces the waste of communication resources, and enhances image restoration results and system performance.
Smart Images

Figure CN118606883B_ABST
Abstract
Description
Technical Field
[0001] This invention belongs to the field of deep learning and Internet of Things communication technology, and particularly relates to a training method, apparatus, device and storage medium for a multi-source semantic communication model. Background Technology
[0002] The deployment of 5G has spurred the emergence of various new Internet of Things (IoT) applications. The massive data exchange generated by interconnected smart devices has brought about a huge communication load, placing more stringent requirements on various technical indicators of communication systems. Therefore, the industry has begun to actively explore 6G mobile communication technology to meet the technical requirements of future IoT applications. However, the current wireless communication architecture based on Shannon's information theory is gradually approaching its theoretical limits. For example, source coding technology has approached the source entropy rate distortion function, and channel coding technologies such as LDPC codes and polar codes have also approached channel capacity. In addition, in multi-source relay communication networks, communication between source nodes and target nodes relies on the assistance of relay nodes to forward data, thereby expanding the communication range of source nodes and reducing signal transmission loss. Current research on relay-based cooperative communication networks mainly focuses on the traditional communication paradigm based on bit transmission. The aim is to optimize key performance indicators (such as throughput, latency, and packet loss rate) to achieve high transmission efficiency and reliability. For example, some schemes propose different relay forwarding strategies, designing how relays process and forward signals for different optimization objectives. Furthermore, many schemes are dedicated to developing relay selection algorithms, selecting the optimal next-hop relay node by evaluating the quality and performance indicators of each relay node (such as signal strength, interference level, and path loss). However, traditional bit-based communication schemes neglect the rich semantic information within the transmitted message. Important semantic and redundant information in the message are encoded together into a bit stream, allocating identical communication resources. This leads to a potential risk of wasted communication resources, as the receiver only needs the key semantic information to achieve the intended communication goal. Therefore, current traditional bit-based communication systems urgently need technological innovation to adapt to the development needs of future 6G mobile communication.
[0003] In recent years, due to the outstanding performance of artificial intelligence-related technologies in various fields, the industry has begun to explore the organic integration of wireless communication and artificial intelligence. Among these, semantic communication, as a new communication paradigm, has received widespread attention and is considered one of the key technologies of 6G. Semantic communication focuses on transmitting the key semantics of messages, enabling the receiver to understand the meaning of the message, thus achieving the ultimate goal of communication—"meaningful communication." Compared with traditional communication, semantic communication effectively reduces the allocation of communication resources to semantically redundant information, thereby improving communication efficiency and potentially overcoming the theoretical limitations of Shannon. Currently, most semantic communication schemes in relay networks focus only on single-source communication scenarios, that is, designing semantic communication models in point-to-point communication systems with a single sender and a single receiver to achieve semantic-level transmission of various types of data (such as text, semantics, images, video, and multimodal data). A few schemes attempt to apply semantic communication to design different relay forwarding strategies in relay cooperative communication networks, but these schemes only consider the single-source case. However, in actual communication scenarios, there may be multiple signal sources. When the forwarding strategies designed by these schemes are adopted, signals from different sources will be forwarded separately in the relay without considering the correlation between signals from different source nodes. Therefore, a new multi-source semantic communication model for multi-source relay cooperative communication networks is urgently needed to solve the above problems. Summary of the Invention
[0004] The purpose of this invention is to provide a training method, apparatus, device, and storage medium for a multi-source semantic communication model, aiming to solve the problems of low communication efficiency and waste of communication resources in multi-source relay communication networks caused by existing technologies.
[0005] On one hand, the present invention provides a training method for a multi-source semantic communication model, the multi-source semantic communication model including a semantic sending module deployed in a source node, a semantic information fusion module deployed in a relay node, and a semantic recovery module deployed in a destination node, the method comprising the following steps:
[0006] Based on a pre-built auxiliary receiver, the semantic sending module is trained on a local image dataset until the pre-set first loss function converges;
[0007] Based on the trained semantic sending module, the semantic information fusion module and the semantic recovery module are trained according to the complete image dataset until the pre-set second loss function converges.
[0008] Preferably, the step of training the semantic transmission module based on a pre-built auxiliary receiver and a local image dataset until a pre-set first loss function converges includes:
[0009] The local image dataset is preprocessed;
[0010] The first training sample is obtained by randomly selecting samples from the preprocessed local image dataset;
[0011] The semantic sending module performs semantic feature processing on the first training sample to obtain a semantic feature signal;
[0012] The received semantic feature signal is reconstructed using the auxiliary receiver to obtain a first restored image;
[0013] Based on the first training sample and the first restored image, calculate the first loss value of the first loss function. Based on the first loss value, update the trainable parameters of the semantic sending module and the auxiliary receiver. When the first loss function fails to converge, jump to the step of randomly sampling samples from the preprocessed local image dataset and continue execution until the first loss function converges.
[0014] Preferably, the trained semantic sending module is deployed to all source nodes, with at least two source nodes. The step of training the semantic information fusion module and the semantic recovery module based on the trained semantic sending module and the complete image dataset until the pre-set second loss function converges includes:
[0015] The complete image dataset is preprocessed;
[0016] Randomly select samples from the preprocessed complete image dataset to obtain the second training samples;
[0017] The semantic sending module in the activated source node performs semantic feature processing on the local image collected by the source node from the second training sample to obtain a local semantic feature signal.
[0018] The semantic information fusion module performs semantic information fusion on the received local semantic feature signals to obtain fused feature signals;
[0019] The semantic recovery module performs semantic recovery on the received fused feature signal to obtain a second recovered image;
[0020] Based on the second training sample and the second restored image, calculate the second loss value of the second loss function. Based on the second loss value, update the trainable parameters of the semantic information fusion module and the semantic restoration module. When the second loss function fails to converge, jump to the step of randomly obtaining samples from the preprocessed complete image dataset and continue execution until the second loss function converges.
[0021] Preferably, the semantic transmission module includes a semantic encoder and a channel encoder, wherein the semantic encoder consists of a two-dimensional convolutional layer and two residual blocks, and the channel encoder consists of a flattening layer, a multilayer perceptron layer and a power normalization layer.
[0022] Preferably, the auxiliary receiver includes a channel decoder and a semantic decoder, wherein the channel decoder consists of a connection layer, a multilayer perceptron layer and a reshaping layer, and the semantic decoder consists of two residual blocks and a two-dimensional deconvolution layer.
[0023] Preferably, the semantic information fusion module includes a connection layer, a position encoding layer, a converter layer, a linear mapping layer, a reshaping layer, and a power normalization layer, and the semantic recovery module includes a reshaping layer, a linear mapping layer, a tag embedding layer, a position encoding layer, and a converter layer.
[0024] On the other hand, the present invention provides an image transmission method in a multi-source relay communication network, which is based on a multi-source semantic communication model obtained by training the multi-source semantic communication model as described in any of the above claims. The method involves deploying the semantic sending module of the multi-source semantic communication model in all source nodes, deploying the semantic information fusion module of the multi-source semantic communication model in relay nodes, and deploying the semantic recovery module of the multi-source semantic communication model in the destination node. The method includes:
[0025] Acquire a complete image of the target area to be observed;
[0026] The semantic sending module in the activated source node performs semantic feature processing on the local image acquired by the source node from the complete target image to obtain the target local semantic signal.
[0027] The received local semantic signals are fused using the semantic information fusion module to obtain the target fused signal.
[0028] The semantic restoration module performs semantic restoration on the received target fusion signal to obtain a target restored image.
[0029] On the other hand, the present invention provides a training apparatus for a multi-source semantic communication model, the multi-source semantic communication model including a semantic sending module deployed in a source node, a semantic information fusion module deployed in a relay node, and a semantic recovery module deployed in a destination node, the apparatus comprising:
[0030] The first training unit is used to train the semantic sending module based on a pre-built auxiliary receiver and a local image dataset until the pre-set first loss function converges.
[0031] The second training unit is used to train the semantic information fusion module and the semantic recovery module based on the trained semantic sending module and the complete image dataset until the pre-set second loss function converges.
[0032] On the other hand, the present invention also provides a communication device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps described in the training method for the multi-source semantic communication model described above.
[0033] On the other hand, the present invention also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps described in the training method for the multi-source semantic communication model above.
[0034] This invention, based on a pre-built auxiliary receiver, trains a semantic sending module using a local image dataset until a pre-set first loss function converges. Based on the trained semantic sending module, it trains a semantic information fusion module and a semantic recovery module using a complete image dataset until a pre-set second loss function converges. The multi-source semantic communication model includes a semantic sending module deployed in the source node, a semantic information fusion module deployed in the relay node, and a semantic recovery module deployed in the destination node. By employing a phased training strategy, this invention improves the training efficiency of the model in multi-source relay communication networks and enhances the communication efficiency of the multi-source semantic communication model in multi-source communication, while reducing the waste of communication resources. Attached Figure Description
[0035] Figure 1 This is a flowchart illustrating the implementation of the training method for the multi-source semantic communication model provided in Embodiment 1 of the present invention.
[0036] Figure 2 This is a schematic diagram of the residual block structure in the training method of the multi-source semantic communication model provided in Embodiment 1 of the present invention;
[0037] Figure 3 This is a schematic diagram of image transmission for the training method of the multi-source semantic communication model provided in Embodiment 1 of the present invention;
[0038] Figure 4 This is a flowchart illustrating the implementation of the training method for the multi-source semantic communication model provided in Embodiment 2 of the present invention.
[0039] Figure 5 This is a schematic diagram of the semantic sending module and auxiliary receiver in the training method of the multi-source semantic communication model provided in Embodiment 2 of the present invention;
[0040] Figure 6 This is a flowchart illustrating the implementation of the training method for the multi-source semantic communication model provided in Embodiment 3 of the present invention.
[0041] Figure 7 This is a schematic diagram of the semantic information fusion module and the semantic recovery module in the training method of the multi-source semantic communication model provided in Embodiment 3 of the present invention;
[0042] Figure 8 This is a flowchart illustrating the implementation of the image transmission method in a multi-source relay communication network provided in Embodiment 4 of the present invention.
[0043] Figure 9 This is a schematic diagram of image transmission in a multi-source relay communication network provided in Embodiment 4 of the present invention.
[0044] Figure 10 This is a schematic diagram of the structure of the training device for the multi-source semantic communication model provided in Embodiment 5 of the present invention;
[0045] Figure 11 This is a schematic diagram of the communication device provided in Embodiment Six of the present invention. Detailed Implementation
[0046] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention.
[0047] The specific implementation of the present invention will be described in detail below with reference to specific embodiments:
[0048] Example 1:
[0049] Figure 1 The implementation flow of the training method for the multi-source semantic communication model provided in Embodiment 1 of the present invention is shown. For ease of explanation, only the parts related to the embodiments of the present invention are shown, and are described in detail below:
[0050] In step S101, the semantic transmission module is trained based on the pre-built auxiliary receiver and the local image dataset until the pre-set first loss function converges.
[0051] The embodiments of the present invention are applicable to communication devices, platforms or systems in multi-source relay communication networks, which include source nodes for sending raw data packets or signals, relay nodes for receiving signals from source nodes and processing and forwarding the signals, and destination nodes for receiving data or signals from source nodes or relay nodes. In this embodiment of the invention, the multi-source semantic communication model includes a semantic sending module, a semantic information fusion module, and a semantic recovery module. The semantic sending module is deployed in the source node, the semantic information fusion module is deployed in the relay node, and the semantic recovery module is deployed in the destination node. An auxiliary receiver is pre-built to assist the training of the semantic sending module during the training phase. Based on the auxiliary receiver, the semantic sending module is trained according to the local image dataset until the pre-set first loss function converges, thereby obtaining a semantic sending module used by all source nodes. The semantic sending module and the auxiliary receiver communicate via a wireless channel. The local images in the local image dataset only contain some features of a complete image. All local images in the local image dataset can come from different complete images. A complete image can use only one local image or multiple local images for the local image dataset. The auxiliary receiver is not used in the testing phase or in actual application. The semantic information fusion module and the semantic recovery module do not participate in the training of this phase.
[0052] In one feasible embodiment, the semantic transmission module includes a semantic encoder for extracting semantic features from a local image and a channel encoder for mapping the extracted semantic features into a signal suitable for transmission in a wireless channel. The semantic encoder consists of a two-dimensional convolutional layer (Conv2D layer) and two residual block layers (ResBlock), and the channel encoder consists of a flatten layer, a multilayer perceptron layer (MLP layer), and a power normalization layer (PowerNorm layer), thereby enabling the semantic transmission module to extract semantic features from the local image and map the signal.
[0053] In another feasible embodiment, the auxiliary receiver, corresponding to the semantic transmission module, includes a channel decoder for signal decoding and a semantic decoder for semantic feature recovery. The channel decoder consists of a concat layer, a multilayer perceptron layer (MLP layer), and a reshape layer. The semantic decoder consists of two residual blocks (ResBlock) and a two-dimensional deconvolution layer (TransConv2D layer), thereby enabling the auxiliary receiver to recover local images.
[0054] In another feasible embodiment, both the semantic sending module and the ResBlock in the auxiliary receiver contain convolutional layers (ConvLayer), batch normalization layers (BatchNorm), and activation functions (ELU), with their specific connection structures as follows: Figure 2 As shown, the convolutional layer in the ResBlock of the semantic sending module is Conv2D, and the convolutional layer in the ResBlock of the auxiliary receiver is TransConv2D.
[0055] As an example, such as Figure 3 As shown, the semantic transmission module extracts semantic features and maps signals from the acquired local image, and transmits the mapped signals to the auxiliary receiver via a wireless channel. The auxiliary receiver then decodes the signals and restores the semantic features to obtain the restored local image.
[0056] In another feasible embodiment, the semantic transmission module is trained based on a pre-built auxiliary receiver and a local image dataset until the pre-set first loss function converges. For details of the specific implementation, please refer to the description of Embodiment 2 below, which will not be repeated here.
[0057] In step S102, based on the trained semantic sending module, the semantic information fusion module and the semantic recovery module are trained according to the complete image dataset until the pre-set second loss function converges.
[0058] In this embodiment of the invention, a sensor is used as the source node to acquire images. The trained semantic sending module is deployed to all sensors, and the parameters of the semantic sending module are frozen. Then, the semantic information fusion module deployed in the relay node and the semantic recovery module deployed in the destination node are trained according to the complete image dataset until the pre-set second loss function converges. The number of sensors is not less than two.
[0059] In a feasible embodiment, the semantic information fusion module is used to perform semantic information fusion on the signals received by the relay node, so that the relay node forwards the fused signal to the destination node. It includes a connection layer (Concat), a position coding layer (PosEmb), a transformer layer (Transformer Layer), a linear mapping layer (LinearProj), a reshaping layer (Reshape), and a power normalization layer (PowerNorm), thereby realizing the joint extraction of the most valuable semantic information from the transmitted signals of multiple sensors and reducing the redundancy of forwarded semantic information.
[0060] In another feasible embodiment, the semantic recovery module is used by the target node to recover semantic information from the signal forwarded by the relay node in order to restore the complete image of the target area to be observed by the target node. It includes a reshape layer, a linear mapping layer, a token embedding layer, a position encoding layer, and a transformer layer, thereby realizing the recovery of the complete image based on limited semantic information, improving the effect and efficiency of image recovery.
[0061] In another feasible embodiment, based on the trained semantic sending module, the semantic information fusion module and the semantic recovery module are trained according to the complete image dataset until the pre-set second loss function converges. For details of the specific implementation, please refer to the description of Embodiment 3 below, which will not be repeated here.
[0062] In this embodiment of the invention, based on a pre-built auxiliary receiver, the semantic sending module is trained on a local image dataset until a pre-set first loss function converges. Based on the trained semantic sending module, the semantic information fusion module and the semantic recovery module are trained on a complete image dataset until a pre-set second loss function converges. The multi-source semantic communication model includes a semantic sending module deployed in the source node, a semantic information fusion module deployed in the relay node, and a semantic recovery module deployed in the destination node. By adopting a phased training strategy, the training efficiency of the model in the multi-source relay communication network is improved, and the communication efficiency of the multi-source semantic communication model in multi-source communication is improved, reducing the waste of communication resources.
[0063] Example 2:
[0064] Figure 4 The implementation flow of the training method for the multi-source semantic communication model provided in Embodiment 2 of the present invention is shown. For ease of explanation, only the parts related to the embodiments of the present invention are shown, and are described in detail below:
[0065] In step S401, the local image dataset is preprocessed.
[0066] In this embodiment of the invention, each local image in the local image dataset is preprocessed. Specifically, the pixel values of each local image are standardized so that the model can better learn the features in the image and improve the performance and generalization ability of the subsequent model.
[0067] In step S402, samples are randomly drawn from the preprocessed local image dataset to obtain the first training samples.
[0068] In this embodiment of the invention, at least one local image is randomly selected from the preprocessed local image dataset, the selected local image is used as a first training sample, and the first training sample is input into the semantic sending module.
[0069] In step S403, the semantic feature processing of the first training sample is performed by the semantic sending module to obtain the semantic feature signal.
[0070] In this embodiment of the invention, when performing semantic feature processing on the first training sample through the semantic sending module, specifically, as follows: Figure 5 As shown, after the first training sample is input into the semantic transmission module, it passes through the Conv2D layer, two stacked ResBlock layers, the Flatten layer, the MLP layer, and the PowerNorm layer in sequence to obtain the semantic feature signal. The semantic feature signal is then input into the wireless channel module to simulate the wireless communication environment in the first hop transmission. The Conv2D layer and ResBlock are used to extract the semantic features of the first training sample. The Flatten layer is used to flatten the semantic feature map output by the semantic encoder into a one-dimensional feature vector. The MLP layer is used to map the flattened vector of the Flatten layer into a signal suitable for transmission in the wireless channel. The PowerNorm layer is used to normalize the mapped signal to keep the signal transmission power consistent.
[0071] In step S404, the received semantic feature signal is reconstructed by an auxiliary receiver to obtain the first restored image.
[0072] In this embodiment of the invention, the semantic feature signal is transmitted to an auxiliary receiver via a wireless channel. When the auxiliary receiver performs image reconstruction on the received semantic feature signal, specifically, as follows: Figure 5 As shown, the semantic feature signal passes through the Concat layer, MLP layer, Reshape layer, two stacked ResBlock layers, and TransConv2D layer in sequence to obtain the first restored image. This first restored image is semantically consistent with the first training sample sent. The Concat layer is used to concatenate the real and imaginary parts of the semantic feature signal transmitted through the wireless channel into a vector. The MLP layer is used to decode the semantic feature vector from the vector concatenated by the Concat layer. The Reshape layer is used to reconstruct the semantic feature vector decoded by the MLP layer into a semantic feature map. The two stacked ResBlock layers are used to learn more complex image features from the semantic feature map. The TransConv2D layer is used to restore the image that is semantically consistent with the original local image based on the features extracted by the ResBlock.
[0073] In step S405, based on the first training sample and the first recovered image, the first loss value of the first loss function is calculated. Based on the first loss value, the trainable parameters of the semantic sending module and the auxiliary receiver are updated. When the first loss function has not converged, the process jumps to step S402 to continue execution until the first loss function converges.
[0074] In this embodiment of the invention, the mean square error (MSE) between the first training sample and the first restored image is calculated, and the calculated mean square error is the first loss function. The first loss value is used as the basis for updating the trainable parameters of the semantic transmitter and the auxiliary receiver using gradient descent. Specifically, the Adam algorithm is used to perform gradient descent on the first loss function to optimize the trainable parameters of the semantic transmitter and the auxiliary receiver. If the first loss function has not converged, the process jumps to step S402 and continues until the first loss function converges. Here, M′ and These represent the first training sample and the first restored image, respectively.
[0075] In this embodiment of the invention, a local image dataset is preprocessed, and samples are randomly selected from the preprocessed local image dataset to obtain a first training sample. The semantic sending module performs semantic feature processing on the first training sample to obtain a semantic feature signal. The received semantic feature signal is then reconstructed by an auxiliary receiver to obtain a first restored image. Based on the first training sample and the first restored image, a first loss value of a first loss function is calculated. Based on the first loss value, the trainable parameters of the semantic sending module and the auxiliary receiver are updated. When the first loss function fails to converge, the process jumps to the step of randomly selecting samples from the preprocessed local image dataset and continues until the first loss function converges. This enables the semantic sending module to learn and extract key semantic features for image restoration, thereby improving the training effect of the semantic sending module.
[0076] Example 3:
[0077] Figure 6 The implementation flow of the training method for the multi-source semantic communication model provided in Embodiment 3 of the present invention is shown. For ease of explanation, only the parts related to the embodiments of the present invention are shown, and are described in detail below:
[0078] In step S601, the complete image dataset is preprocessed.
[0079] In this embodiment of the invention, each complete image in the complete image dataset is preprocessed. Specifically, the pixel values of each complete image are standardized so that the model can better learn the features in the image and improve the performance and generalization ability of the subsequent model.
[0080] In step S602, samples are randomly selected from the preprocessed complete image dataset to obtain the second training samples.
[0081] In this embodiment of the invention, at least one complete image is randomly selected from the preprocessed complete image dataset, and the selected complete image is used as a second training sample.
[0082] In step S603, the semantic sending module in the activated source node performs semantic feature processing on the local image collected by the source node from the second training sample to obtain the local semantic feature signal.
[0083] In this embodiment of the invention, all sensors are deployed at different locations in the target area to observe local images of the target area from different angles. After the semantic transmission module is deployed to all sensors, several of the sensors are randomly activated. Each activated sensor acquires images at different locations of the second training sample to obtain a local image corresponding to that location. The local image is then input into the semantic transmission module, which performs semantic feature processing on the local image acquired by its respective sensor to obtain the corresponding local semantic feature signal. Finally, the activated sensors take turns inputting their respective local semantic feature signals into the wireless channel module to simulate the wireless communication environment in the first hop transmission. The specific implementation of the semantic transmission module performing semantic feature processing on the local image is detailed in step S403 of Embodiment 2, and will not be repeated here.
[0084] In a feasible embodiment, the number of sensors to be activated, k, is first randomly obtained from the range [3n / 4, n]. Then, k sensors are randomly selected from all sensors for activation to observe k different locations in the target area. This allows the multi-source semantic communication model to adapt to different numbers of activated sensors, avoiding the need for retraining when the number of activated sensors is changed in practical applications, thus improving the robustness of the model. Here, n is the number of all sensors.
[0085] In another feasible embodiment, when each activated sensor acquires images at different locations of the second training sample, the second training sample is divided into k equal parts according to the number of activated sensors k, resulting in k local images. Each local image corresponds to a local image acquired by one of the activated sensors from the second training sample.
[0086] In step S604, the received local semantic feature signals are fused using a semantic information fusion module to obtain fused feature signals.
[0087] In this embodiment of the invention, the k local semantic feature signals received in the first hop are input to the semantic information fusion module of the relay node. The semantic information fusion module performs semantic information fusion on each local semantic feature signal to obtain a fused feature signal, where k is the number of activated sensors.
[0088] In a feasible embodiment, when performing semantic information fusion on the received local semantic feature signals through the semantic information fusion module, specifically, as follows: Figure 7 As shown, the first-hop signals (i.e., local semantic feature signals) transmitted by k sensors are first converted into feature vectors through the Concat layer. The k feature vectors are then encoded by the PosEmb layer to embed position information. After that, they pass through multiple Transformer Layers, which use their unique self-attention mechanism to focus on the correlation between different feature vectors and extract semantic information. The LinearProj layer removes semantic redundancy and obtains the most valuable part of this semantic information. Finally, the semantic information is fused through the Reshape layer and the PowerNorm layer, and mapped into a signal suitable for wireless channel transmission, i.e., a fused feature signal. This signal is then input into the wireless channel module to simulate the wireless communication environment in the second-hop transmission.
[0089] In step S605, the received fused feature signal is semantically restored by the semantic restoration module to obtain the second restored image.
[0090] In this embodiment of the invention, the fused feature signal received in the second hop is input to the semantic recovery module of the destination node, and the semantic recovery module performs semantic recovery on the fused feature signal to obtain the second recovered image, which is the restored complete image.
[0091] In a feasible embodiment, when performing semantic recovery on the received fused feature signal through the semantic recovery module, specifically, as follows: Figure 7As shown, the fused feature signal is first mapped into multiple feature vectors by a Reshape layer. These vectors are then linearly mapped by a LinearProj layer to extend their length. Next, multiple identical learnable vectors are embedded by a TokenEmb layer at the locations of the unsent sensors, so the number of embedded vectors is the same as the number of inactive sensors. After passing through multiple Transformer Layers, semantic information is extracted again. Then, another LinearProj layer is used to extend the dimension of the feature vectors through linear mapping. Finally, another Reshape layer is used to directly reshape the feature vectors into a complete image. This allows the destination node to recover the complete image from the limited semantic information even if the signal forwarded by the relay node initially comes from only some sensors (i.e., another part of the sensors are inactive / in hibernation and do not transmit local images). This effectively saves the communication costs of the sensors.
[0092] In another feasible embodiment, the number of Transformer Layers in the semantic information fusion module and the semantic recovery module is determined based on the number and / or size of the acquired local images, or the dimension of the extracted feature vectors. When the number, size, or dimension is large, more layers are needed to capture semantic information. It should be noted that more layers will also increase the model complexity, so it is necessary to find a balance between model complexity and generalization ability.
[0093] In another feasible embodiment, the number of Transformer Layers in the semantic information fusion module and the semantic recovery module is typically set to 6.
[0094] In step S606, based on the second training sample and the second restored image, the second loss value of the second loss function is calculated. Based on the second loss value, the trainable parameters of the semantic information fusion module and the semantic restoration module are updated. When the second loss function fails to converge, the process jumps to step S602 to continue execution until the second loss function converges.
[0095] In this embodiment of the invention, the mean square error between the second training sample and the second restored image is calculated, and the calculated mean square error is the second loss function. The second loss value is used as the basis for updating the trainable parameters of the semantic information fusion module and the semantic recovery module using gradient descent. Specifically, the Adam algorithm is used to perform gradient descent on the second loss function to optimize the trainable parameters of the semantic information fusion module and the semantic recovery module. If the second loss function does not converge, the process jumps to step S602 and continues until the second loss function converges. Here, M and... These represent the second training sample and the second restored image, respectively.
[0096] In this embodiment of the invention, the complete image dataset is preprocessed, and samples are randomly obtained from the preprocessed complete image dataset to obtain second training samples. The semantic sending module in the activated source node performs semantic feature processing on the local images collected from the second training samples by the source node to obtain local semantic feature signals. The semantic information fusion module performs semantic information fusion on the received local semantic feature signals to obtain fused feature signals. The semantic recovery module performs semantic recovery on the received fused feature signals to obtain a second recovered image. Based on the second training samples and the second recovered image, a second loss value of the second loss function is calculated. Based on the second loss value, the trainable parameters of the semantic information fusion module and the semantic recovery module are updated. If the second loss function fails to converge, the process jumps to the step of randomly acquiring samples from the preprocessed complete image dataset and continues until the second loss function converges. Thus, in the second stage of training, by randomly activating sensors, the multi-source semantic communication model adapts to different activation numbers, improving the robustness of the model. Furthermore, the semantic information fusion module enables the joint extraction of the most valuable semantic information from multiple sensor signals, avoiding the allocation of communication resources to redundant semantics, saving communication resources for relays, and improving the performance of the model. At the same time, without requiring all sensors to participate in image acquisition and transmission, the target node can still recover the complete image of the target region based on limited semantic information, effectively saving sensor communication consumption and further improving the performance of the model.
[0097] Example 4:
[0098] Figure 8 The implementation flow of the image transmission method in a multi-source relay communication network provided in Embodiment 4 of the present invention is illustrated. For ease of explanation, only the parts related to the embodiments of the present invention are shown, and are described in detail below:
[0099] In step S801, a complete image of the target area to be observed is obtained;
[0100] In step S802, the semantic sending module in the activated source node performs semantic feature processing on the local image acquired by the source node from the complete target image to obtain the target local semantic signal.
[0101] In step S803, the received local semantic signals are fused using a semantic information fusion module to obtain the target fused signal;
[0102] In step S804, the semantic restoration module performs semantic restoration on the received target fusion signal to obtain the target restored image.
[0103] In embodiments of the present invention, such as Figure 9As shown, based on the multi-source semantic communication model trained in Embodiments 1 to 3 above, the trained semantic sending module is deployed to all sensors, while the semantic information fusion module and semantic recovery module are deployed to the relay and destination nodes, respectively. First, some activated sensors acquire local images of the obtained complete target image. The acquired local images are processed by the semantic sending module in the sensor to obtain the corresponding target local semantic signal. Then, the activated sensors take turns sending the target local semantic signal containing the semantic information of the local image to the relay node in the actual wireless network environment. After one round of transmission, the relay node uses the semantic information fusion module to extract the most valuable semantic information from the received sensor signals and forwards this semantic information to the destination node. The destination node uses the semantic recovery module to restore the complete image of the target area as much as possible, thereby saving the relay communication resources in the image transmission task, improving communication efficiency, enhancing system performance, and improving image recovery effect.
[0104] Example 5:
[0105] Figure 10 The structure of the training device for the multi-source semantic communication model provided in Embodiment 5 of the present invention is shown. For ease of explanation, only the parts related to the embodiments of the present invention are shown, including:
[0106] The first training unit 101 is used to train the semantic sending module based on a pre-built auxiliary receiver and a local image dataset until the pre-set first loss function converges.
[0107] The second training unit 102 is used to train the semantic information fusion module and the semantic recovery module based on the trained semantic sending module and the complete image dataset until the pre-set second loss function converges.
[0108] In this embodiment of the invention, each unit of the training device for the multi-source semantic communication model can be implemented by a corresponding hardware or software unit. Each unit can be an independent hardware or software unit, or it can be integrated into a single hardware or software unit, which is not intended to limit the invention. Specifically, the implementation methods of each unit can be referred to the description of the foregoing method embodiments, and will not be repeated here.
[0109] Example 6:
[0110] Figure 11 The structure of the communication device provided in Embodiment Six of the present invention is shown. For ease of explanation, only the parts related to the embodiments of the present invention are shown.
[0111] The communication device 11 of this embodiment includes a processor 110, a memory 111, and a computer program 112 stored in the memory 111 and executable on the processor 110. When the processor 110 executes the computer program 112, it implements the steps in the above-described training method embodiment of the multi-source semantic communication model, for example... Figure 1 The steps S101 to S102 are shown. Alternatively, when the processor 110 executes the computer program 112, it implements the functions of each unit in the above-described device embodiments, for example... Figure 10 The functions of units 101 to 102 shown.
[0112] In this embodiment of the invention, based on a pre-built auxiliary receiver, the semantic sending module is trained on a local image dataset until a pre-set first loss function converges. Based on the trained semantic sending module, the semantic information fusion module and the semantic recovery module are trained on a complete image dataset until a pre-set second loss function converges. The multi-source semantic communication model includes a semantic sending module deployed in the source node, a semantic information fusion module deployed in the relay node, and a semantic recovery module deployed in the destination node. By adopting a phased training strategy, the training efficiency of the model in the multi-source relay communication network is improved, and the communication efficiency of the multi-source semantic communication model in multi-source communication is improved, reducing the waste of communication resources.
[0113] The steps implemented by the processor 110 in the communication device 11 of this embodiment when executing the computer program 112 to implement the training method of the multi-source semantic communication model can be referred to the description of the foregoing method embodiment, and will not be repeated here.
[0114] Example 7:
[0115] In this embodiment of the invention, a computer-readable storage medium is provided, which stores a computer program. When executed by a processor, the computer program implements the steps in the above-described training method embodiment for the multi-source semantic communication model. For example... Figure 1 The steps S101 to S102 are shown. Alternatively, when the computer program is executed by the processor, it implements the functions of each unit in the above-described device embodiments, for example... Figure 10 The functions of units 101 to 102 shown.
[0116] In this embodiment of the invention, based on a pre-built auxiliary receiver, the semantic sending module is trained on a local image dataset until a pre-set first loss function converges. Based on the trained semantic sending module, the semantic information fusion module and the semantic recovery module are trained on a complete image dataset until a pre-set second loss function converges. The multi-source semantic communication model includes a semantic sending module deployed in the source node, a semantic information fusion module deployed in the relay node, and a semantic recovery module deployed in the destination node. By adopting a phased training strategy, the training efficiency of the model in the multi-source relay communication network is improved, and the communication efficiency of the multi-source semantic communication model in multi-source communication is improved, reducing the waste of communication resources.
[0117] The computer-readable storage medium in embodiments of the present invention may include any entity or device capable of carrying computer program code, a recording medium, such as ROM / RAM, disk, optical disk, flash memory, etc.
[0118] The above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included within the protection scope of the present invention.
Claims
1. A method for training a multi-source semantic communication model, characterized in that, The multi-source semantic communication model includes a semantic transmission module deployed in the source node, a semantic information fusion module deployed in the relay node, and a semantic recovery module deployed in the destination node. The method includes the following steps: Based on a pre-built auxiliary receiver, the semantic sending module is trained on a local image dataset until the pre-set first loss function converges; The trained semantic sending module is deployed to all source nodes. Based on the trained semantic sending module, the semantic information fusion module and the semantic recovery module are trained according to the complete image dataset until the pre-set second loss function converges. The number of source nodes is not less than 2. The step of training the semantic transmission module based on a pre-built auxiliary receiver and a local image dataset until a pre-set first loss function converges includes: The local image dataset is preprocessed; The first training sample is obtained by randomly selecting samples from the preprocessed local image dataset; The semantic sending module performs semantic feature processing on the first training sample to obtain a semantic feature signal; The received semantic feature signal is reconstructed using the auxiliary receiver to obtain a first restored image; Based on the first training sample and the first restored image, calculate the first loss value of the first loss function, update the trainable parameters of the semantic sending module and the auxiliary receiver based on the first loss value, and when the first loss function fails to converge, jump to the step of randomly sampling samples from the preprocessed local image dataset and continue execution until the first loss function converges. The step of training the semantic information fusion module and the semantic recovery module based on the trained semantic sending module using the complete image dataset until the pre-set second loss function converges includes: The complete image dataset is preprocessed; Randomly select samples from the preprocessed complete image dataset to obtain the second training samples; The semantic sending module in the activated source node performs semantic feature processing on the local image collected by the source node from the second training sample to obtain a local semantic feature signal. The semantic information fusion module performs semantic information fusion on the received local semantic feature signals to obtain fused feature signals; The semantic recovery module performs semantic recovery on the received fused feature signal to obtain a second recovered image; Based on the second training sample and the second restored image, calculate the second loss value of the second loss function. Based on the second loss value, update the trainable parameters of the semantic information fusion module and the semantic restoration module. When the second loss function fails to converge, jump to the step of randomly obtaining samples from the preprocessed complete image dataset and continue execution until the second loss function converges.
2. The method of claim 1, wherein, The semantic transmission module includes a semantic encoder and a channel encoder. The semantic encoder consists of a two-dimensional convolutional layer and two residual blocks, and the channel encoder consists of a flattening layer, a multi-layer perceptron layer, and a power normalization layer.
3. The method of claim 1, wherein, The auxiliary receiver includes a channel decoder and a semantic decoder. The channel decoder consists of a connection layer, a multi-layer perceptron layer, and a reshaping layer. The semantic decoder consists of two residual blocks and a two-dimensional deconvolution layer.
4. The method as described in claim 1, characterized in that, The semantic information fusion module includes a connection layer, a position encoding layer, a converter layer, a linear mapping layer, a reshaping layer, and a power normalization layer. The semantic recovery module includes a reshaping layer, a linear mapping layer, a tag embedding layer, a position encoding layer, and a converter layer.
5. An image transmission method in a multi-source relay communication network, characterized in that, Based on the training method of the multi-source semantic communication model as described in any one of claims 1 to 4, the multi-source semantic communication model is obtained by deploying the semantic sending module in all source nodes, deploying the semantic information fusion module in the multi-source semantic communication model in relay nodes, and deploying the semantic recovery module in the multi-source semantic communication model in destination nodes. The method includes: Acquire a complete image of the target area to be observed; The semantic sending module in the activated source node performs semantic feature processing on the local image acquired by the source node from the complete target image to obtain the target local semantic signal. The received local semantic signals are fused using the semantic information fusion module to obtain the target fused signal. The semantic restoration module performs semantic restoration on the received target fusion signal to obtain a target restored image.
6. A training device for a multi-source semantic communication model, characterized in that, The multi-source semantic communication model includes a semantic transmission module deployed in the source node, a semantic information fusion module deployed in the relay node, and a semantic recovery module deployed in the destination node. The device includes: The first training unit is used to train the semantic sending module based on a pre-built auxiliary receiver and a local image dataset until the pre-set first loss function converges. The second training unit is used to deploy the trained semantic sending module to all source nodes, and based on the trained semantic sending module, train the semantic information fusion module and the semantic recovery module according to the complete image dataset until the pre-set second loss function converges. The number of source nodes is not less than 2. The first training unit trains the semantic transmission module based on a pre-built auxiliary receiver and a local image dataset until a pre-set first loss function converges, including: The local image dataset is preprocessed; The first training sample is obtained by randomly selecting samples from the preprocessed local image dataset; The semantic sending module performs semantic feature processing on the first training sample to obtain a semantic feature signal; The received semantic feature signal is reconstructed using the auxiliary receiver to obtain a first restored image; Based on the first training sample and the first restored image, calculate the first loss value of the first loss function, update the trainable parameters of the semantic sending module and the auxiliary receiver based on the first loss value, and when the first loss function fails to converge, jump to the step of randomly sampling samples from the preprocessed local image dataset and continue execution until the first loss function converges. The second training unit, based on the trained semantic sending module, trains the semantic information fusion module and the semantic recovery module according to the complete image dataset until the pre-set second loss function converges, including: The complete image dataset is preprocessed; Randomly select samples from the preprocessed complete image dataset to obtain the second training samples; The semantic sending module in the activated source node performs semantic feature processing on the local image collected by the source node from the second training sample to obtain a local semantic feature signal. The semantic information fusion module performs semantic information fusion on the received local semantic feature signals to obtain fused feature signals; The semantic recovery module performs semantic recovery on the received fused feature signal to obtain a second recovered image; Based on the second training sample and the second restored image, calculate the second loss value of the second loss function. Based on the second loss value, update the trainable parameters of the semantic information fusion module and the semantic restoration module. When the second loss function fails to converge, jump to the step of randomly obtaining samples from the preprocessed complete image dataset and continue execution until the second loss function converges.
7. The apparatus as claimed in claim 6, characterized in that, The semantic transmission module includes a semantic encoder and a channel encoder. The semantic encoder consists of a two-dimensional convolutional layer and two residual blocks, and the channel encoder consists of a flattening layer, a multi-layer perceptron layer, and a power normalization layer.
8. The apparatus as claimed in claim 6, characterized in that, The auxiliary receiver includes a channel decoder and a semantic decoder. The channel decoder consists of a connection layer, a multi-layer perceptron layer, and a reshaping layer. The semantic decoder consists of two residual blocks and a two-dimensional deconvolution layer.
9. A communication device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the steps of the method as described in any one of claims 1 to 5.
10. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the method as described in any one of claims 1 to 5.
Citation Information
Patent Citations
Semantic communication method, device and system for image classification task
CN115761758A
Semantic communication model training method and device
CN117896739A
