Semantic communication method and system, electronic equipment and computer readable storage medium

Abstract

The invention provides a semantic communication method and system, electronic equipment and a computer readable storage medium, and relates to the field of communication, in particular to the field of semantic communication and artificial intelligence. According to the specific implementation scheme, the method comprises the steps of inputting information source data into a pre-trained semantic encoder for semantic encoding, obtaining information source semantic features of the information source data, mapping the information source semantic features to a target codebook, and generating an information source index vector; transmitting the information source index vector from the transmitting end to the receiving end through the communication channel, and obtaining a channel index vector; obtaining a channel semantic feature according to the target codebook and the channel index vector, inputting the channel semantic feature into a pre-trained semantic decoder for semantic decoding, and generating reconstruction data; wherein the target codebook is determined according to the signal quality of the communication channel and an initial codebook; the initial codebook is obtained through joint training with the semantic encoder and the semantic decoder.

Description

Technical Field

[0001] The present disclosure relates to the field of communication technology, in particular to the fields of semantic communication, artificial intelligence, etc. Specifically, the present disclosure relates to a semantic communication method and system, an electronic device, and a computer-readable storage medium. Background Art

[0002] With the rapid development of wireless communication technology, especially in the context of the rapid evolution of 6G (sixth-generation mobile communication standard technology) wireless communication systems, semantic communication has attracted widespread attention due to its efficient data transmission capabilities.

[0003] Traditional communication systems primarily focus on bit-level lossless transmission to ensure data integrity. However, with the surge in data volumes, especially the widespread use of high-dimensional data (such as high-resolution images and videos) in emerging applications (such as autonomous driving, extended reality (XR), and smart cities), the limitations of traditional methods are becoming increasingly apparent in scenarios with limited bandwidth and harsh channel conditions, such as low SNR (Signal to Interference plus Noise Ratio).

[0004] In contrast, semantic communication significantly reduces the transmission of redundant information by extracting and transmitting the core semantic content of data rather than the entire bitstream of the original data, thereby enabling more efficient and intelligent information exchange within limited resources. This approach not only reduces bandwidth requirements but also demonstrates greater robustness under challenging conditions such as low signal-to-noise ratios (SNRs), providing a new development direction for next-generation communication systems. Summary of the Invention

[0005] The present disclosure provides a semantic communication method and system, an electronic device, and a computer-readable storage medium.

[0006] According to a first aspect of the present disclosure, a semantic communication method is provided. The method is used in a semantic communication system, wherein the semantic communication system includes a sending end, a receiving end, and a communication channel connecting the sending end and the receiving end. The method includes:

[0007] Inputting the source data into a pre-trained semantic encoder for semantic encoding, obtaining source semantic features of the source data, mapping the source semantic features to a target codebook, and generating a source index vector;

[0008] Transmitting the source index vector from the transmitting end to the receiving end through the communication channel to obtain a channel index vector;

[0009] Acquire channel semantic features according to the target codebook and the channel index vector, input the channel semantic features into a pre-trained semantic decoder for semantic decoding, and generate reconstructed data;

[0010] The target codebook is determined according to the signal quality of the communication channel and an initial codebook; the initial codebook is obtained by joint training with the semantic encoder and the semantic decoder.

[0011] According to a second aspect of the present disclosure, there is provided a semantic communication system, the system comprising a sending end, a receiving end, and a communication channel connecting the sending end and the receiving end;

[0012] The transmitting end is configured to input the source data into a pre-trained semantic encoder for semantic encoding, obtain source semantic features of the source data, map the source semantic features to a target codebook, and generate a source index vector;

[0013] The communication channel is used to transmit the source index vector from the transmitting end to the receiving end;

[0014] The receiving end is used to obtain a channel index vector, obtain a channel semantic feature according to the target codebook and the channel index vector, input the channel semantic feature into a pre-trained semantic decoder for semantic decoding, and generate reconstructed data;

[0015] The target codebook is determined according to the signal quality of the communication channel and an initial codebook; the initial codebook is obtained by joint training with the semantic encoder and the semantic decoder.

[0016] According to a third aspect of the present disclosure, an electronic device is provided, including:

[0017] at least one processor; and

[0018] A memory communicatively connected to the at least one processor; wherein,

[0019] The memory stores instructions that can be executed by the at least one processor. The instructions are executed by the at least one processor so that the at least one processor can perform the semantic communication method.

[0020] According to a fourth aspect of the present disclosure, a non-transitory computer-readable storage medium storing computer instructions is provided, wherein the computer instructions are used to enable a computer to execute the above-mentioned semantic communication method.

[0021] According to a fifth aspect of the present disclosure, a computer program product is provided, comprising a computer program, which implements the above-mentioned semantic communication method when executed by a processor.

[0022] It should be understood that the contents described in this section are not intended to identify the key or important features of the embodiments of the present disclosure, nor are they intended to limit the scope of the present disclosure. Other features of the present disclosure will become readily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] The accompanying drawings are provided to facilitate a better understanding of the present invention and do not constitute a limitation of the present disclosure.

[0024] Figure 1 is a flowchart of a semantic communication method provided by an embodiment of the present disclosure;

[0025] Figure 2 is a flowchart of some steps of another semantic communication method provided by an embodiment of the present disclosure;

[0026] Figure 3 is a flowchart of some steps of another semantic communication method provided by an embodiment of the present disclosure;

[0027] Figure 4 1 is a schematic diagram of the structure of a codebook optimization network used in another semantic communication method provided by an embodiment of the present disclosure;

[0028] Figure 5 is a flowchart of some steps of another semantic communication method provided by an embodiment of the present disclosure;

[0029] Figure 6 This is a process diagram of a specific embodiment of another semantic communication method provided by an embodiment of the present disclosure;

[0030] Figure 7 is a structural diagram of a semantic communication system provided by an embodiment of the present disclosure;

[0031] Figure 8 It is a block diagram of an electronic device used to implement the semantic communication method according to an embodiment of the present disclosure. DETAILED DESCRIPTION

[0032] The following description of exemplary embodiments of the present disclosure is made in conjunction with the accompanying drawings, including various details of the embodiments of the present disclosure to facilitate understanding. These details should be considered as merely exemplary. Therefore, those skilled in the art will recognize that various changes and modifications may be made to the embodiments described herein without departing from the scope and spirit of the present disclosure. Similarly, for the sake of clarity and conciseness, descriptions of well-known functions and structures are omitted in the following description.

[0033] The terms that may appear in the embodiments of the present disclosure and their corresponding explanations are as follows:

[0034] Semantic coding: Capturing the semantic information contained in business information. Depending on the semantic type, this can be categorized into text semantic coding, image semantic coding, audio semantic coding, video semantic coding, and point cloud semantic coding. The possible types of semantic coding are related to the possible types of business information and can encompass all possible types of semantic coding for communication transmission. The generation of semantic coding also depends on the semantic coding model, which can utilize models from various disciplines, such as artificial intelligence, deep learning, and pattern recognition.

[0035] Transmitter and Receiver: A communication channel connects the two ends. The transmitter (specifically, a transmitter) has an encoder, and the receiver (specifically, a receiver) has a decoder. This disclosure is based on semantic encoding and decoding. Therefore, the transmitter has a semantic encoder, and the receiver has a semantic decoder. Both the transmitter and receiver operate at the semantic layer, with the underlying Shannon physical layer remaining the underlying layer.

[0036] Semantic communication system: includes the sending end, the receiving end and the communication channel connecting the sending end and the receiving end as described above.

[0037] Model: The models referred to in this invention include models from disciplines such as machine learning, artificial intelligence, and neural networks. Models used to semantically encode and decode business information are referred to as semantic encoding models and semantic decoding models. Depending on the type of business information, models can be categorized as text models, audio models, image models, video models, point cloud models, one-dimensional waveform models, radar data models, and so on. The type of model is related to the type of semantic encoding and the type of business information, and can be models of all possible types of communication transmission.

[0038] Codebook: includes multiple available codewords identified by serial numbers, etc. Codewords can be searched from the codebook based on the serial numbers, etc.

[0039] In some related technologies, codebook-based semantic communication methods discretize high-dimensional data into compact and interpretable codewords, improving coding efficiency and enhancing the system's resistance to channel noise.

[0040] Specifically, an encoder can be used to map input data into a feature space, quantize the features mapped to the feature space, and select codewords from the codebook based on their similarity after quantization, thereby effectively compressing and representing semantic information. This approach has demonstrated certain advantages in both theoretical analysis and experimental verification, especially in scenarios such as image transmission, where it can achieve semantic data transmission at a low bandwidth cost.

[0041] However, due to codebook capacity limitations, some semantic information is lost during the quantization process, resulting in reduced image reconstruction quality. Under low SNR conditions, codeword index transmission is susceptible to channel noise, generating semantic noise that further affects reconstruction. Lossless codeword index transmission increases system design complexity and resource overhead, limiting its practicality in noisy channels.

[0042] The semantic communication method, semantic communication system, electronic device, and computer-readable storage medium provided by the embodiments of the present disclosure are intended to solve at least one of the above technical problems in the prior art.

[0043] The semantic communication method provided in the embodiments of the present disclosure can be executed by an electronic device such as a terminal device or a server. The terminal device can be an in-vehicle device, a user equipment (UE), a mobile device, a user terminal, a terminal, a cellular phone, a cordless phone, a personal digital assistant (PDA), a handheld device, a computing device, an in-vehicle device, a wearable device, etc. The method can be implemented by a processor calling computer-readable program instructions stored in a memory. Alternatively, the method can be executed by a server.

[0044] Figure 1 FIG. 1 shows a flow chart of the semantic communication method provided by the embodiment of the present disclosure. Figure 1 As shown in , the semantic communication method provided by the embodiment of the present disclosure is used in the semantic communication system as described above, including step S110, step S120, and step S130.

[0045] In step S110, the source data is input into a pre-trained semantic encoder for semantic encoding, the source semantic features of the source data are obtained, the source semantic features are mapped to the target codebook, and a source index vector is generated;

[0046] In step S120, the source index vector is transmitted from the transmitting end to the receiving end via the communication channel to obtain a channel index vector;

[0047] In step S130, channel semantic features are obtained according to the target codebook and the channel index vector, and the channel semantic features are input into a pre-trained semantic decoder for semantic decoding to generate reconstructed data;

[0048] The target codebook is determined according to the signal quality of the communication channel and the initial codebook; the initial codebook is obtained by joint training with the semantic encoder and the semantic decoder.

[0049] For example, the source data may be image data, voice data, text data, point cloud data, or a mixture of any of the above data.

[0050] In some possible implementations, in step S110 , step S110 may be performed at the transmitting end.

[0051] The semantic encoder is a model used to semantically encode source data. Therefore, inputting the source data into the semantic encoder can obtain the source semantic features of the source data.

[0052] In some possible implementations, the semantic encoder can specifically be a network structure based on Transformer (a deep learning architecture), which can capture the global semantic information of the source data through multi-scale feature extraction.

[0053] In some possible implementations, the semantic encoder may further include a Swin Transformer (Hierarchical Visual Transformer) module, which may be used to model long-range dependencies of data (especially image data).

[0054] The embodiments of the present disclosure do not limit the specific type of semantic encoders. Any model that can implement semantic encoding is within the protection scope of the embodiments of the present disclosure.

[0055] In some possible implementations, the source semantic features may be mapped to a target codebook using a vector quantization method to generate a source index vector.

[0056] In some possible implementations, vector quantization is achieved using the Gumbel-Softmax technique. Specifically, the inner product of the source semantic features and each codeword in the target codebook is calculated, Gumbel noise is added, and then the source index vector is generated using a Softmax (normalized exponential function).

[0057] The target codebook may be generated based on the initial codebook. The initial codebook may be trained. Specifically, the initial codebook, the semantic encoder, and the semantic decoder may be jointly trained using training data.

[0058] After joint training is complete, the initial codebook is used as the target codebook for the first semantic communication. If multiple semantic communications have been performed, the signal quality of the communication channel can be determined based on the semantic communications that have already been performed, and the initial codebook can be adjusted to generate the target codebook. Step S110 is then executed.

[0059] The signal quality of a communication channel can be determined based on the channel's Signal to Interference plus Noise Ratio (SNR). The SNR is the ratio of the strength of the received useful signal to the strength of the received interfering signal (noise and interference), and can be used to measure signal quality.

[0060] In some possible implementations, in step S120, the source index vector can be modulated, and after being modulated into a signal that can be transmitted in a communication channel, it is transmitted from the transmitting end to the receiving end through the communication channel. After being transmitted to the receiving end, the signal is demodulated to obtain the channel index vector.

[0061] The communication channel may be a wireless channel. The embodiments of the present disclosure do not impose any limitation on the signal modulation and signal demodulation methods, and any method that can implement signal modulation and signal demodulation falls within the protection scope of the embodiments of the present disclosure.

[0062] Since the communication channel has communication noise, the source index vector and the channel index vector are not completely identical. If the communication channel does not have communication noise, the source index vector and the channel index vector are identical.

[0063] In some possible implementations, in step S130, after obtaining the channel index vector, the receiving end recovers the channel index vector from the target codebook using inverse vector quantization to convert it into semantic features, i.e., channel semantic features. The channel semantic features are input into a semantic decoder for semantic decoding to generate reconstructed data.

[0064] The reconstructed data generated is the data received by the receiver. Vector quantization and inverse vector quantization are the opposite processes. The semantic decoder is the counterpart of the semantic encoder and is used to decode semantic features. The initial codebook, semantic encoder, and semantic decoder can be jointly trained using training data.

[0065] The embodiments of the present disclosure do not limit the specific type of semantic decoder, and any model that can implement semantic decoding is within the protection scope of the embodiments of the present disclosure.

[0066] The initial codebook, semantic encoder, and semantic decoder are obtained through joint training. Therefore, the semantic features obtained by the semantic encoder are consistent with the initial codebook. After the obtained semantic features are mapped to the target codebook, important semantic features can be preserved, the loss of semantic features can be minimized, and the similarity between the obtained reconstructed data and the source data can be ensured, thereby ensuring the quality of the obtained reconstructed data.

[0067] The initial codebook is shared by the transmitter and receiver, and contains multiple codewords. The dimension of each codeword is consistent with the dimension of the semantic feature, and the initial codebook is synchronously stored by the transmitter and receiver before the first semantic communication.

[0068] In some related semantic communication methods, the static codebook cannot be adaptively adjusted according to the real-time communication channel status, resulting in large fluctuations in the system performance under different communication channel conditions. In particular, in low SNR scenarios, the data reconstruction quality is significantly reduced.

[0069] In the semantic communication method provided in the embodiment of the present disclosure, the target codebook is dynamically adjusted using the signal quality of the communication channel, so that the semantic communication system can adapt to the channel states of different communication channels (such as low SNR and high SNR), thereby improving the robustness of the semantic communication system and the quality of data reconstruction.

[0070] The semantic communication method provided by the embodiment of the present disclosure is described in detail below.

[0071] As described above, in some possible implementations, the signal quality of the communication channel may be determined based on the semantic communication that has been performed, and then the initial codebook may be adjusted to generate a target codebook.

[0072] Figure 2 FIG. 1 shows a flow chart of an implementation method for determining the signal quality of a communication channel, adjusting the initial codebook according to the signal quality of the communication channel, and generating a target codebook. Figure 2 As shown, it may include step S210 and step S220.

[0073] In step S210, the signal quality of the communication channel is calculated based on at least the channel index vector to obtain a signal quality parameter;

[0074] In step S220, the initial codebook is adjusted according to the signal quality parameter to generate a new target codebook.

[0075] In some possible implementations, in step S210, the SNR of the communication channel is calculated as a signal quality parameter based on the signal received by the receiving end from the communication channel before the semantic communication (such as the signal corresponding to the channel index vector).

[0076] In some possible implementations, in step S220 , the common information and individual information in the initial codebook are adjusted according to the signal quality parameter to generate an optimized codebook, ie, a target codebook.

[0077] After executing this semantic communication, the SNR of the communication channel can also be calculated based on the signal received by the receiving end from the communication channel during this semantic communication process (such as the signal corresponding to the channel index vector) as a signal quality parameter. According to the signal quality parameter, the common information and individual information of the initial codebook are adjusted to generate an optimized codebook, that is, a new target codebook.

[0078] Through the above method, public information can be enhanced under low SNR conditions to counteract noise interference in the communication channel, and individual information can be added under high SNR conditions to improve transmission capacity and data reconstruction accuracy, ensuring the robustness of the semantic communication system under complex channels.

[0079] Since some communication channels are very stable, the channel quality will not change in real time. Figure 2 The method shown can be executed at a set execution frequency and executed at a certain frequency, which can be determined according to the specific conditions of the communication channel.

[0080] Figure 3 FIG. 4 shows a flow chart of an implementation method of adjusting the initial codebook according to the signal quality parameter to generate a new target codebook, as shown in FIG. Figure 3 As shown, it may include step S310, step S320, and step S330.

[0081] In step S310, the semantic features of the codewords in the initial codebook are mapped to the latent space to obtain the latent space feature vector;

[0082] In step S320, a modulation factor is generated according to the signal quality parameter, and the feature weight of the latent space feature vector is adjusted according to the modulation factor;

[0083] In step S330, a scaling matrix is ​​generated according to the adjusted latent space eigenvector, and a new target codebook is generated according to the scaling matrix and the initial codebook.

[0084] In some possible implementations, in step S310 , the semantic features of the codewords in the initial codebook are mapped to a higher-level feature space (ie, latent space) to obtain a latent space feature vector corresponding to the codeword.

[0085] In some possible implementations, in step S320 , a modulation factor is generated according to the SNR, the modulation factor is multiplied element-by-element by the latent space feature vector, and the feature weight is dynamically adjusted to balance the ratio of common information and individual information in the codebook.

[0086] In some possible implementations, in step S330, the adjusted latent space feature vector is mapped to a scaling matrix, and the ratio of common information to individual information in the initial codebook is dynamically adjusted through the scaling matrix to generate a new target codebook to adapt to the channel conditions of different communication channels.

[0087] In some possible implementations, Figure 3 The method shown can be implemented based on a codebook optimization network. That is, based on the codebook optimization network, the initial codebook is adjusted according to the signal quality parameter to generate a new target codebook.

[0088] Figure 4 The schematic diagram of the codebook optimization network is shown in FIG. Figure 4 As shown, the codebook optimization network includes multiple SMMs (Semantic Modulation and Mapping modules), multiple AMs (Adaptive Modulation modules) or SNR-AMs, and an activation module.

[0089] Among them, the semantic mapping module and the adaptive modulation module are set alternately; the activation module is connected to the last semantic mapping module; specifically, Figure 4 As shown in Figure 1, the codebook optimization network can include five semantic mapping modules and four adaptive modulation modules. These semantic mapping modules and adaptive modulation modules are arranged alternately, where the input of the first semantic mapping module is the initial codebook, and the last semantic mapping module is connected to the activation module, and the output of the activation module is the optimized codebook.

[0090] The semantic mapping module is used to extract the semantic features of the codewords in the initial codebook, map the semantic features of the codewords to the latent space, and obtain the latent space feature vector.

[0091] Each SMM contains three fully connected layers (FCs), which can be configured in one of the following ways depending on their location in the network:

[0092] The first configuration (D, N, N): The first layer input dimension is D (codeword feature vector dimension), and the number of neurons in the second and third layers is N (latent space feature vector dimension). It is suitable for processing the semantic features of the codewords in the initial codebook and outputting the latent space feature vector (dimension N).

[0093] The second configuration (N,N,N): The number of neurons in each layer is N, and the input and output dimensions are both N. It is used for the conversion of intermediate features to enhance the feature expression capability.

[0094] The third configuration (N,N,D): The number of neurons in the first and second layers is N, the output dimension of the third layer is D, and the feature representation of the optimized codebook is adapted;

[0095] The five SMM modules are arranged in sequence and output feature vectors in sequence. Their dimensions are N, N, N, N, and D according to the module configuration, to match the subsequent AM modulation requirements and generate the feature representation of the optimized codebook.

[0096] The adaptive modulation module is used to generate a modulation factor according to the signal quality parameter and adjust the feature weight of the latent space feature vector according to the modulation factor.

[0097] Each AM adjusts the feature weight according to the signal quality parameter. Its structure includes the following layers: a fully connected layer (FC) with an input dimension of 1 and an output dimension of N, and the activation function is ReLU; two fully connected layers (FC) with an input and output dimension of N in each layer, and the activation function is ReLU; this module generates a modulation factor (dimension N) based on the signal quality parameter, and the modulation factor is multiplied element-by-element with the output feature of the previous SMM, dynamically adjusting the feature weight to balance the ratio of common information and individual information in the codebook.

[0098] The activation module is used to generate a scaling matrix according to the adjusted latent space eigenvector, and to generate a new target codebook according to the scaling matrix and the initial codebook.

[0099] The activation module receives the output of the last SMM and generates a scaling matrix based on the Sigmoid function. It dynamically adjusts the ratio of common information to individual information in the initial codebook and generates an optimized codebook to adapt to different channel conditions.

[0100] As described above, in some possible implementations, residual coding can be used to capture and encode the difference between the original semantic features and the quantized vectors, thereby improving the quality of data reconstruction.

[0101] Figure 5 FIG. 1 is a flow chart showing an implementation method of improving data reconstruction quality by using residual coding, as shown in FIG. Figure 5 As shown, it may include step S510, step S520, and step S530.

[0102] In step S510, according to the target codebook, the source index vector is restored to the quantized semantic feature, the source information difference between the source semantic feature and the quantized semantic feature is calculated, and the source information difference is encoded into source residual information using a pre-trained semantic detail encoder;

[0103] In step S520, the source residual information is transmitted from the transmitting end to the receiving end via the communication channel to obtain the channel residual information;

[0104] In step S530, the channel residual information is input into a pre-trained semantic detail decoder for decoding to obtain channel information difference; the channel semantic features and the channel information difference are input into a pre-trained semantic decoder for semantic decoding to generate reconstructed data;

[0105] Among them, the initial codebook, semantic encoder, semantic decoder, semantic detail encoder, and semantic detail decoder are obtained through joint training.

[0106] In some possible implementations, in step S510, based on the inverse vector quantization technology, according to the target codebook, the source index vector is restored to the quantized semantic feature, and the residual between the quantized semantic feature and the source semantic feature, that is, the source information difference, is calculated; the source information difference is encoded by the semantic detail encoder to generate source residual information.

[0107] Among them, the semantic detail encoder adopts a CNN (convolutional neural network)-based architecture to extract residual information through local features, where the residual information includes the difference between quantized semantic features and source semantic features, which is used to retain the fine-grained details of the data.

[0108] In other words, the source residual information retains the fine-grained details of the data, as well as the information loss caused by quantization.

[0109] In some possible implementations, in step S520, the source residual information may be transmitted from the transmitting end to the receiving end through a communication channel in the same manner as the source index vector.

[0110] In some possible implementations, the source residual information may be transmitted from the transmitting end to the receiving end through a communication channel in a manner different from the source index vector.

[0111] Specifically, the source residual information may be independently channel-coded, and the coded source residual information may be transmitted from the transmitting end to the receiving end via a communication channel to obtain channel residual information.

[0112] Encoding and transmitting the source index vector and the source residual information in different ways can ensure that the receiving end accurately receives both types of data.

[0113] In some possible implementations, in step S530, the channel residual information is input into the semantic detail decoder for decoding, the channel information difference is obtained, the channel semantic features and the channel information difference are added, and the addition result is input into the pre-trained semantic decoder for semantic decoding to generate reconstructed data.

[0114] Residual coding is introduced to capture and encode the differences between the original semantic features and the quantized semantic features, thereby improving the quality of data reconstruction.

[0115] Among them, the initial codebook, semantic encoder, semantic decoder, semantic detail encoder, and semantic detail decoder are obtained through joint training.

[0116] As described above, in some possible implementations, the source residual information may be transmitted from the transmitting end to the receiving end via a communication channel in a manner different from the source index vector.

[0117] The source residual information is independently channel-coded, and the source index vector is transmitted by generating a constellation point set through K-order orthogonal amplitude modulation or K-order phase shift keying.

[0118] Specifically, a constellation point set is generated by K-order quadrature amplitude modulation or K-order phase shift keying for a source index vector; the constellation point set is transmitted from a transmitting end to a receiving end through a communication channel to obtain a modulated signal; and the modulated signal is demodulated to obtain a channel index vector.

[0119] Independent channel coding is performed on the source residual information, the coded source residual information is modulated to generate a modulated signal, the modulated signal is transmitted from the transmitting end to the receiving end through a communication channel, and the signal is demodulated to obtain channel residual information.

[0120] As described above, in some possible implementations, the initial codebook, semantic encoder, and semantic decoder may be trained using training data. The specific training process may include:

[0121] Inputting the training data into a semantic encoder to obtain a first semantic feature corresponding to the training data, mapping the first semantic feature to an initial codebook, and generating a first training index vector;

[0122] Adding channel noise to the first training index vector to generate a second training index vector;

[0123] Obtain a second semantic feature according to the initial codebook and the second training index vector; input the second semantic feature into a semantic decoder to obtain a decoder output;

[0124] According to the decoder output, the parameters of the initial codebook, semantic encoder, and semantic decoder are adjusted through back propagation.

[0125] Adding channel noise to the first training index vector may be performing K-order quadrature amplitude modulation or K-order phase shift keying on the first training index vector to generate a constellation point set, adding channel noise to the constellation point set, and demodulating the constellation points after adding the noise to generate the second training index vector.

[0126] In some possible implementations, the training data may be used to train the initial codebook, the semantic encoder, the semantic decoder, the semantic detail encoder, and the semantic detail decoder. The specific training process may include:

[0127] Inputting the training data into a semantic encoder to obtain a first semantic feature corresponding to the training data, mapping the first semantic feature to an initial codebook, and generating a first training index vector;

[0128] Restoring the first training index vector to a first quantized semantic feature through inverse vector quantization, calculating a residual between the first semantic feature and the first quantized semantic feature, and inputting the calculated residual into a semantic detail encoder to generate first residual information;

[0129] Adding channel noise to the first training index vector and the first residual information to generate a second training index vector and second residual information;

[0130] Obtain a second quantized semantic feature according to the initial codebook and the second training index vector; input the second residual information into the semantic detail decoder for semantic decoding to obtain a residual, add the second quantized semantic feature and the residual to obtain a second semantic feature; input the second semantic feature into the semantic decoder to obtain a decoder output;

[0131] According to the decoder output, the parameters of the initial codebook, semantic encoder, and semantic decoder are adjusted through back propagation.

[0132] Adding channel noise to the first training index vector may be performing K-order quadrature amplitude modulation or K-order phase shift keying on the first training index vector to generate a constellation point set, adding channel noise to the constellation point set, and demodulating the constellation points after adding the noise to generate the second training index vector.

[0133] Adding channel noise to the first residual information to generate the second residual information may be performing channel coding on the first residual information, adding channel noise to the result of signal coding, and performing channel decoding on the result of adding the noise to generate the second residual information.

[0134] The semantic communication method provided by the embodiment of the present disclosure is specifically introduced below with a specific embodiment. Figure 6 A process diagram of a specific embodiment of the semantic communication method provided by the embodiment of the present disclosure is shown as follows: Figure 6 As shown, the semantic communication method provided by the embodiment of the present disclosure can be used for image transmission, specifically including:

[0135] The transmitter and receiver preset an initial codebook M0, generated through pre-training, containing K = 16 codewords, each with dimension D = 128. This initial codebook, a set of trainable parameters, is jointly trained with the semantic codec and the semantic detail codec to ensure it can represent the semantic sub-features of the image. The initial codebook is synchronized to both the transmitter and receiver via a reliable channel before communication begins.

[0136] Input a 3×128×128 image X and pass it through the semantic encoder f based on Swin Transformer θ1 Extract semantic features s. The encoder outputs feature matrix Where n = 56 (number of sub-features) and D = 128 (sub-feature dimensions). Swin Transformer captures the global semantic information of the image through multi-scale windows.

[0137] The semantic feature s is mapped to the current codebook using a vector quantizer (VQ). Since this is the first transmission, the current codebook is the initial codebook M0. The quantization process uses the Gumbel-Softmax technique: the inner product of s and each codeword in M0 is calculated, Gumbel noise is added, and then a discrete index vector z∈{1,2,…,K} is generated through Softmax. n Subsequently, z is converted into a constellation point set c by K-QAM modulation (K-QAM modulation).

[0138] According to the codebook M0 and index vector z, the quantized semantic feature S is restored q =IVQ(z,M0). Calculate the residual f = ss q , and through a CNN-based semantic detail encoder Encode it as residual information Where m is the residual dimension after compression (in this case, m = 512). CNN extracts local detail features in f through convolutional layers to ensure that fine-grained information is preserved.

[0139] The constellation point set c and the residual information r are transmitted to the receiver through a channel, such as an AWGN channel. During the transmission process, the channel gain h1 = h2 = 1, and the noise n1 and n2 obey the normal distribution. The data received by the receiver is and

[0140] The receiving end first demodulates Demodulate into discrete index vector Then, using the semantic detail decoder Will Decoded into residual features Then, through inverse vector quantization Recover quantized semantic features from the current codebook M0 Will and Add and input semantic decoder f θ2 Generate reconstructed image

[0141] The receiver calculates the current channel SNR based on the received signal and sends it as channel feedback information γ to the transmitter via the feedback channel. Both ends adjust the codebook using a feedback-driven codebook optimization network g(γ, M0). The network includes a semantic modulation mapping (SMM) module that projects features into a latent space, a signal-to-noise ratio (SNR) adaptive modulation (SNR-AM) module that adjusts feature weights based on the SNR, and a sigmoid function that generates a scaling matrix. This matrix is ​​multiplied by M0 to generate the optimized codebook M. M will be used in the next transmission.

[0142] Based on Figure 1 The same principle as shown in the method, Figure 7 A structural diagram of a semantic communication system provided by an embodiment of the present disclosure is shown in FIG. Figure 7 As shown, the semantic communication system 70 may include: a sending end 710, a receiving end 730, and a communication channel 720 connecting the sending end 710 and the receiving end 730;

[0143] The transmitter 710 is configured to input the source data into a pre-trained semantic encoder for semantic encoding, obtain source semantic features of the source data, map the source semantic features to a target codebook, and generate a source index vector;

[0144] Communication channel 720, used to transmit the source index vector from the transmitting end to the receiving end;

[0145] The receiving end 730 is configured to obtain a channel index vector, obtain channel semantic features according to the target codebook and the channel index vector, input the channel semantic features into a pre-trained semantic decoder for semantic decoding, and generate reconstructed data;

[0146] The target codebook is determined according to the signal quality of the communication channel and the initial codebook; the initial codebook is obtained by joint training with the semantic encoder and the semantic decoder.

[0147] In the semantic communication system provided by the embodiments of the present disclosure, the target codebook is dynamically adjusted using the signal quality of the communication channel, so that the semantic communication system can adapt to the channel states of different communication channels (such as low SNR and high SNR), thereby improving the robustness of the semantic communication system and the quality of data reconstruction.

[0148] In some possible implementations, the semantic communication system further includes: a codebook adjustment network, configured to calculate the signal quality of the communication channel based on at least the channel index vector to obtain a signal quality parameter; and adjust the initial codebook based on the signal quality parameter to generate a new target codebook.

[0149] In some possible implementations, the codebook adjustment network is further used to: map the semantic features of the codewords of the initial codebook to the latent space to obtain a latent space feature vector; generate a modulation factor according to the signal quality parameter, and adjust the feature weight of the latent space feature vector according to the modulation factor; generate a scaling matrix according to the adjusted latent space feature vector, and generate a new target codebook according to the scaling matrix and the initial codebook.

[0150] In some possible implementations, the codebook adjustment network is used to: adjust the initial codebook according to the signal quality parameter based on the codebook optimization network to generate a new target codebook; wherein the codebook optimization network includes multiple semantic mapping modules, multiple adaptive modulation modules, and an activation module; the semantic mapping modules and the adaptive modulation modules are alternately arranged; the activation module is connected to the last semantic mapping module; the semantic mapping module is used to extract the semantic features of the codewords of the initial codebook, map the semantic features of the codewords to the latent space, and obtain the latent space feature vector; the adaptive modulation module is used to generate a modulation factor according to the signal quality parameter, and adjust the feature weight of the latent space feature vector according to the modulation factor; the activation module is used to generate a scaling matrix according to the adjusted latent space feature vector, and generate a new target codebook according to the scaling matrix and the initial codebook.

[0151] In some possible implementations, the semantic communication system also includes: a semantic detail network, which is used to restore the source index vector to a quantized semantic feature according to the target codebook, calculate the source information difference between the source semantic feature and the quantized semantic feature, and use a pre-trained semantic detail encoder to encode the source information difference into source residual information; transmit the source residual information from the transmitter to the receiver through a communication channel to obtain channel residual information; input the channel residual information into a pre-trained semantic detail decoder for decoding to obtain channel information difference; input the channel semantic feature and the channel information difference into a pre-trained semantic decoder for semantic decoding to generate reconstructed data; wherein the initial codebook, semantic encoder, semantic decoder, semantic detail encoder, and semantic detail decoder are obtained through joint training.

[0152] In some possible implementations, the communication channel is used to: perform channel coding on the source residual information, transmit the coded source residual information from the transmitting end to the receiving end via the communication channel, and obtain the channel residual information.

[0153] In some possible implementations, the transmitter is configured to calculate the inner product of the source semantic features and each codeword in the target codebook, add Gumbel noise, and then generate a source index vector through a normalized exponential function; and the receiver is configured to recover the channel semantic features from the target codebook based on the channel index vector through inverse vector quantization.

[0154] In some possible implementations, the communication channel is used to: generate a constellation point set for a source index vector through K-order orthogonal amplitude modulation or K-order phase shift keying; transmit the constellation point set from a transmitting end to a receiving end through the communication channel to obtain a modulated signal; and demodulate the modulated signal to obtain a channel index vector.

[0155] In some possible implementations, the semantic communication system also includes a training module, which is used to: input training data into a semantic encoder to obtain a first semantic feature corresponding to the training data, map the first semantic feature to an initial codebook, and generate a first training index vector; add channel noise to the first training index vector to generate a second training index vector; obtain a second semantic feature based on the initial codebook and the second training index vector; input the second semantic feature into a semantic decoder to obtain a decoder output; and adjust the parameters of the initial codebook, semantic encoder, and semantic decoder through back propagation based on the decoder output.

[0156] In some possible implementations, the source data is any one of image data, voice data, text data, and point cloud data.

[0157] It is understandable that the above modules of the semantic communication system in the embodiment of the present disclosure have the function of realizing Figure 1 The functions of the corresponding steps of the semantic communication method in the embodiment shown in . This function can be implemented by hardware, or by hardware executing the corresponding software. The hardware or software includes one or more modules corresponding to the above functions. The above modules can be software and / or hardware, and the above modules can be implemented separately or integrated with multiple modules. For the functional description of each module of the above semantic communication system, please refer to Figure 1 The corresponding description of the semantic communication method in the embodiment shown in will not be repeated here.

[0158] In the technical solution disclosed herein, the collection, storage, use, processing, transmission, provision, disclosure and application of user personal information involved comply with the provisions of relevant laws and regulations, take necessary confidentiality measures, and do not violate public order and good morals.

[0159] In the technical solution disclosed herein, the user's authorization or consent is obtained before obtaining or collecting the user's personal information.

[0160] According to an embodiment of the present disclosure, the present disclosure also provides an electronic device, a readable storage medium, and a computer program product.

[0161] The electronic device includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to execute the semantic communication method provided in the embodiment of the present disclosure.

[0162] Compared with the existing technology, this electronic device uses the signal quality of the communication channel to dynamically adjust the target codebook, enabling the semantic communication system to adapt to the channel states of different communication channels (such as low SNR and high SNR), thereby improving the robustness of the semantic communication system and the quality of data reconstruction.

[0163] The readable storage medium is a non-transitory computer-readable storage medium storing computer instructions, wherein the computer instructions are used to enable a computer to execute the semantic communication method provided by the embodiment of the present disclosure.

[0164] Compared with the existing technology, this readable storage medium dynamically adjusts the target codebook using the signal quality of the communication channel, enabling the semantic communication system to adapt to the channel states of different communication channels (such as low SNR and high SNR), thereby improving the robustness of the semantic communication system and the quality of data reconstruction.

[0165] The computer program product includes a computer program, and when the computer program is executed by a processor, it implements the semantic communication method provided in the embodiment of the present disclosure.

[0166] Compared with the existing technology, this computer program product uses the signal quality of the communication channel to dynamically adjust the target codebook, enabling the semantic communication system to adapt to the channel states of different communication channels (such as low SNR and high SNR), thereby improving the robustness of the semantic communication system and the quality of data reconstruction.

[0167] Figure 8 A schematic block diagram of an example electronic device 800 that can be used to implement embodiments of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital assistants, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are provided as examples only and are not intended to limit the implementation of the present disclosure described and / or claimed herein.

[0168] like Figure 8As shown, the device 800 includes a computing unit 801, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 802 or a computer program loaded from a storage unit 808 into a random access memory (RAM) 803. Various programs and data required for the operation of the device 800 can also be stored in the RAM 803. The computing unit 801, the ROM 802, and the RAM 803 are connected to each other via a bus 804. An input / output (I / O) interface 805 is also connected to the bus 804.

[0169] Various components in device 800 are connected to I / O interface 805, including an input unit 806, such as a keyboard, mouse, etc.; an output unit 807, such as various types of displays, speakers, etc.; a storage unit 808, such as a magnetic disk, optical disk, etc.; and a communication unit 809, such as a network card, modem, wireless communication transceiver, etc. The communication unit 809 allows device 800 to exchange information / data with other devices via a computer network such as the Internet and / or various telecommunication networks.

[0170] The computing unit 801 can be a variety of general-purpose and / or specialized processing components with processing and computing capabilities. Some examples of the computing unit 801 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units that run machine learning model algorithms, a digital signal processor (DSP), and any appropriate processor, controller, microcontroller, etc. The computing unit 801 performs the various methods and processes described above, such as the semantic communication method. For example, in some embodiments, the semantic communication method can be implemented as a computer software program that is tangibly contained in a machine-readable medium, such as the storage unit 808. In some embodiments, part or all of the computer program can be loaded and / or installed on the device 800 via the ROM 802 and / or the communication unit 809. When the computer program is loaded into the RAM 803 and executed by the computing unit 801, one or more steps of the semantic communication method described above can be performed. Alternatively, in other embodiments, the computing unit 801 can be configured to perform the semantic communication method by any other appropriate means (e.g., by means of firmware).

[0171] Various embodiments of the systems and techniques described herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), system-on-chip systems (SOCs), programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments can include being implemented in one or more computer programs that are executable and / or interpreted on a programmable system comprising at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0172] The program code for implementing the method of the present disclosure can be written in any combination of one or more programming languages. These program codes can be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device so that when the program code is executed by the processor or controller, the functions / operations specified in the flow chart and / or block diagram are implemented. The program code can be executed entirely on the machine, partially on the machine, as a stand-alone software package, partially on the machine and partially on a remote machine, or entirely on a remote machine or server.

[0173] In the context of the present disclosure, a machine-readable medium can be a tangible medium that can contain or store a program for use by or in conjunction with an instruction execution system, device or equipment. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium can include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0174] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0175] The systems and techniques described herein can be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer having a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network (LAN), a wide area network (WAN), and the Internet.

[0176] A computer system may include a client and a server. The client and server are generally remote from each other and typically interact through a communication network. The client-server relationship arises through computer programs running on the respective computers and having a client-server relationship with each other. The server may be a cloud server, a server in a distributed system, or a server integrated with a blockchain.

[0177] It should be understood that the various forms of the processes shown above can be used to reorder, add, or delete steps. For example, the steps described in this disclosure can be performed in parallel, sequentially, or in a different order, as long as the desired results of the technical solutions disclosed in this disclosure can be achieved. This is not limited herein.

[0178] The above specific embodiments do not constitute a limitation on the scope of protection of this disclosure. Those skilled in the art will appreciate that various modifications, combinations, sub-combinations, and substitutions may be made based on design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this disclosure shall be included within the scope of protection of this disclosure.

Claims

1. A semantic communication method, the method being used in a semantic communication system, the semantic communication system comprising a sending end, a receiving end, and a communication channel connecting the sending end and the receiving end, the method comprising: Inputting the source data into a pre-trained semantic encoder for semantic encoding, obtaining source semantic features of the source data, mapping the source semantic features to a target codebook, and generating a source index vector; Transmitting the source index vector from the transmitting end to the receiving end through the communication channel to obtain a channel index vector; Acquire channel semantic features according to the target codebook and the channel index vector, input the channel semantic features into a pre-trained semantic decoder for semantic decoding, and generate reconstructed data; The target codebook is determined according to the signal quality of the communication channel and an initial codebook; the initial codebook is obtained by joint training with the semantic encoder and the semantic decoder.

2. The method according to claim 1, further comprising: Calculating a signal quality of the communication channel at least according to the channel index vector to obtain a signal quality parameter; The initial codebook is adjusted according to the signal quality parameter to generate a new target codebook.

3. The method according to claim 2, wherein: The adjusting the initial codebook according to the signal quality parameter to generate a new target codebook includes: Mapping the semantic features of the codewords in the initial codebook to a latent space to obtain a latent space feature vector; generating a modulation factor according to the signal quality parameter, and adjusting a feature weight of the latent space feature vector according to the modulation factor; A scaling matrix is ​​generated according to the adjusted latent space eigenvector, and a new target codebook is generated according to the scaling matrix and the initial codebook.

4. The method according to claim 2, wherein: The adjusting the initial codebook according to the signal quality parameter to generate a new target codebook includes: Based on a codebook optimization network, adjusting the initial codebook according to the signal quality parameter to generate a new target codebook; The codebook optimization network includes multiple semantic mapping modules, multiple adaptive modulation modules, and an activation module; the semantic mapping modules and the adaptive modulation modules are alternately arranged; the activation module is connected to the last semantic mapping module; The semantic mapping module is used to extract the semantic features of the codewords in the initial codebook, map the semantic features of the codewords to the latent space, and obtain the latent space feature vector; The adaptive modulation module is used to generate a modulation factor according to the signal quality parameter, and adjust the feature weight of the latent space feature vector according to the modulation factor; The activation module is used to generate a scaling matrix according to the adjusted latent space eigenvector, and generate a new target codebook according to the scaling matrix and the initial codebook.

5. The method according to claim 1, further comprising: According to the target codebook, the source index vector is restored to a quantized semantic feature, a source information difference between the source semantic feature and the quantized semantic feature is calculated, and the source information difference is encoded into source residual information using a pre-trained semantic detail encoder; Transmitting the source residual information from the transmitting end to the receiving end via the communication channel to obtain channel residual information; Inputting the channel residual information into a pre-trained semantic detail decoder for decoding to obtain channel information difference; inputting the channel semantic features and the channel information difference into a pre-trained semantic decoder for semantic decoding to generate reconstructed data; The initial codebook, semantic encoder, semantic decoder, semantic detail encoder, and semantic detail decoder are obtained through joint training.

6. The method according to claim 5, wherein: The transmitting the source residual information from the transmitting end to the receiving end through the communication channel to obtain channel residual information includes: Channel coding is performed on the source residual information, and the coded source residual information is transmitted from the transmitting end to the receiving end via the communication channel to obtain channel residual information.

7. The method according to claim 1, wherein Mapping the information source semantic features to a target codebook to generate an information source index vector includes: Calculating the inner product of the source semantic feature and each codeword in the target codebook, adding Gumbel noise, and generating the source index vector by a normalized exponential function; The acquiring of channel semantic features according to the target codebook and the channel index vector includes: The channel semantic feature is restored from the target codebook based on the channel index vector through inverse vector quantization.

8. The method according to claim 1, wherein The transmitting the information source index vector from the transmitting end to the receiving end through the communication channel to obtain a channel index vector includes: Generating a constellation point set by K-order quadrature amplitude modulation or K-order phase shift keying for the source index vector; Transmitting the constellation point set from the transmitting end to the receiving end via the communication channel to obtain a modulated signal; The modulated signal is demodulated to obtain the channel index vector.

9. The method according to claim 1, further comprising: Inputting training data into the semantic encoder to obtain a first semantic feature corresponding to the training data, mapping the first semantic feature to an initial codebook, and generating a first training index vector; adding channel noise to the first training index vector to generate a second training index vector; Acquire a second semantic feature according to the initial codebook and the second training index vector; Inputting the second semantic feature into the semantic decoder to obtain a decoder output; The parameters of the initial codebook, the semantic encoder, and the semantic decoder are adjusted through back propagation according to the decoder output.

10. The method according to claim 1, wherein The source data is any one of image data, voice data, text data, and point cloud data.

11. A semantic communication system, comprising a sending end, a receiving end, and a communication channel connecting the sending end and the receiving end; The transmitting end is configured to input the source data into a pre-trained semantic encoder for semantic encoding, obtain source semantic features of the source data, map the source semantic features to a target codebook, and generate a source index vector; The communication channel is used to transmit the source index vector from the transmitting end to the receiving end; The receiving end is used to obtain a channel index vector, obtain a channel semantic feature according to the target codebook and the channel index vector, input the channel semantic feature into a pre-trained semantic decoder for semantic decoding, and generate reconstructed data; in, The target codebook is determined according to the signal quality of the communication channel and an initial codebook; the initial codebook is obtained by joint training with the semantic encoder and the semantic decoder.

12. An electronic device comprising: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method according to any one of claims 1 to 10.

13. A non-transitory computer-readable storage medium storing computer instructions, wherein: The computer instructions are used to cause the computer to execute the method according to any one of claims 1-10.

14. A computer program product comprising a computer program, which, when executed by a processor, implements the method according to any one of claims 1 to 10.

Images