A semantic communication transmission method based on text and audio modalities

Abstract

The application belongs to the technical field of semantic communication, and particularly relates to a semantic communication transmission method based on two modalities of text and audio. The application combines deep learning with semantic communication, and comprises a deep learning network, which comprises a text semantic feature extraction module, an audio feature information extraction module and a speech synthesis module. An original speech signal is subjected to the text semantic feature extraction module to obtain a text description of the original speech signal. The audio feature information extraction module is used to obtain audio information of the original speech signal, which comprises duration, pitch and power information. After the two kinds of data information are integrated, source channel joint coding is performed and the data information is sent to a receiving party. The receiving party decodes to obtain the text description and the speech information, and generates a required speech signal through the speech synthesis module. The application can significantly reduce channel resource overhead, improve transmission efficiency, and greatly improve the communication quality and user experience of voice communication.

Description

Technical Field

[0001] This invention belongs to the field of semantic communication technology, specifically relating to a semantic communication transmission method based on text and audio modalities. Background Technology

[0002] In recent years, with the continuous increase in communication demands, wireless data traffic has grown dramatically, placing a heavy burden on current communication infrastructure. Semantic-based communication is one of the promising technologies for next-generation communication because it greatly improves transmission efficiency. Traditional communication systems focus on transmitting symbols while ignoring semantic content, while semantic communication focuses on collecting semantic information from the source and recovering the same semantic information at the receiving end.

[0003] With the rapid development of computer and information technology, we are poised to enter an era of widespread connectivity, even in the realm of voice signals. As more and more applications and devices integrate into voice signal networks, real-time interaction with the cloud will trigger an explosive growth in data volume. Therefore, improving existing voice signal-based communication transmission technologies has become particularly urgent. Summary of the Invention

[0004] The purpose of this invention is to provide an efficient semantic communication transmission method based on both text and audio modalities to meet the ever-expanding demands of voice signal networks.

[0005] The present invention provides an efficient semantic communication transmission method based on both text and audio modalities. This method combines deep learning networks for text semantic feature extraction, audio feature information extraction, and speech synthesis with communication transmission to achieve efficient semantic transmission. It includes constructing a deep learning network comprising a text semantic feature extraction module, an audio feature information extraction module, and a speech synthesis module. The specific steps of the method are as follows:

[0006] (1) Use the dataset to train a deep learning network. The deep learning network includes a text semantic feature extraction module, an audio feature information extraction module, and a speech synthesis module.

[0007] (2) The original speech signal is sent to the text semantic feature extraction module to extract semantic features and obtain a text description of the original speech signal, which contains the semantic information of the speech signal.

[0008] (3) The original speech signal is sent to the audio feature information extraction module to process the speech information and obtain the audio information of the original speech signal, including duration, pitch and power information;

[0009] (4) After integrating the two types of data information, perform source-channel joint coding and send it to the receiver to achieve the purpose of compressing the amount of data and providing anti-interference capability for data transmission;

[0010] (5) The receiver decodes the text description and voice information, processes them through the voice synthesis module, and generates a voice signal that meets the requirements.

[0011] In this invention, the text semantic feature extraction module focuses on extracting the semantic information of the speech signal, rather than all the speech content. In particular, during network training, it focuses on capturing more important text semantic information, thereby reducing the amount of data transmitted.

[0012] In this invention, the audio feature information extraction module extracts specific speech information from the speech signal, such as speech duration, pitch, and power, and transmits it through the channel so that the speech signal can be better reconstructed at the receiving end, thereby improving the quality of speech synthesis.

[0013] In this invention, when performing joint source-channel joint coding, the transmission of text information focuses on the transmission of the meaning expressed by the text. That is, a successful transmission is considered when the meaning expressed by the sentence decoded by the receiver is consistent with the meaning expressed by the sentence before encoding by the sender, rather than focusing on the correctness of the symbols.

[0014] In this invention, when the receiver synthesizes a qualified speech signal based on the decoded text description and speech information, the alignment of the text description and speech information makes the synthesized speech signal closer to the duration, pitch and power information of the original speech signal, thereby making speech recovery more efficient, smooth and natural.

[0015] In this invention, each time the receiver generates a voice signal, it measures whether the generated voice signal meets the quality standard. When the voice signal does not meet the expected quality standard, the receiver generates it again until the quality standard is met. The quality standard is preset according to the transmission requirements.

[0016] In this invention, the method for measuring the quality of the generated speech signal is to convert the speech signal into text using a speech signal reconstruction module consistent with that of the sender, and calculate the semantic similarity between the text and the text decoded by the receiver. The greater the similarity, the more the speech signal conforms to the text conditions.

[0017] The beneficial effects of the efficient semantic communication transmission method of the present invention

[0018] This invention can greatly reduce the amount of data that needs to be transmitted when transmitting audio signals, significantly improve transmission latency, greatly ensure the security of transmitted data, and improve the quality of voice recovery.

[0019] Other advantages and applications of the present invention will be apparent to those skilled in the art. Attached Figure Description

[0020] Figure 1 This is a flowchart illustrating the overall process of the efficient semantic communication transmission method based on text and audio modalities described in this invention.

[0021] Figure 2 This is a flowchart illustrating the text semantic feature extraction module of the efficient semantic communication transmission method based on text and audio modalities described in this invention.

[0022] Figure 3 This is a flowchart illustrating the audio feature information extraction module of the efficient semantic communication transmission method based on text and audio modalities described in this invention.

[0023] Figure 4 This is a flowchart illustrating the speech synthesis module in the efficient semantic communication transmission method based on text and audio modalities described in this invention. Detailed Implementation

[0024] The following embodiments further illustrate the present invention with reference to the accompanying drawings.

[0025] refer to Figure 1 This is a flowchart of the efficient semantic communication transmission method based on text and audio modalities described in this invention. When a device in the Internet of Things (IoT) collects a voice signal for transmission, the original voice signal is first processed by a text semantic feature extraction module to obtain a text feature description of the voice signal, i.e., the semantic information contained in the voice signal. Then, through an audio feature information extraction module, a set of compact voice-related information, including duration, pitch, and power information, is extracted from the voice signal at the transmitting end. Then, after compression by joint source-channel coding, it is transmitted to the receiver, which can be another device or the cloud. After decoding, the receiver obtains the text information and voice information of the voice signal, and then uses a speech synthesis module to recover the voice signal to achieve the purpose of voice signal transmission.

[0026] refer to Figure 2 This diagram illustrates the specific workflow of the text semantic feature extraction module in the efficient semantic communication transmission method based on text and audio modalities described in this invention. The text semantic feature extraction module includes components such as a VGG layer, a BLSTM layer, and an attention mechanism module. Specifically, VGG is a convolutional neural network architecture consisting of 16 convolutional layers; BLSTM is a recurrent neural network architecture used to process sequential data, such as speech and text; and the attention mechanism calculates attention scores to enable the system to focus more on the parts of the input speech signal that are semantically aligned with the text.

[0027] In the text semantic feature extraction module, the speech signal s is first fed into the VGG layer to obtain the semantic feature representation m. Next, the semantic feature representation m is input into the BLSTM layer. This module generates intermediate features H that retain the temporal correlation in the input sequence. Then, it is fed into the attention mechanism module to extract the required semantic feature information Z. At this point, the task of the text semantic feature extraction module is completed.

[0028] Please refer to Figure 3 This is a schematic diagram of the specific process of the audio feature information extraction module in the efficient semantic communication transmission method based on text and audio modalities described in this invention. The audio feature information extraction module includes components such as VGG layer, BLSTM layer, and CTC alignment module. Specifically, VGG is a convolutional neural network architecture composed of 16 convolutional layers; BLSTM is a recurrent neural network architecture used to process sequential data, such as speech and text; the CTC alignment module is based on the dimensional bit algorithm [4] to obtain the duration, pitch and power information of each frame of phonemes.

[0029] The audio feature information extraction module takes the intermediate feature H obtained after processing by VGG and BLSTM layers as the module input and uses the CTC alignment method [1]. First, it finds the probability of the corresponding phoneme in each speech spectrum frame, obtains the peak value when the phoneme changes, and calculates the time between two peak values ​​as the duration of a phoneme. Then, it applies the DIO algorithm [2] to obtain the pitch information of each frame; it obtains the power information by calculating the vector norm of each frame, and then calculates the average power of all frames of the same phoneme. Therefore, we obtain the pitch and power information of each phoneme. Thus, a set of speech-related information containing duration, pitch and power is output and sent to the receiving end for speech signal synthesis. At this time, the task of the audio feature information extraction module is completed.

[0030] Please refer to Figure 4This is a flowchart of the speech synthesis module in the efficient semantic communication transmission method based on text and audio modalities described in this invention. The speech synthesis module includes a text embedding layer, an encoder-decoder layer, a text-speech alignment module, and a pre-trained GAN vocoder [3]. This module takes the predicted transcription (text) from the semantic decoder and the audio feature information (audio) provided by the transmitter as input, and combines the data information of the two modalities to jointly recover the speech signal. This invention adopts a non-autoregressive model text-speech synthesis technology. First, the text embedding layer is used to convert the input text into a text embedding, and then it is fed to a text speech encoder containing four conversion layers to obtain the potential representation of each token in the input text sequence, as well as the prediction of the duration, power, and pitch of the corresponding phoneme. Then, the text-to-speech alignment module is used. This module performs alignment operations on the predicted duration information and the real duration information obtained by the audio feature extraction module, and modifies the frame number of each token in the potential representation output of the text speech encoder, and combines the potential representation with the power information embedding and the pitch information embedding, respectively embedding the real power and pitch information transmitted by the audio feature extraction module, as well as the predicted power and pitch information, such as Figure 4 As shown, a latent representation containing both semantically relevant textual information and speech-related information is generated. This combined information is then input into a text-to-speech decoder, which consists of six Transformer layers, to recover the speech spectrum. Finally, a pre-trained GAN vocoder [3] is used to generate a speech signal from the recovered spectrum. The entire process has a faster training time than traditional speech synthesis, and by combining audio modal information, it also achieves higher quality speech recovery, solving the problem of information loss in speech synthesis.

[0031] References

[0032] [1]GDJForney, "The Viterbi algorithm," Proc.IEEE, vol.61, no.5, pp.268–278, Mar.1993.

[0033] [2]Y.Ren et al., "FastSpeech: Fast, robust and controllable text tospeech," in Proc.Adv.Neural Inf.Process.Syst., vol.32, 2019.

[0034] [3]J.Kong,J.Kim,and J.Bae,“HiFi-GAN:Generative adversarial net worksfor efficient and high fidelity speech synthesis,”in Proc.Adv.NeuralInf.Process.Syst.,vol.33,2020,pp.17022–17033.

[0035] [4]G.D.J.Forney,“The Viterbi algorithm,”Proc.IEEE,vol.61,no.5,pp.268–278,Mar.1993。

Claims

1. A semantic communication transmission method based on both text and audio modalities, characterized in that, This method combines deep learning with semantic communication to achieve efficient semantic transmission. It involves constructing a deep learning network, which includes a text semantic feature extraction module, an audio feature information extraction module, and a speech synthesis module. The specific steps of the semantic communication transmission method are as follows: (1) Use the dataset to train a deep learning network. The deep learning network includes a text semantic feature extraction module, an audio feature information extraction module, and a speech synthesis module. (2) The original speech signal is sent to the text semantic feature extraction module to extract semantic features and obtain a text description of the original speech signal, which contains the semantic information of the speech signal; (3) The original speech signal is sent to the audio feature information extraction module to process the speech information and obtain the audio information of the original speech signal, including duration, pitch and power information; (4) After integrating the two types of data information, perform source-channel joint coding and send it to the receiver to achieve the purpose of compressing the data volume and providing anti-interference capability for data transmission; (5) The receiver decodes the text description and voice information, processes them through the voice synthesis module, and generates a voice signal that meets the requirements. The audio feature information extraction module includes a VGG layer, a BLSTM layer, and a CTC alignment layer. The intermediate feature H obtained after processing by the VGG and BLSTM layers is used as the input to the CTC alignment layer. The alignment process of the CTC alignment layer is as follows: First, find the probability of the corresponding phoneme in each speech spectrum frame, obtain the peak value when the phoneme changes, and calculate the time between two peak values ​​as the duration of a phoneme; then, apply the DIO algorithm to obtain the pitch information of each frame; obtain the power information by calculating the vector norm of each frame, and then calculate the average power of all frames of the same phoneme; obtain the pitch and power information of each phoneme; output a set of speech-related information containing duration, pitch, and power, and send it to the receiving end for speech signal synthesis. At this point, the task of the audio feature information extraction module is completed. The speech synthesis module comprises a text embedding layer, an encoder-decoder layer, a text-to-speech alignment module, and a GAN vocoder. Specifically, it employs a non-autoregressive model text-to-speech synthesis technique. First, the text embedding layer converts the input text into a text embedding, which is then fed to a text-to-speech encoder containing four transformation layers to obtain a latent representation for each token in the input text sequence, along with predictions of the duration, power, and pitch of the corresponding phonemes. Next, the text-to-speech alignment module aligns the predicted duration information received from the receiver with the actual duration information and modifies the frame number of each token in the latent representation output of the text-to-speech encoder. This latent representation is then combined with power and pitch information embeddings, embedding the transmitted power and predicted power and pitch information, respectively, to generate a latent representation that simultaneously contains semantically relevant text information and speech-related information. This latent representation is then input into the text-to-speech decoder, which consists of six Transformer layers, to recover the speech spectrum. Finally, a pre-trained GAN vocoder is used to generate the speech signal from the recovered spectrum.

2. The semantic communication transmission method according to claim 1, characterized in that, When the receiver synthesizes a qualified speech signal based on the decoded text description and speech information, it aligns the text description with the speech information to make the synthesized speech signal closer to the duration, pitch and power information of the original speech signal.

3. The semantic communication transmission method according to claim 1, characterized in that, Each time the receiver generates a voice signal, it measures whether the generated voice signal meets the quality standard. If the voice signal does not meet the expected quality standard, the receiver generates it again until the quality standard is met. The quality standard is preset according to the transmission requirements.

4. The semantic communication transmission method according to claim 1, characterized in that, The quality measurement method for the generated speech signal is to convert the speech signal into text using a speech signal reconstruction module consistent with the sender's speech signal, and calculate the semantic similarity between this text and the text decoded by the receiver. The higher the similarity, the more the speech signal conforms to the text conditions.

5. The semantic communication transmission method according to claim 1, characterized in that, The text semantic feature extraction module includes a VGG layer, a BLSTM layer, and an attention mechanism module. The VGG layer is a convolutional neural network consisting of 16 convolutional layers. The BLSTM layer is a recurrent neural network used to process sequential data, including speech and text. The attention mechanism module calculates attention scores to enable the system to focus more on the parts of the input speech signal that are semantically aligned with the text. The speech signal s first enters the VGG layer to obtain the semantic feature representation m; the semantic feature representation m is input into the BLSTM layer to generate intermediate features H that retain the temporal correlation in the input sequence, and then sent to the attention mechanism module to extract the required semantic feature information Z. At this point, the task of the text semantic feature extraction module is completed.

6. The semantic communication transmission method according to claim 5, characterized in that, The VGG is a convolutional neural network consisting of 16 convolutional layers; the BLSTM is a recurrent neural network used to process sequential data, including speech and text; the attention mechanism is to calculate an attention score, enabling the system to pay more attention to the parts of the input speech signal that are semantically aligned with the text.

7. The semantic communication transmission method according to claim 6, characterized in that, The speech synthesis module takes the predicted transcription from the semantic decoder and the audio feature information provided by the transmitter as input, and combines the two modal data information to jointly recover the speech signal.

Images