A method and system for speech recognition and cloning semantic speech transmission

Abstract

The present invention provides a method and system for speech recognition and semantic cloning speech transmission. First, at the transmitter, the original speech signal is recognized as text by a speech recognition model. Next, the recognized text is encoded into text features by a Transformer-based encoder. A speaker encoder extracts audio semantic features from the original speech signal. Two floating-point semantic features are quantized into OFDM symbols by a quantization layer and transmitted to a receiver. The receiver dequantizes the OFDM symbols into floating-point semantic features through a dequantization module and uses a Transformer-based decoder to recover the text in the original speech signal from the text features. Simultaneously, a speech cloning model uses the recovered text and received audio semantic features to clone the original speech signal, obtaining a speech signal that approximates the speaker's timbre in the original speech. This invention significantly reduces the bandwidth occupied by transmitted speech signals and greatly improves the quality of transmitted speech signals and the user experience.

Description

Technical Field

[0001] The present invention belongs to the technical field of semantic communication, and in particular relates to a method and system for speech recognition and cloning semantic speech transmission. Background Art

[0002] With the rapid development of wireless communication technology, the capacity of traditional communication systems is gradually approaching the limits of Shannon's theorem. At the same time, the massive amount of wireless data has become a burden on the communication infrastructure. Compared to traditional communication, which focuses on the transmission of bit data, semantic communication, as a new communication paradigm, understands the meaning of information at the semantic level and transmits only the semantic content. This significantly reduces the amount of data transmitted and lowers the bandwidth requirements of the system. The transmission of voice signals is a major method of modern communication. They contain not only the textual information the speaker intends to convey, but also the speaker's voice characteristics, such as tone. However, traditional voice signal transmission relies on voice source codecs, which consumes a large amount of bandwidth resources during the transmission process and cannot guarantee accurate voice transmission in poor channel conditions.

[0003] Existing research has only transmitted speech recognition-related features from the speech signal and used these features to recover the text at the receiving end, ensuring accurate transmission of the text within the speech signal with minimal bandwidth. Simultaneously, the recovered text is fed into a separate speech synthesis model to restore the original speech signal. However, these speech synthesis models are specific to a single speaker and require retraining when the speaker changes, resulting in a significant waste of computing resources. Therefore, leveraging current deep learning technologies to design a semantic speech transmission method that consumes little bandwidth, exhibits strong interference resistance, and is applicable to multiple speakers is an urgent challenge. Summary of the Invention

[0004] Purpose of the invention: The present invention provides a method and system for speech recognition and cloning semantic speech transmission, aiming to utilize the advantages of semantic communication to solve the problems of large bandwidth requirements and poor anti-interference ability in the process of speech signal transmission, thereby ensuring the accurate transmission of multi-speaker speech signals with a lower transmission bandwidth.

[0005] Technical solution: The technical solution adopted by the present invention specifically includes the following steps:

[0006] Step 1: Divide the semantics in the speech signal into textual semantics related to the transmitted content and audio semantics related to the speaker's timbre in the speech signal. Then, using the speech recognition model Deep Speech2 and the speaker encoder H / ASP in deep learning, the text T related to the speech content and the audio semantics F related to the speaker's timbre are extracted from the original speech signal. A ;

[0007] Step 2: Recognize all the original speech in the dataset as text to train the text encoder TE based on the Transformer structure en and decoder TE de After the training is completed, the text encoder is placed in the transmitter to complete the conversion of the text T recognized from the original speech signal S into text semantic features F T Function;

[0008] Step 3: Use the quantification module to convert the text semantic features F T and audio semantic features F A quantized into OFDM symbols X and sent to the receiver through the wireless channel; at the receiver, the dequantization module dequantizes the received OFDM symbols X into floating-point semantic features F T and F A ;

[0009] Step 4: Transformer-based text decoder TE de From the received text semantic features F T Recovered text Τ; recovered text Τ and received audio semantic features F A They are fed into the pre-trained speech cloning model YourTTS together to recover a speech whose text content and speaker timbre are close to the original speech signal.

[0010] As a further technical solution of the present invention, in step 1, the speech recognition model Deep Speech 2 and the speaker encoder H / ASP are both pre-trained on the speech dataset and have fixed weight parameters before being directly used in the semantic communication system.

[0011] As a further technical solution of the present invention, in step 2, a speech set with a total duration of 100 hours in the multi-speaker speech dataset Librispecch is used as the training set. The set includes speech samples S and their corresponding text T. The speech samples S are recognized as text T by the Deep Speech 2 model and used to train the text codec, which is expressed as:

[0012]

[0013] Among them, L CE is the cross entropy loss function, Θ en and Θ de are the trainable parameters in the encoder and decoder, respectively, and these parameters are fixed after training is completed.

[0014] At the transmitter, the speech recognition text T is encoded into text semantic features F by the text encoder. T , expressed as:

[0015] F T =TE en (T; Θ TE,en )

[0016] As a further technical solution of the present invention, in step 3, the text semantic feature F is trained. T The process of quantization and dequantization modules is expressed as:

[0017]

[0018] Among them, L MSE is the mean square error (MSE) loss function, Θ Q and Θ deQ are the trainable parameters in the encoder and decoder respectively. Audio semantic features F A The quantization and dequantization of is also trained using the above method, and the parameters of the above structure are fixed after the training is completed.

[0019] As a further technical solution of the present invention, in step 4, the received text semantic feature F T Text decoder TE de Restore to text T and combine with the received audio semantic features F A Input them together into the voice cloning model YourTTS to generate a receiving voice similar to the original voice speaker The whole process is as follows:

[0020] S=SC(F A ,T;Θ SC )=SC(F A ,TE de (F T ;Θ de );Θ SC )

[0021] Among them, SC(·) represents the cloning process of the voice cloning model YourTTS, Θ SC are the parameters in YourTTS, which are pre-trained on the speech dataset and not trained in the semantic communication system.

[0022] The present invention also provides a speech recognition and cloning semantic speech transmission system, comprising a speech recognition model, a speaker encoder, a text encoder, a quantization module, a dequantization module, and a speech cloning model; the speech recognition model and the speaker encoder respectively extract corresponding text content and speaker timbre features from original speech; the text encoder encodes the recognized text into text semantic features at a transmitter; the text decoder decodes the received text semantic features into text at a receiver; the quantization and dequantization modules are responsible for quantizing the semantic features into OFDM symbols and dequantizing the OFDM symbols into semantic features, respectively; and the speech cloning model combines the received timbre features with the decoded text to generate a speech signal close to the original speaker.

[0023] Beneficial effects: Compared with the prior art, the present invention adopts the above technical solution and has the following beneficial effects:

[0024] 1. The present invention divides the semantics in speech signals into textual semantics related to the transmitted content and audio semantics related to the speaker's timbre characteristics in the speech signal. The transmitter uses deep learning methods to extract textual semantic features and audio semantic features from the speech signal for transmission, which greatly reduces the bandwidth resources required for transmission compared to existing methods.

[0025] 2. This invention's Transformer-based text encoding and decoding ensures the accuracy of text semantics in speech signals even in extremely poor channel environments. Furthermore, the voice cloning network, leveraging the recovered text and received audio features, can reconstruct a speech signal close to the speaker's, even in extremely poor channel environments, without requiring network retraining. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] Figure 1 Schematic diagram of the principle of the speech recognition and cloning semantic speech transmission method based on deep learning of the present invention;

[0027] Figure 2 Schematic diagram of the principle of the Transformer-based text codec in the present invention;

[0028] Figure 3 Schematic diagram of the principle of the module for quantizing and dequantizing semantic features in the present invention. DETAILED DESCRIPTION

[0029] The embodiments of the present invention will be described below with reference to the accompanying drawings.

[0030] like Figure 1 As shown, the present invention designs a method for speech recognition and cloning semantic speech transmission, comprising:

[0031] Step 1: Divide the semantics in the speech signal into textual semantics related to the transmitted content and audio semantics related to the speaker's timbre in the speech signal. Then, using the speech recognition model Deep Speech2 and the speaker encoder H / ASP in deep learning, the text T related to the speech content and the audio semantics F related to the speaker's timbre are extracted from the original speech signal S. A ;

[0032] Specifically, the speech recognition model Deep Speech 2 and the speaker encoder H / ASP are both pre-trained on a speech dataset and have their weight parameters fixed before being used directly in the semantic communication system. The process of using Deep Speech 2 to recognize text from raw speech signals is expressed as follows:

[0033] T = ASR(S; Θ ASR )

[0034] Among them, ASR(·) represents the process of Deep Speech 2 speech recognition, Θ ASR For the parameters. Use speaker encoder H / ASP to extract audio semantic features The process is expressed as:

[0035] F A =SE en (S; Θ SE )

[0036] Among them, L audio is the length of the audio semantic feature, SE en (·) represents the process of speaker encoder H / ASP speech recognition, Θ SE For the parameters.

[0037] Step 2: Recognize all the original speech in the dataset as text to train the text encoder TE based on the Transformer structure en and decoder TE de After the training is completed, the text encoder is placed in the transmitter to complete the conversion of the text T recognized from the original speech signal S into text semantic features. function, where L text is the length of the text semantic feature;

[0038] Specifically, a speech collection with a total length of 100 hours in the multi-speaker speech dataset Librispecch is used as the training set for the entire semantic communication system. The collection includes speech samples S and their corresponding texts T. First, the recognized sentence T is padded with 0 to the maximum length L and passed through the text encoder TE. en and decoder TE deComplete the encoding and decoding of the sentence. Then, the embedding layer f embed (·) Using a size of L Ψ The lookup table Ψ of ×E encodes T into a word vector The process is expressed as:

[0039] V=f embed (T,Θ embed )+PE

[0040] Among them, L Ψ is the total number of words in the entire vocabulary, represents the number of words contained in the text T in the entire dataset, E is the size of the word embedding, Θ embed f embed (·) is a trainable parameter, and the matrix PE is the position encoding matrix.

[0041] After the embedding layer, the Transformer encoder TE with the second dimension of size E en Extract the correlation from V and compress it through the fully connected (FC) layer to obtain the text semantic feature F T In the decoder, F T First, it is decompressed by the FC layer, and then en Transformer decoder TE with similar structure de Then, the Argmax layer uses the internal full connection and Softmax activation function to transform TE de The output is converted from size L×E to L×L Ψ , and decode the sentence from the dictionary according to the index of the maximum value of each row to obtain the recovered text T. The entire encoding and decoding process can be described as:

[0042] T=TE de (TE en (T; Θ en );Θ de )=f argmax (T de (f de (f en (T en (f embed (T))))))

[0043] Use the recognized text and the real text to train the text codec. The training process is:

[0044]

[0045] Among them, L CE is the cross entropy loss function, Θ en and Θ deare the trainable parameters in the encoder and decoder, respectively, and these parameters are fixed after training is completed.

[0046] At the transmitter, the speech recognition text T is encoded into text semantic features F by the text encoder. T , expressed as:

[0047] F T =TE en (T; Θ TE,en )

[0048] Step 3: Use the quantification module to convert the text semantic features F T and audio semantic features F A quantized into OFDM symbols X and sent to the receiver through the wireless channel; at the receiver, the dequantization module dequantizes the received OFDM symbols X into floating-point semantic features F T and F A .

[0049] Specifically, before sending to the channel, F T and F A Quantized by the quantization module Q(·) into OFDM symbols X T and X A After transmission through the wireless channel, at the receiver, the dequantization module Q -1 (·) Convert these OFDM symbols back to floating point data. T The QPSK quantization is dequantized. First, F T It will be fed into the FC layer f with sigmoid activation function Q (·) Change the dimension from becomes And the quantization operation is performed to obtain:

[0050]

[0051] Among them, Θ Q is the trainable parameter in the quantization module, L text and 2N text are the length of text semantic features and text OFDM respectively, round(·) is the remainder operation, which maps the output of sigmoid to a vector with a value of 0 or 1, and the subsequent operation maps 0 and 1 to a and b respectively. In addition, the additional R2C module converts the binary vector size into The two parts of the second dimension data are divided into real and imaginary parts and spliced ​​together to obtain X representing the OFDM symbol. T .X T After the transmitter transmits, it reaches the receiver after passing through the wireless channel. The received vector X T Expressed as:

[0052] X T =X T +N

[0053] Among them, N represents the mean is 0 and the variance is Additive white Gaussian noise.

[0054] Then, the received vector X T The C2R module performs the transformation from complex to real and restores it to text features F through the following operations T , expressed as:

[0055]

[0056] Among them, Θ deQ is the trainable parameter in the dequantization module, f deQ (·) is a FC layer. In the training text Q(·) and Q -1 During the back propagation of (·), their derivatives are replaced by the derivatives of the expected value to ensure the normal training of the model. The process is as follows:

[0057]

[0058] Among them, L MSE is the mean square error (MSE) loss function. Audio semantic feature F A The quantization and dequantization of is also trained using the above method, and the parameters of the above structure are fixed after the training is completed.

[0059] Step 4: Transformer-based text decoder TE de From the received text semantic features F T Recovered text Τ; recovered text Τ and received audio semantic features F A They are fed into the pre-trained speech cloning model YourTTS together to recover a speech whose text content and speaker timbre are close to the original speech signal.

[0060] Specifically, the received text semantic feature F T Text decoder TE de Restore to text T and combine with the received audio semantic features F A Input them together into the voice cloning model YourTTS to generate a receiving voice similar to the original voice speaker The whole process is as follows:

[0061] S=SC(F A ,T;Θ SC )=SC(F A ,TEde (F T ;Θ de );Θ SC )

[0062] Among them, SC(·) represents the cloning process of the voice cloning model YourTTS, Θ SC are the parameters in YourTTS, which are pre-trained on the speech dataset and not trained in the semantic communication system.

[0063] like Figure 2 As shown, the text encoder converts the recognized text T into text semantic features F T function, the text decoder converts the text semantic features F T Converted into predicted text T. The maximum length of text in the training dataset is L = 70, the word embedding size E is set to 128, and the length of text semantic features is L text The codec was set to 504, and the Transformer layers of the codec were both 6. The text codec was trained using the Adam optimizer with a batch size of 128, 100 training epochs, and a learning rate of 0.001. Because the text T transmitted was the text after speech recognition, the error between T and T included the error of the speech recognition model. However, compared to the impact of wireless channel interference on the semantics of the text in the speech signal, this error was acceptable.

[0064] like Figure 3 As shown, in the transmitter, the quantization module converts the text semantic features F T and audio semantic features F A Quantized into OFDM symbols. Then, the quantized OFDM symbols are transmitted through the AWGN channel to the receiver. The receiver's dequantization module dequantizes the received OFDM symbols into text semantic features F T and audio semantic features F A The Adam optimizer is used to train the quantization and dequantization modules, with a batch size of 128, 100 epochs, and a learning rate of 0.001.

[0065] Therefore, the present invention provides a new method for voice communication by applying deep learning networks to the semantic processing of voice signals. By dividing the semantics of voice signals into text semantics and audio semantics and quantizing and transmitting them, it demonstrates the voice semantic transmission performance of traditional voice transmission schemes in harsh channel environments and saves a lot of bandwidth resources.

[0066] The present invention also provides a speech recognition and cloning semantic speech transmission system, comprising a speech recognition model, a speaker encoder, a text encoder, a quantization module, a dequantization module, and a speech cloning model; the speech recognition model and the speaker encoder respectively extract corresponding text content and speaker timbre features from original speech; the text encoder encodes the recognized text into text semantic features at a transmitter; the text decoder decodes the received text semantic features into text at a receiver; the quantization and dequantization modules are responsible for quantizing the semantic features into OFDM symbols and dequantizing the OFDM symbols into semantic features, respectively; and the speech cloning model combines the received timbre features with the decoded text to generate a speech signal close to the original speaker.

[0067] What is described above in conjunction with the accompanying drawings is only one of the preferred examples of the present invention and cannot be used to limit the scope of rights included in the present invention. It should be pointed out that any equivalent modifications made in accordance with the principles of the present invention are covered by the protection scope of the claims of the present invention.

Claims

1. A method for speech recognition and cloning semantic speech transmission, characterized in that: The following steps are involved: Step 1: Use the speech recognition model and speaker encoder to extract text related to the speech content from the original speech signal and the audio semantic features F related to the speaker's timbre A ; Step 2: Recognize all the original speech in the dataset as text to train the text encoder TE based on the Transformer structure en and decoder TE de After training is completed, the text encoder is placed in the transmitter to complete the text recognition from the original speech signal S Converted into text semantic features F T Function; Step 3: Use the quantification module to convert the text semantic features F T and audio semantic features F A quantized into OFDM symbols X and sent to the receiver through the wireless channel; at the receiver, the dequantization module converts the received OFDM symbols Dequantized semantic features into floating-point form and Step 4: Transformer-based text decoder TE de From the received text semantic features Recover Text Recovered text and the received audio semantic features They are fed into a pre-trained speech cloning model together to recover a speech whose text content and speaker timbre are close to the original speech signal.

2. The method for speech recognition and cloning semantic speech transmission according to claim 1, characterized in that: In step 1, the speech recognition model and speaker encoder are Deep Speech 2 and H / ASP respectively.

3. The method for speech recognition and cloning semantic speech transmission according to claim 1, characterized in that: Both the speech recognition model and speaker encoder are pre-trained on the speech dataset and the weight parameters are fixed before being used directly in the semantic communication system.

4. The method for speech recognition and cloning semantic speech transmission according to claim 1, characterized in that: In step 2, the process of training the text encoder and decoder is as follows: Among them, L CE is the cross entropy loss function, Θ en and Θ de are the trainable parameters in the encoder and decoder, respectively, and these parameters are fixed after training is completed.

5. The method for speech recognition and cloning semantic speech transmission according to claim 1, characterized in that: In step 3, the training text quantization module Q(·) and the text dequantization module Q -1 The process of (·) is: Among them, L MSE is the mean square error loss function; audio semantic feature F A The quantization and dequantization modules use the same training method as the text quantization and dequantization modules, and are optimized by minimizing the mean square error loss; after training, the parameters of the quantization and dequantization modules of text and audio semantic features will remain fixed.

6. The method for speech recognition and cloning semantic speech transmission according to claim 1, characterized in that: The voice cloning model is YourTTS.

7. The method for speech recognition and cloning semantic speech transmission according to claim 6, characterized in that: In step 4, the received text semantic features Text decoder TE de Restore to text And combined with the received audio semantic features Input them together into the voice cloning model YourTTS to generate a receiving voice similar to the original voice speaker The whole process is as follows: Among them, SC(·) represents the cloning process of the voice cloning model YourTTS, Θ SC are the parameters in YourTTS, which are pre-trained on the speech dataset and not trained in the semantic communication system.

8. A speech recognition and cloning semantic speech transmission system, the system being used to implement the method according to any one of claims 1 to 7, characterized in that: It includes a speech recognition model, a speaker encoder, a text encoder, a text decoder, a quantization module, a dequantization module, and a speech cloning model; the speech recognition model and speaker encoder respectively extract the corresponding text content and speaker timbre features from the original speech; the text encoder encodes the recognized text into text semantic features at the transmitter; the text decoder decodes the received text semantic features into text at the receiver; the quantization module is used to quantize the semantic features into OFDM symbols, and the dequantization module is used to dequantize the OFDM symbols into semantic features; the speech cloning model combines the received timbre features with the decoded text to generate a speech signal close to the original speaker.

Images