Speech recognition and clone semantic speech transmission method and system

Abstract

The invention provides a voice recognition and clone semantic voice transmission method and system, and the method comprises the steps: firstly, recognizing an original voice signal as a text through a voice recognition model at a transmitter; then, the recognized text is encoded into text features by an encoder based on Transform; extracting audio semantic features from the original voice signal by a speaker encoder; the semantic features in the two floating point forms are quantized into OFDM symbols by a quantization layer and are sent to a receiver; the receiver dequantizes the OFDM symbol into a semantic feature in a floating point form through a dequantization module, and recovers a text in the original voice signal from the text feature by using a decoder based on Transform; and meanwhile, the voice cloning model completes cloning of the original voice signal by using the recovered text and the received audio semantic features to obtain a voice signal close to the tone of the speaker in the original voice. According to the invention, the bandwidth occupied by voice signal transmission is obviously reduced, and the quality of voice signal transmission and the user experience are greatly improved.

Description

Technical Field

[0001] The present invention belongs to the technical field of semantic communication, and in particular relates to a method and system for speech recognition and cloning semantic speech transmission. Background Art

[0002] With the rapid development of wireless communication technology, the capacity of traditional communication systems has gradually approached the limit of Shannon's theory. At the same time, massive amounts of wireless data have also become a burden on communication infrastructure. Compared with traditional communication that focuses on the transmission of bit data, semantic communication, as a new communication paradigm, understands the meaning of information at the semantic level and only transmits the semantic content in the information, which greatly reduces the amount of transmitted data and reduces the bandwidth requirements of the system. The transmission of voice signals is the main mode of modern communication. It not only contains the text information that the speaker wants to convey, but also contains the speaker's timbre characteristics. However, the transmission of traditional voice signals depends on the encoding and decoding of voice sources. During the transmission process, a large amount of bandwidth resources are required, and it is difficult to ensure the accurate transmission of voice in poor channel environments.

[0003] Existing research only transmits the features related to speech recognition in the speech signal, and uses the features to restore the text information at the receiving end, ensuring the accurate transmission of the text information in the speech signal with a smaller bandwidth. At the same time, the restored text is sent to an independent speech synthesis model to achieve the restoration of the original speech signal. However, this type of speech synthesis model is only for a single speaker, and when the speaker changes, it needs to be retrained, resulting in a large waste of computing resources. Therefore, how to use the current deep learning technology to design a semantic speech transmission method with low bandwidth occupancy, strong anti-interference ability and suitable for multiple speakers is an urgent problem that needs to be solved. Summary of the invention

[0004] Purpose of the invention: The present invention provides a method and system for speech recognition and cloning semantic speech transmission, aiming to utilize the advantages of semantic communication to solve the problems of large bandwidth requirements and poor anti-interference ability in the process of speech signal transmission, thereby ensuring accurate transmission of multi-speaker speech signals with lower transmission bandwidth.

[0005] Technical solution: The technical solution adopted by the present invention specifically includes the following steps:

[0006] Step 1: Divide the semantics in the speech signal into text semantics related to the transmission content and audio semantics related to the speaker's timbre characteristics in the speech signal. Then, using the speech recognition model Deep Speech2 and speaker encoder H / ASP in deep learning, extract the text T related to the speech content and the audio semantics F related to the speaker's timbre from the original speech signal. A ;

[0007] Step 2: Recognize all the original speech in the dataset as text to train the text encoder TE based on the Transformer structure. en and decoder TE de After training, the text encoder is placed in the transmitter to convert the text T recognized from the original speech signal S into text semantic features F T Functions;

[0008] Step 3: Use the quantization module to convert the text semantic features F T and audio semantic features F A quantized into OFDM symbol X and sent to the receiver through the wireless channel; at the receiver, the dequantization module dequantizes the received OFDM symbol X into a floating-point semantic feature F T and F A ;

[0009] Step 4: Transformer-based text decoder TE de From the received text semantic features, F T Restore text Τ; Restore text Τ and received audio semantic features F A They are sent together to the pre-trained speech cloning model YourTTS to restore the speech whose text content and speaker's timbre are close to the original speech signal.

[0010] As a further technical solution of the present invention, in the step 1, the speech recognition model Deep Speech 2 and the speaker encoder H / ASP are both pre-trained on the speech data set and have their weight parameters fixed before being directly used in the semantic communication system.

[0011] As a further technical solution of the present invention, in step 2, a speech set with a total duration of 100 hours in the multi-speaker speech dataset Librispecch is used as a training set, which includes speech samples S and their corresponding texts T. The speech samples S are recognized as texts T by the Deep Speech 2 model and used to train the text codec, which is expressed as:

[0012]

[0013] Among them, L CE is the cross entropy loss function, Θ en and θ de are the trainable parameters in the encoder and decoder respectively, and these parameters are fixed after training is completed.

[0014] At the transmitter, the speech recognition text T is encoded into text semantic features F by the text encoder. T , expressed as:

[0015] F T =TE en (T; θ TE,en )

[0016] As a further technical solution of the present invention, in step 3, the training text semantic feature F T The process of quantization and dequantization module is expressed as:

[0017]

[0018] Among them, L MSE is the mean square error (MSE) loss function, Θ Q and θ deQ are the trainable parameters in the encoder and decoder respectively. Audio semantic feature F A The quantization and dequantization of is also trained using the above method, and the parameters of the above structure are fixed after the training is completed.

[0019] As a further technical solution of the present invention, in step 4, the received text semantic feature F T Text decoder TE de Restore to text T and combine with the received audio semantic features F A The two are input into the voice cloning model YourTTS to generate a receiving voice similar to the original voice speaker. The whole process is as follows:

[0020] S=SC(F A ,T;Θ SC )=SC(F A ,TE de (F T ; Θ de );Θ SC )

[0021] Among them, SC(·) represents the cloning process of the voice cloning model YourTTS, Θ SC are the parameters in YourTTS, which are pre-trained on the speech dataset and not trained in the semantic communication system.

[0022] The present invention also provides a speech recognition and cloning semantic speech transmission system, comprising a speech recognition model, a speaker encoder, a text encoder, a text encoder, a quantization module, a dequantization module, and a speech cloning model; the speech recognition model and the speaker encoder respectively extract corresponding text content and speaker timbre features from the original speech; the text encoder encodes the recognized text into text semantic features at the transmitter; the text decoder decodes the received text semantic features into text at the receiver; the quantization and dequantization modules are responsible for quantizing the semantic features into OFDM symbols and dequantizing the OFDM symbols into semantic features respectively; the speech cloning model combines the received timbre features with the decoded text to generate a speech signal close to the original speaker.

[0023] Beneficial effects: Compared with the prior art, the present invention adopts the above technical solution and has the following beneficial effects:

[0024] 1. The present invention divides the semantics in the speech signal into text semantics related to the transmission content and audio semantics related to the speaker's timbre characteristics in the speech signal. The transmitter uses a deep learning method to extract text semantic features and audio semantic features from the speech signal for transmission, which greatly reduces the bandwidth resources required for transmission compared to existing methods.

[0025] 2. The text encoding and decoding based on Transformer in the present invention ensures the accuracy of text semantics in speech signals under extremely poor channel environments. At the same time, the speech cloning network, with the help of the restored text and received audio features, can reconstruct speech signals close to the speaker even in extremely poor channel environments without the need to retrain the network. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] Figure 1 It is a schematic diagram of the principle of the speech recognition and cloning semantic speech transmission method based on deep learning of the present invention;

[0027] Figure 2 Schematic diagram of the principle of the text codec based on Transformer in the present invention;

[0028] Figure 3 It is a schematic diagram of the principle of the module for quantizing and dequantizing semantic features in the present invention. DETAILED DESCRIPTION

[0029] The embodiments of the present invention will be described below with reference to the accompanying drawings.

[0030] like Figure 1 As shown, the present invention designs a method for speech recognition and cloning semantic speech transmission, comprising:

[0031] Step 1: Divide the semantics in the speech signal into text semantics related to the transmission content and audio semantics related to the speaker's timbre characteristics in the speech signal. Then, using the speech recognition model Deep Speech2 and speaker encoder H / ASP in deep learning, extract the text T related to the speech content and the audio semantics F related to the speaker's timbre from the original speech signal S respectively. A ;

[0032] Specifically, the speech recognition model Deep Speech 2 and the speaker encoder H / ASP are both pre-trained on the speech dataset and the weight parameters are fixed before being used directly in the semantic communication system. The process of using Deep Speech 2 to recognize text from the original speech signal is expressed as:

[0033] T = ASR(S; θ ASR )

[0034] Among them, ASR(·) represents the process of Deep Speech 2 speech recognition, Θ ASR The parameters are as follows. Use speaker encoder H / ASP to extract audio semantic features The process is expressed as:

[0035] F A =SE en (S; θ SE )

[0036] Among them, L audio is the length of the audio semantic feature, SE en (·) represents the process of speaker encoder H / ASP speech recognition, Θ SE For the parameters.

[0037] Step 2: Recognize all the original speech in the dataset as text to train the text encoder TE based on the Transformer structure. en and decoder TE de After training is completed, the text encoder is placed in the transmitter to complete the conversion of the text T recognized from the original speech signal S into text semantic features. function, where L text is the length of the text semantic feature;

[0038] Specifically, a speech collection with a total length of 100 hours in the multi-speaker speech dataset Librispecch is used as the training set for the entire semantic communication system. The collection includes speech samples S and their corresponding texts T. First, the recognized sentence T is padded with 0 to the maximum length L and passed through the text encoder TE en and decoder TE deComplete the encoding and decoding of the sentence. Then, the embedding layer f embed (·) Using a size L Ψ The lookup table Ψ of ×E encodes T into a word vector The process is expressed as:

[0039] V=f embed (T,Θ embed )+PE

[0040] Among them, L Ψ is the total number of words in the entire vocabulary, represents the number of words contained in the text T in the entire dataset, E is the size of the word embedding, Θ embed f embed (·) is a trainable parameter, and the matrix PE is the position encoding matrix.

[0041] After the embedding layer, the Transformer encoder TE with the second dimension size E en Extract the correlation from V and compress it through the fully connected (FC) layer to obtain the text semantic feature F T In the decoder, F T First, it is decompressed by the FC layer, and then it is transmitted through the TE en Transformer decoder TE with similar structure de Then, the Argmax layer uses the internal full connection and Softmax activation function to transform TE de The output is converted from size L×E to L×L Ψ , and decode the sentence from the dictionary according to the index of the maximum value of each row to obtain the restored text T. The entire encoding and decoding process can be described as:

[0042] T=TE de (TE en (T; θ en );Θ de )=f argmax (T de (f de (f en (T en (f embed (T))))))

[0043] Use the recognized text and the real text to train the text codec. The training process is:

[0044]

[0045] Among them, L CE is the cross entropy loss function, Θ en and θ deare the trainable parameters in the encoder and decoder respectively, and these parameters are fixed after training is completed.

[0046] At the transmitter, the speech recognition text T is encoded into text semantic features F by the text encoder. T , expressed as:

[0047] F T =TE en (T; θ TE,en )

[0048] Step 3: Use the quantization module to convert the text semantic features F T and audio semantic features F A quantized into OFDM symbol X and sent to the receiver through the wireless channel; at the receiver, the dequantization module dequantizes the received OFDM symbol X into a floating-point semantic feature F T and F A .

[0049] Specifically, before sending to the channel, F T and F A Quantized by quantization module Q(·) into OFDM symbol X T and X A After transmission through the wireless channel, at the receiver, the dequantization module Q -1 (·) Convert these OFDM symbols back to floating point data. T The QPSK quantization is dequantized. First, F T It will be sent to the FC layer f with sigmoid activation function Q (·) Change the dimension from becomes And the quantization operation is performed to obtain:

[0050]

[0051] Among them, Θ Q is the trainable parameter in the quantization module, L text and 2N text are the length of the text semantic feature and the text OFDM, respectively. round(·) is the modulo operation, which maps the output of sigmoid to a vector with a value of 0 or 1. The subsequent operation maps 0 and 1 to a and b, respectively. In addition, the additional R2C module converts the binary vector size into The two parts of the second dimension data are divided into real and imaginary parts and concatenated to obtain X representing the OFDM symbol. T .X T After the transmitter transmits, it reaches the receiver after passing through the wireless channel. The received vector X T It is expressed as:

[0052] X T =X T +N

[0053] Among them, N represents the mean is 0 and the variance is Additive white Gaussian noise.

[0054] Then, the received vector X T The C2R module performs the transformation from complex to real and restores it to text features F through the following operations T , expressed as:

[0055]

[0056] Among them, Θ deQ is the trainable parameter in the dequantization module, f deQ (·) is a FC layer. In the training text Q(·) and Q -1 During the back propagation of (·), their derivatives are replaced by the derivatives of the expected value to ensure the normal training of the model. The process is as follows:

[0057]

[0058] Among them, L MSE is the mean square error (MSE) loss function. Audio semantic feature F A The quantization and dequantization of is also trained using the above method, and the parameters of the above structure are fixed after the training is completed.

[0059] Step 4: Transformer-based text decoder TE de From the received text semantic features, F T Restore text Τ; Restore text Τ and received audio semantic features F A They are sent together to the pre-trained speech cloning model YourTTS to restore the speech whose text content and speaker's timbre are close to the original speech signal.

[0060] Specifically, the received text semantic feature F T Text decoder TE de Restore to text T and combine with the received audio semantic features F A The two are input into the voice cloning model YourTTS to generate a receiving voice similar to the original voice speaker. The whole process is as follows:

[0061] S=SC(F A ,T;Θ SC )=SC(F A ,TEde (F T ; Θ de );Θ SC )

[0062] Among them, SC(·) represents the cloning process of the voice cloning model YourTTS, Θ SC are the parameters in YourTTS, which are pre-trained on the speech dataset and not trained in the semantic communication system.

[0063] like Figure 2 As shown, the text encoder converts the recognized text T into text semantic features F T The text decoder converts the text semantic features F T Transformed into predicted text T. The maximum length of text in the training dataset is L = 70, the word embedding size E is set to 128, and the length of text semantic features is L text The codec is set to 504, and the number of layers of the Transformer of the codec is 6. The Adam optimizer is used to train the text codec, the batch size is set to 128, the training is 100 rounds, and the learning rate is set to 0.001. Because the text T after speech recognition is transmitted, the error between T and T includes the error of the speech recognition model, but compared with the impact of wireless channel interference on the text semantics in the speech signal, this error is acceptable.

[0064] like Figure 3 As shown, in the transmitter, the quantization module converts the text semantic features F T and audio semantic features F A Quantized into OFDM symbols. Then, the quantized OFDM symbols are transmitted through the AWGN channel to the receiver. The receiver's dequantization module dequantizes the received OFDM symbols into text semantic features F T and audio semantic features F A The Adam optimizer is used to train the quantization and dequantization modules, with the batch size set to 128, 100 rounds of training, and the learning rate set to 0.001.

[0065] Therefore, the present invention provides a new method for voice communication by applying a deep learning network to the semantic processing of voice signals. By dividing the semantics of voice signals into text semantics and audio semantics and quantizing and transmitting them, it demonstrates the voice semantic transmission performance of traditional voice transmission schemes under harsh channel environments and saves a lot of bandwidth resources.

[0066] The present invention also provides a speech recognition and cloning semantic speech transmission system, comprising a speech recognition model, a speaker encoder, a text encoder, a text encoder, a quantization module, a dequantization module, and a speech cloning model; the speech recognition model and the speaker encoder respectively extract corresponding text content and speaker timbre features from the original speech; the text encoder encodes the recognized text into text semantic features at the transmitter; the text decoder decodes the received text semantic features into text at the receiver; the quantization and dequantization modules are responsible for quantizing the semantic features into OFDM symbols and dequantizing the OFDM symbols into semantic features respectively; the speech cloning model combines the received timbre features with the decoded text to generate a speech signal close to the original speaker.

[0067] What is described above in conjunction with the accompanying drawings is only one of the preferred examples of the present invention and cannot be used to limit the scope of rights included in the present invention. It should be pointed out that any equivalent modifications made in accordance with the principles of the present invention are covered by the protection scope of the claims of the present invention.

Claims

1. A method for speech recognition and cloning semantic speech transmission, characterized in that: The following steps are involved: Step 1: Use the speech recognition model and speaker encoder to extract the text T related to the speech content and the audio semantic features F related to the speaker's timbre from the original speech signal respectively. A ; Step 2: Recognize all the original speech in the dataset as text to train the text encoder TE based on the Transformer structure. en and decoder TE de After the training is completed, the text encoder is placed in the transmitter to complete the conversion of the text T recognized from the original speech signal S into text semantic features F T Functions; Step 3: Use the quantization module to convert the text semantic features F T and audio semantic features F A quantized into OFDM symbol X and sent to the receiver through the wireless channel; at the receiver, the dequantization module dequantizes the received OFDM symbol X into a floating-point semantic feature F T and F A ; Step 4: Transformer-based text decoder TE de From the received text semantic features, F T Restore text Τ; Restore text Τ and received audio semantic features F A They are sent together into the pre-trained speech cloning model to restore speech with text content and speaker timbre close to the original speech signal.

2. The method for speech recognition and cloning semantic speech transmission according to claim 1, characterized in that: In the step 1, the speech recognition model and the speaker encoder are Deep Speech 2 and H / ASP respectively.

3. The method for speech recognition and cloning semantic speech transmission according to claim 1, characterized in that: Both the speech recognition model and the speaker encoder are pre-trained on the speech dataset and the weight parameters are fixed before being used directly in the semantic communication system.

4. The method for speech recognition and cloning semantic speech transmission according to claim 1, characterized in that: In step 2, the process of training the text encoder and decoder is: Among them, L CE is the cross entropy loss function, Θ en and θ de are the trainable parameters in the encoder and decoder respectively, and these parameters are fixed after training is completed.

5. The method for speech recognition and cloning semantic speech transmission according to claim 1, characterized in that: In step 3, the text quantization module Q() and the text dequantization module Q are trained. -1 The process of () is: Among them, L MSE is the mean square error loss function; audio semantic feature F A The quantization and dequantization of is also trained using the above method, and the parameters of the above structure are fixed after the training is completed.

6. The method for speech recognition and cloning semantic speech transmission according to claim 1, characterized in that: The voice cloning model is YourTTS.

7. The method for speech recognition and cloning semantic speech transmission according to claim 6, characterized in that: In step 4, the received text semantic feature F T Text decoder TE de Restore to text T and combine with the received audio semantic features F A The two are input into the voice cloning model YourTTS to generate a receiving voice similar to the original voice speaker. The whole process is as follows: S=SC(F A ,T;Θ SC )=SC(F A ,TE de (F T ;Θ de );Θ SC ) Among them, SC() represents the cloning process of the voice cloning model YourTTS, Θ SC are the parameters in YourTTS, which are pre-trained on the speech dataset and not trained in the semantic communication system.

8. A speech recognition and cloning semantic speech transmission system, characterized in that: It includes a speech recognition model, a speaker encoder, a text encoder, a text decoder, a quantization module, a dequantization module, and a speech cloning model; the speech recognition model and the speaker encoder respectively extract the corresponding text content and the speaker's timbre features from the original speech; the text encoder encodes the recognized text into text semantic features at the transmitter; the text decoder decodes the received text semantic features into text at the receiver; the quantization and dequantization modules are responsible for quantizing the semantic features into OFDM symbols and dequantizing the OFDM symbols into semantic features, respectively; the speech cloning model combines the received timbre features with the decoded text to generate a speech signal close to the original speaker.

Images