Variable-bit-rate redundancy-eliminating voice semantic coding method and device

Abstract

The invention relates to the technical field of voice coding, in particular to a variable-bit-rate redundancy-eliminating voice semantic coding method and device, and the method comprises the steps: obtaining the original audio of a target speaker based on a preset deep neural network model; extracting linguistic discrete semantic representation in the original audio, and obtaining voiceprint features and compressed spectrum features of the original audio; and respectively carrying out reconstruction processing on the linguistic discrete semantic representation, the voiceprint feature and the compressed spectrum feature to obtain a target audio acoustic intermediate representation, and obtaining a target audio to be output by utilizing the target audio acoustic intermediate representation so as to carry out redundancy elimination voice semantic coding on the original audio. Therefore, the problems that in the related technology, a voice synthesis model is a pre-training model with fixed parameters, speaker features in voice are difficult to change and recover according to different speakers, part of voice information is distorted, the quality and efficiency of voice recognition are reduced, and the accuracy of voice recovery is reduced are solved.

Description

Technical Field

[0001] The present invention relates to the technical field of speech coding, and in particular to a variable bit rate redundant speech semantic coding method and device. Background Art

[0002] In the related technology, most of the speech semantic coding algorithms based on DNN (Deep Neural Network) perform nonlinear transformation on the speech waveform through the convolution layer and then perform vector quantization. The purpose of this method is to separate the irrelevant components in the speech signal through a pure deep neural network method. Among them, the deep neural network requires a certain training scale to perform speech semantic coding based on the deep neural network.

[0003] However, the degree of de-redundancy of speech semantic coding obtained in related technologies is limited by the scale of the model, and the existing sound synthesis model is a pre-trained model with fixed parameters, which makes it difficult to change the speaker characteristics in the restored speech according to different speakers, resulting in distortion of some speech information, reducing the quality and efficiency of speech recognition, and reducing the accuracy of speech recovery, which urgently needs to be solved. Summary of the invention

[0004] The present invention provides a variable bit rate de-redundant speech semantic coding method and device to solve the problem that the de-redundancy degree of speech semantic coding obtained in the related art is limited by the model scale, and the existing sound synthesis model is a pre-trained model with fixed parameters, which is difficult to change the speaker characteristics in the restored speech according to different speakers, resulting in distortion of part of the speech information, reducing the quality and efficiency of speech recognition, and reducing the accuracy of speech recovery.

[0005] The first aspect of the present invention provides a variable bit rate de-redundant speech semantic coding method, comprising the following steps: based on a preset deep neural network model, obtaining the original audio of the target speaker; extracting the linguistic discrete semantic representation in the original audio, and obtaining the voiceprint features and compressed spectrum features of the original audio; reconstructing the linguistic discrete semantic representation, the voiceprint features and the compressed spectrum features respectively to obtain an acoustic intermediate representation of the target audio, and using the acoustic intermediate representation of the target audio to obtain the target audio to be output, so as to perform de-redundant speech semantic coding on the original audio.

[0006] Optionally, in one embodiment of the present invention, the extracting the linguistic discrete semantic representation of the original audio includes: obtaining a feature vector of a speech signal corresponding to the original audio; and obtaining the linguistic code using the feature vector to extract the linguistic discrete semantic representation corresponding to the linguistic code.

[0007] Optionally, in one embodiment of the present invention, the speech signal is represented as:

[0008]

[0009] Among them, h(t) is the speech signal, G w is the lip, tooth and tongue filter bank, f(t) is the basic waveform, G r is the vocal cord and resonance filter bank, n r is the number of vocal cords and resonance filters, n w is the number of lip, tooth and tongue filters, i is the lip, tooth and tongue filter number, and j is the vocal cord and resonance filter number.

[0010] Optionally, in one embodiment of the present invention, the acquiring of the voiceprint features and compressed spectrum features of the original audio includes: converting the speech waveform corresponding to the original audio into a Mel spectrum representing acoustic features; extracting the acoustic information latent code corresponding to the Mel spectrum; acquiring the voiceprint feature code using the acoustic information latent code, so as to obtain the voiceprint feature using the voiceprint feature code; based on the acoustic information latent code, acquiring the corresponding compressed spectrum code, so as to obtain the compressed spectrum feature using the compressed spectrum code.

[0011] Optionally, in one embodiment of the present invention, the reconstructing the linguistic discrete semantic representation, the voiceprint feature and the compressed spectral feature respectively to obtain the target audio acoustic intermediate representation includes: reconstructing the linguistic discrete semantic representation to obtain a semantic embedding; reconstructing the voiceprint feature to obtain a voiceprint embedding; reconstructing the compressed spectral feature to obtain a spectral embedding; performing additive aggregation using the semantic embedding, the voiceprint embedding, the spectral embedding and the Mel spectrum of the target time step to obtain an aggregated embedding vector; converting the aggregated embedding vector into a Mel spectrum of the current time step; and concatenating the Mel spectrum of the target time step and the Mel spectrum of the current time step to obtain the target audio acoustic intermediate representation.

[0012] The second aspect of the present invention provides a variable bit rate de-redundant speech semantic coding device, including: an acquisition module, used to acquire the original audio of the target speaker based on a preset deep neural network model; a determination module, used to extract the linguistic discrete semantic representation in the original audio, and obtain the voiceprint features and compressed spectrum features of the original audio; a processing module, used to reconstruct the linguistic discrete semantic representation, the voiceprint features and the compressed spectrum features respectively to obtain an acoustic intermediate representation of the target audio, and use the acoustic intermediate representation of the target audio to obtain the target audio to be output, so as to perform de-redundant speech semantic coding on the original audio.

[0013] Optionally, in one embodiment of the present invention, the determination module includes: a first acquisition unit, used to acquire a feature vector of a speech signal corresponding to the original audio; and a first extraction unit, used to acquire the linguistic code using the feature vector to extract the linguistic discrete semantic representation corresponding to the linguistic code.

[0014] Optionally, in one embodiment of the present invention, the speech signal is represented as:

[0015]

[0016] Among them, h(t) is the speech signal, G w is the lip, tooth and tongue filter bank, f(t) is the basic waveform, G r is the vocal cord and resonance filter bank, n r is the number of vocal cords and resonance filters, n w is the number of lip, tooth and tongue filters, i is the lip, tooth and tongue filter number, and j is the vocal cord and resonance filter number.

[0017] Optionally, in one embodiment of the present invention, the determination module includes: a conversion unit, used to convert the speech waveform corresponding to the original audio into a Mel spectrum representing the acoustic feature; a second extraction unit, used to extract the acoustic information latent code corresponding to the Mel spectrum; a second acquisition unit, used to obtain the voiceprint feature code using the acoustic information latent code, so as to obtain the voiceprint feature using the voiceprint feature code; a first determination unit, used to obtain the corresponding compressed spectrum code based on the acoustic information latent code, so as to obtain the compressed spectrum feature using the compressed spectrum code.

[0018] Optionally, in one embodiment of the present invention, the processing module includes: a first processing unit, used to reconstruct the linguistic discrete semantic representation to obtain a semantic embedding; a second processing unit, used to reconstruct the voiceprint feature to obtain a voiceprint embedding; a third processing unit, used to reconstruct the compressed spectral feature to obtain a spectral embedding; a second determination unit, used to perform additive aggregation using the semantic embedding, the voiceprint embedding, the spectral embedding and the Mel spectrum of the target time step to obtain an aggregated embedding vector; a fourth processing unit, used to convert the aggregated embedding vector into a Mel spectrum of the current time step; a third acquisition unit, used to concatenate the Mel spectrum of the target time step and the Mel spectrum of the current time step to obtain the target audio acoustic intermediate representation.

[0019] The third aspect of the present invention provides an electronic device, comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the variable bit rate de-redundant speech semantic coding method as described in the above embodiment.

[0020] A fourth aspect of the present invention provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, which, when executed by a processor, implements the above variable bit rate de-redundant speech semantic coding method.

[0021] A fifth aspect of the present invention provides a computer program product, including a computer program, which, when executed, is used to implement the above variable bit rate de-redundant speech semantic coding method.

[0022] The embodiment of the present invention can reconstruct the linguistic discrete semantic representation, voiceprint features and compressed spectrum features obtained from the original audio based on the deep neural network model, obtain the target audio acoustic intermediate representation, and use the target audio acoustic intermediate representation to obtain the target audio to be output, so as to perform redundancy speech semantic coding on the original audio, effectively improve the quality and efficiency of speech recognition, and improve the accuracy of speech recovery. Thus, the problem that the degree of redundancy of speech semantic coding obtained in the related art is limited by the scale of the model, and the existing sound synthesis model is a pre-trained model with fixed parameters, which makes it difficult to change the speaker features in the restored speech according to different speakers, resulting in partial distortion of speech information, reducing the quality and efficiency of speech recognition, and reducing the accuracy of speech recovery is solved.

[0023] Additional aspects and advantages of the present invention will be given in part in the following description and in part will be obvious from the following description, or will be learned through practice of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] The above and / or additional aspects and advantages of the present invention will become apparent and easily understood from the following description of the embodiments in conjunction with the accompanying drawings, in which:

[0025] Figure 1 A flowchart of a variable bit rate de-redundant speech semantic coding method provided according to an embodiment of the present invention;

[0026] Figure 2 A schematic diagram of a de-redundant speech semantic coding system according to a specific embodiment of the present invention;

[0027] Figure 3 A schematic diagram of the structure of a semantic extraction module according to a specific embodiment of the present invention;

[0028] Figure 4This is a structural diagram of a voiceprint feature extraction module according to a specific embodiment of the present invention;

[0029] Figure 5 It is a structural schematic diagram of an acoustic synthesis module of a specific embodiment of the present invention;

[0030] Figure 6 It is a structural schematic diagram of a vocoder according to a specific embodiment of the present invention;

[0031] Figure 7 A schematic diagram of the structure of a variable bit rate redundant speech semantic coding device provided according to an embodiment of the present invention;

[0032] Figure 8 It is a schematic diagram of the structure of an electronic device provided according to an embodiment of the present invention. DETAILED DESCRIPTION

[0033] Embodiments of the present invention are described in detail below, examples of which are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are intended to be used to explain the present invention, and should not be construed as limiting the present invention.

[0034] The following describes the variable bit rate de-redundant speech semantic coding method and device of the embodiment of the present invention with reference to the accompanying drawings. In view of the problem that the de-redundancy degree of the speech semantic coding obtained in the related art mentioned in the above background technology is limited by the model scale, and the existing sound synthesis model is a pre-trained model with fixed parameters, which is difficult to change the speaker characteristics in the restored speech according to different speakers, resulting in the distortion of part of the speech information, reducing the quality and efficiency of speech recognition, and reducing the accuracy of speech recovery, the present invention provides a variable bit rate de-redundant speech semantic coding method, in which the linguistic discrete semantic representation, voiceprint features and compressed spectrum features obtained in the original audio can be reconstructed and processed respectively based on a deep neural network model to obtain the target audio acoustic intermediate representation, and the target audio to be output is obtained by using the target audio acoustic intermediate representation to perform de-redundant speech semantic coding on the original audio, which effectively improves the quality and efficiency of speech recognition, and improves the accuracy of speech recovery. Thereby, the problem that the degree of de-redundancy of speech semantic coding obtained in the related technology is limited by the model scale, and the existing sound synthesis model is a pre-trained model with fixed parameters, which makes it difficult to change the speaker characteristics in the restored speech according to different speakers, resulting in distortion of part of the speech information, reducing the quality and efficiency of speech recognition, and reducing the accuracy of speech recovery is solved.

[0035] Specifically, Figure 1A schematic flow chart of a variable bit rate de-redundant speech semantic coding method provided by an embodiment of the present invention.

[0036] like Figure 1 As shown, the variable bit rate de-redundant speech semantic coding method comprises the following steps:

[0037] In step S101, the original audio of the target speaker is obtained based on a preset deep neural network model.

[0038] In an embodiment of the present invention, the preset deep neural network model is a modular and multi-step deep neural network model, that is, a variable bit rate de-redundant speech semantic coding algorithm model, wherein the de-redundant speech semantic coding algorithm is composed of an encoder and a decoder, the encoder includes a semantic extraction module and a voiceprint feature extraction module, and the decoder includes an acoustic synthesis module and a vocoder. This model will be used as an example for detailed explanation below.

[0039] It can be understood that the embodiment of the present invention can obtain the original audio of the target speaker, that is, the original audio to be converted, through the above-mentioned model. The embodiment of the present invention can perform de-redundant speech semantic coding on the original audio to be converted through a deep neural network model, which effectively improves the executability of de-redundant speech semantic coding.

[0040] In step S102, the linguistic discrete semantic representation in the original audio is extracted, and the voiceprint features and compressed spectrum features of the original audio are obtained.

[0041] It can be understood that the embodiment of the present invention can extract the linguistic discrete semantic representation in the original audio in the following steps. For example, the embodiment of the present invention can process the original audio through the semantic extraction module in the above-mentioned encoder, that is, the semantic extraction module inputs the original audio to be converted, outputs the linguistic discrete semantic representation contained in the original audio, and obtains the voiceprint features and compressed spectrum features of the original audio in the following steps. For example, the embodiment of the present invention can use the voiceprint feature extraction module in the above-mentioned encoder to input the acoustic spectrum of the original audio, and output the voiceprint features of the original audio speaker and the compressed spectrum feature encoding of the original audio, which effectively improves the accuracy of de-redundant speech semantic encoding.

[0042] For example, Figure 2 As shown, the semantic extraction module in the encoder is a self-attention pre-trained linguistics large model with a vector quantization layer; the semantic extraction module can use the pre-trained Hubert algorithm to discretize the linguistic discrete semantic representation into an integer sequence of 1 to 100 after vector quantization.

[0043] For example, Figure 2As shown, the voiceprint feature extraction module can be a deep neural network model. The voiceprint feature extraction module can use a Mel filter to convert the speech waveform from the time domain to the frequency domain, and then input the Mel spectrum obtained in the following steps into a three-layer convolutional neural network and a layer of long short-term memory neural network. The output of the last step is taken, and after a linear layer, it is converted into a fixed-length voiceprint feature vector. In addition, a nonlinear transformation layer is added after the convolution to convert the spectrum features into a low-dimensional compressed spectrum feature vector.

[0044] Among them, in one embodiment of the present invention, extracting the linguistic discrete semantic representation of the original audio includes: obtaining the feature vector of the speech signal corresponding to the original audio; using the feature vector to obtain the linguistic code to extract the linguistic discrete semantic representation corresponding to the linguistic code.

[0045] During the actual implementation process, the embodiment of the present application can obtain the feature vector of the speech signal corresponding to the original audio based on a specific convolutional layer, and then, based on a specific self-attention layer, obtain the linguistic code corresponding to the feature vector. Thirdly, based on a specific vector quantization layer, obtain the linguistic discrete semantic representation corresponding to the linguistic code, thereby effectively improving the efficiency of de-redundant speech semantic coding.

[0046] For example, the embodiments of the present invention can be based on the acoustic principle of human speech, that is, the fundamental tone of the vocal cords resonates when the airflow passes through the vocal cords, and then the sound resonates in the natural resonators such as the oral cavity and nasal cavity to obtain overtones. Using the filter as an analogy, the sound produced by a person can be regarded as a basic waveform f(t) that first passes through a vocal cord and resonance filter group G that determines the characteristics of the speaker. r , and then through a series of lips, teeth and tongue filters G that change dynamically according to the content of the speech w , and the final speech signal h(t) is obtained.

[0047] Among them, the speech signal is expressed as:

[0048]

[0049] Among them, h(t) is the speech signal, G w is the lip, tooth and tongue filter bank, f(t) is the basic waveform, G r is the vocal cord and resonance filter bank, n r is the number of vocal cords and resonance filters, n w is the number of lip, tooth and tongue filters, i is the lip, tooth and tongue filter number, and j is the vocal cord and resonance filter number.

[0050] Since the acoustic features in the embodiment of the present invention are largely repeated in the speech signal, redundant features that need to be removed are input to improve the efficiency of de-redundant speech semantic coding.

[0051] Among them, in one embodiment of the present invention, the voiceprint features and compressed spectrum features of the original audio are obtained, including: converting the speech waveform corresponding to the original audio into a Mel spectrum representing the acoustic features; extracting the acoustic information latent code corresponding to the Mel spectrum; using the acoustic information latent code to obtain the voiceprint feature code, and using the voiceprint feature code to obtain the voiceprint feature; based on the acoustic information latent code, obtaining the corresponding compressed spectrum code, and using the compressed spectrum code to obtain the compressed spectrum feature.

[0052] As a possible implementation method, an embodiment of the present invention can convert the speech waveform corresponding to the original audio into a Mel spectrum representing the acoustic features, and then, based on a specific convolutional layer, obtain the acoustic information latent code corresponding to the Mel spectrum, secondly, based on a specific long short-term memory layer, obtain the voiceprint feature code corresponding to the acoustic information latent code, and then, based on a specific linear mapping layer, obtain the voiceprint feature vector corresponding to the voiceprint feature code, and finally, based on a specific nonlinear mapping layer, obtain the corresponding compressed spectrum code from the acoustic information latent code, thereby obtaining compressed spectrum features using compressed spectrum coding, which effectively improves the efficiency and quality of de-redundant speech semantic coding.

[0053] In step S103, the linguistic discrete semantic representation, voiceprint features and compressed spectral features are reconstructed respectively to obtain the target audio acoustic intermediate representation, and the target audio acoustic intermediate representation is used to obtain the target audio to be output, so as to perform de-redundant speech semantic encoding on the original audio.

[0054] It can be understood that the embodiments of the present invention can reconstruct the linguistic discrete semantic representation, voiceprint features and compressed spectral features respectively. For example, the embodiments of the present invention can process the linguistic discrete semantic representation, voiceprint features and compressed spectral features through the acoustic synthesis module in the decoder, so as to obtain the target audio acoustic intermediate representation corresponding to the linguistic discrete semantic representation, voiceprint features and compressed spectral features, and use the vocoder in the decoder to process the target audio acoustic intermediate representation to obtain the target audio to be output, and output the target audio, and then perform redundancy speech semantic encoding on the original audio, thereby effectively improving the quality of speech recognition.

[0055] For example, the acoustic synthesis module can be a deep neural network autoregressive model. The acoustic synthesis module can use a nonlinear layer to match the Mel spectrum of the target speaker's timbre generated in the previous time step, the input linguistic discrete speech representation sequence, the input voiceprint feature vector and the length of the compressed spectrum feature vector, and add and aggregate the data, input the multi-layer residual long short-term memory neural network, and finally obtain a Mel spectrum with the characteristics of the target speaker.

[0056] Among them, the embodiment of the present invention can determine the amount of semantic information at the current time step based on the linguistic discrete semantic representation extracted by the semantic extraction module, and determine the length of the compressed acoustic spectral features that should be used for actual transmission based on the amount of semantic information. For different time steps, compressed acoustic spectral features of different lengths are transmitted to achieve variable bit rate speech semantic encoding.

[0057] In addition, the embodiments of the present invention can regard the voiceprint features as the inherent features of the signal, extract the voiceprint features of any long speech signal as a voiceprint feature vector of fixed length, and use it as a global constraint to guide the recovery and synthesis of speech in the decoding stage; compress the spectral features and use them as a local constraint to further guide the recovery and synthesis of speech in the decoding stage, so that the recovered speech has higher quality.

[0058] Optionally, in one embodiment of the present invention, the linguistic discrete semantic representation, voiceprint features and compressed spectral features are respectively reconstructed to obtain a target audio acoustic intermediate representation, including: reconstructing the linguistic discrete semantic representation to obtain a semantic embedding; reconstructing the voiceprint features to obtain a voiceprint embedding; reconstructing the compressed spectral features to obtain a spectral embedding; additively aggregating the semantic embedding, voiceprint embedding, spectral embedding and the Mel spectrum of the target time step to obtain an aggregated embedding vector; converting the aggregated embedding vector into a Mel spectrum of the current time step; and concatenating the Mel spectrum of the target time step and the Mel spectrum of the current time step to obtain the target audio acoustic intermediate representation.

[0059] For example, Figure 2 As shown, the embodiment of the present invention can convert linguistic discrete semantic representation into semantic embedding through a specific embedding layer, can convert voiceprint features, that is, voiceprint feature vectors, into voiceprint embedding through a specific nonlinear mapping layer, and can convert compressed spectrum coding into spectrum embedding through a specific nonlinear mapping layer. Therefore, the embodiment of the present invention can perform additive aggregation on semantic embedding, voiceprint embedding, spectrum embedding and Mel spectrum of the previous time step to obtain the aggregated embedding vector, and based on the specific residual long short-term memory layer, convert the aggregated embedding vector into the Mel spectrum of the current time step, and then concatenate the Mel spectrum of all time steps to obtain the target audio acoustic intermediate representation, thereby effectively improving the efficiency of speech recognition.

[0060] For example, Figure 2As shown, the embodiment of the present invention can input the linguistic discrete semantic representation, voiceprint features and compressed spectrum feature encoding of the original audio through the acoustic synthesis module in the decoder to output the acoustic spectrum of the target audio. The vocoder in the decoder can use the HiFi-GAN algorithm to convert the input Mel spectrum into an audio waveform of a speech with the characteristics of the target speaker to output the speech with the timbre of the target speaker. Therefore, the embodiment of the present invention can encode the original audio using the linguistic discrete speech representation, effectively remove the speaker features in the original audio, thereby removing acoustic redundancy and improving the accuracy of original speech recovery.

[0061] The choice of the vocoder depends on the actual application scenario. HiFi-GAN can be used as the vocoder, or WaveNet2 or other models can be used as the vocoder, and no specific limitation is made here.

[0062] For example, the de-redundant speech semantic coding algorithm in the embodiment of the present invention may be composed of an encoder and a decoder, wherein the encoder includes a semantic extraction module and a voiceprint feature extraction module, and the decoder includes an acoustic synthesis module and a vocoder. Figure 3 As shown in the figure, the semantic extraction module can input the speech content that needs to be retained and remove the original speech waveform of the voiceprint features. The convolution layer is used to convert the PCM (Pulse Code Modulation) code of the audio into a high-dimensional vector, and the random mask and position information encoding layer are used for additive embedding. The token is input into the BERT (Bidirectional Encoder Representations from Transformers) model. The output of BERT passes through the linear mapping layer and the clustering quantization layer to become a linguistic discrete speech representation, which is input into the acoustic synthesis module.

[0063] Then, if Figure 4 As shown, the voiceprint feature extraction module can input the original audio waveform, use the Mel filter group to obtain the frequency domain representation of the original audio, extract the implicit vector through the first spectrum convolution and the second spectrum convolution, input it into the long short-term memory layer to extract the voiceprint features, and use a linear mapping layer to input it into the acoustic synthesis module. In addition, the nonlinear layer can also be used to extract the compressed acoustic spectrum features to obtain a maximum of 8-dimensional acoustic feature vector sequence.

[0064] Secondly, if Figure 5As shown, the acoustic synthesis module can receive the features of the semantic extraction module through the embedding layer, the nonlinear layer and the convolution upsampling layer as input, the voiceprint features and the compressed acoustic spectrum features of the voiceprint feature extraction module as input, and receive the earlier output of the acoustic synthesis module through the nonlinear transformation layer as the input of the current time step, and concatenate and embed the spectrum features with the voiceprint semantics after addition and embedding, and then pass through the residual long short-term memory layer, and output the Mel spectrum of the current time step through the output mapping layer.

[0065] Again, if Figure 6 As shown, the vocoder can receive the Mel spectrum of the acoustic synthesis module, pass through the convolution layer, a series of convolution upsampling layers and multi-receptive field fusion modules, and finally pass through the output mapping layer to output the waveform of the target audio.

[0066] Among them, the above-mentioned model in the embodiment of the present invention can adopt Hubert with pre-trained parameters as the semantic extraction module, or can autonomously train the parameters of the semantic extraction module based on a given data set; and the training of the voiceprint feature extraction module and the acoustic synthesis module is joint training, using the trained semantic extraction module on a given data set to obtain input, and using the L2 norm as the output target loss function.

[0067] Therefore, the embodiment of the present invention can achieve a compression ratio of 2% for the extracted semantic coding and a subjective MUSHRA score of 88 based on the de-redundancy and high sound quality characteristics of the variable bit rate de-redundant speech semantic coding.

[0068] According to the variable bit rate de-redundant speech semantic coding method proposed in the embodiment of the present invention, the linguistic discrete semantic representation, voiceprint features and compressed spectrum features obtained from the original audio can be reconstructed based on the deep neural network model to obtain the target audio acoustic intermediate representation, and the target audio to be output is obtained by using the target audio acoustic intermediate representation to de-redundant speech semantic coding of the original audio, which effectively improves the quality and efficiency of speech recognition and improves the accuracy of speech recovery. Thus, the problem that the sound synthesis model in the related art is a pre-trained model with fixed parameters, which makes it difficult to change the speaker features in the restored speech according to different speakers, resulting in distortion of part of the speech information, reducing the quality and efficiency of speech recognition, and reducing the accuracy of speech recovery is solved.

[0069] Next, the variable bit rate redundant speech semantic coding device proposed in accordance with the embodiment of the present invention will be described with reference to the accompanying drawings.

[0070] Figure 7 It is a block diagram of a variable bit rate redundant speech semantic coding device according to an embodiment of the present invention.

[0071] like Figure 7As shown, the variable bit rate redundant speech semantic coding device 10 includes: an acquisition module 100, a determination module 200 and a processing module 300.

[0072] Specifically, the acquisition module 100 is used to acquire the original audio of the target speaker based on a preset deep neural network model.

[0073] The determination module 200 is used to extract the linguistic discrete semantic representation in the original audio and obtain the voiceprint features and compressed spectrum features of the original audio.

[0074] The processing module 300 is used to reconstruct the linguistic discrete semantic representation, voiceprint features and compressed spectrum features respectively to obtain the target audio acoustic intermediate representation, and use the target audio acoustic intermediate representation to obtain the target audio to be output, so as to perform redundancy speech semantic encoding on the original audio.

[0075] Optionally, in one embodiment of the present invention, the determination module includes: a first acquisition unit and a first extraction unit.

[0076] The first acquisition unit is used to acquire a feature vector of a speech signal corresponding to the original audio.

[0077] The first extraction unit is used to obtain the linguistic code by using the feature vector to extract the linguistic discrete semantic representation corresponding to the linguistic code.

[0078] Optionally, in one embodiment of the present invention, the speech signal is represented as:

[0079]

[0080] Among them, h(t) is the speech signal, G w is the lip, tooth and tongue filter bank, f(t) is the basic waveform, G r is the vocal cord and resonance filter bank, n r is the number of vocal cords and resonance filters, n w is the number of lip, tooth and tongue filters, i is the lip, tooth and tongue filter number, and j is the vocal cord and resonance filter number.

[0081] Optionally, in one embodiment of the present invention, the determination module includes: a conversion unit, a second extraction unit, a second acquisition unit and a first determination unit.

[0082] The conversion unit is used to convert the speech waveform corresponding to the original audio into a Mel spectrum representing the acoustic features;

[0083] The second extraction unit is used to extract the acoustic information potential code corresponding to the Mel spectrum.

[0084] The second acquisition unit is used to obtain the voiceprint feature code by using the acoustic information latent code, so as to obtain the voiceprint feature by using the voiceprint feature code.

[0085] The first determining unit is used to obtain corresponding compressed spectrum coding based on the acoustic information potential coding, so as to obtain compressed spectrum features by using the compressed spectrum coding.

[0086] Optionally, in one embodiment of the present invention, the processing module includes: a first processing unit, a second processing unit, a third processing unit, a second determining unit, a fourth processing unit and a third acquiring unit.

[0087] The first processing unit is used to reconstruct the linguistic discrete semantic representation to obtain semantic embedding.

[0088] The second processing unit is used to reconstruct the voiceprint feature to obtain the voiceprint embedding.

[0089] The third processing unit is used to reconstruct the compressed spectrum features to obtain spectrum embedding.

[0090] The second determination unit is used to perform additive aggregation using the semantic embedding, the voiceprint embedding, the spectrum embedding and the Mel spectrum of the target time step to obtain an aggregated embedding vector.

[0091] The fourth processing unit is used to convert the aggregated embedding vector into a Mel spectrum of the current time step.

[0092] The third acquisition unit concatenates the Mel spectrum of the target time step and the Mel spectrum of the current time step to obtain an intermediate acoustic representation of the target audio.

[0093] It should be noted that the aforementioned explanation of the variable bit rate de-redundant speech semantic coding method embodiment is also applicable to the variable bit rate de-redundant speech semantic coding device of this embodiment, and will not be repeated here.

[0094] According to the variable bit rate de-redundant speech semantic coding device proposed in the embodiment of the present invention, the linguistic discrete semantic representation, voiceprint features and compressed spectrum features obtained from the original audio can be reconstructed based on the deep neural network model to obtain the target audio acoustic intermediate representation, and the target audio to be output is obtained by using the target audio acoustic intermediate representation to de-redundant speech semantic coding of the original audio, which effectively improves the quality and efficiency of speech recognition and improves the accuracy of speech recovery. Thus, the problem that the sound synthesis model in the related art is a pre-trained model with fixed parameters, which makes it difficult to change the speaker features in the restored speech according to different speakers, resulting in distortion of part of the speech information, reducing the quality and efficiency of speech recognition, and reducing the accuracy of speech recovery is solved.

[0095] Figure 8 A schematic diagram of the structure of an electronic device provided by an embodiment of the present invention. The electronic device may include:

[0096] A memory 801 , a processor 802 , and a computer program stored in the memory 801 and executable on the processor 802 .

[0097] When the processor 802 executes the program, the variable bit rate de-redundant speech semantic coding method provided in the above embodiment is implemented.

[0098] Furthermore, the electronic device further comprises:

[0099] The communication interface 803 is used for communication between the memory 801 and the processor 802 .

[0100] The memory 801 is used to store computer programs that can be executed on the processor 802 .

[0101] The memory 801 may include a high-speed RAM memory, and may also include a non-volatile memory (non-volatile memory), such as at least one disk memory.

[0102] If the memory 801, the processor 802 and the communication interface 803 are implemented independently, the communication interface 803, the memory 801 and the processor 802 can be connected to each other through a bus and communicate with each other. The bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus. The bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 8 Only one thick line is used in the diagram, but this does not mean that there is only one bus or only one type of bus.

[0103] Optionally, in a specific implementation, if the memory 801, the processor 802 and the communication interface 803 are integrated on a chip, the memory 801, the processor 802 and the communication interface 803 can communicate with each other through an internal interface.

[0104] The processor 802 may be a central processing unit (CPU), or an application specific integrated circuit (ASIC), or one or more integrated circuits configured to implement the embodiments of the present invention.

[0105] This embodiment also provides a computer-readable storage medium on which a computer program is stored. When the program is executed by a processor, the above variable bit rate de-redundant speech semantic coding method is implemented.

[0106] This embodiment also provides a computer program product, including a computer program. When the computer program is executed, it is used to implement the above variable bit rate de-redundant speech semantic coding method.

[0107] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" etc. means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or N embodiments or examples in a suitable manner. In addition, those skilled in the art may combine and combine the different embodiments or examples described in this specification and the features of the different embodiments or examples, without contradiction.

[0108] In addition, the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Therefore, the features defined as "first" and "second" may explicitly or implicitly include at least one of the features. In the description of the present invention, the meaning of "N" is at least two, such as two, three, etc., unless otherwise clearly and specifically defined.

[0109] Any process or method description in a flowchart or otherwise described herein may be understood to represent a module, fragment or portion of code comprising one or N executable instructions for implementing the steps of a custom logical function or process, and the scope of the preferred embodiments of the present invention includes alternative implementations in which functions may not be performed in the order shown or discussed, including performing functions in a substantially simultaneous manner or in reverse order depending on the functions involved, which should be understood by technicians in the technical field to which the embodiments of the present invention belong.

[0110] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as an ordered list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by an instruction execution system, device or apparatus (such as a computer-based system, a system including a processor, or other system that can fetch instructions from an instruction execution system, device or apparatus and execute instructions), or in combination with these instruction execution systems, devices or apparatuses. For the purpose of this specification, "computer-readable medium" can be any device that can contain, store, communicate, propagate or transmit a program for use by an instruction execution system, device or apparatus, or in combination with these instruction execution systems, devices or apparatuses. More specific examples of computer-readable media (a non-exhaustive list) include the following: an electrical connection with one or N wirings (electronic devices), a portable computer disk box (magnetic device), a random access memory (RAM), a read-only memory (ROM), an erasable and programmable read-only memory (EPROM or flash memory), a fiber optic device, and a portable compact disk read-only memory (CDROM). In addition, the computer-readable medium may even be paper or other suitable medium on which the program is printed, since the program may be obtained electronically by optically scanning the paper or other medium and then editing, interpreting or processing in other suitable ways as necessary and then storing it in a computer memory.

[0111] It should be understood that the various parts of the present invention can be implemented by hardware, software, firmware or a combination thereof. In the above embodiment, the N steps or methods can be implemented by software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented by hardware, as in another embodiment, it can be implemented by any one or a combination of the following technologies known in the art: a discrete logic circuit having a logic gate circuit for implementing a logic function for a data signal, a dedicated integrated circuit having a suitable combination of logic gate circuits, a programmable gate array (PGA), a field programmable gate array (FPGA), etc.

[0112] A person skilled in the art may understand that all or part of the steps in the method for implementing the above-mentioned embodiment may be completed by instructing related hardware through a program, and the program may be stored in a computer-readable storage medium, which, when executed, includes one or a combination of the steps of the method embodiment.

[0113] In addition, each functional unit in each embodiment of the present invention may be integrated into a processing module, or each unit may exist physically separately, or two or more units may be integrated into one module. The above-mentioned integrated module may be implemented in the form of hardware or in the form of a software functional module. If the integrated module is implemented in the form of a software functional module and sold or used as an independent product, it may also be stored in a computer-readable storage medium.

[0114] The storage medium mentioned above may be a read-only memory, a magnetic disk or an optical disk, etc. Although the embodiments of the present invention have been shown and described above, it can be understood that the above embodiments are exemplary and cannot be understood as limiting the present invention. A person of ordinary skill in the art may change, modify, replace and modify the above embodiments within the scope of the present invention.

Claims

1. A variable bit rate de-redundant speech semantic coding method, characterized in that: The following steps are involved: Based on the preset deep neural network model, the original audio of the target speaker is obtained; Extracting linguistic discrete semantic representations from the original audio, and obtaining voiceprint features and compressed spectrum features of the original audio; The linguistic discrete semantic representation, the voiceprint feature and the compressed spectral feature are respectively reconstructed to obtain a target audio acoustic intermediate representation, and the target audio to be output is obtained using the target audio acoustic intermediate representation to perform de-redundant speech semantic encoding on the original audio.

2. The variable bit rate de-redundant speech semantic coding method according to claim 1, characterized in that: The step of extracting the linguistic discrete semantic representation of the original audio comprises: Obtaining a feature vector of a speech signal corresponding to the original audio; The linguistic code is obtained by using the feature vector to extract the linguistic discrete semantic representation corresponding to the linguistic code.

3. The variable bit rate de-redundant speech semantic coding method according to claim 2, characterized in that: The speech signal is expressed as: Among them, h(t) is the speech signal, G w is the lip, tooth and tongue filter bank, f(t) is the basic waveform, G r is the vocal cord and resonance filter bank, n r is the number of vocal cords and resonance filters, n w is the number of lip, tooth and tongue filters, i is the lip, tooth and tongue filter number, and j is the vocal cord and resonance filter number.

4. The variable bit rate de-redundant speech semantic coding method according to claim 1, characterized in that: The obtaining of the voiceprint features and compressed spectrum features of the original audio includes: Converting the speech waveform corresponding to the original audio into a Mel spectrum representing acoustic features; Extracting the acoustic information latent coding corresponding to the Mel spectrum; Obtaining a voiceprint feature code using the acoustic information latent code, so as to obtain the voiceprint feature using the voiceprint feature code; Based on the acoustic information latent coding, a corresponding compressed spectrum coding is obtained, so as to obtain the compressed spectrum feature by using the compressed spectrum coding.

5. The variable bit rate de-redundant speech semantic coding method according to claim 1, characterized in that: The step of reconstructing the linguistic discrete semantic representation, the voiceprint feature, and the compressed spectrum feature to obtain a target audio acoustic intermediate representation includes: Reconstructing the linguistic discrete semantic representation to obtain semantic embedding; Reconstructing the voiceprint feature to obtain voiceprint embedding; Reconstructing the compressed spectrum feature to obtain spectrum embedding; Performing additive aggregation using the semantic embedding, the voiceprint embedding, the spectrum embedding, and the Mel spectrum of the target time step to obtain an aggregated embedding vector; Convert the aggregated embedding vector to a Mel frequency spectrum of the current time step; The Mel spectrum of the target time step and the Mel spectrum of the current time step are concatenated to obtain the target audio acoustic intermediate representation.

6. A variable bit rate de-redundant speech semantic coding device, characterized in that: include: An acquisition module, used to acquire the original audio of the target speaker based on a preset deep neural network model; A determination module, used to extract linguistic discrete semantic representations from the original audio, and obtain voiceprint features and compressed spectrum features of the original audio; A processing module is used to reconstruct the linguistic discrete semantic representation, the voiceprint feature and the compressed spectral feature respectively to obtain a target audio acoustic intermediate representation, and use the target audio acoustic intermediate representation to obtain the target audio to be output, so as to perform de-redundant speech semantic encoding on the original audio.

7. The variable bit rate de-redundant speech semantic coding device according to claim 6, characterized in that: The determination module comprises: A first acquisition unit, used to acquire a feature vector of a speech signal corresponding to the original audio; The first extraction unit is used to obtain the linguistic code by using the feature vector to extract the linguistic discrete semantic representation corresponding to the linguistic code.

8. An electronic device, characterized in that: include: A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the variable bit rate de-redundant speech semantic coding method as described in any one of claims 1 to 5.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: The program is executed by a processor to implement the variable bit rate de-redundant speech semantic coding method as described in any one of claims 1-5.

10. A computer program product, comprising a computer program, characterized in that The computer program is executed by a processor to implement the variable bit rate de-redundant speech semantic coding method as described in any one of claims 1-5.

Images