Variable bit rate de-redundant speech semantic coding method and device

Abstract

The present invention relates to the field of speech coding technology, and more particularly to a variable bit rate de-redundant speech semantic encoding method and apparatus. The method comprises: obtaining original audio of a target speaker based on a preset deep neural network model; extracting linguistic discrete semantic representations from the original audio, and obtaining voiceprint features and compressed spectrum features of the original audio; reconstructing the linguistic discrete semantic representations, voiceprint features, and compressed spectrum features to obtain an acoustic intermediate representation of the target audio, and using the acoustic intermediate representation of the target audio to obtain the target audio to be output, thereby de-redundanting speech semantic encoding of the original audio. This solves the problem in related arts that the sound synthesis model is a pre-trained model with fixed parameters, making it difficult to modify the speaker features in the restored speech according to different speakers, resulting in partial distortion of speech information, reduced speech recognition quality and efficiency, and reduced accuracy of speech recovery.

Description

Technical Field

[0001] The present invention relates to the field of speech coding technology, and in particular to a variable bit rate de-redundant speech semantic coding method and device. Background Art

[0002] In related technologies, most speech semantic coding algorithms based on DNN (Deep Neural Network) perform nonlinear transformation on the speech waveform through the convolution layer and then perform vector quantization. The purpose of this type of method is to separate the irrelevant components in the speech signal through a pure deep neural network method. Among them, the deep neural network requires a certain training scale to perform speech semantic coding based on the deep neural network.

[0003] However, the degree of de-redundancy of the speech semantic coding obtained in the related technology is limited by the model size, and the existing sound synthesis model is a pre-trained model with fixed parameters. It is difficult to change and restore the speaker characteristics in the speech according to different speakers, resulting in distortion of some speech information, reducing the quality and efficiency of speech recognition, and reducing the accuracy of speech recovery, which needs to be solved urgently. Summary of the Invention

[0004] The present invention provides a variable bit rate de-redundant speech semantic coding method and device to solve the problem that the degree of de-redundancy of speech semantic coding obtained in the related art is limited by the model scale, and the existing sound synthesis model is a pre-trained model with fixed parameters, which makes it difficult to change the speaker characteristics in the restored speech according to different speakers, resulting in distortion of part of the speech information, reducing the quality and efficiency of speech recognition, and reducing the accuracy of speech recovery.

[0005] The first aspect of the present invention provides a variable bit rate de-redundant speech semantic coding method, comprising the following steps: obtaining the original audio of a target speaker based on a preset deep neural network model; extracting the linguistic discrete semantic representation in the original audio, and obtaining the voiceprint features and compressed spectrum features of the original audio; reconstructing the linguistic discrete semantic representation, the voiceprint features, and the compressed spectrum features respectively to obtain an acoustic intermediate representation of the target audio, and using the acoustic intermediate representation of the target audio to obtain the target audio to be output, so as to perform de-redundant speech semantic coding on the original audio.

[0006] Optionally, in one embodiment of the present invention, the extracting the linguistic discrete semantic representation of the original audio includes: obtaining a feature vector of a speech signal corresponding to the original audio; and using the feature vector to obtain the linguistic code to extract the linguistic discrete semantic representation corresponding to the linguistic code.

[0007] Optionally, in one embodiment of the present invention, the speech signal is represented as:

[0008]

[0009] Among them, h(t) is the speech signal, G w is the lip, tooth and tongue filter bank, f(t) is the basic waveform, G r is the vocal cord and resonance filter bank, n r is the number of vocal cords and resonance filters, n w is the number of lip, tooth and tongue filters, i is the lip, tooth and tongue filter number, and j is the vocal cord and resonance filter number.

[0010] Optionally, in one embodiment of the present invention, obtaining the voiceprint features and compressed spectrum features of the original audio includes: converting the speech waveform corresponding to the original audio into a Mel spectrum representing acoustic features; extracting the acoustic information latent code corresponding to the Mel spectrum; using the acoustic information latent code to obtain the voiceprint feature code, and using the voiceprint feature code to obtain the voiceprint feature; based on the acoustic information latent code, obtaining the corresponding compressed spectrum code, and using the compressed spectrum code to obtain the compressed spectrum feature.

[0011] Optionally, in one embodiment of the present invention, the linguistic discrete semantic representation, the voiceprint feature and the compressed spectral feature are respectively reconstructed to obtain a target audio acoustic intermediate representation, including: reconstructing the linguistic discrete semantic representation to obtain a semantic embedding; reconstructing the voiceprint feature to obtain a voiceprint embedding; reconstructing the compressed spectral feature to obtain a spectral embedding; additively aggregating the semantic embedding, the voiceprint embedding, the spectral embedding and the Mel spectrum of the target time step to obtain an aggregated embedding vector; converting the aggregated embedding vector into a Mel spectrum of the current time step; and splicing the Mel spectrum of the target time step and the Mel spectrum of the current time step to obtain the target audio acoustic intermediate representation.

[0012] The second aspect of the present invention provides a variable bit rate de-redundant speech semantic coding device, including: an acquisition module for acquiring the original audio of a target speaker based on a preset deep neural network model; a determination module for extracting the linguistic discrete semantic representation in the original audio and obtaining the voiceprint features and compressed spectrum features of the original audio; a processing module for reconstructing the linguistic discrete semantic representation, the voiceprint features and the compressed spectrum features respectively to obtain an acoustic intermediate representation of the target audio, and using the acoustic intermediate representation of the target audio to obtain the target audio to be output, so as to perform de-redundant speech semantic coding on the original audio.

[0013] Optionally, in one embodiment of the present invention, the determination module includes: a first acquisition unit, used to obtain a feature vector of a speech signal corresponding to the original audio; and a first extraction unit, used to obtain the linguistic code using the feature vector to extract the linguistic discrete semantic representation corresponding to the linguistic code.

[0014] Optionally, in one embodiment of the present invention, the speech signal is represented as:

[0015]

[0016] Among them, h(t) is the speech signal, G w is the lip, tooth and tongue filter bank, f(t) is the basic waveform, G r is the vocal cord and resonance filter bank, n r is the number of vocal cords and resonance filters, n w is the number of lip, tooth and tongue filters, i is the lip, tooth and tongue filter number, and j is the vocal cord and resonance filter number.

[0017] Optionally, in one embodiment of the present invention, the determination module includes: a conversion unit, used to convert the speech waveform corresponding to the original audio into a Mel spectrum representing the acoustic feature; a second extraction unit, used to extract the acoustic information latent code corresponding to the Mel spectrum; a second acquisition unit, used to obtain the voiceprint feature code using the acoustic information latent code, so as to obtain the voiceprint feature using the voiceprint feature code; a first determination unit, used to obtain the corresponding compressed spectrum code based on the acoustic information latent code, so as to obtain the compressed spectrum feature using the compressed spectrum code.

[0018] Optionally, in one embodiment of the present invention, the processing module includes: a first processing unit for reconstructing the linguistic discrete semantic representation to obtain a semantic embedding; a second processing unit for reconstructing the voiceprint feature to obtain a voiceprint embedding; a third processing unit for reconstructing the compressed spectral feature to obtain a spectral embedding; a second determination unit for performing additive aggregation using the semantic embedding, the voiceprint embedding, the spectral embedding and the Mel spectrum of the target time step to obtain an aggregated embedding vector; a fourth processing unit for converting the aggregated embedding vector into a Mel spectrum of the current time step; a third acquisition unit for splicing the Mel spectrum of the target time step and the Mel spectrum of the current time step to obtain the target audio acoustic intermediate representation.

[0019] The third aspect of the present invention provides an electronic device, comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the variable bit rate de-redundant speech semantic coding method as described in the above embodiment.

[0020] A fourth aspect of the present invention provides a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the above variable bit rate de-redundant speech semantic coding method.

[0021] A fifth aspect of the present invention provides a computer program product, comprising a computer program, which, when executed, is used to implement the above variable bit rate de-redundant speech semantic coding method.

[0022] The embodiment of the present invention can reconstruct the linguistic discrete semantic representation, voiceprint features, and compressed spectrum features obtained from the original audio based on a deep neural network model to obtain an acoustic intermediate representation of the target audio, and use the acoustic intermediate representation of the target audio to obtain the target audio to be output, so as to perform redundancy speech semantic encoding on the original audio, effectively improving the quality and efficiency of speech recognition, and improving the accuracy of speech recovery. This solves the problem that the degree of redundancy of speech semantic encoding obtained in related technologies is limited by the scale of the model, and that the existing sound synthesis model is a pre-trained model with fixed parameters, which makes it difficult to change the speaker features in the restored speech according to different speakers, resulting in distortion of some speech information, reducing the quality and efficiency of speech recognition, and reducing the accuracy of speech recovery.

[0023] Additional aspects and advantages of the present invention will be set forth in part in the description which follows and, in part, will be obvious from the description which follows, or may be learned through practice of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] The above and / or additional aspects and advantages of the present invention will become apparent and readily understood from the following description of the embodiments with reference to the accompanying drawings, in which:

[0025] Figure 1 A flowchart of a variable bit rate de-redundant speech semantic coding method provided according to an embodiment of the present invention;

[0026] Figure 2 A schematic diagram of a de-redundant speech semantic coding system according to a specific embodiment of the present invention;

[0027] Figure 3 This is a schematic diagram of the structure of a semantic extraction module according to a specific embodiment of the present invention;

[0028] Figure 4This is a structural diagram of a voiceprint feature extraction module according to a specific embodiment of the present invention;

[0029] Figure 5 This is a schematic structural diagram of an acoustic synthesis module according to a specific embodiment of the present invention;

[0030] Figure 6 A schematic structural diagram of a vocoder according to a specific embodiment of the present invention;

[0031] Figure 7 A schematic structural diagram of a variable bit rate de-redundant speech semantic coding device according to an embodiment of the present invention;

[0032] Figure 8 A schematic structural diagram of an electronic device provided according to an embodiment of the present invention. DETAILED DESCRIPTION

[0033] The following describes embodiments of the present invention in detail, examples of which are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are intended to be used to explain the present invention, and are not to be construed as limiting the present invention.

[0034] The following describes a variable bit rate de-redundant speech semantic coding method and apparatus according to an embodiment of the present invention with reference to the accompanying drawings. In view of the problem that the degree of de-redundancy of speech semantic coding obtained in the related art mentioned in the above background technology is limited by the scale of the model, and the existing sound synthesis model is a pre-trained model with fixed parameters, which makes it difficult to change the speaker characteristics in the restored speech according to different speakers, resulting in distortion of part of the speech information, reducing the quality and efficiency of speech recognition, and reducing the accuracy of speech recovery, the present invention provides a variable bit rate de-redundant speech semantic coding method, in which the linguistic discrete semantic representation, voiceprint features and compressed spectrum features obtained from the original audio can be reconstructed based on a deep neural network model to obtain an acoustic intermediate representation of the target audio, and the target audio to be output is obtained by using the acoustic intermediate representation of the target audio to perform de-redundant speech semantic coding on the original audio, thereby effectively improving the quality and efficiency of speech recognition, and improving the accuracy of speech recovery. This solves the problem that the degree of de-redundancy of speech semantic coding obtained in related technologies is limited by the model size, and the existing sound synthesis model is a pre-trained model with fixed parameters, which makes it difficult to change and restore speaker features in the speech according to different speakers, resulting in distortion of some speech information, reducing the quality and efficiency of speech recognition, and reducing the accuracy of speech recovery.

[0035] Specifically, Figure 1A flow chart of a variable bit rate de-redundant speech semantic coding method provided by an embodiment of the present invention.

[0036] like Figure 1 As shown, the variable bit rate de-redundant speech semantic coding method includes the following steps:

[0037] In step S101, the original audio of the target speaker is obtained based on a preset deep neural network model.

[0038] In an embodiment of the present invention, the preset deep neural network model is a modular and multi-step deep neural network model, that is, a variable bit rate de-redundant speech semantic coding algorithm model, wherein the de-redundant speech semantic coding algorithm is composed of an encoder and a decoder, the encoder includes a semantic extraction module and a voiceprint feature extraction module, and the decoder includes an acoustic synthesis module and a vocoder. This model will be used as an example for detailed explanation below.

[0039] It can be understood that the embodiment of the present invention can obtain the original audio of the target speaker, that is, the original audio to be converted, through the above-mentioned model. The embodiment of the present invention can perform de-redundant speech semantic encoding on the original audio to be converted through a deep neural network model, effectively improving the executability of de-redundant speech semantic encoding.

[0040] In step S102, the linguistic discrete semantic representation in the original audio is extracted, and the voiceprint features and compressed spectrum features of the original audio are obtained.

[0041] It can be understood that the embodiment of the present invention can extract the linguistic discrete semantic representation in the original audio in the following steps. For example, the embodiment of the present invention can process the original audio through the semantic extraction module in the above-mentioned encoder, that is, the semantic extraction module inputs the original audio to be converted, outputs the linguistic discrete semantic representation contained in the original audio, and obtains the voiceprint features and compressed spectrum features of the original audio in the following steps. For example, the embodiment of the present invention can use the voiceprint feature extraction module in the above-mentioned encoder to input the acoustic spectrum of the original audio, output the voiceprint features of the original audio speaker and the compressed spectrum feature encoding of the original audio, which effectively improves the accuracy of de-redundant speech semantic encoding.

[0042] For example, Figure 2 As shown in FIG, the semantic extraction module in the encoder is a self-attention pre-trained linguistics large model with a vector quantization layer; the semantic extraction module can use the pre-trained Hubert algorithm and, after vector quantization, discretize the linguistic discrete semantic representation into an integer sequence of 1 to 100.

[0043] For example, Figure 2As shown, the voiceprint feature extraction module can be a deep neural network model. The voiceprint feature extraction module can use a Mel filter to convert the speech waveform from the time domain to the frequency domain, and then input the Mel spectrum obtained in the following steps into a three-layer convolutional neural network and a long short-term memory neural network. The output of the last step is taken and converted into a fixed-length voiceprint feature vector through a linear layer. In addition, a nonlinear transformation layer is added after the convolution to convert the spectrum features into a low-dimensional compressed spectrum feature vector.

[0044] In one embodiment of the present invention, extracting the linguistic discrete semantic representation of the original audio includes: obtaining the feature vector of the speech signal corresponding to the original audio; and using the feature vector to obtain the linguistic code to extract the linguistic discrete semantic representation corresponding to the linguistic code.

[0045] During the actual implementation process, the embodiment of the present application can obtain the feature vector of the speech signal corresponding to the original audio based on a specific convolutional layer, and then, based on a specific self-attention layer, obtain the linguistic code corresponding to the feature vector. Thirdly, based on a specific vector quantization layer, obtain the linguistic discrete semantic representation corresponding to the linguistic code, thereby effectively improving the efficiency of de-redundant speech semantic coding.

[0046] For example, embodiments of the present invention can be based on the acoustic principle of human speech, that is, the fundamental tone of the vocal cords is formed by the vibration of the airflow through the vocal cords, and then the sound resonates in natural resonators such as the mouth and nasal cavity to obtain overtones. Using the filter as an analogy, the sound produced by a person can be regarded as a basic waveform f(t) that first passes through a vocal cord and resonance filter group G that determines the speaker's characteristics. r , and then through a series of lips, teeth and tongue filter groups G that change dynamically according to the content of the speech w , and the final speech signal h(t) is obtained.

[0047] Among them, the speech signal is expressed as:

[0048]

[0049] Among them, h(t) is the speech signal, G w is the lip, tooth and tongue filter bank, f(t) is the basic waveform, G r is the vocal cord and resonance filter bank, n r is the number of vocal cords and resonance filters, n w is the number of lip, tooth and tongue filters, i is the lip, tooth and tongue filter number, and j is the vocal cord and resonance filter number.

[0050] Since the acoustic features in the embodiment of the present invention are largely repeated in the speech signal, redundant features that need to be removed are input to improve the efficiency of de-redundant speech semantic coding.

[0051] Among them, in one embodiment of the present invention, obtaining the voiceprint features and compressed spectrum features of the original audio includes: converting the speech waveform corresponding to the original audio into a Mel spectrum representing the acoustic features; extracting the acoustic information latent code corresponding to the Mel spectrum; using the acoustic information latent code to obtain the voiceprint feature code, and using the voiceprint feature code to obtain the voiceprint feature; based on the acoustic information latent code, obtaining the corresponding compressed spectrum code, and using the compressed spectrum code to obtain the compressed spectrum feature.

[0052] As a possible implementation method, an embodiment of the present invention can convert the speech waveform corresponding to the original audio into a Mel spectrum representing the acoustic features. Then, based on a specific convolutional layer, the acoustic information latent code corresponding to the Mel spectrum is obtained. Secondly, based on a specific long-short-term memory layer, the voiceprint feature code corresponding to the acoustic information latent code is obtained. Again, based on a specific linear mapping layer, the voiceprint feature vector corresponding to the voiceprint feature code is obtained. Finally, based on a specific nonlinear mapping layer, the corresponding compressed spectrum code is obtained from the acoustic information latent code, thereby obtaining compressed spectrum features using compressed spectrum coding, effectively improving the efficiency and quality of de-redundant speech semantic coding.

[0053] In step S103, the linguistic discrete semantic representation, voiceprint features and compressed spectral features are reconstructed respectively to obtain the target audio acoustic intermediate representation, and the target audio acoustic intermediate representation is used to obtain the target audio to be output, so as to perform de-redundant speech semantic encoding on the original audio.

[0054] It can be understood that the embodiments of the present invention can reconstruct and process the linguistic discrete semantic representation, voiceprint features and compressed spectral features respectively. For example, the embodiments of the present invention can process the linguistic discrete semantic representation, voiceprint features and compressed spectral features through the acoustic synthesis module in the decoder, so as to obtain the target audio acoustic intermediate representation corresponding to the linguistic discrete semantic representation, voiceprint features and compressed spectral features, and use the vocoder in the decoder to process the target audio acoustic intermediate representation to obtain the target audio to be output, and output the target audio, and then perform de-redundant speech semantic encoding on the original audio, thereby effectively improving the quality of speech recognition.

[0055] For example, the acoustic synthesis module can be a deep neural network autoregressive model. The acoustic synthesis module can use a nonlinear layer to match the Mel spectrum of the target speaker's timbre generated in the previous time step, the input linguistic discrete speech representation sequence, the input voiceprint feature vector and the length of the compressed spectrum feature vector, and add and aggregate the data, input it into a multi-layer residual long short-term memory neural network, and finally obtain a Mel spectrum with the characteristics of the target speaker.

[0056] Among them, the embodiment of the present invention can determine the amount of semantic information at the current time step based on the linguistic discrete semantic representation extracted by the semantic extraction module, and determine the length of the compressed acoustic spectrum features that should be used for actual transmission based on the amount of semantic information. For different time steps, compressed acoustic spectrum features of different lengths are transmitted to achieve variable bit rate speech semantic encoding.

[0057] In addition, the embodiments of the present invention can regard the voiceprint features as the inherent characteristics of the signal, extract the voiceprint features of any long speech signal as a fixed-length voiceprint feature vector, and use it as a global constraint to guide the recovery and synthesis of the speech in the decoding stage; compress the spectral features and use them as a local constraint to further guide the recovery and synthesis of the speech in the decoding stage, so that the recovered speech has higher quality.

[0058] Optionally, in one embodiment of the present invention, the linguistic discrete semantic representation, voiceprint features and compressed spectral features are respectively reconstructed to obtain a target audio acoustic intermediate representation, including: reconstructing the linguistic discrete semantic representation to obtain a semantic embedding; reconstructing the voiceprint features to obtain a voiceprint embedding; reconstructing the compressed spectral features to obtain a spectral embedding; additively aggregating the semantic embedding, voiceprint embedding, spectral embedding and the Mel spectrum of the target time step to obtain an aggregated embedding vector; converting the aggregated embedding vector into the Mel spectrum of the current time step; and concatenating the Mel spectrum of the target time step and the Mel spectrum of the current time step to obtain the target audio acoustic intermediate representation.

[0059] For example, Figure 2 As shown, the embodiment of the present invention can convert linguistic discrete semantic representation into semantic embedding through a specific embedding layer, can convert voiceprint features, that is, voiceprint feature vectors, into voiceprint embedding through a specific nonlinear mapping layer, and can convert compressed spectrum coding into spectrum embedding through a specific nonlinear mapping layer. Therefore, the embodiment of the present invention can additively aggregate the semantic embedding, voiceprint embedding, spectrum embedding and the Mel spectrum of the previous time step to obtain the aggregated embedding vector, and based on the specific residual long short-term memory layer, convert the aggregated embedding vector into the Mel spectrum of the current time step, and then splice the Mel spectrum of all time steps to obtain the target audio acoustic intermediate representation, thereby effectively improving the efficiency of speech recognition.

[0060] For example, Figure 2As shown, the embodiment of the present invention can input the linguistic discrete semantic representation, voiceprint features and compressed spectrum feature encoding of the original audio through the acoustic synthesis module in the decoder to output the acoustic spectrum of the target audio. The vocoder in the decoder can use the HiFi-GAN algorithm to convert the input Mel spectrum into an audio waveform of a speech with the characteristics of the target speaker to output a speech with the timbre of the target speaker. Therefore, the embodiment of the present invention can use the linguistic discrete speech representation to encode the original audio, effectively remove the speaker features in the original audio, thereby removing acoustic redundancy and improving the accuracy of original speech recovery.

[0061] The choice of vocoder depends on the actual application scenario. HiFi-GAN, WaveNet2 or other models can be used as the vocoder, and no specific limitation is made here.

[0062] For example, the de-redundant speech semantic coding algorithm in the embodiment of the present invention may be composed of an encoder and a decoder, wherein the encoder includes a semantic extraction module and a voiceprint feature extraction module, and the decoder includes an acoustic synthesis module and a vocoder. Figure 3 As shown in the figure, the semantic extraction module can input the speech content that needs to be retained and remove the original speech waveform with voiceprint features. The convolutional layer is used to convert the audio PCM (Pulse Code Modulation) code into a high-dimensional vector, which is then additively embedded through a random mask and position information encoding layer. The token is input into the BERT (Bidirectional Encoder Representations from Transformers) model. The output of BERT passes through a linear mapping layer and a cluster quantization layer to become a linguistic discrete speech representation, which is then input into the acoustic synthesis module.

[0063] Then, if Figure 4 As shown in the figure, the voiceprint feature extraction module can input the original audio waveform, use the Mel filter group to obtain the frequency domain representation of the original audio, extract the implicit vector through the first spectrum convolution and the second spectrum convolution, input it into the long short-term memory layer to extract the voiceprint features, and use a linear mapping layer to input it into the acoustic synthesis module. In addition, it can also extract compressed acoustic spectrum features through a nonlinear layer to obtain a maximum of 8-dimensional acoustic feature vector sequence.

[0064] Secondly, if Figure 5As shown in the figure, the acoustic synthesis module can receive the features of the semantic extraction module through the embedding layer, the nonlinear layer and the convolution upsampling layer as input, the voiceprint features and the compressed acoustic spectrum features of the voiceprint feature extraction module as input, and receive the earlier output of the acoustic synthesis module through the nonlinear transformation layer as the input of the current time step, and concatenate and embed the spectrum features with the voiceprint semantic features after the addition and embedding, and then pass through the residual long short-term memory layer and output the Mel spectrum of the current time step through the output mapping layer.

[0065] Again, as Figure 6 As shown in the figure, the vocoder can receive the Mel spectrum of the acoustic synthesis module, pass through the convolution layer, a series of convolution upsampling layers and multi-receptive field fusion modules, and finally pass through the output mapping layer to output the waveform of the target audio.

[0066] Among them, the above-mentioned model in the embodiment of the present invention can adopt Hubert with pre-trained parameters as the semantic extraction module, or can autonomously train the parameters of the semantic extraction module based on a given data set; and the training of the voiceprint feature extraction module and the acoustic synthesis module is joint training, using the trained semantic extraction module on the given data set to obtain input, and using the L2 norm as the output target loss function.

[0067] Therefore, the embodiment of the present invention can achieve a compression ratio of 2% for the extracted semantic coding and a subjective MUSHRA score of 88 based on the de-redundancy and high sound quality characteristics of the variable bit rate de-redundant speech semantic coding.

[0068] The variable bit rate de-redundant speech semantic coding method proposed in an embodiment of the present invention can reconstruct the linguistic discrete semantic representation, voiceprint features, and compressed spectral features obtained from the original audio based on a deep neural network model to obtain an intermediate acoustic representation of the target audio. The intermediate acoustic representation of the target audio is then used to obtain the target audio to be output, thereby de-redundanting speech semantic coding of the original audio. This effectively improves the quality and efficiency of speech recognition and the accuracy of speech recovery. This solves the problem in the related art that the sound synthesis model is a pre-trained model with fixed parameters, which makes it difficult to modify the speaker features in the recovered speech according to different speakers, resulting in distortion of some speech information, reduced quality and efficiency of speech recognition, and reduced accuracy of speech recovery.

[0069] Next, a variable bit rate de-redundant speech semantic coding apparatus according to an embodiment of the present invention will be described with reference to the accompanying drawings.

[0070] Figure 7 It is a block diagram of a variable bit rate redundant speech semantic coding apparatus according to an embodiment of the present invention.

[0071] like Figure 7As shown, the variable bit rate de-redundant speech semantic coding device 10 includes: an acquisition module 100 , a determination module 200 and a processing module 300 .

[0072] Specifically, the acquisition module 100 is used to obtain the original audio of the target speaker based on a preset deep neural network model.

[0073] The determination module 200 is used to extract the linguistic discrete semantic representation in the original audio and obtain the voiceprint features and compressed spectrum features of the original audio.

[0074] The processing module 300 is used to reconstruct the linguistic discrete semantic representation, voiceprint features and compressed spectral features respectively to obtain the target audio acoustic intermediate representation, and use the target audio acoustic intermediate representation to obtain the target audio to be output, so as to perform redundancy speech semantic encoding on the original audio.

[0075] Optionally, in one embodiment of the present invention, the determination module includes: a first acquisition unit and a first extraction unit.

[0076] The first acquisition unit is used to acquire the feature vector of the speech signal corresponding to the original audio.

[0077] The first extraction unit is configured to obtain a linguistic code using the feature vector to extract a linguistic discrete semantic representation corresponding to the linguistic code.

[0078] Optionally, in one embodiment of the present invention, the speech signal is represented as:

[0079]

[0080] Among them, h(t) is the speech signal, G w is the lip, tooth and tongue filter bank, f(t) is the basic waveform, G r is the vocal cord and resonance filter bank, n r is the number of vocal cords and resonance filters, n w is the number of lip, tooth and tongue filters, i is the lip, tooth and tongue filter number, and j is the vocal cord and resonance filter number.

[0081] Optionally, in one embodiment of the present invention, the determination module includes: a conversion unit, a second extraction unit, a second acquisition unit and a first determination unit.

[0082] The conversion unit is used to convert the speech waveform corresponding to the original audio into a Mel spectrum representing acoustic features;

[0083] The second extraction unit is used to extract the latent coding of acoustic information corresponding to the Mel spectrum.

[0084] The second acquiring unit is configured to acquire a voiceprint feature code by utilizing the acoustic information latent coding, so as to obtain a voiceprint feature by utilizing the voiceprint feature code.

[0085] The first determining unit is configured to obtain corresponding compressed spectrum coding based on the acoustic information latent coding, so as to obtain compressed spectrum features by using the compressed spectrum coding.

[0086] Optionally, in one embodiment of the present invention, the processing module includes: a first processing unit, a second processing unit, a third processing unit, a second determining unit, a fourth processing unit and a third acquiring unit.

[0087] The first processing unit is used to reconstruct the linguistic discrete semantic representation to obtain semantic embedding.

[0088] The second processing unit is used to reconstruct the voiceprint features to obtain voiceprint embedding.

[0089] The third processing unit is configured to reconstruct the compressed spectrum features to obtain spectrum embedding.

[0090] The second determination unit is used to perform additive aggregation using the semantic embedding, the voiceprint embedding, the spectrum embedding, and the Mel spectrum of the target time step to obtain an aggregated embedding vector.

[0091] The fourth processing unit is used to convert the aggregated embedding vector into the Mel spectrum of the current time step.

[0092] The third acquisition unit concatenates the Mel spectrum of the target time step and the Mel spectrum of the current time step to obtain an intermediate acoustic representation of the target audio.

[0093] It should be noted that the above explanation of the embodiment of the variable bit rate de-redundant speech semantic coding method is also applicable to the variable bit rate de-redundant speech semantic coding device of this embodiment, and will not be repeated here.

[0094] The variable bit rate de-redundant speech semantic coding device proposed in an embodiment of the present invention can reconstruct the linguistic discrete semantic representation, voiceprint features, and compressed spectrum features obtained from the original audio based on a deep neural network model to obtain an intermediate acoustic representation of the target audio. The target audio to be output is then obtained using the intermediate acoustic representation of the target audio to perform de-redundant speech semantic coding on the original audio, effectively improving the quality and efficiency of speech recognition and the accuracy of speech recovery. This solves the problem in the related art that the sound synthesis model is a pre-trained model with fixed parameters, which makes it difficult to modify the speaker features in the recovered speech according to different speakers, resulting in distortion of some speech information, reduced quality and efficiency of speech recognition, and reduced accuracy of speech recovery.

[0095] Figure 8 This is a schematic diagram of the structure of an electronic device provided by an embodiment of the present invention. The electronic device may include:

[0096] A memory 801 , a processor 802 , and a computer program stored in the memory 801 and executable on the processor 802 .

[0097] When the processor 802 executes the program, the variable bit rate de-redundant speech semantic coding method provided in the above embodiment is implemented.

[0098] Furthermore, the electronic device further includes:

[0099] The communication interface 803 is used for communication between the memory 801 and the processor 802 .

[0100] The memory 801 is used to store computer programs that can be run on the processor 802.

[0101] The memory 801 may include a high-speed RAM memory, and may also include a non-volatile memory (non-volatile memory), such as at least one disk memory.

[0102] If the memory 801, the processor 802, and the communication interface 803 are implemented independently, the communication interface 803, the memory 801, and the processor 802 can be connected to each other via a bus and communicate with each other. The bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus. The bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 8 Only one thick line is used in the diagram, but this does not mean that there is only one bus or one type of bus.

[0103] Optionally, in a specific implementation, if the memory 801, the processor 802 and the communication interface 803 are integrated on a chip, the memory 801, the processor 802 and the communication interface 803 can communicate with each other through an internal interface.

[0104] The processor 802 may be a central processing unit (CPU), an application specific integrated circuit (ASIC), or one or more integrated circuits configured to implement the embodiments of the present invention.

[0105] This embodiment further provides a computer-readable storage medium having a computer program stored thereon. When the program is executed by a processor, the above variable bit rate de-redundant speech semantic coding method is implemented.

[0106] This embodiment further provides a computer program product, including a computer program. When the computer program is executed, it is used to implement the above variable bit rate de-redundant speech semantic coding method.

[0107] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic expressions of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described can be combined in any one or N embodiments or examples in a suitable manner. In addition, those skilled in the art can combine and combine different embodiments or examples described in this specification and the features of different embodiments or examples without contradiction.

[0108] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be understood to indicate or imply relative importance or implicitly specify the number of technical features indicated. Thus, a feature specified as "first" or "second" may explicitly or implicitly include at least one such feature. In the description of the present invention, "N" means at least two, such as two, three, etc., unless otherwise specifically defined.

[0109] Any process or method description in a flowchart or otherwise described herein may be understood to represent a module, segment or portion of code comprising one or N executable instructions for implementing a custom logical function or step of a process, and the scope of the preferred embodiments of the present invention includes alternative implementations in which functions may be performed out of the order shown or discussed, including performing functions in a substantially simultaneous manner or in the reverse order depending on the functions involved, which should be understood by those skilled in the art to which the embodiments of the present invention pertain.

[0110] The logic and / or steps represented in the flowcharts or otherwise described herein, for example, can be considered as a sequenced list of executable instructions for implementing the logical functions, and can be embodied in any computer-readable medium for use by, or in conjunction with, an instruction execution system, apparatus, or device (e.g., a computer-based system, a system including a processor, or other system that can fetch and execute instructions from an instruction execution system, apparatus, or device). For purposes of this specification, a "computer-readable medium" can be any device that can contain, store, communicate, propagate, or transport a program for use by, or in conjunction with, an instruction execution system, apparatus, or device. More specific examples (a non-exhaustive list) of computer-readable media include the following: an electrical connection with one or N wires (electronic devices), a portable computer disk cartridge (magnetic device), random access memory (RAM), read-only memory (ROM), erasable and programmable read-only memory (EPROM or flash memory), fiber optic devices, and a portable compact disc read-only memory (CDROM). In addition, the computer-readable medium may even be paper or other suitable medium on which the program is printed, since the program can be obtained electronically by optically scanning the paper or other medium and then editing, interpreting or processing it in other suitable ways as necessary, and then storing it in a computer memory.

[0111] It should be understood that various parts of the present invention can be implemented using hardware, software, firmware, or a combination thereof. In the above embodiment, the N steps or methods can be implemented using software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented using hardware, as in another embodiment, it can be implemented using any one or a combination of the following technologies known in the art: a discrete logic circuit having logic gate circuits for implementing logical functions on data signals, an application-specific integrated circuit having suitable combinational logic gate circuits, a programmable gate array (PGA), a field programmable gate array (FPGA), etc.

[0112] Those skilled in the art will understand that all or part of the steps in the method of the above embodiment can be completed by instructing related hardware through a program, and the program can be stored in a computer-readable storage medium. When the program is executed, it includes one or a combination of the steps of the method embodiment.

[0113] In addition, the functional units in the various embodiments of the present invention may be integrated into a single processing module, or each unit may exist physically separately, or two or more units may be integrated into a single module. The aforementioned integrated modules may be implemented in the form of hardware or in the form of software functional modules. If the integrated modules are implemented in the form of software functional modules and sold or used as independent products, they may also be stored in a computer-readable storage medium.

[0114] The storage medium mentioned above may be a read-only memory, a magnetic disk, or an optical disk, etc. Although the embodiments of the present invention have been shown and described above, it is understood that the above embodiments are exemplary and are not to be construed as limiting the present invention. Persons skilled in the art may make changes, modifications, substitutions, and variations to the above embodiments within the scope of the present invention.

Claims

1. A variable bit rate de-redundant speech semantic coding method, characterized in that: The following steps are involved: Based on the preset deep neural network model, the original audio of the target speaker is obtained; Extracting linguistic discrete semantic representations from the original audio, and obtaining voiceprint features and compressed spectrum features of the original audio; Reconstructing the linguistic discrete semantic representation, the voiceprint feature, and the compressed spectral feature to obtain a target audio acoustic intermediate representation, and using the target audio acoustic intermediate representation to obtain a target audio to be output, so as to perform redundancy-free speech semantic encoding on the original audio; Among them, the linguistic discrete semantic representation, the voiceprint feature and the compressed spectral feature are respectively reconstructed to obtain an acoustic intermediate representation of the target audio, and the target audio to be output is obtained by using the acoustic intermediate representation of the target audio to perform de-redundant speech semantic encoding on the original audio, including: judging the amount of semantic information of the current time step according to the linguistic discrete semantic representation, judging the length of the compressed acoustic spectral feature actually used for transmission based on the semantic information, and transmitting compressed acoustic spectral features of different lengths for different time steps to achieve speech semantic encoding with variable bit rate.

2. The variable bit rate de-redundant speech semantic coding method according to claim 1, characterized in that: The extracting of the linguistic discrete semantic representation of the original audio includes: Obtaining a feature vector of a speech signal corresponding to the original audio; The feature vector is used to obtain a linguistic code to extract a linguistic discrete semantic representation corresponding to the linguistic code.

3. The variable bit rate de-redundant speech semantic coding method according to claim 2, characterized in that: The speech signal is expressed as: in, is the speech signal, is the lip, tooth and tongue filter group, As the basic waveform, for the vocal cords and resonance filter banks, is the number of vocal cords and resonance filters, is the number of lip, dentate and tongue filters, Number the lip, dentate and tongue filters. Number the vocal cords and resonance filters.

4. The variable bit rate de-redundant speech semantic coding method according to claim 1, characterized in that: The obtaining of the voiceprint features and compressed spectrum features of the original audio includes: Converting the speech waveform corresponding to the original audio into a Mel spectrum representing acoustic features; Extracting the acoustic information latent code corresponding to the Mel spectrum; Obtaining a voiceprint feature code using the acoustic information latent code, and obtaining the voiceprint feature using the voiceprint feature code; Based on the acoustic information latent code, a corresponding compressed spectrum code is obtained, so as to obtain the compressed spectrum feature by using the compressed spectrum code.

5. The variable bit rate de-redundant speech semantic coding method according to claim 1, characterized in that: The reconstructing the linguistic discrete semantic representation, the voiceprint feature, and the compressed spectrum feature to obtain a target audio acoustic intermediate representation includes: Reconstructing the linguistic discrete semantic representation to obtain a semantic embedding; Reconstructing the voiceprint feature to obtain a voiceprint embedding; Reconstructing the compressed spectrum features to obtain spectrum embedding; performing additive aggregation using the semantic embedding, the voiceprint embedding, the spectral embedding, and the Mel spectrum of the target time step to obtain an aggregated embedding vector; Convert the aggregated embedding vector into a Mel spectrum of the current time step; The mel spectrum of the target time step and the mel spectrum of the current time step are concatenated to obtain an intermediate acoustic representation of the target audio.

6. A variable bit rate de-redundant speech semantic coding device, characterized in that: include: An acquisition module, used to acquire the original audio of the target speaker based on a preset deep neural network model; a determination module, configured to extract linguistic discrete semantic representations from the original audio, and obtain voiceprint features and compressed spectrum features of the original audio; a processing module, configured to reconstruct the linguistic discrete semantic representation, the voiceprint feature, and the compressed spectral feature to obtain a target audio acoustic intermediate representation, and use the target audio acoustic intermediate representation to obtain a target audio to be output, so as to perform redundancy-free speech semantic encoding on the original audio; In which, the processing module is further used to determine the amount of semantic information of the current time step based on the linguistic discrete semantic representation, and determine the length of the compressed acoustic spectrum features actually used for transmission based on the amount of semantic information. For different time steps, compressed acoustic spectrum features of different lengths are transmitted to achieve variable bit rate speech semantic encoding.

7. The variable bit rate de-redundant speech semantic coding device according to claim 6, characterized in that: The determination module includes: A first acquiring unit, configured to acquire a feature vector of a speech signal corresponding to the original audio; The first extraction unit is configured to obtain a linguistic code using the feature vector to extract a linguistic discrete semantic representation corresponding to the linguistic code.

8. An electronic device, characterized in that: include: A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the variable bit rate de-redundant speech semantic coding method according to any one of claims 1 to 5.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: The program is executed by a processor to implement the variable bit rate de-redundant speech semantic coding method according to any one of claims 1 to 5.

10. A computer program product comprising a computer program, characterized in that The computer program is executed by a processor to implement the variable bit rate de-redundant speech semantic coding method according to any one of claims 1 to 5.

Images