A voice interaction method, system and storage medium for adaptive voice enhancement
Through the adaptive self-attention voice conversion scheme, combined with StarGAN and AdaIN, the noise adaptive self-attention StarGAN network is used to solve the problem of low voice comprehensibility in noisy environments, realize voice style conversion and voice interaction, and improve voice in noise Intelligibility in the environment.
Patent Information
- Application Number
- CN202111393902.2
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2021-11-23
- Publication Date
- 2024-11-29
- Estimated Expiration
- 2041-11-23
AI Technical Summary
In noisy environments, the prior art is difficult to effectively improve the intelligibility of speech, especially under the conditions of low signal-to-noise ratio, the Lombardian voice conversion effect is poor, resulting in the ineffective voice style conversion in voice calls.
Adaptive self-attention speech conversion scheme is adopted, combining StarGAN generator, self-attention mechanism, adaptive instance normalization (AdaIN) and convolutional neural network to design noise adaptive self-attention StarGAN network, and perform speech style by obtaining the ambient noise level. Transform and combine speech recognition to achieve voice interaction.
It effectively improves the intelligibility of speech in noisy environments. Through the conversion between different Lombardy speeches and normal speeches, the adaptive speech intelligibility enhancement in the noise environment is achieved, solving the problem of low training efficiency.
Smart Images

Figure CN114464207B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of voice interaction technology, and in particular to a voice interaction method, device and storage medium for adaptive voice enhancement. Background Art
[0002] In recent years, how to improve the intelligibility of speech in noisy environments has attracted increasing attention. This topic is called speech intelligibility enhancement or near-end hearing enhancement, which is a perceptual enhancement technology that reproduces clean (noise-free) speech in noisy environments. When exposed to adverse noise environments, even if the device outputs clear speech, it is difficult for the listener to obtain the information.
[0003] Previous studies have introduced a conversion from noise-free normal speech to Lombard speech at the near end using the Lombard effect. However, the effect is very poor in strong near-end noise interference with very low signal-to-noise ratio. In some voice calls, the speaker and the receiver are in different noise environments, causing the speaker to use Lombard speech. In this case, the implementation of language style conversion from normal to Lombard speech is not as effective as expected. Summary of the invention
[0004] This patent proposes an adaptive self-attention speech conversion scheme, designs a speech style conversion scheme that adapts to environmental noise, realizes the conversion between different Lombard voices and normal voices, and combines it with speech recognition to realize speech interaction with machines. The StarGAN generator combines the self-attention mechanism and adaptive instance normalization AdaIN with a convolutional neural network, named AdaSAStarGAN. It can effectively solve the problem of low training efficiency.
[0005] In view of this, the first aspect of the present application provides a voice interaction method with adaptive voice enhancement, characterized in that the method includes: obtaining a voice file input by a user, and obtaining a corresponding text file through voice recognition; translating the text file according to a seq2seq end-to-end model to obtain a text file replied by a machine, wherein the seq2seq end-to-end model is trained based on the THCHS-30 Chinese database; performing text analysis on the text file replied by the machine, and sending it to an acoustic model for training to obtain voice features, and synthesizing voice through a vocoder; obtaining a Lombard audio file containing normal voice audio and two noise levels. A German corpus of speech audio is used as a training data set, and the training data set is used to extract characteristic spectrum envelopes MCCs and fundamental frequency f0 in speech through a vocoder; the current environmental noise level is obtained in real time through a sound pressure meter; the characteristic spectrum envelope MCCs features and the current environmental noise level are input into a noise adaptive self-attention StarGAN network model to test the model, and the corresponding speech style conversion is performed according to the real-time noise level, and the speech after speech enhancement is output, and the loss function of the adaptive self-attention StarGAN network is calculated; speech intelligibility and short-time objective intelligibility are used as evaluation indicators of the intelligibility of speech after speech enhancement and the perceptual effect.
[0006] Optionally, in combination with the first aspect, in a possible implementation, obtaining the voice file input by the user specifically includes: obtaining a wav format file based on device recording; and converting the wav format file into a pcm format file through an ffmpeg program.
[0007] Optionally, in combination with the first aspect, in a possible implementation method, the translating the text file according to the seq2seq end-to-end model to obtain the text file of the machine reply specifically includes: training with a long short-term memory recurrent network structure to obtain the real-time response of the machine, and outputting the text file of the machine reply.
[0008] Optionally, in combination with the first aspect, in a possible implementation, the training data set is obtained by selecting part of speech from a German corpus for experiments, and the German corpus includes normal speech recorded under 0 decibel background noise, Lombard language recorded under 55 decibel noise, and Lombard language recorded under 70 decibel background noise.
[0009] Optionally, in combination with the first aspect, in a possible implementation, the vocoder is a world vocoder, the world vocoder uses three algorithms to extract three parameters, and uses a synthesis algorithm to take the three parameters as input, and the use of the training data set to extract the characteristic spectrum envelope MCCs and fundamental frequency f0 in the speech through the vocoder specifically includes: inputting wave to estimate the fundamental frequency f0 through the DIO algorithm; taking the fundamental frequency f0 and wave as input, and estimating the spectrum envelope by cheap trick; inputting f0 / sp / wave, and using PLATINUM to estimate the extracted signal to obtain periodic parameters.
[0010] Optionally, in combination with the first aspect, in a possible implementation method, the real-time acquisition of the current environmental noise level through a sound pressure meter specifically includes: using a sound pressure meter to obtain the decibel value of the current environmental noise in real time, wherein the decibel value of the measured audio signal is between 0 decibels and 100 decibels; classifying the decibel value of the current environmental noise obtained in real time, wherein speech under noise of 0 to 55 decibels is classified as normal speech, speech under noise of 55 to 70 decibels is classified as Lombard language level 1, and speech under noise above 70 decibels is classified as Lombard language level 2.
[0011] Optionally, in combination with the first aspect, in a possible implementation, the loss function of the adaptive self-attention StarGAN network is specifically calculated as follows:
[0012] Compute the target-conditioned adversarial loss:
[0013] L st-adv =E (x,n)~P(x,n),n′~P(n′) [logD(x,n′,n)]+E (x,n)~P(x,n),n′~P(n′) [logD(G(x,n,n′),n,n′)];
[0014] Among them, n′~P(n′) is randomly sampled independently of the real data, and D is the target condition discriminator;
[0015] Calculate cycle consistency loss:
[0016] L cyc =E (x,n)~P(x,n),n′~P(n′) [||xG(G(x,n′),n)||1];
[0017] Identity Mapping Loss:
[0018] L id =E (x,n)~P(x,n) [||G(x,n)-x||];
[0019] The adaptive self-attention StarGAN minimizes the overall goal of the generator and discriminator as follows:
[0020]
[0021] L D =-L st-adv ;
[0022] in, and are regularization parameters, respectively, which measure the importance of cycle consistency loss and identity mapping loss relative to adversarial loss. The generator and discriminator minimize L G and L D To optimize.
[0023] Optionally, in combination with the first aspect, in a possible implementation, the setting of parameters and training specifically includes: pre-setting the number of hidden layers, hidden nodes and learning rate of the noise adaptive self-attention StarGAN network; using the vocoder world to extract the spectrum envelope and fundamental frequency f0 extracted from the training corpus as features input into the noise adaptive self-attention StarGAN network for network training; the inputting the characteristic spectrum envelope MCCs and the current environmental noise level into the noise adaptive self-attention StarGAN network model for model testing specifically includes: using the vocoder world to extract the characteristic spectrum envelope MCCs features of the speech after speech synthesis and the current environmental noise input into the trained network for testing.
[0024] The second aspect of the present application provides an adaptive speech enhancement voice interaction system, characterized in that the adaptive speech enhancement voice interaction system comprises: a memory and at least one processor, the memory storing instructions, the memory and the at least one processor being interconnected through lines; the at least one processor calling the instructions in the memory so that the adaptive speech enhancement voice interaction system executes the adaptive speech enhancement voice interaction method as described in any one of the first aspect to the first aspect of the present application.
[0025] A third aspect of the present application provides a computer-readable storage medium having a computer program stored thereon, wherein when the computer program is executed by a processor, the voice interaction method for adaptive voice enhancement as described in any one of the first aspect to the first aspect of the present application is implemented.
[0026] The present application provides a voice interaction method, system and storage medium for adaptive speech enhancement. The method includes: obtaining a voice file input by a user, and obtaining a corresponding text file through speech recognition; translating the text file according to a seq2seq end-to-end model to obtain a text file replied by a machine, wherein the seq2seq end-to-end model is trained based on the THCHS-30 Chinese database; performing text analysis on the text file replied by the machine, and sending it to an acoustic model for training to obtain speech features, and synthesizing speech through a vocoder; obtaining a German corpus containing normal speech audio and Lombard speech audio under two noise levels as a training data set, using The training data set extracts the characteristic spectrum envelope MCCs and fundamental frequency f0 in the speech through the vocoder; obtains the current environmental noise level in real time through the sound pressure meter; inputs the characteristic spectrum envelope MCCs features and the current environmental noise level into the noise adaptive self-attention StarGAN network model to test the model, performs corresponding speech style conversion according to the real-time noise level, outputs the speech after speech enhancement, and calculates the loss function of the adaptive self-attention StarGAN network; uses speech intelligibility and short-term objective intelligibility as evaluation indicators of the intelligibility and perceptual effect of the speech after speech enhancement. Through the conversion between different Lombard voices and normal voices, and combined with speech recognition, speech interaction with the machine is realized. The problem of low training efficiency can be effectively solved. At the same time, the conversion of speech style types can be realized through a noise adaptive self-attention StarGAN network, so as to achieve the effect of enhancing the speech intelligibility adaptive to the noise environment. BRIEF DESCRIPTION OF THE DRAWINGS
[0027] Figure 1 A schematic diagram of a voice interaction process of adaptive voice enhancement in an embodiment of the present application;
[0028] Figure 2 A schematic diagram of a voice interaction framework for adaptive voice enhancement provided in this application;
[0029] Figure 3 A noise adaptive self-attention StarGAN network training process diagram provided in an embodiment of the present application;
[0030] Figure 4 A schematic diagram of a voice interaction system with adaptive voice enhancement in an embodiment of the present application. DETAILED DESCRIPTION
[0031] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of this application.
[0032] The term "and / or" in this application can be a description of the association relationship of associated objects, indicating that there can be three relationships. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. In addition, the character " / " in this application generally indicates that the associated objects before and after are in an "or" relationship.
[0033] The terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments described herein can be implemented in an order other than that illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or modules is not necessarily limited to those steps or modules that are clearly listed, but may include other steps or modules that are not clearly listed or inherent to these processes, methods, products or devices.
[0034] How to improve the intelligibility of speech in noisy environments has attracted increasing attention. This topic is called speech intelligibility enhancement or near-end hearing enhancement, which is a perceptual enhancement technology that reproduces clean (noise-free) speech in noisy environments. When exposed to adverse noise environments, even if the device outputs clear speech, it is difficult for the listener to obtain the information.
[0035] Previous studies have introduced a conversion from noise-free normal speech to Lombard speech at the near end using the Lombard effect. However, the effect is very poor in strong near-end noise interference with very low signal-to-noise ratio. In some voice calls, the speaker and the receiver are in different noise environments, causing the speaker to use Lombard speech. In this case, the implementation of language style conversion from normal to Lombard speech is not as effective as expected.
[0036] Therefore, the purpose of the present invention is to provide a method for adaptive self-attention speech conversion, a speech style conversion scheme adapted to environmental noise, and to achieve speech style conversion and intelligibility enhancement by acquiring Lombard speech under environmental noise. And to realize a speech interaction system in combination with speech recognition.
[0037] The technical problem to be solved by the present invention is to provide a voice interaction system for adaptive voice enhancement in a noisy environment, comprising:
[0038] Get the voice file input by the user, which is a wav format file obtained after recording by the device and needs to be converted into a pcm format file through the ffmpeg program.
[0039] The seq2seq model uses the long short-term memory recurrent network structure to train the machine to get the response it should make at the moment and output the text file of the machine's reply. The obtained series of text files are sent to the acoustic model through text analysis to obtain speech features, and the speech is synthesized through the vocoder to realize the interaction between the machine and the human.
[0040] The training data set is obtained by selecting some speech from the German corpus for experiments. The database used includes normal speech recorded under 0dB background noise, Lombard language recorded under 55dB background noise, and Lombard language recorded under 70dB background noise.
[0041] The spectral envelope features and fundamental frequency f0 in speech are extracted using the WORLD vocoder, which uses three algorithms to extract three parameters and then uses a synthesis algorithm to take them as input to achieve high-quality speech synthesis.
[0042] 1) Input wave to estimate the fundamental frequency f0 through DIO algorithm;
[0043] 2) Taking the fundamental frequency f0 and wave as input, the spectrum envelope is estimated by the cheap trick;
[0044] 3) Input f0 / sp / wave, use PLATINUM to estimate the extracted signal and obtain the non-periodic parameters.
[0045] Use a sound pressure meter to obtain the decibel value of the current environmental noise in real time. The decibel value of the measured audio signal is between 0dB and 100dB. The measured decibel value is used for classification. Speech in noise of 0-55dB is classified as normal speech, speech in noise of 55-70dB is classified as Lombard level 1, and speech in noise above 70dB is classified as Lombard level 2.
[0046] The loss function of the noise adaptive self-attention StarGAN network is composed of adversarial loss, cycle consistency loss and identity mapping loss.
[0047] Target Conditioned Adversarial Loss:
[0048] L st-adv =E (x,n)~P(x,n),n′~P(n′) [logD(x,n′,n)]+E (x,n)~P(x,n),n′~P(n′)[logD(G(x,n,n′),n,n′)]
[0049] where n′~P(n′) are randomly sampled independently of the real data. Where D is the target-conditioned discriminator. By maximizing this loss, the discriminator attempts to learn the best decision boundary between the transformed acoustic features and the real acoustic features conditioned on the target domain code (n and n′). Conversely, the generator attempts to make the transformed features indistinguishable from the real acoustic features conditioned on n′ by minimizing this loss.
[0050] Cycle consistency loss:
[0051] L cyc =E (x,n)~P(x,n),n′~P(n′) [||xG(G(x,n′),n)||1]
[0052] This loop constraint encourages the generator to find an optimal source and target pair that does not harm the composition.
[0053] Identity Mapping Loss:
[0054] L id =E (x,n)~P(x,n) [||G(x,n)-x||]
[0055] In summary, the noise adaptive self-attention StarGAN network minimizes the overall objectives of the generator, discriminator, and classifier as follows:
[0056]
[0057] L D =-L st-adv
[0058] in and are regularization parameters, respectively, which measure the importance of cycle consistency loss and identity mapping loss relative to adversarial loss. The discriminator and generator are respectively minimized by L D and L G To optimize.
[0059] Train the noise adaptive self-attention StarGAN network, test it, and analyze the evaluation indicators.
[0060] 1) Setting parameters: pre-set the number of hidden layers, number of hidden layer nodes and learning rate of the noise adaptive self-attention StarGAN network;
[0061] 2) The spectrum envelope and fundamental frequency f0 extracted from the vocoder world training corpus are used as features to input into the noise adaptive self-attention StarGAN network for network training. Then the synthesized speech and environmental noise are input into the trained network for testing.
[0062] 3) SIIB is an estimate of the amount of information shared between the speaker and the listener, measured in bits per second. Intelligibility is quantified by counting the number of correctly recognized words or phonemes. Short-term objective intelligibility (STOI) predicts the human perception and intelligibility of speech, ranging from 0 to 1.
[0063] Furthermore, the technical solution adopted by the present invention is a voice interaction system for adaptive voice enhancement in a noisy environment. The present invention will be further described in detail below in conjunction with the accompanying drawings and embodiments.
[0064] See also Figure 1 and Figure 2 , introduced the framework of this experiment. The specific steps are as follows:
[0065] Step 110: Acquire the voice file input by the user, convert the voice file type, and obtain a text file through voice recognition.
[0066] Get the voice file input by the user, which is a wav format file obtained after recording by the device and needs to be converted into a pcm format file through the ffmpeg program.
[0067] Step 120: Obtain the THCHS-30 Chinese data set to train a seq2seq model, use the seq2seq end-to-end model for translation, obtain a text file of the machine reply, and realize voice interaction.
[0068] The seq2seq model uses the long short-term memory recurrent network structure to train the machine to get the response it should make at the moment and output the text file of the machine's reply. The obtained series of text files are sent to the acoustic model through text analysis to obtain speech features, and the speech is synthesized through the vocoder to realize the interaction between the machine and the human.
[0069] Step 130: Obtain the text file output by the seq2seq model, perform text analysis and send it into the acoustic model for training, obtain speech features, and synthesize speech through a vocoder.
[0070] The training data set is obtained by selecting some speech from the German corpus for experiments. The database used includes normal speech recorded under 0dB background noise, Lombard language recorded under 55dB background noise, and Lombard language recorded under 70dB background noise.
[0071] Step 140: Acquire a German corpus including normal speech audio and Lombard speech audio at two noise levels as a training data set.
[0072] The spectral envelope features and fundamental frequency f0 in speech are extracted using the WORLD vocoder, which uses three algorithms to extract three parameters and then uses a synthesis algorithm to take them as input to achieve high-quality speech synthesis.
[0073] 1) Input wave to estimate the fundamental frequency f0 through DIO algorithm;
[0074] 2) Taking the fundamental frequency f0 and wave as input, the spectrum envelope is estimated by the cheap trick;
[0075] 3) Input f0 / sp / wave, use PLATINUM to estimate the extracted signal and obtain the non-periodic parameters.
[0076] Step 1.5: The training data set is used to extract the characteristic spectrum envelope MCCs and fundamental frequency f0 in the speech through the vocoder.
[0077] Set the loss function of the noise adaptive self-attention StarGAN network. The loss function of the noise adaptive self-attention StarGAN network is composed of adversarial loss, cycle consistency loss and identity mapping loss.
[0078] Step 1.6. Calculate the loss function of the adaptive self-attention StarGAN network, set the parameters, and train the noisy adaptive self-attention StarGAN network.
[0079] Training a noise-adaptive self-attention StarGAN network.
[0080] 1) Setting parameters: pre-set the number of hidden layers, number of hidden layer nodes and learning rate of the noise adaptive self-attention StarGAN network;
[0081] 2) The spectral envelope and fundamental frequency f0 of speech with three attributes extracted from the training corpus of the vocoder world are used as speech signal features and input into the noise adaptive self-attention StarGAN network for network training.
[0082] 3) Input the real source speech signal features and the target speech attribute label into the generator in the AdaSAStarGAN network framework to generate a false speech signal feature with the target speech attribute label, and input the target speech attribute label into the discriminator in the AdaSAStarGAN network framework. At the same time, the real target speech signal features and the target speech attribute label are also input into the discriminator, and the discriminator determines whether the input speech signal features are the real target speech signal features. If the discriminator determines that the input speech signal features belong to the target attribute label, it outputs true, otherwise it outputs false. The generated false target speech signal features and the source speech attribute label are input into the generator to generate false source speech signal features. See Figure 3 .
[0083] 4) The network training process optimizes the model through the calculation results of adversarial loss, cyclic consistency loss and identity mapping loss, and performs back propagation. After multiple rounds of propagation, the model training is completed when the training requirement of 20,000 iterations is met.
[0084] Target Conditioned Adversarial Loss:
[0085] L st-adv =E (x,n)~P(x,n),n′~P(n′) [logD(x,n′,n)]+E (x,n)~P(x,n),n′~P(n′) [logD(G(x,n,n′),n,n′)]
[0086] where n′~P(n′) are randomly sampled independently of the real data. Where D is the target-conditioned discriminator. By maximizing this loss, the discriminator attempts to learn the best decision boundary between the transformed acoustic features and the real acoustic features conditioned on the target domain code (n and n′). Conversely, the generator attempts to make the transformed features indistinguishable from the real acoustic features conditioned on n′ by minimizing this loss.
[0087] Cycle consistency loss:
[0088] L cyc =E (x,n)~P(x,n),n′~P(n′) [||xG(G(x,n′),n)||1]
[0089] This loop constraint encourages the generator to find an optimal source and target pair that does not harm the composition.
[0090] Identity Mapping Loss:
[0091] L id =E (x,n)~P(x,n) [||G(x,n)-x||]
[0092] In summary, the adaptive self-attention StarGAN minimizes the overall objectives of the generator, discriminator, and classifier as follows:
[0093]
[0094] L D =-L st-adv
[0095] in and are regularization parameters, respectively, which measure the importance of cycle consistency loss and identity mapping loss relative to adversarial loss. The discriminator and generator are respectively minimized by L D and L G To optimize.
[0096] Step 170: A built-in sound pressure meter is used to obtain the current environmental noise level in real time.
[0097] Use a sound pressure meter to obtain the decibel value of the current environmental noise in real time. The decibel value of the measured audio signal is between 0dB and 100dB. The measured decibel value is used for classification. Speech in noise of 0-55dB is classified as normal speech, speech in noise of 55-70dB is classified as Lombard level 1, and speech in noise above 70dB is classified as Lombard level 2.
[0098] Step 180, test the network, use the spectrum envelope MCCs features extracted from the vocoder test corpus and the current environmental noise, input them into the noise adaptive self-attention StarGAN network model to test the model, perform corresponding speech style conversion according to the real-time noise level, and output the speech after speech enhancement.
[0099] The synthesized speech is input into the WORLD vocoder to extract the characteristic spectrum envelope MCCs and fundamental frequency F0, and the extracted features and environmental noise are input into the noise adaptive self-attention StarGAN network. Finally, the enhanced speech is synthesized by the WORLD vocoder and the enhanced speech is output to make it easier to obtain the information in the speech.
[0100] Finally, SIIB and STOI are used to evaluate the intelligibility and naturalness of speech. SIIB is an estimate of the amount of information shared between the speaker and the listener, measured in bits per second. Intelligibility is quantified by counting the number of correctly recognized words or phonemes. Short-time objective intelligibility (STOI) predicts the human perception and intelligibility of speech, ranging from 0 to 1.
[0101] See also Figure 2 , Figure 2A schematic diagram of a voice interaction framework for adaptive speech enhancement is provided. The voice interaction process of the adaptive speech enhancement includes: user speaking, denoising, speech recognition, seq2seq model, speech synthesis, world vocoder feature extraction, noise adaptive self-attention StarGAN mapping, world vocoder audio synthesis, and enhanced speech. The noise adaptive self-attention StarGAN mapping can also include environmental noise and obtain noise level.
[0102] See also Figure 3 , the present application embodiment provides a noise adaptive self-attention StarGAN network training process diagram. Please refer to Figure 3 , the source speech attribute label enters the generator, and the generator generates a false source speech signal feature. At the same time, the target speech attribute label and the source speech signal feature enter the generator to generate a false target speech signal feature. The target speech attribute label and the target speech signal feature enter the discriminator, and the generated speech signal feature is judged to be whether it belongs to the target speech attribute.
[0103] Figure 4 4 is a schematic diagram of the structure of a voice interaction system with adaptive speech enhancement provided by an embodiment of the present application. The voice interaction system 400 with adaptive speech enhancement may have relatively large differences due to different configurations or performances, and may include one or more processors (central processing units, CPU) 410 (for example, one or more processors) and a memory 420, and one or more storage media 430 (for example, one or more mass storage devices) storing application programs 433 or data 432. Among them, the memory 420 and the storage medium 430 may be short-term storage or permanent storage. The program stored in the storage medium 430 may include one or more modules (not shown in the figure), and each module may include a series of instruction operations in the voice interaction system 400 with adaptive speech enhancement. Furthermore, the processor 410 may be configured to communicate with the storage medium 430 to execute a series of instruction operations in the storage medium 430 on the voice interaction system 400 with adaptive speech enhancement.
[0104] The adaptive speech enhanced speech interaction system 400 may also include one or more power supplies 440, one or more wired or wireless network interfaces 430, one or more input and output interfaces 460, and / or one or more operating systems 431, such as Windows Serve, Mac OS X, Unix, Linux, FreeBSD, etc. It will be appreciated by those skilled in the art that Figure 4The structure of the adaptive speech enhancement voice interaction system shown does not constitute a limitation of the adaptive speech enhancement voice interaction system, and may include more or fewer components than shown in the figure, or a combination of certain components, or a different arrangement of components.
[0105] The present application also provides a computer-readable storage medium, which may be a non-volatile computer-readable storage medium or a volatile computer-readable storage medium, wherein instructions are stored in the computer, and when the instructions are executed on a computer, the computer executes the steps of the adaptive speech enhanced speech interaction.
[0106] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.
[0107] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM), random access memory (RAM), disk or optical disk and other media that can store program code.
[0108] In the embodiments provided in this application, it should be understood that the disclosed methods can be implemented in other ways without exceeding the spirit and scope of this application. The current embodiment is only an illustrative example and should not be used as a limitation. The specific content given should not limit the purpose of this application. For example, some features can be ignored or not executed.
[0109] The technical means disclosed in the present application are not limited to the technical means disclosed in the above-mentioned implementation methods, but also include technical solutions composed of any combination of the above technical features. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of this application, and these improvements and modifications are also regarded as the protection scope of this application.
[0110] The above is a detailed introduction to the voice interaction, system and storage medium for adaptive voice enhancement provided by the embodiments of the present application. Specific examples are used in this article to illustrate the principles and implementation methods of the present application. The description of the above embodiments is only used to help understand the method and its core idea of the present application; at the same time, for those of ordinary skill in the art, according to the idea of the present application, there will be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as a limitation on the present application. Although the present application is described in detail with reference to the aforementioned embodiments, those of ordinary skill in the art should understand that they can still modify the technical solutions recorded in the aforementioned embodiments, or replace some of the technical features therein by equivalents; and these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application.
Claims
1. A voice interaction method with adaptive voice enhancement, It is characterized in that The method comprises: Obtain the voice file input by the user and obtain the corresponding text file through voice recognition; Translating the text file according to a seq2seq end-to-end model to obtain a text file replied by the machine, wherein the seq2seq end-to-end model is trained according to the THCHS-30 Chinese database; Performing text analysis on the text file replied by the machine, and sending it to the acoustic model for training, obtaining speech features, and synthesizing speech through a vocoder; A German corpus including normal speech audio and Lombard speech audio at two noise levels is obtained as a training data set, and characteristic spectrum envelopes MCCs and fundamental frequency f0 in speech are extracted by a vocoder using the training data set; Obtain the current ambient noise level in real time through the sound pressure meter; Input the characteristic spectrum envelope MCCs feature and the current environmental noise level into the noise adaptive self-attention StarGAN network model to test the model, perform corresponding speech style conversion according to the real-time noise level, output the speech after speech enhancement, and calculate the loss function of the adaptive self-attention StarGAN network; Speech intelligibility and short-term objective intelligibility are used as evaluation indicators of the intelligibility and perceptual effect of speech after speech enhancement.
2. The voice interaction method of adaptive voice enhancement according to claim 1, It is characterized in that The step of obtaining the voice file input by the user specifically includes: WAV format file obtained from the device recording; The wav format file is converted into a pcm format file through the ffmpeg program.
3. The voice interaction method of adaptive voice enhancement according to claim 1, It is characterized in that The translating of the text file according to the seq2seq end-to-end model to obtain a text file replied by the machine specifically includes: The long short-term memory recurrent network structure is used for training to obtain the real-time response of the machine and output the text file of the machine's reply.
4. The voice interaction method of adaptive voice enhancement according to claim 1, It is characterized in that The training data set is obtained by selecting part of speech from the German corpus for experiment. The German corpus includes normal speech recorded under 0 decibel background noise, Lombard speech recorded under 55 decibel noise, and Lombard speech recorded under 70 decibel background noise.
5. The voice interaction method of adaptive voice enhancement according to claim 1, It is characterized in that The vocoder is a world vocoder, which uses three algorithms to extract three parameters, and uses a synthesis algorithm to take the three parameters as input, and the method of extracting the characteristic spectrum envelope MCCs and fundamental frequency f0 in the speech through the vocoder using the training data set specifically includes: The input wave is used to estimate the fundamental frequency f0 through the DIO algorithm; The fundamental frequency f0 and wave are used as input, and the spectrum envelope is estimated by the cheap trick; Input f0 / sp / wave, use PLATINUM to estimate the extracted signal and obtain the periodic parameters.
6. The voice interaction method of adaptive voice enhancement according to claim 1, It is characterized in that The real-time acquisition of the current environmental noise level by the sound pressure meter specifically includes: A sound pressure meter is used to obtain the decibel value of the current environmental noise in real time, wherein the decibel value of the measured audio signal is between 0 decibel and 100 decibel; The decibel value of the current environmental noise obtained in real time is classified, wherein speech under a noise level of 0 to 55 decibels is classified as normal speech, speech under a noise level of 55 to 70 decibels is classified as Lombard level 1, and speech under a noise level of more than 70 decibels is classified as Lombard level 2.
7. The voice interaction method of adaptive voice enhancement according to claim 1, It is characterized in that The specific calculation method of the loss function of the adaptive self-attention StarGAN network is as follows: Compute the target-conditional adversarial loss: L st-adv JE (χ,n)~P(χ,n),n′~P(n) [log D(χ,n4n)]+E (x,n)~P(x,n),n′~P(n′) [log D(G(x,n,n′),n,n′)] Among them, n′~P(n′) is randomly sampled independently of the real data, and D is the target condition discriminator; Calculate cycle consistency loss: L cyc =E (x,n)~P(x,n),n′~P(n′) [|χ-G(G(χ,n),n)| / 1 ]; Identity Mapping Loss: L id =E (x,n)~P(x,n) [|G(x,n)-x|]; The adaptive self-attention StarGAN minimizes the overall goal of the generator and discriminator as follows: L D =-L st-adv ; in, and are regularization parameters, respectively, which measure the importance of cycle consistency loss and identity mapping loss relative to adversarial loss. The generator and discriminator minimize L G and L D To optimize.
8. The voice interaction method of adaptive voice enhancement according to claim 5, It is characterized in that Setting parameters and training specifically include: Preset the number of hidden layers, hidden nodes, and learning rate of the noise adaptive self-attention StarGAN network; The spectrum envelope and fundamental frequency f0 extracted from the vocoder world extraction training corpus are used as feature input to the noise adaptive self-attention StarGAN network for network training; The step of inputting the characteristic spectrum envelope MCCs and the current environmental noise level into the noise adaptive self-attention StarGAN network model to test the model specifically includes: The vocoder world is used to extract the characteristic spectrum envelope MCCs features of the synthesized speech and the current environmental noise is input into the trained network for testing.
9. A voice interaction system with adaptive voice enhancement, It is characterized in that The adaptive speech enhanced speech interaction system comprises: a memory and at least one processor, the memory stores instructions, and the memory and the at least one processor are interconnected via a line; The at least one processor calls the instructions in the memory so that the adaptive speech enhancement speech interaction system executes the adaptive speech enhancement speech interaction method as described in any one of claims 1 to 8.
10. A computer-readable storage medium having a computer program stored thereon, It is characterized in that When the computer program is executed by a processor, the speech interaction method for adaptive speech enhancement according to any one of claims 1 to 8 is implemented.
Citation Information
Patent Citations
Voice definition enhancing method based on acoustic character conversion
CN110085245A
Speech enhancement method and system
CN112581973A
Model training method and apparatus, voice conversion method, device, and storage medium
WO2022121180A1
