Acoustic modeling method, device, equipment and storage medium

Abstract

The present disclosure provides an acoustic modeling method, a speech conversion method, an apparatus, a device, a storage medium, and a computer program product, which relate to the field of artificial intelligence technology, specifically the field of deep learning and speech synthesis technology, and can be applied to scenarios such as acoustic modeling and speech conversion. The specific implementation scheme is as follows: obtaining text training samples and corresponding speech spectrum training samples; performing spectrum quality calibration on the speech spectrum training samples to obtain calibrated speech spectrum training samples; training an initial acoustic model based on the text training samples and the calibrated speech spectrum training samples to obtain a target acoustic model. First, the spectrum quality of the speech spectrum training samples is calibrated, and then training is performed based on the text training samples and the calibrated speech spectrum training samples to make the speech spectrum output by the target acoustic model more accurate.

Description

Technical Field

[0001] The present disclosure relates to the field of artificial intelligence technology, specifically the field of deep learning and speech synthesis technology, and can be applied to scenarios such as acoustic modeling and speech conversion. In particular, it relates to an acoustic modeling method, a speech conversion method, an apparatus, a device, a storage medium, and a computer program product. Background Art

[0002] Currently, speech synthesis usually synthesizes standard speech, but cannot synthesize whispered speech. Summary of the Invention

[0003] The present disclosure provides an acoustic modeling method, a speech conversion method, an apparatus, a device, a storage medium, and a computer program product, which make the speech spectrum output by the target acoustic model more accurate.

[0004] According to one aspect of the present disclosure, an acoustic modeling method is provided, comprising: obtaining text training samples and corresponding speech spectrum training samples; performing spectrum quality calibration on the speech spectrum training samples to obtain calibrated speech spectrum training samples; and training an initial acoustic model based on the text training samples and the calibrated speech spectrum training samples to obtain a target acoustic model.

[0005] According to another aspect of the present disclosure, a speech conversion method is provided, comprising: acquiring a text to be converted; obtaining a speech spectrum corresponding to the text to be converted based on a target acoustic model; converting the speech spectrum into speech, and outputting the speech.

[0006] According to another aspect of the present disclosure, an acoustic modeling device is provided, including: a first acquisition module, configured to acquire text training samples and corresponding speech spectrum training samples; a calibration module, configured to perform spectrum quality calibration on the speech spectrum training samples to obtain calibrated speech spectrum training samples; and a training module, configured to train an initial acoustic model based on the text training samples and the calibrated speech spectrum training samples to obtain a target acoustic model.

[0007] According to another aspect of the present disclosure, a speech conversion device is provided, including: a second acquisition module, configured to acquire text to be converted; a third acquisition module, configured to obtain a speech spectrum corresponding to the text to be converted based on a target acoustic model; and a conversion module, configured to convert the speech spectrum into speech and output the speech.

[0008] According to another aspect of the present disclosure, an electronic device is provided, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the acoustic modeling method and the speech conversion method.

[0009] According to another aspect of the present disclosure, a non-transitory computer-readable storage medium storing computer instructions is provided, wherein the computer instructions are used to enable the computer to execute the acoustic modeling method and the speech conversion method.

[0010] According to another aspect of the present disclosure, a computer program product is provided, including a computer program, which implements the acoustic modeling method and speech conversion method when executed by a processor.

[0011] It should be understood that the contents described in this section are not intended to identify the key or important features of the embodiments of the present disclosure, nor are they intended to limit the scope of the present disclosure. Other features of the present disclosure will become readily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] The accompanying drawings are provided to facilitate a better understanding of the present invention and do not constitute a limitation of the present disclosure.

[0013] Figure 1 is an exemplary system architecture diagram in which the present disclosure may be applied;

[0014] Figure 2 is a flow chart of an embodiment of an acoustic modeling method according to the present disclosure;

[0015] Figure 3 is a flowchart of another embodiment of the acoustic modeling method according to the present disclosure;

[0016] Figure 4 is a flowchart of another embodiment of the acoustic modeling method according to the present disclosure;

[0017] FIG5( a ) is a schematic diagram of a speech spectrum training sample before calibration according to the present disclosure;

[0018] FIG5( b ) is a schematic diagram of a speech spectrum training sample after calibration according to the present disclosure;

[0019] Figure 6 is a flow chart of an embodiment of a voice conversion method according to the present disclosure;

[0020] Figure 7 is a flow chart of another embodiment of the voice conversion method according to the present disclosure;

[0021] Figure 8 is a schematic structural diagram of an embodiment of an acoustic modeling device according to the present disclosure;

[0022] Figure 9 is a structural diagram of an embodiment of a speech conversion device according to the present disclosure;

[0023] Figure 10 The block diagram is a block diagram of an electronic device used to implement the acoustic modeling method or speech conversion method of the embodiment of the present disclosure. DETAILED DESCRIPTION

[0024] The following description of exemplary embodiments of the present disclosure is made in conjunction with the accompanying drawings, including various details of the embodiments of the present disclosure to facilitate understanding. These details should be considered as merely exemplary. Therefore, those skilled in the art will recognize that various changes and modifications may be made to the embodiments described herein without departing from the scope and spirit of the present disclosure. Similarly, for the sake of clarity and conciseness, descriptions of well-known functions and structures are omitted in the following description.

[0025] Figure 1 An exemplary system architecture 100 is shown to which embodiments of the acoustic modeling method or speech conversion method or acoustic modeling device or speech conversion device of the present disclosure can be applied.

[0026] like Figure 1 As shown, system architecture 100 may include terminal devices 101, 102, 103, a network 104, and a server 105. Network 104 is a medium for providing communication links between terminal devices 101, 102, 103 and server 105. Network 104 may include various connection types, such as wired or wireless communication links or fiber optic cables.

[0027] Users can use terminal devices 101, 102, 103 to interact with server 105 via network 104 to obtain target acoustic models, etc. Various client applications, such as speech spectrum processing applications, can be installed on terminal devices 101, 102, 103.

[0028] Terminal devices 101, 102, and 103 can be either hardware or software. When terminal devices 101, 102, and 103 are hardware, they can be various electronic devices, including but not limited to smartphones, tablet computers, laptop computers, and desktop computers. When terminal devices 101, 102, and 103 are software, they can be installed in the aforementioned electronic devices. They can be implemented as multiple software programs or software modules, or as a single software program or software module. This is not specifically limited here.

[0029] The server 105 can provide various services based on acoustic modeling. For example, the server 105 can analyze and process text training samples and corresponding speech spectrum training samples obtained from the terminal devices 101, 102, and 103, and generate processing results (such as determining a target acoustic model, etc.).

[0030] It should be noted that the server 105 can be hardware or software. When the server 105 is hardware, it can be implemented as a distributed server cluster consisting of multiple servers, or it can be implemented as a single server. When the server 105 is software, it can be implemented as multiple software or software modules (for example, to provide distributed services), or it can be implemented as a single software or software module. No specific limitations are given here.

[0031] It should be noted that the acoustic modeling method or speech conversion method provided in the embodiments of the present disclosure is generally executed by the server 105 , and accordingly, the acoustic modeling device or speech conversion device is generally provided in the server 105 .

[0032] It should be understood that Figure 1 The number of terminal devices, networks and servers in the embodiment is merely illustrative. Any number of terminal devices, networks and servers may be provided as required.

[0033] Continue to refer Figure 2 , which shows a process 200 of an embodiment of an acoustic modeling method according to the present disclosure. The acoustic modeling method includes the following steps:

[0034] Step 201: Obtain text training samples and corresponding speech spectrum training samples.

[0035] In this embodiment, the execution body of the acoustic modeling method (eg Figure 1 The server 105 shown can obtain text training samples and corresponding speech spectrum training samples. The text training samples can be text of any length and content. The text can be in Chinese, English, or Korean, and this disclosure does not limit this. A text segment can be extracted from a book or other existing text and used as a text training sample, or a segment can be written as a text training sample, and this disclosure does not limit this. A speech spectrum is a mel spectrum that can be used to represent short-term audio information. The pitch of a sound heard by the human ear is not linearly related to the actual frequency. The mel spectrum conforms to the auditory characteristics of the human ear, i.e., it is linearly distributed below 1000 Hz and grows logarithmically above 1000 Hz. Therefore, the original speech signal can be converted into a mel spectrum for analysis. The speech spectrum training samples correspond to the text training samples. After obtaining the text training samples, corresponding speech can be generated based on the content of the text training samples. The recorded speech is used as the speech corresponding to the text training samples, and then the speech is converted into a mel spectrum to obtain the corresponding speech spectrum training samples. The loudness and pitch of the voice may be any loudness and pitch, which is not limited in the present disclosure. For example, the voice may be a low-loud whisper.

[0036] Step 202: calibrate the spectrum quality of the speech spectrum training sample to obtain a calibrated speech spectrum training sample.

[0037] In this embodiment, after obtaining the speech spectrum training samples, the aforementioned execution entity may perform spectrum quality calibration on the speech spectrum training samples to obtain calibrated speech spectrum training samples. Spectrum quality calibration involves data processing of the spectrum to remove noise and other artifacts from the spectrum, thereby enabling the spectrum to better restore the original speech. Specifically, the speech spectrum training samples may be input into a noise recognition model to identify noise and artifacts in the speech spectrum training samples, remove signals marked as noise and artifacts, and obtain calibrated speech spectrum training samples.

[0038] Step 203: Based on the text training samples and the calibrated speech spectrum training samples, an initial acoustic model is trained to obtain a target acoustic model.

[0039] In this embodiment, after obtaining the text training samples and the calibrated speech spectrum training samples, the above-mentioned execution subject can train the initial acoustic model based on the text training samples and the calibrated speech spectrum training samples to obtain the target acoustic model. Specifically, the following training steps can be performed: input the text training samples into the initial acoustic model, obtain the speech spectrum corresponding to each text training sample output by the initial acoustic model, compare the output speech spectrum with the calibrated speech spectrum training sample, obtain the prediction accuracy of the initial acoustic model, compare the prediction accuracy of the initial acoustic model with a pre-set accuracy threshold, illustratively, the pre-set accuracy threshold is 80%, if the prediction accuracy of the initial acoustic model is greater than the pre-set accuracy threshold, the initial acoustic model is determined as the target acoustic model, if the prediction accuracy of the initial acoustic model is less than the pre-set accuracy threshold, the parameters of the initial acoustic model are adjusted and training continues. The initial acoustic model can be an autoregressive speech synthesis model or a non-autoregressive speech synthesis model, and this disclosure does not limit this.

[0040] In some optional implementations of this embodiment, a text training sample can be input into an initial acoustic model. First, a semantic analysis is performed on the text training sample to obtain pronunciation prompts. Then, prosody processing is performed to obtain the pitch, length, intensity, etc. of each pronunciation to make the synthesized speech more natural. Finally, acoustic processing is performed to obtain a synthesized speech spectrum, which is output. The output speech spectrum is compared with the calibrated speech spectrum training sample to obtain the prediction accuracy of the initial acoustic model. The prediction accuracy of the initial acoustic model is compared with a pre-set accuracy threshold. If the prediction accuracy of the initial acoustic model is greater than the pre-set accuracy threshold, the initial acoustic model is determined as the target acoustic model. If the prediction accuracy of the initial acoustic model is less than the pre-set accuracy threshold, the parameters of the initial acoustic model are adjusted and training continues.

[0041] The acoustic modeling method provided by the embodiments of the present disclosure first obtains text training samples and corresponding speech spectrum training samples, then performs spectrum quality calibration on the speech spectrum training samples to obtain calibrated speech spectrum training samples, and finally trains an initial acoustic model based on the text training samples and the calibrated speech spectrum training samples to obtain a target acoustic model, so that the speech spectrum output by the target acoustic model is more accurate.

[0042] Further reference Figure 3 , which shows a process 300 of another embodiment of the acoustic modeling method according to the present disclosure. The acoustic modeling method includes the following steps:

[0043] Step 301: Obtain text training samples and corresponding speech spectrum training samples.

[0044] In this embodiment, the specific operation of step 301 has been Figure 2 Step 201 is described in detail in the illustrated embodiment and will not be repeated here.

[0045] Step 302: Perform spectrum quality calibration on the speech spectrum training sample to obtain a calibrated speech spectrum training sample, wherein the spectrum quality calibration includes a low-frequency attenuation operation and a spectrum correction operation.

[0046] In this embodiment, the execution subject can perform spectrum quality calibration on the speech spectrum training sample to obtain the calibrated speech spectrum training sample. The specific operation of performing spectrum quality calibration on the speech spectrum training sample to obtain the calibrated speech spectrum training sample has been described in detail. Figure 2Step 202 is described in detail in the illustrated embodiment and will not be repeated here. Specifically, spectrum quality calibration may include a low-frequency attenuation operation and a spectrum correction operation. The low-frequency attenuation operation is an operation for attenuating low-frequency signals in the spectrum. Low-frequency signals in the spectrum generally represent noise and hum. Attenuating low-frequency signals can reduce the noise and hum in the spectrum and improve the quality of the spectrum. The spectrum correction operation is an operation for correcting the energy of the spectrum. During the process of generating original speech, the loudness and pitch of some single tones may change due to environmental influences, thereby reducing the quality of the original speech. After the original speech is converted into a speech spectrum, the quality of the corresponding speech spectrum is also reduced. If the speech spectrum is then converted into speech, it may cause inaccurate speech translation. After performing the spectrum correction operation on the speech spectrum, the problematic spectrum can be repaired, so that when the repaired speech spectrum is converted into speech again, the accuracy of speech translation can be improved.

[0047] Step 303: Based on the text training samples and the calibrated speech spectrum training samples, an initial acoustic model is trained to obtain a target acoustic model.

[0048] In this embodiment, the specific operation of step 303 has been Figure 2 Step 203 is described in detail in the illustrated embodiment and will not be repeated here.

[0049] It should be noted that the speech spectrum training sample can be a speech spectrum with a total speech duration of any value. In some optional implementations of this embodiment, the total speech duration corresponding to the speech spectrum training sample can be less than a preset duration threshold, wherein the preset duration threshold can be a smaller value, so that the sample data used for model training is small sample data. For example, the preset duration threshold is one hour. Specifically, based on the need to use small samples for model training, the model structure of the initial acoustic model can be modified, and the modified initial acoustic model can be trained using speech spectrum training samples and corresponding text training samples that are less than the preset duration threshold. After the training is completed, the obtained initial acoustic model can achieve stable acoustic modeling using only a small number of training samples.

[0050] from Figure 3 It can be seen that Figure 2 Compared with the corresponding embodiments, the acoustic modeling method in this embodiment uses low-frequency attenuation operation and spectrum correction operation to calibrate the spectrum quality of the speech spectrum training samples, thereby improving the quality of the speech spectrum training samples, thereby making the speech spectrum output by the target acoustic model more accurate. At the same time, the model structure is improved so that the improved model can achieve stable modeling with only training samples less than a preset time length threshold, thereby improving the efficiency of acoustic modeling.

[0051] Further reference Figure 4 , which shows a process 400 of another embodiment of the acoustic modeling method according to the present disclosure. The acoustic modeling method includes the following steps:

[0052] Step 401: Obtain text training samples and corresponding speech spectrum training samples.

[0053] In this embodiment, the specific operation of step 401 has been Figure 2 Step 201 is described in detail in the illustrated embodiment and will not be repeated here.

[0054] Step 402: Obtain low-frequency signals in the speech spectrum training sample.

[0055] In this embodiment, the above-mentioned execution subject can obtain low-frequency signals in the speech spectrum training samples. Usually, speech signals below 500 Hz are low-frequency signals, and the sound frequency range of low-frequency noise is 20 Hz-200 Hz. Low-frequency noise in the speech spectrum training samples can be obtained, that is, low-frequency signals of 20 Hz-200 Hz are mainly obtained. A low-frequency threshold can be arbitrarily set within the range of 20 Hz-200 Hz, and the signal below the low-frequency threshold in the speech spectrum training sample is obtained as a low-frequency signal. It is also possible to first obtain the historical value of the low-frequency signal, average all the historical values, and use the signal below the average value as a low-frequency signal. It is also possible to use a default value as the basis and use the signal below the default value as a low-frequency signal. The present disclosure does not limit this.

[0056] Step 403: Attenuate the low-frequency signal to obtain an attenuated speech spectrum training sample.

[0057] In this embodiment, after obtaining the low-frequency signal, the execution entity may attenuate the low-frequency signal to obtain an attenuated speech spectrum training sample. Attenuation refers to the removal of low-frequency signals. Specifically, all low-frequency signals in the speech spectrum training sample may be obtained and their values ​​replaced with pre-calculated values. Alternatively, low-frequency signals in the speech spectrum training sample may be obtained and then smoothed to remove noise and other artifacts from the speech spectrum training sample. Attenuating the low-frequency signals in the speech spectrum training sample to obtain an attenuated speech spectrum training sample can prevent random noise from being generated in the synthesized speech, thereby improving the accuracy of the synthesized speech.

[0058] Step 404: Obtain spectrum boundaries in the attenuated speech spectrum training sample.

[0059] In this embodiment, the execution entity can obtain spectrum boundaries in the attenuated speech spectrum training sample. The attenuated speech spectrum training sample can be presented in the form of a spectrogram, where whiter colors represent greater speech signal energy, and darker colors represent lower speech signal energy. The boundaries of all white bright spots in the spectrogram can be found and determined as spectrum boundaries in the attenuated speech spectrum training sample.

[0060] Step 405: Correct the spectrum boundary to obtain a calibrated speech spectrum training sample.

[0061] In this embodiment, after obtaining the spectrum boundary, the above-mentioned execution subject can correct the spectrum boundary to obtain a calibrated speech spectrum training sample. If the speech spectrum training sample is a low-frequency speech spectrum between 20 Hz and 500 Hz, for example, the speech spectrum training sample represents a soft voice in the form of a whisper at 100 Hz, then the soft voice is very sensitive to the spectrum boundary. If the spectrum boundary is not corrected, it may cause inaccurate synthesized speech. Therefore, it is necessary to correct the spectrum boundary to improve the stability of acoustic modeling. Specifically, the attenuated speech spectrum training sample can be input into the spectrum correction program, and based on the obtained spectrum boundary, the energy within the white bright spot range is flattened as a whole, and the spectrum boundary is finely trimmed to obtain a calibrated speech spectrum training sample. Converting the calibrated speech spectrum training sample into speech can make the converted speech more accurate.

[0062] Step 406: Obtain an optimized loss function, wherein the optimized loss function includes loss functions for intermediate frequency signals and high frequency signals of the speech spectrum.

[0063] In this embodiment, the above-mentioned execution entity can obtain an optimized loss function. The loss function is a function used to evaluate the output results of the model. Exemplarily, the loss function can be a first loss function plus a second loss function, wherein the first loss function is the sum of all absolute differences between the calibrated speech spectrum training sample and the speech spectrum output by the initial acoustic model, and the second loss function is the sum of all square differences between the calibrated speech spectrum training sample and the speech spectrum output by the initial acoustic model. The optimized loss function is a loss function that includes the intermediate frequency signal and the high frequency signal for the speech spectrum, that is, the optimized loss function is the first loss function plus the second loss function plus the loss function for the intermediate frequency signal and the high frequency signal for the speech spectrum, wherein the intermediate frequency signal is a speech signal of 500 Hz-2000 Hz, and the high frequency signal is a speech signal of 2000 Hz-16000 Hz. Specifically, the calibrated speech spectrum training samples and the speech spectrum output by the initial acoustic model are both a Mel spectrum that can be used to represent short-term audio information. The Mel spectrum uses short-time Fourier transform to map the speech signal from a linear frequency scale to a logarithmic Mel scale to obtain a multidimensional feature vector. Generally, an 80-dimensional feature vector can be used to represent the Mel spectrum, because the 80-dimensional feature vector can roughly represent the distribution of signal energy on the Mel scale frequency, where the 30th to 70th dimensions represent the intermediate frequency signal and the high frequency signal. The optimization loss function is the first loss function of the 1st to 80th dimensions plus the second loss function, plus the loss function of the 30th to 70th dimensions.

[0064] Step 407: Based on the text training samples, the calibrated speech spectrum training samples and the optimized loss function, an initial acoustic model is trained to obtain a target acoustic model.

[0065] In this embodiment, the above-mentioned execution subject can train the initial acoustic model based on the text training samples, the calibrated speech spectrum training samples and the optimized loss function to obtain the target acoustic model. Specifically, the text training samples can be input into the initial acoustic model to obtain the speech spectrum corresponding to each text training sample output by the initial acoustic model. Based on the output speech spectrum and the calibrated speech spectrum training samples, the value corresponding to the optimized loss function is calculated, and the value corresponding to the optimized loss function is compared with a pre-set accuracy threshold. For example, the pre-set accuracy threshold is 80%. If the value corresponding to the optimized loss function is greater than the pre-set accuracy threshold, the initial acoustic model is determined as the target acoustic model. If the value corresponding to the optimized loss function is less than the pre-set accuracy threshold, the parameters of the initial acoustic model are adjusted and training continues.

[0066] from Figure 4 It can be seen that Figure 3Compared with the corresponding embodiments, the acoustic modeling method in this embodiment attenuates the low-frequency signals in the speech spectrum training samples, so that the attenuated speech spectrum training samples can avoid random noise in the synthesized speech, improve the accuracy of the synthesized speech, correct the spectrum boundaries, improve the stability of the acoustic modeling, and construct an optimized loss function, thereby further improving the accuracy of the speech spectrum output by the target acoustic model.

[0067] Further referring to Figure 5, Figure 5(a) is a schematic diagram of the speech spectrum training sample before calibration of the present invention, and Figure 5(b) is a schematic diagram of the speech spectrum training sample after calibration of the present invention. As shown in Figure 5, the whiter the color in the figure, the greater the energy of the speech signal, and the darker the color, the smaller the energy of the speech signal. Figure 5 represents the speech spectrum from high frequency to low frequency from top to bottom. Comparing the low-frequency spectra in the boxes below Figure 5(a) and Figure 5(b), it can be seen that the calibrated speech spectrum training sample in Figure 5(b) attenuates the low-frequency signal. Compared with Figure 5(a), it can avoid the generation of noise in the synthesized speech. Further comparing the distribution of white bright spots in Figure 5(a) and Figure 5(b), it can be seen that after the spectrum boundary is corrected, Figure 5(b) achieves the correction of the spectrum boundary and energy, thereby improving the stability of the modeling.

[0068] Further reference Figure 6 , which shows a process 600 of an embodiment of a voice conversion method according to the present disclosure. The voice conversion method includes the following steps:

[0069] Step 601: Obtain the text to be converted.

[0070] In this embodiment, the execution entity can obtain a text to be converted. The text to be converted can be of any length and content, and can be in Chinese, English, or Korean, without limitation in this disclosure. A text can be extracted from a book or other existing text as the text to be converted, or a text can be written as the text to be converted, without limitation in this disclosure.

[0071] Step 602: Based on the target acoustic model, obtain the speech spectrum corresponding to the text to be converted.

[0072] In this embodiment, after the execution entity obtains the text to be converted, it can obtain the speech spectrum corresponding to the text to be converted based on the target acoustic model. The target acoustic model is a pre-trained acoustic model that can obtain the correspondence between text and speech spectrum. Specifically, based on the target acoustic model, the speech spectrum corresponding to each word in the text to be converted can be obtained, and the speech spectrum corresponding to all words in the text to be converted can be determined as the speech spectrum corresponding to the text to be converted.

[0073] Step 603: Convert the speech spectrum into speech and output the speech.

[0074] In this embodiment, the execution entity can convert the speech spectrum into speech and output the speech. Specifically, a speech decoding program can be called to convert the speech spectrum into speech. After obtaining the speech, a play instruction can be sent to the speaker control program, and the converted speech signal can be transmitted to the playback program. After receiving the play instruction, the speaker control program controls the playback program to play the converted speech.

[0075] from Figure 6 As can be seen, the speech conversion method in this embodiment first obtains the text to be converted, then obtains the speech spectrum corresponding to the text to be converted based on the target acoustic model, and finally converts the speech spectrum into speech and outputs the speech. Converting text to speech using the target acoustic model improves the accuracy of the text-to-speech conversion.

[0076] Further reference Figure 7 , which shows a process 700 of another embodiment of the voice conversion method according to the present disclosure. The voice conversion method includes the following steps:

[0077] Step 701: Obtain the text to be converted.

[0078] In this embodiment, the specific operation of step 701 has been Figure 6 Step 601 is described in detail in the illustrated embodiment and will not be repeated here.

[0079] Step 702: Input the text to be converted into the target acoustic model to obtain the corresponding speech spectrum, or construct a speech spectrum library based on the target acoustic model and obtain the speech spectrum corresponding to the text to be converted from the speech spectrum library.

[0080] In this embodiment, the above-mentioned execution subject can input the text to be converted into the target acoustic model to obtain the corresponding speech spectrum. Specifically, the text to be converted is input as input data to the input end of the target acoustic model, and the corresponding speech spectrum is output from the output end of the target acoustic model. The above-mentioned execution subject can also construct a speech spectrum library based on the target acoustic model, and obtain the speech spectrum corresponding to the text to be converted from the speech spectrum library. Specifically, by inputting any text into the pre-trained target acoustic model, the corresponding speech spectrum can be accurately output. Therefore, multiple different texts can be input into the pre-trained target acoustic model in advance, and the correspondence between different texts and speech spectra can be obtained from the pre-trained target acoustic model. The correspondence is determined as a speech spectrum library. After obtaining the text to be converted, the speech spectrum corresponding to each word of the text to be converted is found from the speech spectrum library, and the speech spectra of all words of the text to be converted are spliced ​​to obtain the speech spectrum corresponding to the text to be converted.

[0081] Step 703: Convert the speech spectrum into speech and output the speech.

[0082] In this embodiment, the specific operation of step 703 has been Figure 6 Step 603 is described in detail in the illustrated embodiment and will not be repeated here.

[0083] from Figure 7 It can be seen that Figure 6 Compared with the corresponding embodiments, the speech conversion method in this embodiment inputs the text to be converted into the target acoustic model to obtain the corresponding speech spectrum, or builds a speech spectrum library based on the target acoustic model, obtains the speech spectrum corresponding to the text to be converted from the speech spectrum library, and then converts the speech spectrum into speech, making the speech conversion method more flexible.

[0084] Further references Figure 8 As an implementation of the above acoustic modeling method, the present disclosure provides an embodiment of an acoustic modeling device. Figure 2 Corresponding to the method embodiment shown, the device can be specifically applied to various electronic devices.

[0085] like Figure 8 As shown, the acoustic modeling device 800 of this embodiment may include a first acquisition module 801, a calibration module 802, and a training module 803. The first acquisition module 801 is configured to acquire text training samples and corresponding speech spectrum training samples; the calibration module 802 is configured to perform spectrum quality calibration on the speech spectrum training samples to obtain calibrated speech spectrum training samples; and the training module 803 is configured to train an initial acoustic model based on the text training samples and the calibrated speech spectrum training samples to obtain a target acoustic model.

[0086] In this embodiment, the acoustic modeling device 800: the specific processing of the first acquisition module 801, the calibration module 802 and the training module 803 and the technical effects thereof can be referred to respectively. Figure 2 The relevant descriptions of steps 201-203 in the corresponding embodiment are not repeated here.

[0087] In some optional implementations of this embodiment, the spectrum quality calibration in the calibration module 802 includes a low-frequency attenuation operation and a spectrum correction operation.

[0088] In some optional implementations of this embodiment, the calibration module 802 includes: a first acquisition submodule, configured to acquire a low-frequency signal in a speech spectrum training sample; an attenuation submodule, configured to attenuate the low-frequency signal to obtain an attenuated speech spectrum training sample; a second acquisition submodule, configured to acquire a spectrum boundary in the attenuated speech spectrum training sample; and a correction submodule, configured to correct the spectrum boundary to obtain a calibrated speech spectrum training sample.

[0089] In some optional implementations of this embodiment, the training module 803 includes: a third acquisition submodule, configured to obtain an optimized loss function, wherein the optimized loss function includes loss functions for intermediate frequency signals and high frequency signals of the speech spectrum; a training submodule, configured to train an initial acoustic model based on text training samples, calibrated speech spectrum training samples and the optimized loss function to obtain a target acoustic model.

[0090] In some optional implementations of this embodiment, the total speech duration corresponding to the speech spectrum training samples in the acoustic modeling device is less than a preset duration threshold.

[0091] Further references Figure 9 As an implementation of the above-mentioned voice conversion method, the present disclosure provides an embodiment of a voice conversion device. Figure 6 Corresponding to the method embodiment shown, the device can be specifically applied to various electronic devices.

[0092] like Figure 9 As shown, the speech conversion apparatus 900 of this embodiment may include a second acquisition module 901, a third acquisition module 902, and a conversion module 903. The second acquisition module 901 is configured to acquire the text to be converted; the third acquisition module 902 is configured to obtain the speech spectrum corresponding to the text to be converted based on the target acoustic model; and the conversion module 903 is configured to convert the speech spectrum into speech and output the speech.

[0093] In this embodiment, the speech conversion device 900: the specific processing of the second acquisition module 901, the third acquisition module 902 and the conversion module 903 and the technical effects thereof can be referred to respectively. Figure 6 The relevant descriptions of steps 601-603 in the corresponding embodiment are not repeated here.

[0094] In some optional implementations of this embodiment, the third acquisition module 902 includes: an input submodule, configured to input the text to be converted into the target acoustic model to obtain the corresponding speech spectrum; or, a construction submodule, configured to construct a speech spectrum library based on the target acoustic model, and obtain the speech spectrum corresponding to the text to be converted from the speech spectrum library.

[0095] According to an embodiment of the present disclosure, the present disclosure also provides an electronic device, a readable storage medium, and a computer program product.

[0096] Figure 10 A schematic block diagram of an example electronic device 1000 that can be used to implement embodiments of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital assistants, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present disclosure described and / or claimed herein.

[0097] like Figure 10 As shown, the device 1000 includes a computing unit 1001, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 1002 or a computer program loaded from a storage unit 1008 into a random access memory (RAM) 1003. Various programs and data required for the operation of the device 1000 can also be stored in the RAM 1003. The computing unit 1001, the ROM 1002, and the RAM 1003 are connected to each other via a bus 1004. An input / output (I / O) interface 1005 is also connected to the bus 1004.

[0098] Various components in device 1000 are connected to I / O interface 1005, including an input unit 1006, such as a keyboard, mouse, etc.; an output unit 1007, such as various types of displays, speakers, etc.; a storage unit 1008, such as a magnetic disk, optical disk, etc.; and a communication unit 1009, such as a network card, modem, wireless communication transceiver, etc. The communication unit 1009 allows device 1000 to exchange information / data with other devices via a computer network such as the Internet and / or various telecommunication networks.

[0099] The computing unit 1001 can be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the computing unit 1001 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units that run machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc. The computing unit 1001 performs the various methods and processes described above, such as the acoustic modeling method or the voice conversion method. For example, in some embodiments, the acoustic modeling method or the voice conversion method can be implemented as a computer software program that is tangibly contained in a machine-readable medium, such as the storage unit 1008. In some embodiments, part or all of the computer program can be loaded and / or installed on the device 1000 via the ROM 1002 and / or the communication unit 1009. When the computer program is loaded into the RAM 1003 and executed by the computing unit 1001, one or more steps of the acoustic modeling method or the voice conversion method described above can be performed. Alternatively, in other embodiments, the computing unit 1001 may be configured to execute the acoustic modeling method or the speech conversion method in any other appropriate manner (for example, by means of firmware).

[0100] Various embodiments of the systems and techniques described herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), system-on-chip systems (SOCs), programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments can include being implemented in one or more computer programs that are executable and / or interpreted on a programmable system that includes at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0101] The program code for implementing the method of the present disclosure can be written in any combination of one or more programming languages. These program codes can be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device so that when the program code is executed by the processor or controller, the functions / operations specified in the flow chart and / or block diagram are implemented. The program code can be executed entirely on the machine, partially on the machine, as a stand-alone software package, partially on the machine and partially on a remote machine, or entirely on a remote machine or server.

[0102] In the context of the present disclosure, a machine-readable medium can be a tangible medium that can contain or store a program for use by or in conjunction with an instruction execution system, device or equipment. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium can include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0103] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0104] The systems and techniques described herein can be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer having a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network (LAN), a wide area network (WAN), and the Internet.

[0105] A computer system may include a client and a server. The client and server are generally remote from each other and typically interact via a communication network. The client-server relationship arises through computer programs running on the respective computers and having a client-server relationship with each other. The server may be a server in a distributed system or a server integrated with blockchain. The server may also be a cloud server, or an intelligent cloud computing server or intelligent cloud host with artificial intelligence technology. The server may be a server in a distributed system or a server integrated with blockchain. The server may also be a cloud server, or an intelligent cloud computing server or intelligent cloud host with artificial intelligence technology.

[0106] It should be understood that the various forms of the processes shown above can be used to reorder, add, or delete steps. For example, the steps described in this disclosure can be performed in parallel, sequentially, or in a different order, as long as the desired results of the technical solutions disclosed in this disclosure can be achieved. This is not a limitation herein.

[0107] The above specific embodiments do not constitute a limitation on the scope of protection of this disclosure. Those skilled in the art will appreciate that various modifications, combinations, sub-combinations, and substitutions may be made based on design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this disclosure shall be included within the scope of protection of this disclosure.

Claims

1. An acoustic modeling method, comprising: Obtain text training samples and corresponding speech spectrum training samples; Acquiring a low-frequency signal from the speech spectrum training sample; Attenuating the low-frequency signal to obtain an attenuated speech spectrum training sample; Obtaining spectrum boundaries in the attenuated speech spectrum training sample; Correcting the spectrum boundary to obtain a calibrated speech spectrum training sample; Based on the text training samples and the calibrated speech spectrum training samples, an initial acoustic model is trained to obtain a target acoustic model.

2. The method according to claim 1, wherein The training of the initial acoustic model based on the text training sample and the calibrated speech spectrum training sample to obtain the target acoustic model includes: Obtaining an optimized loss function, wherein the optimized loss function includes loss functions for intermediate frequency signals and high frequency signals of a speech spectrum; Based on the text training samples, the calibrated speech spectrum training samples and the optimized loss function, the initial acoustic model is trained to obtain the target acoustic model.

3. The method according to any one of claims 1 to 2, wherein: The total speech duration corresponding to the speech spectrum training sample is less than a preset duration threshold.

4. A voice conversion method, comprising: Get the text to be converted; Obtaining a speech spectrum corresponding to the text to be converted based on a target acoustic model, wherein the target acoustic model is constructed by the acoustic modeling method according to any one of claims 1 to 3; The speech spectrum is converted into speech, and the speech is output.

5. The method according to claim 4, wherein The obtaining of a speech spectrum corresponding to the text to be converted based on the target acoustic model includes: Input the text to be converted into the target acoustic model to obtain the corresponding speech spectrum; or A speech spectrum library is constructed based on the target acoustic model, and a speech spectrum corresponding to the text to be converted is obtained from the speech spectrum library.

6. An acoustic modeling device, comprising: A first acquisition module is configured to acquire a text training sample and a corresponding speech spectrum training sample; The calibration module includes: a first acquisition submodule configured to acquire a low-frequency signal from the speech spectrum training sample; an attenuation submodule configured to attenuate the low-frequency signal to obtain an attenuated speech spectrum training sample; a second acquisition submodule configured to acquire a spectrum boundary from the attenuated speech spectrum training sample; and a correction submodule configured to correct the spectrum boundary to obtain a calibrated speech spectrum training sample. The training module is configured to train an initial acoustic model based on the text training sample and the calibrated speech spectrum training sample to obtain a target acoustic model.

7. The device according to claim 6, wherein The training module includes: A third acquisition submodule is configured to acquire an optimized loss function, wherein the optimized loss function includes loss functions for intermediate frequency signals and high frequency signals of the speech spectrum; The training submodule is configured to train the initial acoustic model based on the text training sample, the calibrated speech spectrum training sample and the optimization loss function to obtain the target acoustic model.

8. The device according to any one of claims 6 to 7, wherein: The total speech duration corresponding to the speech spectrum training sample is less than a preset duration threshold.

9. A speech conversion device, comprising: A second acquisition module is configured to acquire the text to be converted; a third acquisition module, configured to obtain a speech spectrum corresponding to the to-be-converted text based on a target acoustic model, wherein the target acoustic model is constructed by the acoustic modeling method according to any one of claims 1 to 3; The conversion module is configured to convert the speech spectrum into speech and output the speech.

10. The device according to claim 9, wherein The third acquisition module includes: An input submodule is configured to input the text to be converted into the target acoustic model to obtain a corresponding speech spectrum; or The construction submodule is configured to construct a speech spectrum library based on the target acoustic model, and obtain the speech spectrum corresponding to the text to be converted from the speech spectrum library.

11. An electronic device comprising: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method according to any one of claims 1 to 5.

12. A non-transitory computer-readable storage medium storing computer instructions, wherein: The computer instructions are used to cause the computer to execute the method according to any one of claims 1 to 5.

13. A computer program product comprising a computer program, which, when executed by a processor, implements the method according to any one of claims 1 to 5.

Images