Speech synthesis method combining emotion intensity, electronic device and readable storage medium

Abstract

The present application relates to the field of artificial intelligence technology, and provides a speech synthesis method, device, electronic device, and computer-readable storage medium that combine emotional intensity. The method includes: obtaining a speech signal and a synthesized text corresponding to the speech signal; performing retrieval processing on the synthesized text to obtain a phoneme sequence and note pitch features; inputting the text features into a pitch encoding model for conversion processing to obtain text pitch information; inputting the second pitch feature information and the text pitch information into a residual pitch prediction network model for a first iterative prediction process to obtain effective pitch information; and inputting the second duration information into a pre-trained residual duration prediction network model for a second iterative prediction process to obtain an effective duration; decoding the synthesized pitch information to obtain a synthesized speech mel-spectrogram. The above technical solution can combine emotional expression in the process of speech synthesis, making the speech synthesis effect better.

Description

Technical Field

[0001] The embodiments of the present application relate to, but are not limited to, the field of artificial intelligence technology, and in particular to a speech synthesis method, device, electronic device, and computer-readable storage medium that combine emotion intensity. Background Art

[0002] In the existing technology, speech synthesis technology has been rapidly developed and has been widely used in fields such as smart speakers, virtual robots and telephone robots. In the financial industry, virtual robots have been increasingly used to answer questions asked by customers. In traditional technical speech synthesis systems, audio emotion features are extracted from reference audio and added to the system, so as to control the emotion of the synthesized audio to be close to the emotion of the reference audio. However, in the current speech synthesis process, people often only pay more attention to whether the speech synthesis results are clear and accurate, and often cannot adjust the synthesized speech according to the speaker's emotions, lacking emotional expression. This will make users feel that the speech synthesis effect is not perfect and the emotional expression granularity is not fine enough. Summary of the Invention

[0003] The following is a summary of the subject matter described in detail herein. This summary is not intended to limit the scope of the claims.

[0004] In order to solve the problems mentioned in the above background technology, the embodiments of the present application provide a speech synthesis method, device, electronic device and computer-readable storage medium that combine emotional intensity, which can combine emotional expression in the process of speech synthesis, making the speech synthesis effect better.

[0005] In a first aspect, an embodiment of the present application provides a speech synthesis method incorporating emotion intensity, the method comprising:

[0006] Acquire a speech signal and a synthesized text corresponding to the speech signal;

[0007] Performing a search process on the synthesized text to obtain a phoneme sequence and a note pitch feature; and performing a feature extraction process on the speech signal to obtain first pitch feature information and first duration information;

[0008] Combining the phoneme sequence and the note pitch feature to obtain a text feature; and embedding the first pitch feature information and the first duration information with emotion information corresponding to the speech signal to obtain second pitch feature information corresponding to the first pitch feature information and second duration information corresponding to the first duration information;

[0009] Inputting the text features into a pre-trained pitch coding model for conversion processing to obtain text pitch information;

[0010] Inputting the second pitch feature information and the text pitch information into a pre-trained residual pitch prediction network model for a first iterative prediction process to obtain effective pitch information; and inputting the second duration information into a pre-trained residual duration prediction network model for a second iterative prediction process to obtain effective duration;

[0011] Combining the effective pitch information and the effective duration to obtain synthesized pitch information;

[0012] The synthesized pitch information is decoded to obtain a synthesized speech mel-spectrogram.

[0013] According to some embodiments of the present application, the step of performing retrieval processing on the synthesized text to obtain a phoneme sequence and a note pitch feature includes:

[0014] Performing matching processing on the synthesized text based on a preset embedding table to obtain text matching information;

[0015] The phoneme sequence and the note pitch feature are extracted from the embedding table according to the text matching information.

[0016] According to some embodiments of the present application, performing feature extraction processing on the speech signal to obtain first pitch feature information and first duration information includes:

[0017] Performing pitch feature quantization processing on the speech signal to obtain the first pitch feature information;

[0018] The pitch duration calculation process is performed on the speech signal to obtain the first duration information.

[0019] According to some embodiments of the present application, inputting the text features into a pre-trained pitch coding model for conversion processing to obtain text pitch information includes:

[0020] Regularizing the text features based on the pitch coding model to obtain regularized text;

[0021] Performing pitch conversion processing on the regularized text to obtain the text pitch information.

[0022] According to some embodiments of the present application, before decoding the synthesized pitch information to obtain a synthesized speech mel-spectrogram, the method further includes:

[0023] The synthesized pitch information is length-normalized according to a preset Mel-frame length.

[0024] According to some embodiments of the present application, after embedding the emotion information corresponding to the speech signal into the first pitch feature information and the first duration information respectively to obtain second pitch feature information corresponding to the first pitch feature information and second duration information corresponding to the first duration information, the method further includes:

[0025] Acquiring pitch information of the speech signal;

[0026] The tone information is embedded in the second pitch feature information and the second duration information respectively.

[0027] According to some embodiments of the present application, after embedding the emotion information corresponding to the speech signal into the first pitch feature information and the first duration information respectively to obtain second pitch feature information corresponding to the first pitch feature information and second duration information corresponding to the first duration information, the method further includes:

[0028] Acquiring energy information of the speech signal;

[0029] The energy information is embedded into the second pitch feature information and the second duration information respectively.

[0030] In a second aspect, an embodiment of the present application further provides a speech synthesis device incorporating emotion intensity, the device comprising:

[0031] A first processing module, configured to obtain a speech signal and a synthesized text corresponding to the speech signal;

[0032] A second processing module is configured to perform a search process on the synthesized text to obtain a phoneme sequence and a note pitch feature; and perform a feature extraction process on the speech signal to obtain first pitch feature information and first duration information;

[0033] a third processing module, configured to combine the phoneme sequence and the note pitch feature to obtain a text feature; and embed the first pitch feature information and the first duration information into the emotion information corresponding to the speech signal to obtain second pitch feature information corresponding to the first pitch feature information and second duration information corresponding to the first duration information;

[0034] A fourth processing module is used to input the text features into a pre-trained pitch coding model for conversion processing to obtain text pitch information;

[0035] a fifth processing module, configured to input the second pitch feature information and the text pitch information into a pre-trained residual pitch prediction network model for a first iterative prediction process to obtain effective pitch information; and input the second duration information into a pre-trained residual duration prediction network model for a second iterative prediction process to obtain effective duration;

[0036] a sixth processing module, configured to combine the effective pitch information and the effective duration to obtain synthesized pitch information;

[0037] The seventh processing module is used to decode the synthesized pitch information to obtain a synthesized speech mel-spectrogram.

[0038] In a third aspect, an embodiment of the present application further provides an electronic device comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the speech synthesis method combined with emotion intensity as described in the first aspect above is implemented.

[0039] In a fourth aspect, an embodiment of the present application further provides a computer-readable storage medium storing computer-executable instructions, wherein the computer-executable instructions are used to execute the speech synthesis method combined with emotion intensity as described in the first aspect above.

[0040] The speech synthesis method combined with emotion intensity according to the embodiment provided by the present application has at least the following beneficial effects: first, a speech signal and a synthesized text corresponding to the speech signal are obtained; then, the synthesized text is retrieved and processed to obtain a phoneme sequence and a note pitch feature; and a feature extraction process is performed on the speech signal to obtain first pitch feature information and first duration information; then, the phoneme sequence and the note pitch feature are combined and processed to obtain text features; and the first pitch feature information and the first duration information are respectively embedded with the emotion information corresponding to the speech signal to obtain second pitch feature information corresponding to the first pitch feature information and second duration information corresponding to the first duration information; then, the text features are input into a pre-trained pitch encoding model for conversion processing to obtain text pitch information; then, the second pitch feature information and the text pitch information are input into a pre-trained residual pitch prediction network model for a first iterative prediction process to obtain effective pitch information; and the second duration information is input into a pre-trained residual duration prediction network model for a second iterative prediction process to obtain effective duration; then, the effective pitch information and the effective duration are combined and processed to obtain synthesized pitch information; and finally, the synthesized pitch information is decoded to obtain a synthesized speech mel-spectrogram. Through the above technical solution, emotional expression can be combined in the process of speech synthesis, making the speech synthesis effect better. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] The accompanying drawings are used to provide a further understanding of the technical solution of the present application and constitute a part of the specification. Together with the embodiments of the present application, they are used to explain the technical solution of the present application and do not constitute a limitation on the technical solution of the present application.

[0042] Figure 1 This is a flowchart of a speech synthesis method combining emotional intensity provided by an embodiment of the present application;

[0043] Figure 2 This is a flowchart of performing retrieval processing on synthesized text in a speech synthesis method combined with emotion intensity provided by an embodiment of the present application;

[0044] Figure 3 This is a flowchart of extracting features from a speech signal in a speech synthesis method combined with emotion intensity provided by an embodiment of the present application;

[0045] Figure 4 This is a flowchart of converting text features in a speech synthesis method combined with emotion intensity provided by an embodiment of the present application;

[0046] Figure 5 Another embodiment of the present application provides a speech synthesis method combining emotional intensity;

[0047] Figure 6 Another embodiment of the present application provides a speech synthesis method combining emotional intensity;

[0048] Figure 7 Another embodiment of the present application provides a speech synthesis method combining emotional intensity;

[0049] Figure 8 This is a schematic diagram of a speech synthesis device incorporating emotional intensity provided by one embodiment of the present application;

[0050] Figure 9 This is a schematic diagram of an electronic device provided by an embodiment of the present application. DETAILED DESCRIPTION

[0051] In order to make the purpose, technical solutions and advantages of this application more clear, the following further describes this application in detail with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain this application and are not intended to limit this application.

[0052] It should be noted that although the device schematics illustrate functional module divisions and the flowcharts illustrate logical sequences, in certain circumstances, the steps shown or described may be performed in a sequence that differs from the module divisions in the device or the sequence in the flowcharts. The terms "first," "second," and so on, used in the specification, claims, and drawings, are used to distinguish similar items and are not necessarily used to describe a specific sequence or precedence.

[0053] It should be noted that, unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by those skilled in the art to which this application belongs. The terms used herein are only for the purpose of describing the embodiments of this application and are not intended to limit this application.

[0054] The embodiments of the present application can acquire and process relevant data based on artificial intelligence technology. Artificial Intelligence (AI) is the theory, method, technology, and application system that uses digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceive the environment, acquire knowledge, and use knowledge to achieve optimal results.

[0055] AI is a new technical discipline that studies and develops theories, methods, technologies, and application systems for simulating, extending, and expanding human intelligence. Artificial intelligence is a branch of computer science that seeks to understand the essence of intelligence and produce new intelligent machines that can respond in a manner similar to human intelligence. Research in this field includes robotics, speech recognition, image recognition, natural language processing, and expert systems. Artificial intelligence can simulate the information processes of human consciousness and thinking. It also refers to the theories, methods, technologies, and application systems that use digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceive the environment, acquire knowledge, and use that knowledge to achieve optimal results.

[0056] Fundamental AI technologies generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing, operating / interaction systems, and mechatronics. AI software technologies primarily encompass computer vision, robotics, biometrics, speech processing, natural language processing, and machine learning / deep learning.

[0057] Artificial intelligence, or AI, is a theory, method, technology, and application system that uses digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceive the environment, acquire knowledge, and use knowledge to achieve optimal results.

[0058] The servers involved in artificial intelligence technology can be independent servers or cloud servers that provide basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content delivery networks (CDNs), as well as big data and artificial intelligence platforms.

[0059] The present application provides a speech synthesis method, device, electronic device and computer-readable storage medium that combine emotion intensity. The method comprises the following steps: first, obtaining a speech signal and a synthesized text corresponding to the speech signal; then performing a search process on the synthesized text to obtain a phoneme sequence and a note pitch feature; and performing a feature extraction process on the speech signal to obtain first pitch feature information and first duration information; then combining the phoneme sequence and the note pitch feature to obtain text features; and respectively embedding the first pitch feature information and the first duration information into the emotion information corresponding to the speech signal to obtain second pitch feature information corresponding to the first pitch feature information and second duration information corresponding to the first duration information; then inputting the text features into a pre-trained pitch encoding model for conversion processing to obtain text pitch information; then inputting the second pitch feature information and the text pitch information into a pre-trained residual pitch prediction network model for a first iterative prediction process to obtain effective pitch information; and inputting the second duration information into a pre-trained residual duration prediction network model for a second iterative prediction process to obtain effective duration; then combining the effective pitch information and the effective duration to obtain synthesized pitch information; and finally, decoding the synthesized pitch information to obtain a synthesized speech mel-spectrogram. Through the above technical solution, emotional expression can be combined in the process of speech synthesis, making the speech synthesis effect better.

[0060] The speech synthesis method combined with emotional intensity provided in the embodiment of the present application relates to the field of artificial intelligence technology. The speech synthesis method combined with emotional intensity provided in the embodiment of the present application can be applied in a terminal, can be applied in a server side, and can also be software running in a terminal or a server side. In some embodiments, the terminal can be a smart phone, a tablet computer, a laptop computer, a desktop computer, etc.; the server side can be configured as an independent physical server, or as a server cluster or distributed system composed of multiple physical servers, or as a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms; the software can be an application that implements the speech synthesis method combined with emotional intensity, etc., but is not limited to the above forms.

[0061] The present application can be used in many general or special computer system environments or configurations. For example: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputers, mainframe computers, distributed computing environments including any of the above systems or devices, and the like. The present application can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, and the like that perform specific tasks or implement specific abstract data types. The present application can also be practiced in distributed computing environments in which tasks are performed by remote processing devices connected via a communication network. In a distributed computing environment, program modules can be located in local and remote computer storage media, including storage devices.

[0062] It should be noted that in each specific embodiment of the present application, when it comes to the need to perform relevant processing based on data related to the user's identity or characteristics, such as user information, user behavior data, user historical data, and user location information, the user's permission or consent will be obtained first, and the collection, use, and processing of such data will comply with relevant laws, regulations, and standards. In addition, when the embodiment of the present application needs to obtain the user's sensitive personal information, the user's separate permission or consent will be obtained through a pop-up window or by jumping to a confirmation page. After clearly obtaining the user's separate permission or consent, the necessary user-related data for the normal operation of the embodiment of the present application will be obtained.

[0063] The embodiments of the present application are further described below with reference to the accompanying drawings.

[0064] like Figure 1 As shown, Figure 1 This is a flowchart of a speech synthesis method combined with emotion intensity provided by an embodiment of the present application. The speech synthesis method combined with emotion intensity includes but is not limited to steps S100 to S700.

[0065] Step S100, obtaining a speech signal and a synthesized text corresponding to the speech signal;

[0066] Step S200: performing a search process on the synthesized text to obtain a phoneme sequence and a note pitch feature; and performing a feature extraction process on the speech signal to obtain first pitch feature information and first duration information;

[0067] Step S300: Combining the phoneme sequence and the note pitch feature to obtain a text feature; and embedding the first pitch feature information and the first duration information with emotion information corresponding to the speech signal to obtain second pitch feature information corresponding to the first pitch feature information and second duration information corresponding to the first duration information;

[0068] Step S400: Inputting text features into a pre-trained pitch coding model for conversion processing to obtain text pitch information;

[0069] Step S500: Input the second pitch feature information and the text pitch information into a pre-trained residual pitch prediction network model for a first iterative prediction process to obtain effective pitch information; and input the second duration information into a pre-trained residual duration prediction network model for a second iterative prediction process to obtain effective duration;

[0070] Step S600, combining the effective pitch information and the effective duration to obtain synthesized pitch information;

[0071] Step S700: Decode the synthesized pitch information to obtain a synthesized speech mel-spectrogram.

[0072] It should be noted that, first, a speech signal and a synthetic text corresponding to the speech signal are obtained; then the synthetic text is retrieved and processed to obtain a phoneme sequence and note pitch features; and the speech signal is feature extracted to obtain first pitch feature information and first duration information; then the phoneme sequence and note pitch features are combined to obtain text features; and the first pitch feature information and the first duration information are respectively embedded with the emotional information corresponding to the speech signal to obtain second pitch feature information corresponding to the first pitch feature information and second duration information corresponding to the first duration information; then the text features are input into a pre-trained pitch encoding model for conversion processing to obtain text pitch information; then the second pitch feature information and the text pitch information are input into a pre-trained residual pitch prediction network model for a first iterative prediction processing to obtain effective pitch information; and the second duration information is input into a pre-trained residual duration prediction network model for a second iterative prediction processing to obtain effective duration; then the effective pitch information and the effective duration are combined to obtain synthetic pitch information; finally, the synthetic pitch information is decoded to obtain a synthetic speech Mel-spectrogram. Through the above technical solution, emotional expression can be combined in the process of speech synthesis, making the speech synthesis effect better.

[0073] It should be noted that the voice signal can be obtained through a voice collection device, wherein the voice collection device can be a voice recorder, a mobile phone, a microphone, or other electronic device with a voice collection function. The synthesized text corresponding to the voice signal can be a musical score of a song, which includes lyrics and the corresponding notes of the lyrics.

[0074] It's worth noting that a phoneme is the smallest unit of speech derived from a syllable's timbre. It's the most singular sound produced during speech, and it can't be further broken down. Phonemes are generally divided into two categories: vowels and consonants. Pitch is determined by the vibration frequency of an object, and the vibration distribution of modern pitch is based on the twelve-tone equal temperament.

[0075] It can be understood that the emotion information corresponding to the speech signal is embedded in the first pitch feature information and the first duration information respectively, so that the second pitch feature information and the second duration information both contain information related to the emotion intensity.

[0076] It is worth noting that the text pitch information can be obtained by inputting the text features into a pre-trained pitch coding model for conversion processing; the pitch coding model can be a neural network model, and its internal network parameters are corrected after training processing to facilitate subsequent conversion processing of the text features.

[0077] It's worth noting that both the residual pitch prediction network and the residual duration prediction network models can be residual neural network models. To address the degradation problem in deep networks, some layers of the neural network can be artificially skipped, allowing connections between neurons in the next layer to be made, thus weakening the strong connections between layers. This type of neural network is called a residual network. Consider the following: Consider a shallow network and want to build a deeper network by stacking new layers. An extreme case is that these added layers learn nothing but simply replicate the features of the shallow network, meaning that the new layers are identity mappings. In this case, the deeper network should perform at least as well as the shallow network and should not exhibit degradation. For deep neural networks, if the added layers can be trained to the identity mapping, the new model will be as effective as the original. Furthermore, since the new model may yield a better solution to the training dataset, adding layers seems to be more likely to reduce training error. The core idea of ​​residual networks is that each additional layer should easily contain the original function as one of its elements.

[0078] It is worth noting that the Mel spectrum is a general spectrum graph plus a Mel filter function. The Mel spectrum is intended to simulate the sensitivity of the human ear to actual frequencies. The importance of the Mel spectrum for speech synthesis is particularly important in today's era when deep learning is widely popular. Most mainstream speech synthesis systems based on deep learning generally use the Mel spectrum as an intermediate representation for model generation. In signal processing, the Mel inverse spectrum is a spectrum that can be used to represent short-term audio signals. Its principle is based on the logarithmic spectrum represented by the nonlinear Mel scale and its linear cosine transform. It is worth noting that the first Mel spectrum and the second Mel spectrum in the embodiment of the present application are only used to distinguish different objects in order to facilitate the explanation of the embodiment of the present application, and do not mean that the composition of the two is different.

[0079] It is worth noting that a person's emotional intensity is mainly expressed through subtle changes in pitch. Therefore, by combining the effective pitch information and the effective duration, synthetic pitch information can be obtained to realize speech synthesis processing. The synthetic pitch information contains emotional intensity information.

[0080] It is worth noting that in the process of obtaining voice signals, the consent of the detection personnel is required, and voice signals will not be collected without the consent of the detection personnel. When it comes to the need to perform relevant processing based on data related to user identity or characteristics such as user information, user behavior data, user historical data, and user location information, the user's permission or consent will be obtained first, and the collection, use and processing of these data will comply with relevant laws, regulations and standards. In addition, when the embodiment of the present application needs to obtain the user's sensitive personal information, the user's separate permission or consent will be obtained through a pop-up window or by jumping to a confirmation page. After clearly obtaining the user's separate permission or consent, the necessary user-related data for the normal operation of the embodiment of the present application will be obtained.

[0081] In some embodiments, as Figure 2 As shown, the above step S200 may include but is not limited to steps S210 to S220.

[0082] Step S210, performing matching processing on the synthesized text based on a preset embedding table to obtain text matching information;

[0083] Step S220: extracting a phoneme sequence and a note pitch feature from the embedding table according to the text matching information.

[0084] It should be noted that in the process of extracting phoneme sequences and note pitch features, the synthesized text is first matched based on a preset embedding table to obtain text matching information; then the phoneme sequence and note pitch features are extracted from the embedding table based on the text matching information.

[0085] It is worth noting that the embedding table stores elements such as phoneme sequences and note pitch features. Matching processing is first performed to obtain text matching information; then the phoneme sequence and note pitch features can be extracted from the embedding table based on the text matching information.

[0086] In some embodiments, as Figure 3 As shown, the above step S200 may also include but is not limited to steps S230 to S240.

[0087] Step S230, performing pitch feature quantization processing on the speech signal to obtain first pitch feature information;

[0088] Step S240: performing pitch duration calculation processing on the speech signal to obtain first duration information.

[0089] It should be noted that the first pitch feature information can be obtained by quantizing the pitch feature of the speech signal; then the first duration information can be obtained by calculating the pitch duration of the speech signal, which is a prerequisite for subsequent speech synthesis.

[0090] It is worth noting that the first pitch feature information can be obtained by performing pitch feature quantization processing on the speech signal, that is, the pitch feature quantization processing on the speech signal can be performed to obtain the pitch size information of the speech signal.

[0091] In some embodiments, as Figure 4 As shown, the above step S400 may include but is not limited to step S410 and step S420.

[0092] Step S410, performing regularization processing on text features based on the pitch coding model to obtain regularized text;

[0093] Step S420 , performing pitch conversion processing on the regularized text to obtain text pitch information.

[0094] It should be noted that in the process of inputting text features into the pre-trained pitch coding model for conversion processing, the text features must first be regularized based on the pitch coding model to obtain regularized text; then the regularized text is pitch converted to obtain the text pitch information.

[0095] It is worth noting that when there is not enough training data, overfitting often occurs. Regularization method is a general term for a class of methods that introduce additional information to the original model to prevent overfitting and improve the generalization performance of the model. In actual deep learning scenarios, it is found that the best fitting model is a large model with appropriate regularization.

[0096] In some embodiments, as Figure 5 As shown, after executing the above step S600, it may also include but is not limited to step S610.

[0097] Step S610 , performing length normalization processing on the synthesized pitch information according to a preset Mel-frame length.

[0098] It should be noted that before decoding the synthesized pitch information to obtain the synthesized speech Mel spectrum, the synthesized pitch information can also be length-normalized according to the preset Mel frame length to eliminate the impact caused by different voice inputs, making the subsequent speech synthesis more accurate.

[0099] In some embodiments, as Figure 6 As shown, after executing the above step S300, it may also include but not limited to step S310 and step S320.

[0100] Step S310, obtaining pitch information of the speech signal;

[0101] Step S320: embed the tone information into the second pitch feature information and the second duration information respectively.

[0102] It should be noted that the pitch information of the speech signal is first obtained, and then the pitch information is embedded in the second pitch feature information and the second duration information respectively. The pitch information is used as a relevant influencing factor as input to make the subsequent speech synthesis result more accurate. Among them, the pitch information represents the frequency of the speech signal.

[0103] In some embodiments, as Figure 7 As shown, after executing the above step S300, it may also include but not limited to step S330 and step S340.

[0104] Step S330, obtaining energy information of the speech signal;

[0105] Step S340: embed the energy information into the second pitch feature information and the second duration information respectively.

[0106] It should be noted that the energy information of the speech signal is first obtained, and then the energy information is embedded into the second pitch feature information and the second duration information respectively. The energy information is used as a relevant influencing factor as input to make the subsequent speech synthesis result more accurate. The energy information represents the strength of the speech signal.

[0107] In addition, if Figure 8 As shown, an embodiment of the present application further provides a speech synthesis device 10 combining emotion intensity, comprising:

[0108] The first processing module 100 is used to obtain a speech signal and a synthesized text corresponding to the speech signal;

[0109] The second processing module 200 is used to perform a search process on the synthesized text to obtain a phoneme sequence and a note pitch feature; and perform a feature extraction process on the speech signal to obtain first pitch feature information and first duration information;

[0110] The third processing module 300 is configured to combine the phoneme sequence and the note pitch feature to obtain a text feature; and to embed the emotion information corresponding to the speech signal into the first pitch feature information and the first duration information, respectively, to obtain second pitch feature information corresponding to the first pitch feature information and second duration information corresponding to the first duration information;

[0111] The fourth processing module 400 is used to input the text features into a pre-trained pitch coding model for conversion processing to obtain text pitch information;

[0112] A fifth processing module 500 is configured to input the second pitch feature information and the text pitch information into a pre-trained residual pitch prediction network model for a first iterative prediction process to obtain effective pitch information; and input the second duration information into a pre-trained residual duration prediction network model for a second iterative prediction process to obtain effective duration;

[0113] A sixth processing module 600 is configured to combine the effective pitch information and the effective duration to obtain synthesized pitch information;

[0114] The seventh processing module 700 is used to decode the synthesized pitch information to obtain a synthesized speech mel-spectrogram.

[0115] It should be noted that, first, a speech signal and a synthetic text corresponding to the speech signal are obtained; then the synthetic text is retrieved and processed to obtain a phoneme sequence and note pitch features; and the speech signal is feature extracted to obtain first pitch feature information and first duration information; then the phoneme sequence and note pitch features are combined to obtain text features; and the first pitch feature information and the first duration information are respectively embedded with the emotional information corresponding to the speech signal to obtain second pitch feature information corresponding to the first pitch feature information and second duration information corresponding to the first duration information; then the text features are input into a pre-trained pitch encoding model for conversion processing to obtain text pitch information; then the second pitch feature information and the text pitch information are input into a pre-trained residual pitch prediction network model for a first iterative prediction processing to obtain effective pitch information; and the second duration information is input into a pre-trained residual duration prediction network model for a second iterative prediction processing to obtain effective duration; then the effective pitch information and the effective duration are combined to obtain synthetic pitch information; finally, the synthetic pitch information is decoded to obtain a synthetic speech Mel-spectrogram. Through the above technical solution, emotional expression can be combined in the process of speech synthesis, making the speech synthesis effect better.

[0116] The specific implementation of the speech synthesis device 10 combined with emotion intensity is basically the same as the specific embodiment of the speech synthesis method combined with emotion intensity described above, and will not be repeated here.

[0117] In addition, if Figure 9 As shown, an embodiment of the present application further provides an electronic device 700 , which includes: a memory 720 , a processor 710 , and a computer program stored in the memory 720 and executable on the processor 710 .

[0118] The processor 710 and the memory 720 may be connected via a bus or other means.

[0119] The non-transient software program and instructions required to implement the speech synthesis method combined with emotion intensity of the above embodiments are stored in the memory 720. When executed by the processor 710, the speech synthesis method combined with emotion intensity of the above embodiments is executed, for example, the above described speech synthesis method is executed. Figure 1 Method steps S100 to S700, Figure 2 Method steps S210 to S220, Figure 3 Method steps S230 to S240, Figure 4 Steps S410 to S420 of the method, Figure 5 Step S610 of the method, Figure 6 Method steps S310 to S320 and Figure 7Method steps S330 to S340 in .

[0120] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, i.e., they may be located in one place or distributed across multiple network units. Some or all of the modules may be selected based on actual needs to achieve the objectives of this embodiment.

[0121] In addition, an embodiment of the present application further provides a computer-readable storage medium, which stores computer-executable instructions. The computer-executable instructions are executed by a processor 710 or a controller, for example, by a processor 710 in the above-mentioned device embodiment, so that the processor 710 can execute the speech synthesis method combined with emotion intensity in the above-mentioned embodiment, for example, execute the above-mentioned Figure 1 Method steps S100 to S700, Figure 2 Method steps S210 to S220, Figure 3 Method steps S230 to S240, Figure 4 Steps S410 to S420 of the method, Figure 5 Step S610 of the method, Figure 6 Method steps S310 to S320 and Figure 7 Method steps S330 to S340 in .

[0122] The above embodiments may be used in combination, and modules with the same name in different embodiments may be the same or different.

[0123] The foregoing description describes specific embodiments of the present application, and other embodiments are within the scope of the appended claims. In some cases, the actions or steps recited in the claims may be performed in an order different from that described in the embodiments and still achieve the desired results. In addition, the processes depicted in the accompanying drawings do not necessarily require the specific order shown or the sequential order to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0124] The various embodiments in this application are described in a progressive manner. Similar portions between the various embodiments can be referenced to each other, and each embodiment focuses on the differences from the other embodiments. In particular, the device, equipment, and computer-readable storage medium embodiments are generally similar to the method embodiments, so their descriptions are relatively simple. For relevant portions, refer to the descriptions of the method embodiments.

[0125] The apparatus, device, computer-readable storage medium and method provided in the embodiments of the present application correspond to each other. Therefore, the apparatus, device and non-volatile computer storage medium also have similar beneficial technical effects as the corresponding method. Since the beneficial technical effects of the method have been described in detail above, the beneficial technical effects of the corresponding apparatus, device and computer storage medium will not be repeated here.

[0126] In the 1990s, technological improvements could be clearly distinguished as either hardware improvements (for example, improvements to circuit structures like diodes, transistors, and switches) or software improvements (improvements to process flows). However, with the advancement of technology, many process flow improvements today can now be considered direct improvements to hardware circuit structures. Designers almost always create the corresponding hardware circuit structure by programming the improved process flow into the hardware circuit. Therefore, it cannot be said that a process flow improvement cannot be implemented using hardware modules. For example, a programmable logic device (PLD), such as a field programmable gate array (FPGA), is an integrated circuit whose logical function is determined by user programming. Designers can "integrate" a digital system on a PLD through their own programming, without having to hire a chip manufacturer to design and manufacture a dedicated integrated circuit chip. Moreover, nowadays, instead of manually fabricating integrated circuit chips, this programming is mostly done using "logic compiler" software. This is similar to the software compiler used when developing programs. Before compilation, the original code must also be written in a specific programming language, called a hardware description language (HDL). There is not just one HDL, but many, such as ABEL (Advanced Boolean Expression Language), AHDL (Altera Hardware Description Language), Confluence, CUPL (Cornell University Programming Language), HDCal, JHDL (Java Hardware Description Language), Lava, Lola, MyHDL, PALASM, RHDL (Ruby Hardware Description Language), etc. The most commonly used ones are VHDL (Very-High-Speed ​​Integrated Circuit Hardware Description Language) and Verilog. Those skilled in the art will also understand that by simply programming the method flow in one of these hardware description languages ​​and then programming it into an integrated circuit, a hardware circuit that implements the logic method flow can be easily obtained.

[0127] The controller can be implemented in any suitable manner. For example, the controller can take the form of a microprocessor or processor and a computer-readable medium storing computer-readable program code (e.g., software or firmware) executable by the (micro)processor, logic gates, switches, application-specific integrated circuits (ASICs), programmable logic controllers, and embedded microcontrollers. Examples of controllers include, but are not limited to, the following microcontrollers: ARC 625D, Atmel AT91SAM, Microchip PIC18F26K20, and Silicone Labs C8051F320. The memory controller can also be implemented as part of the control logic of the memory. Those skilled in the art will also know that in addition to implementing the controller in a purely computer-readable program code format, the controller can be implemented in the form of logic gates, switches, application-specific integrated circuits, programmable logic controllers, and embedded microcontrollers by logically programming the method steps. Therefore, such a controller can be considered a hardware component, and the devices included therein for implementing various functions can also be considered as structures within the hardware component. Or even, the devices for implementing various functions can be considered as both software modules implementing the method and structures within the hardware component.

[0128] The systems, devices, modules, or units described in the above embodiments may be implemented by computer chips or entities, or by products having certain functions. A typical implementation device is a computer. Specifically, the computer may be, for example, a personal computer, a laptop computer, a cellular phone, a camera phone, a smartphone, a personal digital assistant, a media player, a navigation device, an email device, a game console, a tablet computer, a wearable device, or a combination of any of these devices.

[0129] For the convenience of description, the above devices are described as being divided into various units according to their functions. Of course, when implementing the embodiments of the present application, the functions of each unit can be implemented in the same or multiple software and / or hardware.

[0130] Those skilled in the art will appreciate that the embodiments of the present application can be provided as methods, systems, or computer program products. Therefore, the embodiments of the present application can adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the embodiments of the present application can adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to magnetic disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0131] This specification is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the steps in the flowchart and / or block diagram. Figure 1 a process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0132] These computer program instructions may also be stored in a computer readable memory that can direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 a process or multiple processes and / or boxes Figure 1 The function specified in one or more boxes.

[0133] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operational steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing the instructions executed on the computer or other programmable device for implementing the process. Figure 1 a process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0134] In a typical configuration, a computing device includes one or more processors (CPUs), input / output interfaces, network interfaces, and memory.

[0135] Memory may include non-permanent storage in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. Memory is an example of a computer-readable medium.

[0136] Computer-readable media includes permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. Information can be computer-readable instructions, data structures, program modules, or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disc read-only memory (CD-ROM), digital versatile disc (DVD) or other optical storage, magnetic cassettes, magnetic disk storage or other magnetic storage devices, or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer-readable media does not include transitory computer-readable media (transitory media), such as modulated data signals and carrier waves.

[0137] It should also be noted that the terms "comprises," "includes," or any other variations thereof are intended to encompass non-exclusive inclusion, such that a process, method, commodity, or apparatus that includes a series of elements includes not only those elements but also other elements not explicitly listed, or includes elements inherent to such process, method, commodity, or apparatus. In the absence of further limitations, an element defined by the phrase "comprises a ..." does not exclude the presence of other identical elements in the process, method, commodity, or apparatus that includes the element.

[0138] In the embodiments of the present application, "at least one" refers to one or more, and "more" refers to two or more. "And / or" describes the association relationship of associated objects, indicating that three relationships may exist. For example, A and / or B can represent the existence of A alone, the existence of A and B at the same time, and the existence of B alone. Among them, A and B can be singular or plural. The character " / " generally indicates that the previous and next associated objects are in an "or" relationship. "At least one of the following" and similar expressions refer to any combination of these items, including any combination of single or plural items. For example, at least one of a, b and c can represent: a, b, c, a and b, a and c, b and c or a and b and c, where a, b, c can be single or multiple.

[0139] Embodiments of the present application may be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, and the like that perform specific tasks or implement specific abstract data types. Embodiments of the present application may also be practiced in distributed computing environments where tasks are performed by remote processing devices connected via a communications network. In a distributed computing environment, program modules may be located in local and remote computer storage media, including storage devices.

[0140] The various embodiments in this application are described in a progressive manner. Similar parts between the various embodiments can be referred to in conjunction with each other. Each embodiment focuses on the differences between the other embodiments. In particular, the system embodiment is generally similar to the method embodiment, so the description is relatively simple. For relevant parts, refer to the partial description of the method embodiment.

[0141] The foregoing is merely an embodiment of the present application and is not intended to limit the present application. For those skilled in the art, the present application may have various modifications and variations. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present application should be included within the scope of the claims of the present application.

Claims

1. A speech synthesis method combining emotional intensity, characterized in that: The method comprises: Acquire a speech signal and a synthesized text corresponding to the speech signal; Performing a search process on the synthesized text to obtain a phoneme sequence and a note pitch feature; and performing a feature extraction process on the speech signal to obtain first pitch feature information and first duration information; Combining the phoneme sequence and the note pitch feature to obtain a text feature; and embedding the first pitch feature information and the first duration information with emotion information corresponding to the speech signal to obtain second pitch feature information corresponding to the first pitch feature information and second duration information corresponding to the first duration information; Inputting the text features into a pre-trained pitch coding model for conversion processing to obtain text pitch information; Inputting the second pitch feature information and the text pitch information into a pre-trained residual pitch prediction network model for a first iterative prediction process to obtain effective pitch information; and inputting the second duration information into a pre-trained residual duration prediction network model for a second iterative prediction process to obtain effective duration; Combining the effective pitch information and the effective duration to obtain synthesized pitch information; The synthesized pitch information is decoded to obtain a synthesized speech mel-spectrogram.

2. The speech synthesis method according to claim 1, characterized in that: The retrieval process of the synthesized text to obtain a phoneme sequence and a note pitch feature includes: Performing matching processing on the synthesized text based on a preset embedding table to obtain text matching information; The phoneme sequence and the note pitch feature are extracted from the embedding table according to the text matching information.

3. The speech synthesis method according to claim 1, characterized in that: The performing feature extraction processing on the speech signal to obtain first pitch feature information and first duration information includes: Performing pitch feature quantization processing on the speech signal to obtain the first pitch feature information; The pitch duration calculation process is performed on the speech signal to obtain the first duration information.

4. The speech synthesis method according to claim 1, characterized in that: The step of inputting the text features into a pre-trained pitch coding model for conversion to obtain text pitch information includes: Regularizing the text features based on the pitch coding model to obtain regularized text; Performing pitch conversion processing on the regularized text to obtain the text pitch information.

5. The speech synthesis method combining emotional intensity according to claim 1, characterized in that: Before decoding the synthesized pitch information to obtain a synthesized speech mel-spectrogram, the method further includes: The synthesized pitch information is length-normalized according to a preset Mel-frame length.

6. The speech synthesis method combining emotional intensity according to claim 1, characterized in that: After embedding the emotion information corresponding to the speech signal into the first pitch feature information and the first duration information respectively to obtain second pitch feature information corresponding to the first pitch feature information and second duration information corresponding to the first duration information, the method further includes: Acquiring pitch information of the speech signal; The tone information is embedded in the second pitch feature information and the second duration information respectively.

7. The speech synthesis method combining emotional intensity according to claim 1, characterized in that: After embedding the emotion information corresponding to the speech signal into the first pitch feature information and the first duration information respectively to obtain second pitch feature information corresponding to the first pitch feature information and second duration information corresponding to the first duration information, the method further includes: Acquiring energy information of the speech signal; The energy information is embedded into the second pitch feature information and the second duration information respectively.

8. A speech synthesis device combining emotional intensity, characterized in that The device comprises: A first processing module, configured to obtain a speech signal and a synthesized text corresponding to the speech signal; A second processing module is configured to perform a search process on the synthesized text to obtain a phoneme sequence and a note pitch feature; and perform a feature extraction process on the speech signal to obtain first pitch feature information and first duration information; a third processing module, configured to combine the phoneme sequence and the note pitch feature to obtain a text feature; and embed the first pitch feature information and the first duration information into the emotion information corresponding to the speech signal to obtain second pitch feature information corresponding to the first pitch feature information and second duration information corresponding to the first duration information; A fourth processing module is used to input the text features into a pre-trained pitch coding model for conversion processing to obtain text pitch information; a fifth processing module, configured to input the second pitch feature information and the text pitch information into a pre-trained residual pitch prediction network model for a first iterative prediction process to obtain effective pitch information; and input the second duration information into a pre-trained residual duration prediction network model for a second iterative prediction process to obtain effective duration; a sixth processing module, configured to combine the effective pitch information and the effective duration to obtain synthesized pitch information; The seventh processing module is used to decode the synthesized pitch information to obtain a synthesized speech mel-spectrogram.

9. An electronic device comprising: A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the speech synthesis method combined with emotion intensity as described in any one of claims 1 to 7 is implemented.

10. A computer-readable storage medium storing computer-executable instructions, characterized in that: The computer-executable instructions are used to execute the speech synthesis method combined with emotion intensity as described in any one of claims 1 to 7.

Images