Method and apparatus for generating spoken language acoustic features, electronic device, and storage medium

Abstract

The application discloses a spoken language acoustic feature generation method and device, electronic equipment and a storage medium. The method comprises the following steps: generating a speaker vector based on obtained speaker information, generating a style vector based on obtained speech style information, and generating a phoneme vector based on phoneme information of a text to be converted; the text to be converted represents a text to be converted into speech; inputting the speaker vector, the style vector and the phoneme vector into an encoder model to generate a target phoneme feature sequence representing the text to be converted; and inputting the target phoneme feature sequence into a decoder model to obtain target acoustic features of the text to be converted. The technical scheme provided by the application can improve the adaptability of a speech synthesis system to different application scenarios to a certain extent.

Description

Technical Field

[0001] This invention relates to the field of speech synthesis technology, and more specifically to a method, apparatus, electronic device, and storage medium for generating spoken-language acoustic features. Background Technology

[0002] Speech synthesis technology provides the ability to convert readable text into audible sound, and has been widely used in many fields, such as voice assistants, intelligent education, and intelligent customer service. However, in current technologies, the tone of synthesized speech is invariable in human-computer interaction scenarios, thus current speech synthesis systems cannot produce natural, human-like speech in these situations. Summary of the Invention

[0003] In view of this, embodiments of the present invention provide a method, apparatus, electronic device and storage medium for generating spoken acoustic features, which can improve the adaptability of speech synthesis systems to different application scenarios to a certain extent.

[0004] This invention provides a method for generating spoken acoustic features, the method comprising: generating a speaker vector based on acquired speaker information, generating a style vector based on acquired speech style information, and generating a phoneme vector based on phoneme information of the text to be converted; wherein the text to be converted represents text to be converted into speech; inputting the speaker vector, the style vector, and the phoneme vector into an encoder model to generate a target phoneme feature sequence representing the text to be converted; and inputting the target phoneme feature sequence into a decoder model to obtain the target acoustic features of the text to be converted.

[0005] In conjunction with the first aspect, in one embodiment, generating a phoneme vector based on the phoneme information of the text to be converted includes: performing a convolution operation on the phoneme information according to a preset dimension to obtain the phoneme features of the text to be converted; and inputting the phoneme features into a bidirectional recurrent neural network to obtain the phoneme vector.

[0006] In conjunction with the first aspect, in one embodiment, the speaker vector, the style vector, and the phoneme vector are input into an encoder model to generate a target phoneme feature sequence representing the text to be converted, including: inputting the target vector into a prosody predictor to obtain prosodic features of the text to be converted; the target vector is obtained by concatenating the speaker vector, the style vector, and the phoneme vector; inputting the target vector and the prosody features into a pitch predictor to obtain a pitch vector representing the text to be converted; and inputting the target vector and the prosody features into a volume predictor to obtain a volume vector representing the text to be converted; and determining the target phoneme feature sequence representing the text to be converted based on the phoneme vector, the pitch vector, and the volume vector.

[0007] In conjunction with the first aspect and other corresponding implementation methods, in one implementation method, determining a target phoneme feature sequence representing the text to be converted based on the phoneme vector, the pitch vector, and the volume vector includes: superimposing the phoneme vector, the pitch vector, and the volume vector to obtain a first target phoneme vector; inputting the first target phoneme vector into a duration predictor to obtain a pronunciation duration sequence representing the pronunciation duration of each phoneme in the text to be converted; and mapping the first target phoneme feature vector to a vector space with the same dimension as the target acoustic features based on the pronunciation duration sequence to obtain a target phoneme feature sequence.

[0008] In one embodiment, combining the first aspect and other corresponding implementation methods, the phoneme vector, the pitch vector, and the volume vector are superimposed to obtain a first target phoneme vector; the first target phoneme vector is concatenated with the speaker vector and the style vector to obtain a second target phoneme vector; the second target phoneme vector is input into a duration predictor to obtain a pronunciation duration sequence representing the pronunciation duration of each phoneme in the text to be converted; based on the pronunciation duration sequence, the second target phoneme feature vector is mapped to a vector space with the same dimension as the target acoustic features to obtain a target phoneme feature sequence.

[0009] In conjunction with the first aspect, in one embodiment, prior to the step of generating phoneme vectors based on the phoneme information of the text to be converted, the method for generating colloquial acoustic features further includes: normalizing the text to be converted to obtain target text; the normalization process includes at least one of the following: punctuation processing, number processing, unit processing, fraction processing, and score processing; marking the prosody of the characters in the target text to obtain prosodic marking information of the target text; determining the pronunciation information of each character in the target text; and generating phoneme information of the text to be converted based on the prosodic marking information and the pronunciation information.

[0010] In conjunction with the first aspect, in one embodiment, the method for generating colloquial acoustic features further includes: inputting the target acoustic features into a vocoder to obtain a target sound signal characterizing the speech of the text to be converted.

[0011] A second aspect of the present invention also provides an apparatus for generating spoken acoustic features, the apparatus comprising: a vector generation unit for generating a speaker vector based on acquired speaker information, a style vector based on acquired speech style information, and a phoneme vector based on phoneme information of the text to be converted; the text to be converted represents text to be converted into speech; a target phoneme feature sequence generation unit for inputting the speaker vector, the style vector, and the phoneme vector into an encoder model to generate a target phoneme feature sequence characterizing the text to be converted; and a target acoustic feature generation unit for inputting the target phoneme feature sequence into a decoder model to obtain the target acoustic features of the text to be converted.

[0012] In another aspect, the present invention provides an electronic device comprising a processor and a memory, the memory being used to store a computer program, which, when executed by the processor, implements the above-described method for generating spoken acoustic features.

[0013] In another aspect, the present invention provides a computer-readable storage medium for storing a computer program that, when executed by a processor, implements the above-described method for generating colloquial acoustic features.

[0014] By acquiring the text to be converted, speaker information, and speech style, and generating phoneme vectors based on phoneme information generated from the text to be converted, speaker vectors based on speaker information, and speech style vectors based on speech style information; then, inputting the speaker vectors, the style vectors, and the phoneme vectors into an encoder model to generate a target phoneme feature sequence representing the text to be converted; inputting the target phoneme feature sequence into a decoder model to obtain the target acoustic features of the text to be converted; finally, the target acoustic features can be converted into speech signals through a vocoder, thereby improving the adaptability of the speech synthesis system to different application scenarios to a certain extent. Attached Figure Description

[0015] The features and advantages of the invention will be more clearly understood by referring to the accompanying drawings, which are schematic and should not be construed as limiting the invention in any way. In the drawings:

[0016] Figure 1 A schematic diagram illustrating the steps of a method for generating colloquial acoustic features according to one embodiment of this disclosure is shown.

[0017] Figure 2 A schematic diagram of the encoder structure in one embodiment of this disclosure is shown;

[0018] Figure 3 A schematic diagram of an apparatus for generating colloquial acoustic features is shown in one embodiment of this disclosure;

[0019] Figure 4 A schematic diagram of the structure of an electronic device according to one embodiment of the present disclosure is shown. Detailed Implementation

[0020] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of this disclosure will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this disclosure, and not all of them. Based on the embodiments of this disclosure, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this disclosure.

[0021] Speech synthesis is the task of converting input text into standard, fluent human speech. Speech synthesis technology provides the ability to transform readable text into audible sound and has been widely applied in numerous fields, such as voice assistants, intelligent education, and intelligent customer service.

[0022] In scenarios requiring human-computer interaction, such as voice assistants, current standard speech synthesis systems can achieve clear expression and a relatively comfortable listening experience. However, synthesized speech often represents the average rhythm of a database, making it difficult to adapt to different scenarios such as novels, news, customer service, and broadcasting. For example, in intelligent interaction scenarios, existing intelligent customer service cannot empathize with people's emotions; even when people are angry, the synthesized speech still provides a response with the same tone. Or, in novel reading scenarios, the narrator's dialogue is ambiguous, and the synthesized speech conveys the same tone regardless of the character's different emotions. In other words, the sound signals obtained through existing speech synthesis technology are audibly machine-generated, not spoken by a real person.

[0023] Therefore, it is necessary to propose a conversational speech synthesis method. The speech information obtained by this method can be more natural and human-like, making it difficult for people to distinguish whether it is machine-synthesized or a real person speaking. In other words, this speech synthesis method can be applied to various different application scenarios.

[0024] Please see Figure 1 This disclosure provides a method for generating colloquial acoustic features according to one embodiment, which may include the following steps.

[0025] S110: Generate a speaker vector based on the acquired speaker information, generate a style vector based on the acquired speech style information, and generate a phoneme vector based on the phoneme information of the text to be converted; the text to be converted represents the text to be converted into speech.

[0026] In this embodiment, for the speech processing model, the input text first needs to be converted into linguistic features such as phonemes required by the acoustic model. These linguistic features are then input into the neural network to generate corresponding acoustic features. Finally, the vocoder can reconstruct the sound signal based on these acoustic features. Therefore, the phoneme features of the text to be converted into speech must first be extracted, that is, the phoneme information of the text to be converted is converted into phoneme vectors. Furthermore, considering that in related technologies, if only the phonemes of the text are converted into vectors, the intonation of the speech obtained by the speech synthesis model is unchanging and not applicable to various tasks and scenarios, the speaker's information and the speech scene information can also be used to generate corresponding vectors. These three vectors are then concatenated and used as input to the neural network model to generate the corresponding acoustic features.

[0027] In this embodiment, the voice style information can be for live-streaming e-commerce, video broadcasting, story reading, etc. Voice style information can also be for novels, news, customer service, interactive, conversational, etc., and is not limited here. The voice style information can be selected from several preset style information. Specifically, the style category can be selected according to the usage scenario. The speaker information is the speaker's information; different speaker information has different speaker identifiers. This speaker identifier is used to distinguish the timbre of different speakers so that, during the speech synthesis process, synthesized speech with the expected timbre can be synthesized based on the speaker identifier.

[0028] In this embodiment, the text to be converted can be pre-input by the user. For example, in a novel reading scenario, text such as narration or dialogue can be pre-input as the text to be converted; similarly, in a news scenario, news text can be pre-input as the text to be converted. Other cases can follow the same logic. The text to be synthesized can also be predicted. For example, in a customer service scenario, corresponding response interaction text can be generated based on the user's interaction text as the text to be synthesized. It should be noted that the user interaction text can be input by the user or obtained by recognizing the user's speech. Furthermore, a question-answering model based on an Encoder-Decoder structure can be used to predict the user interaction text and obtain the corresponding response interaction text. Other cases can follow the same logic, and will not be listed here.

[0029] S120: Input the speaker vector, the style vector and the phoneme vector into the encoder model to generate a target phoneme feature sequence representing the text to be converted.

[0030] In this embodiment, the speaker vector, style vector and phoneme vector obtained in the above steps are concatenated to obtain a new vector, and this new vector is input into the encoder model to obtain the target phoneme feature sequence of the text to be converted.

[0031] In this embodiment, the encoder model is used to extract features from a new vector obtained by concatenating the speaker vector, style vector, and phoneme vector, thereby obtaining the target phoneme feature sequence of the text to be converted. The encoder model includes, but is not limited to, AE (AutoEncoder), VAE (Variational AutoEncoder), CVAE (Conditional Variational AutoEncoder), GST (Global Style Tokens), etc., and is not limited here.

[0032] S130: Input the target phoneme feature sequence into the decoder model to obtain the target acoustic features of the text to be converted.

[0033] In this embodiment, after obtaining the target phoneme feature sequence, it needs to be input into a decoder for decoding to synthesize acoustic features. Then, based on the synthesized acoustic features, the target speech signal of the text to be converted can be further synthesized.

[0034] In this embodiment, the decoder is used to decode the target phoneme feature sequence generated by the encoder, thereby obtaining the target acoustic features of the text to be converted. The decoder network includes a locally optimized recurrent network (PRN) composed of fully connected layers combined with ReLU nonlinear activation and Dropout, a unidirectional GRU (Gate Recurrent Unit) module, and finally a fully connected layer as the output layer. It should be noted that the network model of the decoder is not limited in this embodiment; the network model of the decoder can also be a neural network model such as a Long Short-Term Memory (LSTM) network model.

[0035] In this embodiment, the target acoustic features are used to represent the speech acoustic characteristics of the text to be converted. For example, they can represent the energy concentration area of ​​the timbre, formant frequency, formant intensity, and bandwidth, as well as the duration, fundamental frequency, and average speech power representing the prosodic characteristics. The target acoustic features can be represented using Mel-Spectrum. Of course, the target acoustic features can also be represented using FFT, Pitch, MFCC, Fbank, PCEN, etc.

[0036] In one specific implementation, the speaker information and speech style information corresponding to the target speech signal, as well as the text to be converted, are first determined through user selection or other feasible implementation methods. Then, a speaker vector is extracted based on the speaker information, a speech style vector is extracted based on the speech style information, and a phoneme vector is extracted based on the text to be converted. The speaker vector, speech style vector, and phoneme vector are then concatenated to obtain a target phoneme vector. Next, using an encoder-decoder model, the target phoneme vector is first input into the encoder model to obtain the target phoneme sequence features of the text to be converted, and then the target phoneme sequence features are input into the decoder model to obtain the Mel spectrum of the text to be converted. Finally, the Mel spectrum is converted into the target speech signal using a vocoder. Because the synthesized target speech signal includes speaker information and speech style information, it can adapt to various application scenarios.

[0037] In one implementation, generating a phoneme vector based on the phoneme information of the text to be converted may include: performing a convolution operation on the phoneme information according to a preset dimension to obtain the phoneme features of the text to be converted; and inputting the phoneme features into a bidirectional recurrent neural network to obtain the phoneme vector.

[0038] In this embodiment, the long original time-series data is converted into a shorter sequence composed of abstract features through convolution calculation and max pooling. This sequence is then used as the input of the bidirectional GRU network layer for analysis and processing. This fully combines the speed advantage of temporal convolutional neural networks in processing time-series data with the temporal sensitivity of gated recurrent units.

[0039] In this embodiment, since phoneme information has temporal characteristics, 128 temporal convolution kernels of size 24 are used in the convolution calculation of the temporal convolutional neural network processing layer, and the max pooling operator of the max pooling layer has a size of 3. The activation function of the temporal convolutional neural network processing layer is the Rectified Linear Unit (ReLU), also known as the corrected linear unit. The ReLU function can make some neurons output 0 to improve the sparsity of the network and reduce the interdependence of parameters to avoid overfitting. Max pooling reduces the dimensionality of the output data while retaining the most significant feature information of the input data. The significance of max pooling is mainly twofold: firstly, by reducing the dimensionality of the feature map, it effectively reduces the parameters required by subsequent layers; secondly, it maintains translation invariance, that is, when the input data undergoes a small displacement in the neighborhood, the output of the max pooling layer remains unchanged, thereby enhancing the robustness of the neural network and providing a certain degree of anti-perturbation effect.

[0040] Because temporal information decays as it propagates forward in a GRU network, the decay is more severe in earlier sequences. To overcome this decay problem, this paper improves the GRU network by training two GRUs in opposite directions to form a bidirectional GRU model. The bidirectional GRU model consists of two unidirectional GRUs stacked together. The input at each time step t is simultaneously provided to both GRU network layers in opposite directions for learning, and the final output of the model is determined by the combined outputs of these two unidirectional GRU network layers. In the bidirectional GRU model, the outputs of the forward and backward sub-GRU networks are superimposed to obtain the bidirectional GRU model output. In the bidirectional GRU neural network processing layer, the forward and backward sub-layers can each have 32 neurons.

[0041] In one implementation, inputting the speaker vector, the style vector, and the phoneme vector into an encoder model to generate a target phoneme feature sequence representing the text to be converted may include: inputting the target vector into a prosody predictor to obtain prosodic features of the text to be converted; the target vector is obtained by concatenating the speaker vector, the style vector, and the phoneme vector; inputting the target vector and the prosody features into a pitch predictor to obtain a pitch vector representing the text to be converted; and inputting the target vector and the prosody features into a volume predictor to obtain a volume vector representing the text to be converted; and determining the target phoneme feature sequence representing the text to be converted based on the phoneme vector, the pitch vector, and the volume vector.

[0042] In this embodiment, the phonemes that vary in sound include pitch, volume, time, and timbre, ultimately resulting in different spectrograms generated for the same text. Therefore, various acoustic features can be generated using an amplitude adapter, and these acoustic features can then be superimposed to enrich the different types of acoustic features. Finally, these acoustic features can be combined to obtain the target phoneme feature sequence.

[0043] Please see Figure 2 In this embodiment, the speaker embedding, style embedding, and phoneme embedding obtained in the above embodiments are first concatenated to obtain the target vector. Then, the target vector is input into a phone-level prosody predictor to obtain the prosodic features of the text to be converted. Next, the target vector, combined with the prosodic features, is input into a pitch predictor to obtain a pitch vector representing the text to be converted; and the target vector, combined with the prosodic features, is input into an energy predictor to obtain a volume vector representing the text to be converted. Finally, the target phoneme feature sequence representing the text to be converted can be determined based on these three vectors.

[0044] It should be noted that in this embodiment, the pitch predictor and the volume predictor have the same network structure. This may include a FeedForward Sequential Memory Networks (FSMN) Encoder, a bidirectional Long Short-Term Memory (LSTM) network, and a fully-connected layer. In this case, the phoneme vector, pitch vector, and volume vector all have the same length. The prosody predictor may consist primarily of a convolutional neural network and a fully-connected layer.

[0045] In one implementation, determining a target phoneme feature sequence representing the text to be converted based on the phoneme vector, the pitch vector, and the volume vector may include: superimposing the phoneme vector, the pitch vector, and the volume vector to obtain a first target phoneme vector; inputting the first target phoneme vector into a duration predictor to obtain a pronunciation duration sequence representing the pronunciation duration of each phoneme in the text to be converted; and mapping the first target phoneme feature vector to a vector space with the same dimension as the target acoustic features based on the pronunciation duration sequence to obtain a target phoneme feature sequence.

[0046] In this implementation, in addition to predicting pitch and volume, it is also necessary to predict the duration of each phoneme's pronunciation. This allows for a more accurate reproduction of the natural tone, intonation, rhythm, and cadence of human speech, as well as natural features such as breathing and pauses, resulting in a more immersive human-computer interaction experience. Please refer to [link / reference]. Figure 2 The process involves first concatenating the phoneme vector, pitch vector, and volume vector to obtain the first target phoneme vector. Then, this first target phoneme vector is input into an AR-Duration Predictor to obtain a duration sequence of the pronunciation duration of each phoneme in the text to be converted. Next, the first target phoneme vector and the duration vector are aligned using a length regulator, thus mapping the first target phoneme vector to a vector space with the same dimension as the target acoustic features. Specifically, for example, the phoneme sequence can be extended to the same dimension as the Mel spectrum using a length regulator.

[0047] In this embodiment, superimposing the phoneme vector, the pitch vector, and the volume vector can be achieved by adding the values ​​of the phoneme vector, the pitch vector, and the volume vector that are located in the same dimension. Specifically, for example, if the phoneme vector is represented as [1,2,3,4,5,2,1], the pitch vector is represented as [2,3,5,1,6,0,3], and the volume vector is represented as [6,1,2,5,2,1,4], then the first target phoneme vector can be obtained by adding the values ​​of these vectors in the corresponding dimensions, resulting in the first target phoneme vector being [9,6,10,10,13,3,8].

[0048] In this embodiment, the network structure of the AR-Duration Predictor may include PreNet (Fully-Connected Layer + ReLU + dropout), a unidirectional long short-term memory network (LSTM), and a fully-connected layer.

[0049] In one implementation, determining a target phoneme feature sequence representing the text to be converted based on the phoneme vector, the pitch vector, and the volume vector may include: superimposing the phoneme vector, the pitch vector, and the volume vector to obtain a first target phoneme vector; concatenating the first target phoneme vector with the speaker vector and the style vector to obtain a second target phoneme vector; inputting the second target phoneme vector into a duration predictor to obtain a pronunciation duration sequence representing the pronunciation duration of each phoneme in the text to be converted; and mapping the second target phoneme feature vector to a vector space with the same dimension as the target acoustic features based on the pronunciation duration sequence to obtain a target phoneme feature sequence.

[0050] In this embodiment, in addition to adding duration, pitch and energy information to the speech model, the amplitude variation adapter can also incorporate style information, emotion information, speaker information and so on.

[0051] In this embodiment, the first target phoneme vector, style vector, and speaker vector obtained in the above steps can be concatenated to obtain the second target phoneme vector. Then, the second target phoneme vector is input into an AR-Duration Predictor to obtain a duration sequence of the pronunciation duration of each phoneme in the text to be converted. Next, the second target phoneme vector and the pronunciation duration vector are aligned using a Length Regulator based on the duration sequence, thereby mapping the second target phoneme vector to a vector space with the same dimension as the target acoustic features.

[0052] In this embodiment, concatenating the first target phoneme vector with the speaker vector and the style vector to obtain the second target phoneme vector can expand the vector's dimensions. Specifically, for example, if the first target phoneme vector is [9,6,10,10,13,3,8], the speaker vector is [1,21], and the style vector is [23,1], then the concatenated second target phoneme vector would be [9,6,10,10,13,3,8,1,21,23,1].

[0053] In one embodiment, before the step of generating a phoneme vector based on the phoneme information of the text to be converted, the method for generating colloquial acoustic features may further include: normalizing the text to be converted to obtain a target text; the normalization process includes at least one of the following: punctuation processing, number processing, unit processing, fraction processing, score processing; marking the prosody of the characters in the target text to obtain the prosody marking information of the target text; determining the pronunciation information of each character in the target text; generating the phoneme information of the text to be converted based on the prosody marking information and the pronunciation information.

[0054] In this embodiment, before inputting the text to be processed into the speech synthesis model, the preprocessing process of the front end is also extremely important. First, the process of converting irregular text into regular text, using the processing of special punctuation marks, number processing, and the processing of units, fractions, scores, etc. The normalization process described in the embodiments of this specification is to regularize the text according to how the audio is read, and during this process, ASR can be used to assist in the processing, which will not be specifically elaborated here.

[0055] In this embodiment, the normalization process can be a process of eliminating ambiguity. For example, the number 1234 can be read as one two three four, yao er san si, or one thousand two hundred and thirty-four, etc. However, if there is audio, the reading method is fixed, and it can be converted into the Chinese reading method of specific audio reading to avoid errors in subsequent processing. The text information obtained after normalizing the text to be processed is the target text.

[0056] In this embodiment, it is also necessary to mark the prosody of the target text and eliminate polyphonic characters in the text to be converted. Determine the pronunciation information of each character in the target text, that is, determine the pronunciation of each character in the target text. When the character in the target text is a polyphonic character, it is necessary to determine the specific pronunciation of the character in this text in combination with the context of the target text. Specifically, for example, the character "乐" in "音乐" should be pronounced as "yuè", while the character "乐" in "快乐" should be pronounced as "lè".

[0057] In one embodiment, the method for generating colloquial acoustic features may further include: inputting the target acoustic features into a vocoder to obtain a target sound signal representing the speech of the text to be converted.

[0058] In this embodiment, after obtaining the target acoustic features, it is also necessary to convert the acoustic features into speech signals. Therefore, the target acoustic features can be converted into speech signals through a vocoder. Specifically, for example, the target acoustic features are Mel spectrograms, and the Mel spectrograms are input into the HiFi-GAN vocoder to obtain a target sound signal representing the text to be processed.

[0059] Please refer to Figure 3 One embodiment of this disclosure also provides an apparatus for generating colloquial acoustic features, which may include: a vector generation unit, a target phoneme feature sequence generation unit, and a target acoustic feature generation unit.

[0060] The vector generation unit is used to generate a speaker vector based on the acquired speaker information, a style vector based on the acquired speech style information, and a phoneme vector based on the phoneme information of the text to be converted; the text to be converted represents the text to be converted into speech.

[0061] The target phoneme feature sequence generation unit is used to input the speaker vector, the style vector and the phoneme vector into the encoder model to generate a target phoneme feature sequence representing the text to be converted.

[0062] The target acoustic feature generation unit is used to input the target phoneme feature sequence into the decoder model to obtain the target acoustic features of the text to be converted.

[0063] The specific functions and effects of the spoken language acoustic feature generation device can be explained by referring to other embodiments in this specification, and will not be repeated here. Each module in the spoken language acoustic feature generation device can be implemented entirely or partially through software, hardware, or a combination thereof. Each module can be embedded in or independent of the processor in a computer device in hardware form, or stored in the memory of a computer device in software form, so that the processor can call and execute the operations corresponding to each module.

[0064] Please see Figure 4 One embodiment of this disclosure also provides an electronic device, which includes a processor and a memory, the memory being used to store a computer program, which, when executed by the processor, implements the above-described method for generating colloquial acoustic features.

[0065] The processor can be a central processing unit (CPU). It can also be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, or combinations thereof.

[0066] Memory, as a non-transitory computer-readable storage medium, can be used to store non-transitory software programs, non-transitory computer-executable programs, and modules, such as the program instructions / modules corresponding to the methods in the embodiments of this invention. The processor executes various functional applications and data processing by running the non-transitory software programs, instructions, and modules stored in the memory, thereby implementing the methods described in the above embodiments.

[0067] The memory may include a program storage area and a data storage area. The program storage area may store the operating system and applications required for at least one function; the data storage area may store data created by the processor, etc. Furthermore, the memory may include high-speed random access memory and non-transitory memory, such as at least one disk storage device, flash memory device, or other non-transitory solid-state storage device. In some embodiments, the memory may optionally include memory remotely located relative to the processor, which can be connected to the processor via a network. Examples of such networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.

[0068] One embodiment of this disclosure also provides a computer-readable storage medium for storing a computer program that, when executed by a processor, implements the above-described method for generating colloquial acoustic features.

[0069] Those skilled in the art will understand that implementing all or part of the processes in the methods described in this specification can be accomplished by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments described. Any references to memory, storage, databases, or other media used in the embodiments provided in this specification can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, or optical storage, etc. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM), etc.

[0070] It should be understood that each block of a flowchart and / or block diagram, and combinations of blocks in a flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing device to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing device, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0071] This specification describes various embodiments in a progressive manner. Different embodiments focus on describing the parts that differ from other embodiments. Those skilled in the art, upon reading this specification, will realize that the various embodiments and the technical features disclosed in these embodiments can be combined in numerous ways. For the sake of brevity, not all possible combinations of the technical features in the described embodiments are described. However, any combination of these technical features that does not contradict each other should be considered within the scope of this specification.

[0072] It should also be noted that the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0073] The various embodiments described in this specification emphasize the parts that differ from other embodiments, and these embodiments can be explained by comparison with each other. Any combination of the various embodiments described in this specification, based on general technical knowledge, is covered within the scope of this specification.

[0074] The above description is merely an embodiment of this invention and is not intended to limit the scope of protection of the claims. Various modifications and variations are possible for those skilled in the art. Any modifications, equivalent substitutions, or improvements made within the spirit and principle of this invention should be included within the scope of the claims.

Claims

1. A method for generating colloquial acoustic features, characterized in that, The method includes: A speaker vector is generated based on the acquired speaker information, a style vector is generated based on the acquired speech style information, and a phoneme vector is generated based on the phoneme information of the text to be converted; the text to be converted represents the text to be converted into speech. The speaker vector, the style vector, and the phoneme vector are input into the encoder model to generate a target phoneme feature sequence representing the text to be converted. The target phoneme feature sequence is input into the decoder model to obtain the target acoustic features of the text to be converted; The step of inputting the speaker vector, the style vector, and the phoneme vector into the encoder model to generate a target phoneme feature sequence representing the text to be converted includes: The target vector is input into the prosody predictor to obtain the prosodic features of the text to be converted; the target vector is obtained by concatenating the speaker vector, the style vector and the phoneme vector. The target vector and the prosodic feature are input into the pitch predictor to obtain a pitch vector representing the text to be converted; and the target vector and the prosodic feature are input into the volume predictor to obtain a volume vector representing the text to be converted. Based on the phoneme vector, the pitch vector, and the volume vector, a target phoneme feature sequence representing the text to be converted is determined, including: The phoneme vector, the pitch vector, and the volume vector are superimposed to obtain the first target phoneme vector; The first target phoneme vector is input into the duration predictor to obtain a pronunciation duration sequence that represents the pronunciation duration of each phoneme in the text to be converted. Based on the pronunciation duration sequence, the first target phoneme feature vector is mapped to a vector space with the same dimension as the target acoustic feature to obtain the target phoneme feature sequence.

2. The method according to claim 1, characterized in that, Generate phoneme vectors based on the phoneme information of the text to be converted, including: The phoneme information is convolved according to a preset dimension to obtain the phoneme features of the text to be converted; The phoneme features are input into a bidirectional recurrent neural network to obtain the phoneme vector.

3. The method according to claim 1, characterized in that, Based on the phoneme vector, the pitch vector, and the volume vector, a target phoneme feature sequence representing the text to be converted is determined, including: The phoneme vector, the pitch vector, and the volume vector are superimposed to obtain the first target phoneme vector; The first target phoneme vector is concatenated with the speaker vector and the style vector to obtain the second target phoneme vector; The second target phoneme vector is input into the duration predictor to obtain a pronunciation duration sequence representing the pronunciation duration of each phoneme in the text to be converted; Based on the pronunciation duration sequence, the second target phoneme feature vector is mapped to a vector space with the same dimension as the target acoustic feature to obtain the target phoneme feature sequence.

4. The method according to claim 1, characterized in that, Prior to the step of generating phoneme vectors based on the phoneme information of the text to be converted, the method further includes: The text to be converted is normalized to obtain the target text; the normalization process includes at least one of the following: punctuation mark processing, number processing, unit processing, fraction processing, and score processing; The rhythm of the text in the target text is marked to obtain the rhythm marking information of the target text; Determine the pronunciation information of each character in the target text; Based on the prosodic marker information and the pronunciation information, the phoneme information of the text to be converted is generated.

5. The method according to claim 1, characterized in that, The method further includes: The target acoustic features are input into a vocoder to obtain a target sound signal that represents the speech of the text to be converted.

6. A device for generating colloquial acoustic features, characterized in that, The device for generating the colloquial acoustic features includes: The vector generation unit is used to generate a speaker vector based on the acquired speaker information, a style vector based on the acquired speech style information, and a phoneme vector based on the phoneme information of the text to be converted; the text to be converted represents the text to be converted into speech. The target phoneme feature sequence generation unit is used to input the speaker vector, the style vector and the phoneme vector into the encoder model to generate a target phoneme feature sequence representing the text to be converted. The target acoustic feature generation unit is used to input the target phoneme feature sequence into the decoder model to obtain the target acoustic features of the text to be converted. Specifically, the target phoneme feature sequence generation unit is used to input the target vector into the prosody predictor to obtain the prosodic features of the text to be converted; the target vector is obtained by concatenating the speaker vector, the style vector and the phoneme vector. The target vector and the prosodic feature are input into the pitch predictor to obtain a pitch vector representing the text to be converted; and the target vector and the prosodic feature are input into the volume predictor to obtain a volume vector representing the text to be converted. Based on the phoneme vector, the pitch vector, and the volume vector, a target phoneme feature sequence representing the text to be converted is determined. The target phoneme feature sequence generation unit is specifically used to superimpose the phoneme vector, the pitch vector, and the volume vector to obtain a first target phoneme vector; The first target phoneme vector is input into the duration predictor to obtain a pronunciation duration sequence that represents the pronunciation duration of each phoneme in the text to be converted. Based on the pronunciation duration sequence, the first target phoneme feature vector is mapped to a vector space with the same dimension as the target acoustic feature to obtain the target phoneme feature sequence.

7. An electronic device, characterized in that, The electronic device includes a processor and a memory, the memory being used to store a computer program that, when executed by the processor, implements the method as described in any one of claims 1 to 5.

8. A computer-readable storage medium, characterized in that, The computer-readable storage medium is used to store a computer program that, when executed by a processor, implements the method as described in any one of claims 1 to 5.

Images