Speech synthesis method, device, electronic device and storage medium

Abstract

The present disclosure provides a speech synthesis method, apparatus, electronic device, and storage medium. In some embodiments of the present disclosure, the speech synthesis method may include: obtaining a first text and predetermined pitch control data; obtaining a first acoustic feature of the first text; obtaining a prosodic control feature having a spectral length equal to the spectral length of the first acoustic feature based on the pitch control data; superimposing the prosodic control feature with the first acoustic feature to obtain a second acoustic feature; and synthesizing a second speech of the first text using the second acoustic feature. The present disclosure can improve the sound quality of synthesized speech while achieving flexible pitch display control.

Description

Technical Field

[0001] The present disclosure relates to a speech synthesis method, apparatus, electronic device and storage medium. Background Art

[0002] With the advancement of speech synthesis technology, many scenarios require adjusting the pitch of audio based on user or scenario requirements. Currently, pitch control in speech synthesis is primarily achieved through a separate reference encoder. This approach not only requires separate training of complex models, suffers from low accuracy, and has limited applicability. Furthermore, this approach struggles to achieve character-level pitch control. Summary of the Invention

[0003] In order to solve at least one of the above technical problems, the present disclosure provides a speech synthesis method, apparatus, electronic device and storage medium.

[0004] A first aspect of the present disclosure provides a speech synthesis method, comprising:

[0005] obtaining a first text and predetermined pitch control data;

[0006] Acquiring a first acoustic feature of the first text;

[0007] obtaining, according to the pitch control data, a prosody control feature having a spectrum length identical to that of the first acoustic feature;

[0008] superimposing the prosody control feature and the first acoustic feature to obtain a second acoustic feature;

[0009] A second speech of the first text is synthesized using the second acoustic feature.

[0010] In some embodiments of the present disclosure, the pitch control data includes a predetermined audio position and its corresponding fundamental frequency control value; the fundamental frequency value corresponding to the predetermined audio position in the second acoustic feature is determined by the fundamental frequency value corresponding to the predetermined audio position in the first acoustic feature and the fundamental frequency control value.

[0011] In some embodiments of the present disclosure, before obtaining a prosody control feature having a spectrum length identical to that of the first acoustic feature based on the pitch control data, the method further includes adjusting a fundamental frequency control value in the pitch control data based on a pre-acquired fundamental frequency adjustment parameter of a selected speaker.

[0012] In some embodiments of the present disclosure, obtaining a prosody control feature having a spectrum length identical to that of the first acoustic feature based on the pitch control data includes performing an interpolation operation on the pitch control data, the result of the interpolation operation being the prosody control feature.

[0013] In some embodiments of the present disclosure, before obtaining the prosody control feature having the same spectrum length as the spectrum length of the first acoustic feature according to the pitch control data, the method further includes: synthesizing the first speech of the first text using the first acoustic feature;

[0014] The method of obtaining a prosody control feature having a spectrum length identical to that of the first acoustic feature based on the pitch control data includes: aligning the first text and the first speech in a time dimension to obtain word-level phoneme duration information; and performing an interpolation operation on the phoneme duration information and the pitch control data, the result of the interpolation operation being the prosody control feature.

[0015] In some embodiments of the present disclosure, before superimposing the prosody control feature with the first acoustic feature to obtain the second acoustic feature, it also includes: performing decompression processing on the first acoustic feature; after superimposing the prosody control feature with the first acoustic feature to obtain the second acoustic feature, it also includes: performing compression processing on the second acoustic feature so that the value of the element in the second acoustic feature is within a predetermined range.

[0016] In some embodiments of the present disclosure, the speech synthesis method is implemented by a speech synthesis model, and the speech synthesis model is trained in the following manner:

[0017] Acquire corpus for training the speech synthesis model, the corpus comprising a plurality of speech-text pairs, each of the speech-text pairs comprising a second text and its corresponding real speech;

[0018] Screening out speech-text pairs whose fundamental frequency characteristics of real speech meet predetermined conditions from the corpus;

[0019] Performing compression processing on the fundamental frequency features of the real speech in each screened speech-text pair to obtain the original fundamental frequency features of the speech-text pair;

[0020] The speech synthesis model is obtained by training the screened speech-text pairs and their original fundamental frequency features;

[0021] The updating of the parameters of the speech synthesis model by comparing the second acoustic feature with the original fundamental frequency feature includes:

[0022] Obtaining a second acoustic feature corresponding to a second text in the speech-text pair by running the speech synthesis model, and performing compression processing on the second acoustic feature corresponding to the second text so that element values ​​in the second acoustic feature are within a predetermined interval;

[0023] The parameters of the speech synthesis model are updated by comparing the second acoustic feature with the original fundamental frequency feature of the speech-text pair.

[0024] A first aspect of the present disclosure provides a speech synthesis device, comprising:

[0025] an acquiring unit, configured to acquire a first text and predetermined pitch control data;

[0026] an acoustic feature extraction unit, configured to obtain a first acoustic feature of the first text;

[0027] a prosody control feature extraction unit, configured to obtain, based on the pitch control data, a prosody control feature having a spectrum length identical to that of the first acoustic feature;

[0028] a feature superposition unit, configured to superimpose the prosody control feature and the first acoustic feature to obtain a second acoustic feature;

[0029] A speech generation unit is configured to synthesize a second speech of the first text using the second acoustic feature.

[0030] A third aspect of the present disclosure provides an electronic device, including:

[0031] a memory storing execution instructions; and

[0032] A processor executes the execution instructions stored in the memory, so that the processor executes the above-mentioned speech synthesis method.

[0033] A fourth aspect of the present disclosure provides a readable storage medium, wherein the readable storage medium stores execution instructions, and the execution instructions are used to implement the above-mentioned speech synthesis method when executed by a processor.

[0034] The disclosed embodiments achieve prosody control of synthesized speech by simply setting pitch control data, without the need to reference rhythmic features of additional audio or train a complex model dedicated to pitch control. This effectively addresses the inability of conventional speech synthesis to flexibly display and control pitch. Furthermore, the disclosed embodiments achieve speech synthesis by superimposing prosody control features derived from pitch control data onto acoustic features. This allows for flexible pitch display control while improving the sound quality of synthesized speech, resulting in a more natural and robust sound. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] The accompanying drawings illustrate exemplary embodiments of the present disclosure and together with the description serve to explain the principles of the present disclosure. These drawings are included to provide a further understanding of the present disclosure and are incorporated in and constitute a part of this specification.

[0036] Figure 1 It is a flowchart of a speech synthesis method according to some embodiments of the present disclosure.

[0037] Figure 2 It is a schematic diagram of the speech synthesis model training process according to some embodiments of the present disclosure.

[0038] Figure 3 It is a structural diagram of a speech synthesis model according to some embodiments of the present disclosure.

[0039] Figure 4 It is a schematic block diagram of the structure of a speech synthesis device using a hardware implementation of a processing system according to an embodiment of the present disclosure.

[0040] Figure 5 It is a schematic block diagram of the structure of a model training device using a hardware implementation of a processing system according to an embodiment of the present disclosure.

[0041] Description of Reference Numerals

[0042] 200 Speech Synthesis Device

[0043] 300 Model Training Device

[0044] 400 bus

[0045] 500 processors

[0046] 600 Memory

[0047] 700 Various other circuits. DETAILED DESCRIPTION

[0048] The present disclosure will be further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only intended to explain the relevant content and are not intended to limit the present disclosure. It should also be noted that, for ease of description, only the portions relevant to the present disclosure are shown in the accompanying drawings.

[0049] It should be noted that, in the absence of conflict, the embodiments and features of the embodiments in the present disclosure can be combined with each other. The technical solution of the present disclosure will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.

[0050] Unless otherwise stated, the exemplary embodiments / examples shown are to be understood as providing exemplary features of various details of some ways in which the technical concepts of the present disclosure can be implemented in practice. Therefore, unless otherwise stated, the features of the various embodiments / examples may be further combined, separated, interchanged, and / or rearranged without departing from the technical concepts of the present disclosure.

[0051] The use of cross hatching and / or shading in the accompanying drawings is generally used to make the boundaries between adjacent components clear. As such, unless otherwise indicated, the presence or absence of cross hatching or shading does not convey or indicate any preference or requirement for the specific materials, material properties, dimensions, proportions, commonalities between the components shown, and / or any other characteristics, attributes, properties, etc. of the components. In addition, in the accompanying drawings, the sizes and relative sizes of the components may be exaggerated for clarity and / or descriptive purposes. When the exemplary embodiments can be implemented differently, the specific process sequence can be performed in a different order than described. For example, two successively described processes can be performed substantially simultaneously or in an order opposite to the order described. In addition, the same figure numbers represent the same components.

[0052] When a component is referred to as being “on,” “over,” “connected to,” or “coupled to” another component, the component may be directly on, directly connected to, or directly coupled to the other component, or intervening components may be present. However, when a component is referred to as being “directly on,” “directly connected to,” or “directly coupled to” another component, there are no intervening components present. For this purpose, the term “connected” may refer to a physical connection, an electrical connection, etc., with or without intervening components.

[0053] The terms used herein are for the purpose of describing specific embodiments and are not intended to be restrictive. As used herein, unless the context clearly indicates otherwise, the singular forms "one (kind, person)" and "said (the)" are also intended to include plural forms. In addition, when the terms "comprise" and / or "include" and their variations are used in this specification, the features, integral bodies, steps, operations, parts, assemblies and / or their groups stated are indicated, but the presence or addition of one or more other features, integral bodies, steps, operations, parts, assemblies and / or their groups is not excluded. It should also be noted that, as used herein, the terms "substantially", "approximately" and other similar terms are used as approximate terms and not as degree terms, and as such, they are used to explain the inherent deviations of the measured values, calculated values ​​and / or values ​​provided that will be recognized by those of ordinary skill in the art.

[0054] Explanation of terms in this article:

[0055] Text-to-speech (TTS): technology that converts computer-generated or externally input text information into understandable and fluent human speech and outputs it.

[0056] Pitch: Fundamental frequency, also known as fundamental frequency, is the vibration frequency of the vocal cords. When a person produces a voiced sound, the reciprocal of the period of airflow through the vocal tract (i.e., the period of vocal cord vibration) is the fundamental frequency. The fundamental frequency primarily characterizes the speaker's timbre and is crucial in speech emotion recognition. It varies from approximately 50 to 500 Hz.

[0057] Pitch display is controllable: the pitch of the audio synthesized by the model can be adjusted by manually inputting a numerical value.

[0058] Encoder: Able to perform processing such as character embedding on text, so that the text is encoded into a character vector.

[0059] Decoder: It is an autoregressive recurrent neural network, which mainly includes a pre-processing module (Pre-Net), a location-sensitive attention module (Location Sensitive Attention), two stacked LSTM layers (2 LSTM Layers) and a linear projection layer (Linear Projection). The input features of the decoder include character vectors encoded by the text, and the output features of the decoder include the target spectrum frame corresponding to the text. The target spectrum frame of the text can constitute the acoustic features such as the Mel-spectrogram of the text. The processing process of the decoder may include: the character vector is input to the 2 LSTM Layers, the output features of the 2 LSTM Layers are spliced ​​together with the attention vector of the position-sensitive attention module, and processed by the linear projection layer (Linear Projection) to obtain a preliminary Mel-spectrogram frame and return it to the Pre-Net. The output features of the Pre-Net are then spliced ​​with the attention vector of the position-sensitive attention module and passed to the 2 LSTM Layers. The output features of the 2 LSTM Layers are then spliced ​​together with the attention vector of the position-sensitive attention module and processed by the linear projection layer (Linear Projection). This cycle is repeated until it stops, and the target spectrum frame is finally obtained.

[0060] Post-processing network (Post-Net): Consists of five convolutional layers, it can correct acoustic features such as the Mel-spectrogram output by the decoder. The post-processing network's processing includes: after the target spectrum frame output by the decoder passes through the post-processing network, it first obtains the residual term of the target spectrum frame, and then superimposes the residual term on the corresponding target spectrum frame, thereby improving the spectrum reconstruction result.

[0061] Neural Network Vocoder (LPCNet): A neural network-based vocoder that can synthesize acoustic features such as mel-spectrograms into audio.

[0062] Montreal Forced Aligner (MFA): A tool for aligning audio files and text transcripts in the time dimension to obtain phoneme durations.

[0063] The second text refers to the text used to train the speech synthesis model, and the first text refers to the text of the speech to be synthesized.

[0064] Figure 1 Schematic diagram showing the flow of speech synthesis methods according to some embodiments of the present disclosure. Figure 1 As shown, the speech synthesis method of the embodiment of the present disclosure may include:

[0065] Step S12, obtaining a first text and predetermined pitch control data;

[0066] In some implementations, the pitch control data may include a predetermined audio position and its corresponding fundamental frequency control value.

[0067] In some embodiments, the pitch control data can be pre-configured, manually input, or predetermined in other ways. For example, a user can input the relative position of a predetermined audio frequency for which pitch adjustment is desired and the corresponding fundamental frequency control value through a human-computer interaction interface or other means. The electronic device can generate the pitch control data upon receiving this information input by the user.

[0068] For example, the input method of the pitch control data can be, but is not limited to: 0.6:0.1-1.3:0.3-1.1:0.6-0.7:0.7-0.5:0.8-1.2:1.0, where "-" is used to separate different audios. For each predetermined audio, its pitch control information can be in the form of "x:y", where the number "x" before the colon represents the fundamental frequency control value, and the number "y" after the colon represents the relative position of the predetermined audio to be adjusted in pitch. For example, in "0.6:0.1", "0.6" means adjusting the fundamental frequency value to 0.6 times the original value, and "0.1" means that the relative position of the predetermined audio to be adjusted in pitch in the entire speech is "the 1st second", that is, "0.6:0.1" means adjusting the fundamental frequency value of the audio whose relative position in the entire speech is "the 1st second" to 0.6 times the original fundamental frequency value. The other parts are similar to this and will not be repeated here.

[0069] Step S14, obtaining a first acoustic feature of the first text;

[0070] Specifically, step S14 may include: first encoding the first text into a first character vector through an encoder, and the first character vector can be processed by the decoder mentioned above to obtain a first acoustic feature, and the first acoustic feature can be but is not limited to a Mel-spectrogram.

[0071] Step S16, obtaining a prosody control feature having a spectrum length identical to that of the first acoustic feature according to the pitch control data;

[0072] In some implementations, step S16 may include performing an interpolation operation on the pitch control data, with the result of this interpolation being the prosody control feature. This allows for global pitch control of the synthesized speech without the need to train a deep learning model dedicated to pitch control. This reduces computational complexity, simplifies implementation, improves processing efficiency, and reduces hardware costs.

[0073] In some embodiments, before step S16, the speech synthesis method may further include: step S15, synthesizing a first speech of the first text using the first acoustic feature. Specifically, step S16 may include: first optimizing the first acoustic feature using a post-processing network (Post-Net), and then synthesizing the optimized first acoustic feature using an LPCNet to obtain the first speech, where the first speech is synthesized speech without pitch control.

[0074] In some implementations, step S16 may include: step a1, aligning the first text and the first speech along the time dimension to obtain word-level phoneme duration information; and step a2, performing an interpolation operation on the phoneme duration information and the pitch control data, with the result of this interpolation operation being a prosodic control feature. In this manner, by first aligning words and phonemes and then obtaining prosodic control features through interpolation, word-level pitch control (i.e., fine-grained pitch control) can be achieved for synthesized speech without the need to train a dedicated model. This not only reduces computational complexity but also allows for flexible word-level pitch control, thereby further improving the sound quality of the synthesized speech while increasing processing efficiency and reducing hardware costs.

[0075] In step S16, by performing processing such as interpolation on the pitch control data, a control vector having a length equal to the first acoustic feature spectrum is obtained. This control vector is the prosodic control feature of the first text. In this prosodic control feature, the element value corresponding to the predetermined audio position in the pitch control data is determined by the fundamental frequency control value corresponding to the predetermined audio position.

[0076] In some implementations, step a1 can be implemented by tools such as MAF.

[0077] Step S18, superimposing the prosody control feature and the first acoustic feature to obtain a second acoustic feature;

[0078] In some embodiments, step S110 may include performing a multiplication process on the prosody control feature and the first acoustic feature, with the result of the multiplication being a second acoustic feature. Thus, the fundamental frequency value corresponding to the predetermined audio position in the second acoustic feature can be determined by the fundamental frequency value corresponding to the predetermined audio position in the first acoustic feature and the fundamental frequency control value, thereby achieving prosody controllability of the synthesized speech by modifying or adjusting the fundamental frequency.

[0079] For example, the fundamental frequency value corresponding to a predetermined audio position in the second acoustic feature can be the product of the fundamental frequency value corresponding to the predetermined audio position in the first acoustic feature and the corresponding element value in the prosody control feature. Taking the example of "0.6:0.1" mentioned above, the fundamental frequency value at the first second in the second acoustic feature can be the product of the fundamental frequency value at the first second in the first acoustic feature and the element value corresponding to the first second in the prosody control feature, where the element value corresponding to the first second in the prosody control feature is interpolated from the fundamental frequency control value "0.6."

[0080] Step S110: synthesize a second speech of the first text using the second acoustic feature.

[0081] Specifically, step S110 may include: first optimizing the second acoustic feature through a post-processing network (Post-Net), and then synthesizing the optimized second acoustic feature through LPCNet to obtain a second speech, where the second speech is a synthesized speech with pitch control.

[0082] In some implementations, prior to step S16, the pitch control data may be preprocessed based on the characteristics of the current speaker. Specifically, prior to step S16, the speech synthesis method may further include: step S13, adjusting the fundamental frequency control value in the pitch control data based on the pre-acquired fundamental frequency adjustment parameter of the selected speaker. This ensures that the fundamental frequency remains within a reasonable range, thereby achieving pitch display control while improving the sound quality of the synthesized speech.

[0083] In a specific application, fundamental frequency adjustment parameters of a plurality of different speakers may be pre-configured. After a speaker is selected, the fundamental frequency adjustment parameters of the selected speaker may be directly read and the fundamental frequency control value of the pitch control data may be adjusted accordingly.

[0084] Each speaker's fundamental frequency adjustment parameters may include: an upper reference value, a lower reference value, an upper fundamental frequency limit, and a lower fundamental frequency limit. The upper and lower reference values ​​are fixed constants that can be determined through statistical experiments. The upper and lower fundamental frequency limits can characterize the speaker's audio characteristics and can be different or the same for different speakers. In specific applications, the upper and lower fundamental frequency limits can be empirical values, set values, or other numerical values.

[0085] In some implementations, step 13 may include the following two situations:

[0086] 1) When the fundamental frequency control value of the pitch control data is greater than a preset threshold, the fundamental frequency control value is adjusted according to the upper limit reference value and the fundamental frequency upper limit. In this way, when the fundamental frequency control value is high, the fundamental frequency control value can be adjusted to be within a reasonable range, thereby adjusting the pitch control value within the control threshold without affecting the sound quality. This can effectively avoid situations such as excessively sharp voiced sounds in the second speech.

[0087] For example, when the base frequency control value represents the base frequency adjustment ratio and the preset threshold is set to 1, when the base frequency control value is greater than 1, the value of the base frequency control value can be reset by the following formula (1):

[0088] (1)

[0089] in, represents the fundamental frequency control value, kmax represents the upper limit reference value (for example, it can be set to 2), Indicates the upper limit of the fundamental frequency.

[0090] 2) When the fundamental frequency control value of the pitch control data is less than or equal to a preset threshold, the fundamental frequency control value is adjusted based on the lower limit reference value and the fundamental frequency lower limit. In this way, when the fundamental frequency control value is low, the fundamental frequency control value can be adjusted to be within a reasonable range, thereby adjusting the pitch control value within the control threshold without affecting the sound quality. This can effectively avoid situations such as excessively low voiced sounds in the second speech.

[0091] For example, when the base frequency control value represents the base frequency adjustment ratio and the preset threshold is set to 1, when the base frequency control value is less than or equal to 1, the value of the base frequency control value can be reset by the following formula (2):

[0092] (2)

[0093] in, represents the fundamental frequency control value, kmin represents the lower limit reference value (for example, it can be set to 0.5), Indicates the lower limit of the fundamental frequency.

[0094] In some embodiments, if the speech synthesis method is implemented using a speech synthesis model, such as the one described below, and since the acoustic features are compressed during training, the first acoustic feature obtained in step S14 is a compressed acoustic feature. To enhance the superposition of the first acoustic feature with the prosody control feature, the method may further include, before step S18, decompressing the first acoustic feature. Similarly, after step S18, the method may further include compressing the second acoustic feature so that the element values ​​in the second acoustic feature are within a predetermined range (e.g., between -4 and 4).

[0095] For example, if the compression algorithm used in the training of the speech synthesis model is the MinMax algorithm, then the decompression process can be implemented by the DeMinMax algorithm, and the compression process can be implemented by the MinMax algorithm. The predetermined interval in step S19 can be, but is limited to, (-4, 4).

[0096] In some implementations, the speech synthesis method of the embodiments of the present disclosure may be implemented by embedding a speech synthesis model in hardware such as electronic devices, chips, etc. The speech synthesis model may be obtained through pre-training.

[0097] Figure 2 FIG. 1 shows an exemplary training flow diagram of a speech synthesis model. In some implementations, Figure 2 As shown, the speech synthesis model can be trained through the following steps:

[0098] Step S22, obtaining corpus for training a speech synthesis model, the corpus including a plurality of speech-text pairs, each speech-text pair including a second text and its corresponding real speech;

[0099] Preferably, the audio corpus with a relatively steady speaking speed, high volume consistency, and no long pauses in the sentences is selected as the corpus for training the speech synthesis model.

[0100] Step S24, filtering out speech-text pairs whose fundamental frequency characteristics of real speech meet predetermined conditions from the corpus;

[0101] In some implementations, in step S24 , a speech-text pair may be selected whose fundamental frequency feature of the real speech satisfies the threshold value less than 0.01 in the following formula (3), that is, the predetermined condition may be the following formula (3).

[0102] (3)

[0103] Where n is the base frequency sequence length, cur is the base frequency value of the current frame, pre is the base frequency value of the previous frame, T is set to 0.2, and the rate threshold is set to 0.01.

[0104] It should be noted that the setting of formula (3) and its rate threshold can be flexibly adjusted according to actual needs.

[0105] In this way, the corpus can be filtered and speech-text pairs with better audio data quality can be selected as training corpus for the speech synthesis model, thereby obtaining a sound and stable speech synthesis model, avoiding sudden changes in the synthesized speech due to errors in F0 and thus causing the speech synthesis model to be imperfect.

[0106] Step S26, performing compression processing on the fundamental frequency features of the real speech in each screened speech-text pair to obtain the original fundamental frequency features of the speech-text pair;

[0107] Specifically, a compression algorithm such as minmax can be used to compress the fundamental frequency feature between 0 and 255. The minmax algorithm is shown in the following formula (4):

[0108]

[0109] Among them, cout represents the currently extracted F0 value, F0xmin, Represents the lower limit value related to the fundamental frequency, F0xmax, Represents the upper limit of the fundamental frequency. For example, you can set F0max=256, F0min=0, F0xmax=63, and F0xmin=468.

[0110] Step S28, using the selected speech-text pairs and their original fundamental frequency features to train a speech synthesis model;

[0111] Among them, step S28 may include: obtaining the second acoustic feature corresponding to the second text in the speech-text pair by running the speech synthesis model, and performing compression processing on the second acoustic feature corresponding to the second text so that the value of the element value in the second acoustic feature is within a predetermined range; and updating the parameters of the speech synthesis model by comparing the compressed second acoustic feature and the original fundamental frequency feature of the speech-text pair.

[0112] Specifically, the second acoustic feature can be compressed into the range of (-4, 4) by using a compression algorithm such as minmax, so that features of different dimensions can be numerically comparable.

[0113] Figure 3 FIG. 1 shows an exemplary structural diagram of the speech synthesis model. Figure 3 As shown, the speech synthesis model may include: a decoder, an encoder, a post-processing network, a vocoder, a prosody control feature extraction unit and a feature superposition unit.

[0114] like Figure 3As shown, the decoder, encoder, post-processing network, and vocoder are sequentially connected. The vocoder, prosody control feature extraction unit, feature superposition unit, and post-processing network are sequentially connected. The input data of the prosody control feature extraction unit includes predetermined pitch control data. The prosody control feature extraction unit is configured to obtain a prosody control feature having a spectral length equal to the spectral length of a first acoustic feature based on the pitch control data. The feature superposition unit is configured to superimpose the prosody control feature with the first acoustic feature to obtain a second acoustic feature. For example, the prosody control feature extraction unit can be implemented as an interpolation unit, and the feature superposition unit can be implemented as a multiplication unit.

[0115] In some implementations, the speech synthesis model may further include an alignment unit configured to align the first text and the first speech in a temporal dimension to obtain word-level phoneme duration information. For example, the alignment unit may be implemented as a tool such as MFA.

[0116] Figure 4 1 is a schematic block diagram of the structure of a speech synthesis apparatus 200 implemented in hardware using a processing system according to an embodiment of the present disclosure.

[0117] Figure 5 A block diagram illustrating the structure of a speech synthesis apparatus 300 implemented in hardware using a processing system according to an embodiment of the present disclosure.

[0118] The device may include corresponding modules for executing each or several steps in the above flowchart. Therefore, each step or several steps in the above flowchart may be executed by a corresponding module, and the device may include one or more of these modules. The module may be one or more hardware modules specifically configured to execute the corresponding steps, or implemented by a processor configured to execute the corresponding steps, or stored in a computer-readable medium for execution by a processor, or implemented by some combination thereof.

[0119] The hardware structure can be implemented using a bus architecture. The bus architecture can include any number of interconnecting buses and bridges, depending on the specific application and overall design constraints of the hardware. Bus 400 connects various circuits including one or more processors 500, memory 600, and / or hardware modules. Bus 400 can also connect various other circuits 700 such as peripherals, voltage regulators, power management circuits, external antennas, etc.

[0120] Bus 400 may be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Component Architecture (EISA) bus. Buses can be classified as address buses, data buses, control buses, and the like. For ease of illustration, this figure shows only one connecting line, but this does not imply that there is only one bus or only one type of bus.

[0121] Any process or method description in the flowchart or otherwise described herein can be understood to represent a module, fragment or portion of code including one or more executable instructions for implementing the steps of a specific logical function or process, and the scope of the preferred embodiments of the present disclosure includes alternative implementations in which the functions may not be performed in the order shown or discussed, including performing the functions in a substantially simultaneous manner or in a reverse order depending on the functions involved, which should be understood by those skilled in the art to which the embodiments of the present disclosure belong. The processor performs the various methods and processes described above. For example, the method embodiments of the present disclosure can be implemented as a software program that is tangibly contained in a machine-readable medium, such as a memory. In some embodiments, part or all of the software program can be loaded and / or installed via a memory and / or a communication interface. When the software program is loaded into the memory and executed by the processor, one or more steps in the method described above can be performed. Alternatively, in other embodiments, the processor can be configured to perform one of the above methods in any other appropriate manner (e.g., by means of firmware).

[0122] The logic and / or steps represented in the flowchart or otherwise described herein may be embodied in any readable storage medium for use by, or in conjunction with, an instruction execution system, apparatus, or device (such as a computer-based system, a system including a processor, or other system that can fetch and execute instructions from an instruction execution system, apparatus, or device).

[0123] For the purposes of this specification, a "readable storage medium" can be any device that can contain, store, communicate, propagate or transmit a program for use with or in conjunction with an instruction execution system, device or apparatus. More specific examples (a non-exhaustive list) of readable storage media include the following: an electrical connection having one or more wires (electronic device), a portable computer disk cartridge (magnetic device), a random access memory (RAM), a read-only memory (ROM), an erasable and editable read-only memory (EPROM or flash memory), a fiber optic device, and a portable read-only memory (CDROM). In addition, the readable storage medium can even be paper or other suitable medium on which the program can be printed, since the program can be obtained electronically, for example, by optically scanning the paper or other medium and then editing, interpreting or processing it in other suitable ways as necessary, and then storing it in a memory.

[0124] It should be understood that various parts of the present disclosure can be implemented using hardware, software, or a combination thereof. In the above-described embodiments, multiple steps or methods can be implemented using software stored in a memory and executed by a suitable instruction execution system. For example, if implemented using hardware, as in another embodiment, any one of the following technologies known in the art or a combination thereof can be used to implement the hardware: a discrete logic circuit having a logic gate circuit for implementing a logic function on a data signal, an application-specific integrated circuit having a suitable combination of logic gate circuits, a programmable gate array (PGA), a field programmable gate array (FPGA), etc.

[0125] Those skilled in the art will understand that all or part of the steps of the above-mentioned implementation method can be accomplished by instructing related hardware through a program, and the program can be stored in a readable storage medium. When the program is executed, it includes one or a combination of the steps of the method implementation method.

[0126] Furthermore, the functional units in the various embodiments of the present disclosure may be integrated into a single processing module, each unit may exist physically separately, or two or more units may be integrated into a single module. The aforementioned integrated modules may be implemented in the form of hardware or software functional modules. If the integrated modules are implemented as software functional modules and sold or used as independent products, they may also be stored in a readable storage medium. The storage medium may be a read-only memory, a magnetic disk, or an optical disk, etc.

[0127] Figure 4 FIG is a structural diagram of a speech synthesis device 200 according to an embodiment of the present disclosure. Figure 4 As shown, a speech synthesis apparatus 200 according to an embodiment of the present disclosure may include:

[0128] An acquiring unit 202 is configured to acquire a first text and predetermined pitch control data;

[0129] an acoustic feature extraction unit 204, configured to obtain a first acoustic feature of the first text;

[0130] a prosody control feature extraction unit 206 for obtaining, according to the pitch control data, a prosody control feature having a spectrum length identical to that of the first acoustic feature;

[0131] a feature superposition unit 208, configured to superimpose the prosody control feature and the first acoustic feature to obtain a second acoustic feature;

[0132] The speech generation unit 210 is configured to synthesize a second speech of the first text using the second acoustic feature.

[0133] In some embodiments, the pitch control data includes a predetermined audio position and its corresponding fundamental frequency control value; the fundamental frequency value corresponding to the predetermined audio position in the second acoustic feature is determined by the fundamental frequency value corresponding to the predetermined audio position in the first acoustic feature and the fundamental frequency control value.

[0134] In some implementations, the speech synthesis apparatus 200 may further include an adjusting unit 212 configured to adjust the fundamental frequency control value in the pitch control data according to a pre-acquired fundamental frequency adjustment parameter of the selected speaker.

[0135] In some implementations, the prosody control feature extraction unit 206 may be configured to perform an interpolation operation on the pitch control data, where a result of the interpolation operation is the prosody control feature.

[0136] In some embodiments, the speech generation unit 210 can also be used to synthesize the first speech of the first text using the first acoustic feature; the prosody control feature extraction unit 206 can be specifically used to: align the first text and the first speech in the time dimension to obtain word-level phoneme duration information; and perform an interpolation operation on the phoneme duration information and the pitch control data, and the result of the interpolation operation is the prosody control feature.

[0137] In some embodiments, the speech synthesis device 200 may further include: a decompression unit 214 and a compression unit 216; wherein the decompression unit 214 can be used to perform decompression processing on the first acoustic feature; the compression unit 216 can be used to perform compression processing on the second acoustic feature so that the value of the element in the second acoustic feature is within a predetermined range.

[0138] like Figure 5As shown, the embodiment of the present disclosure further provides a model training device 300, which can be used to train a speech synthesis model for implementing the above-mentioned speech synthesis method. Figure 5 As shown, the model training device 300 may include:

[0139] A corpus unit 302 is used to obtain corpus for training a speech synthesis model, the corpus including a plurality of speech-text pairs, each speech-text pair including a second text and its corresponding real speech;

[0140] A screening unit 304 is configured to screen out speech-text pairs whose fundamental frequency characteristics of real speech meet predetermined conditions from the corpus;

[0141] The fundamental frequency compression unit 306 performs compression processing on the fundamental frequency features of the real speech in each screened speech-text pair to obtain the original fundamental frequency features of the speech-text pair;

[0142] The training execution unit 308 is used to train a speech synthesis model using the screened speech-text pairs and their original fundamental frequency features.

[0143] Among them, the training execution unit 308 is specifically used to: obtain the second acoustic feature corresponding to the second text in the speech-text pair by running the speech synthesis model, and update the parameters of the speech synthesis model by comparing the compressed second acoustic feature and the original fundamental frequency feature of the speech-text pair; the fundamental frequency compression unit 306 can also be used to perform compression processing on the second acoustic feature corresponding to the second text so that the value of the element value in the second acoustic feature is within a predetermined range.

[0144] The present disclosure also provides an electronic device, including: a memory, the memory storing execution instructions; and a processor or other hardware module, the processor or other hardware module executing the execution instructions stored in the memory, so that the processor or other hardware module performs the above-mentioned speech synthesis method.

[0145] The present disclosure also provides a readable storage medium, in which execution instructions are stored. When the execution instructions are executed by a processor, they are used to implement the above-mentioned speech synthesis method.

[0146] In the description of this specification, the description with reference to the terms "one embodiment / method", "some embodiments / methods", "example", "specific example", or "some examples" means that the specific features, structures, materials or characteristics described in conjunction with the embodiment / method or example are included in at least one embodiment / method or example of the present application. In this specification, the schematic representations of the above terms are not necessarily the same embodiment / method or example. Moreover, the specific features, structures, materials or characteristics described may be combined in a suitable manner in any one or more embodiments / methods or examples. In addition, those skilled in the art may combine and combine the different embodiments / methods or examples described in this specification and the features of the different embodiments / methods or examples, unless they are mutually inconsistent.

[0147] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of the technical features being referred to. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one of such features. Throughout the description of this application, "plurality" means at least two, for example, two, three, etc., unless otherwise specifically defined.

[0148] Those skilled in the art will appreciate that the above embodiments are merely intended to clearly illustrate the present disclosure and are not intended to limit the scope of the present disclosure. Other changes or modifications may be made based on the above disclosure, and such changes or modifications are still within the scope of the present disclosure.

Claims

1. A speech synthesis method, characterized in that: include: obtaining a first text and predetermined pitch control data; Acquiring a first acoustic feature of the first text; obtaining a prosody control feature having a spectrum length identical to that of the first acoustic feature according to the pitch control data, wherein the pitch control data includes a predetermined audio position and a corresponding fundamental frequency control value; superimposing the prosody control feature and the first acoustic feature to obtain a second acoustic feature, wherein a fundamental frequency value corresponding to the predetermined audio position in the second acoustic feature is determined by the fundamental frequency value corresponding to the predetermined audio position in the first acoustic feature and the fundamental frequency control value; synthesizing a second speech of the first text using the second acoustic feature, Before obtaining the prosody control feature having the same spectrum length as the first acoustic feature according to the pitch control data, the method further includes: synthesizing the first speech of the first text using the first acoustic feature; Obtaining a prosody control feature having the same spectrum length as the first acoustic feature according to the pitch control data, including: aligning the first text and the first speech in the time dimension to obtain word-level phoneme duration information; performing an interpolation operation on the phoneme duration information and the pitch control data, the result of the interpolation operation being the prosody control feature.

2. The speech synthesis method according to claim 1, wherein: Before obtaining the prosody control feature having the same spectrum length as the first acoustic feature according to the pitch control data, the method further includes adjusting the fundamental frequency control value in the pitch control data according to the pre-acquired fundamental frequency adjustment parameter of the selected speaker.

3. The speech synthesis method according to claim 1, wherein: The obtaining of a prosody control feature having a spectrum length identical to that of the first acoustic feature according to the pitch control data includes: performing an interpolation operation on the pitch control data, wherein a result of the interpolation operation is the prosody control feature.

4. The speech synthesis method according to claim 1, wherein: Before superimposing the prosody control feature with the first acoustic feature to obtain a second acoustic feature, the method further includes: performing a decompression process on the first acoustic feature; After superimposing the prosody control feature and the first acoustic feature to obtain a second acoustic feature, the method further includes: performing compression processing on the second acoustic feature so that the values ​​of elements in the second acoustic feature are within a predetermined interval.

5. The speech synthesis method according to claim 4, characterized in that: The speech synthesis method is implemented by a speech synthesis model, and the speech synthesis model is trained in the following manner: Acquire corpus for training the speech synthesis model, the corpus comprising a plurality of speech-text pairs, each of the speech-text pairs comprising a second text and its corresponding real speech; Screening out speech-text pairs whose fundamental frequency characteristics of real speech meet predetermined conditions from the corpus; Performing compression processing on the fundamental frequency features of the real speech in each screened speech-text pair to obtain the original fundamental frequency features of the speech-text pair; The speech synthesis model is obtained by training using the screened speech-text pairs and their original fundamental frequency features, wherein the second acoustic feature corresponding to the second text in the speech-text pair is obtained by running the speech synthesis model, and compression processing is performed on the second acoustic feature corresponding to the second text so that the value of the element in the second acoustic feature is within a predetermined interval; and the parameters of the speech synthesis model are updated by comparing the second acoustic feature with the original fundamental frequency feature of the speech-text pair.

6. A speech synthesis device, characterized in that: include: an acquiring unit, configured to acquire a first text and predetermined pitch control data; an acoustic feature extraction unit, configured to obtain a first acoustic feature of the first text; a prosody control feature extraction unit, configured to obtain a prosody control feature having a spectrum length identical to that of the first acoustic feature based on the pitch control data, wherein the pitch control data includes a predetermined audio position and a corresponding fundamental frequency control value; a feature superposition unit, configured to superimpose the prosody control feature and the first acoustic feature to obtain a second acoustic feature, wherein a fundamental frequency value corresponding to the predetermined audio position in the second acoustic feature is determined by the fundamental frequency value corresponding to the predetermined audio position in the first acoustic feature and the fundamental frequency control value; a speech generation unit, configured to synthesize a second speech of the first text using the second acoustic feature, Before obtaining the prosody control feature having the same spectrum length as the first acoustic feature according to the pitch control data, the method further includes: synthesizing a first speech of the first text by using the first acoustic feature through a speech generation unit; Obtaining a prosody control feature having the same spectrum length as the first acoustic feature according to the pitch control data, including: aligning the first text and the first speech in the time dimension to obtain word-level phoneme duration information; performing an interpolation operation on the phoneme duration information and the pitch control data, the result of the interpolation operation being the prosody control feature.

7. An electronic device, characterized in that: include: a memory storing execution instructions; as well as A processor, wherein the processor executes the execution instruction stored in the memory, so that the processor executes the speech synthesis method according to any one of claims 1 to 5.

8. A readable storage medium, characterized in that: The readable storage medium stores execution instructions, which are used to implement the speech synthesis method according to any one of claims 1 to 5 when executed by a processor.

Images