Speech synthesis method, speech editing model training method, speech synthesis device and speech editing model training equipment

Abstract

The invention provides a speech synthesis method, a speech editing model training device and equipment, and belongs to the technical field of computers. According to the method, on the basis of not modifying the original speech synthesis model, the speech signal containing the sublanguage fragment can be generated through the speech editing model by cooperating with any type of speech synthesis model, so that the naturalness and expressive force of speech synthesis are remarkably improved, and the speech synthesis effect is improved.

Description

Technical Field

[0001] The present disclosure relates to the field of computer technologies, and particularly to a method for speech synthesis, training of a speech editing model, an apparatus, and a device. Background Art

[0002] Text-to-Speech (TTS) technology is a technology that converts text information into speech signals. Currently, end-to-end models based on deep learning can generate natural speech close to the human level. However, existing technologies lack the ability to model paralinguistic events (such as laughter, crying, sighing, coughing, etc.) that contain rich emotional and interaction information, resulting in poor speech synthesis effects. Therefore, a new speech synthesis scheme is needed to solve the above problems. Summary of the Invention

[0003] The present disclosure provides a method for speech synthesis, training of a speech editing model, an apparatus, and a device. This solution can work in cooperation with any type of speech synthesis model without modifying the original speech synthesis model, and generate speech signals containing paralinguistic segments through a speech editing model, significantly improving the naturalness and expressiveness of speech synthesis and enhancing the effect of speech synthesis.

[0004] According to one aspect of the embodiments of the present disclosure, a method for speech synthesis is provided, and the method includes:

[0005] Processing a first text through a speech synthesis model to obtain a first speech signal, where the speech synthesis model is used to generate a corresponding speech signal according to the input text, and the first text includes paralinguistic tags;

[0006] Masking the speech segment corresponding to the paralinguistic tag in the first speech signal to obtain a second speech signal;

[0007] Processing the first text and the second speech signal through a speech editing model to obtain a third speech signal, where the speech editing model is used to generate paralinguistic segments, and the third speech signal includes paralinguistic segments corresponding to the paralinguistic tags.

[0008] According to another aspect of the embodiments of the present disclosure, a method for training a speech editing model is provided, and the method includes:

[0009] Processing a sample text through a speech synthesis model to obtain an intermediate speech signal, where the speech synthesis model is used to generate a corresponding speech signal according to the input text, and the sample text includes sample paralinguistic tags;

[0010] Upsample the sample text to obtain an aligned text sequence, where the length of the aligned text sequence is consistent with the number of frames of the intermediate speech signal. For any morpheme in the sample text, the number of occurrences of the morpheme in the aligned text sequence is consistent with the number of frames in the intermediate speech signal;

[0011] Mask the speech segment corresponding to the sample paralinguistic label in the intermediate speech signal to obtain a masked speech signal;

[0012] Train a speech editing model based on the aligned text sequence, the masked speech signal, and the intermediate speech signal. The speech editing model is used to generate paralinguistic segments.

[0013] According to another aspect of the embodiments of the present disclosure, a speech synthesis device is provided. The device includes:

[0014] A text processing unit configured to process a first text through a speech synthesis model to obtain a first speech signal. The speech synthesis model is used to generate a corresponding speech signal according to the input text, and the first text includes paralinguistic labels;

[0015] A signal masking unit configured to mask the speech segment corresponding to the paralinguistic label in the first speech signal to obtain a second speech signal;

[0016] A signal editing unit configured to process the first text and the second speech signal through a speech editing model to obtain a third speech signal. The speech editing model is used to generate paralinguistic segments, and the third speech signal includes paralinguistic segments corresponding to the paralinguistic labels.

[0017] In some embodiments, the signal masking unit is configured to determine the start frame and end frame of the speech segment corresponding to the paralinguistic label through a duration alignment model. The duration alignment model is used to predict the duration of phonemes; replace the speech segment between the start frame and the end frame in the first speech signal with the masking signal to obtain the second speech signal.

[0018] In some embodiments, the signal editing unit is configured to upsample the first text to obtain a second text. The length of the second text is consistent with the number of frames of the second speech signal. The first text includes at least one morpheme, and the number of occurrences of any morpheme in the first text in the second text is consistent with the number of frames of the morpheme in the second speech signal; splice the second text and the second speech signal to obtain a spliced signal; process the spliced signal through the speech editing model to obtain the third speech signal.

[0019] According to another aspect of the embodiments of the present disclosure, there is provided a training apparatus for a speech editing model, and the method includes:

[0020] A text processing unit configured to process a sample text through a speech synthesis model to obtain an intermediate speech signal, where the speech synthesis model is used to generate a corresponding speech signal according to the input text, and the sample text includes sample paralinguistic tags;

[0021] An upsampling unit configured to upsample the sample text to obtain an aligned text sequence, where the length of the aligned text sequence is consistent with the number of frames of the intermediate speech signal, the sample text includes at least one morpheme, and the number of any morpheme in the sample text in the aligned text sequence is consistent with the number of frames of the morpheme in the intermediate speech signal;

[0022] A signal masking unit configured to mask a speech segment corresponding to the sample paralinguistic tag in the intermediate speech signal to obtain a masked speech signal;

[0023] A model training unit configured to train a speech editing model according to the aligned text sequence, the masked speech signal, and the intermediate speech signal, where the speech editing model is used to generate paralinguistic segments.

[0024] In some embodiments, the signal masking unit is configured to determine a start frame and an end frame of a speech segment corresponding to the sample paralinguistic tag through a duration alignment model, where the duration alignment model is used to predict the duration of phonemes; replace the speech segment between the start frame and the end frame in the intermediate speech signal with the sample masking signal to obtain the masked speech signal.

[0025] In some embodiments, the model training unit is configured to process the aligned text sequence and the masked speech signal through a speech editing model to obtain an output speech signal, where the output speech signal includes a paralinguistic segment corresponding to the paralinguistic tag; determine a mean square error loss based on the intermediate speech signal and the output speech signal; and train the speech editing model based on the mean square error loss.

[0026] According to another aspect of the embodiments of the present disclosure, there is provided an electronic device, and the electronic device includes:

[0027] One or more processors;

[0028] A memory for storing executable program code of the processor;

[0029] Wherein, the processor is configured to execute the program code to implement the above-mentioned speech synthesis method or implement the above-mentioned training method for the speech editing model.

[0030] According to another aspect of the embodiments of the present disclosure, a computer-readable storage medium is provided. When instructions in the computer-readable storage medium are executed by a processor of an electronic device, the electronic device can execute the above voice synthesis method or the above training method of the voice editing model.

[0031] According to another aspect of the embodiments of the present disclosure, a computer program product is provided, including a computer program which, when executed by a processor, implements the above voice synthesis method or implements the training of the above voice editing model.

[0032] The embodiments of the present disclosure provide a voice synthesis solution. Without modifying the original voice synthesis model, this solution can work in cooperation with any type of voice synthesis model, and generate a voice signal containing a paralinguistic segment through a voice editing model, significantly improving the naturalness and expressiveness of voice synthesis and enhancing the effect of voice synthesis.

[0033] It should be understood that the above general description and the following detailed description are only exemplary and explanatory, and cannot limit the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] The drawings herein are incorporated into the specification and constitute a part of this specification, showing embodiments consistent with the present disclosure, and are used together with the specification to explain the principles of the present disclosure, and do not constitute an improper limitation to the present disclosure.

[0035] Figure 1 is a schematic diagram of an implementation environment of a voice synthesis method shown according to an exemplary embodiment.

[0036] Figure 2 is a flowchart of a voice synthesis method shown according to an exemplary embodiment.

[0037] Figure 3 is a flowchart of another voice synthesis method shown according to an exemplary embodiment.

[0038] Figure 4 is a schematic diagram of a voice synthesis solution provided according to an exemplary embodiment.

[0039] Figure 5 is a flowchart of a training method of a voice editing model shown according to an exemplary embodiment.

[0040] Figure 6 is a flowchart of another training method of a voice editing model shown according to an exemplary embodiment.

[0041] Figure 7It is a schematic diagram of a training scheme for a speech editing model provided according to an exemplary embodiment.

[0042] Figure 8 It is a block diagram of a speech synthesis device shown according to an exemplary embodiment.

[0043] Figure 9 It is a block diagram of a training device for a speech editing model shown according to an exemplary embodiment.

[0044] Figure 10 It is a block diagram of an electronic device shown according to an exemplary embodiment. Detailed implementation manners

[0045] In order to enable those of ordinary skill in the art to better understand the technical solutions of the present disclosure, the technical solutions in the embodiments of the present disclosure will be clearly and completely described below with reference to the accompanying drawings.

[0046] It should be noted that the terms "first", "second", etc. in the specification and claims of the present disclosure and the above-mentioned drawings are used to distinguish similar objects, and do not necessarily have to be used to describe a specific order or sequence. It should be understood that such used data can be interchanged under appropriate circumstances so that the embodiments of the present disclosure described herein can be implemented in an order other than those illustrated or described herein. The implementation manners described in the following exemplary embodiments do not represent all implementation manners consistent with the present disclosure. On the contrary, they are merely examples of devices and methods consistent with some aspects of the present disclosure as detailed in the appended claims.

[0047] It should be noted that the information (including but not limited to user device information, user personal information, etc.), data (including but not limited to data for analysis, stored data, displayed data, etc.) and signals involved in the present disclosure are all authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data need to comply with the relevant laws, regulations and standards of relevant countries and regions. For example, various data signals involved in the present disclosure are obtained under full authorization.

[0048] Figure 1 It is a schematic diagram of an implementation environment of a speech synthesis method shown according to an exemplary embodiment. Refer to Figure 1 , the implementation environment specifically includes: an electronic device 101 and a server 102. The electronic device 101 can be connected to the server 102 through a wireless network or a wired network.

[0049] The electronic device 101 can be at least one of devices such as a smart phone, a smart watch, a desktop computer, a laptop computer, an MP3 player (Moving Picture Experts Group Audio Layer III), an MP4 (Moving Picture Experts Group Audio Layer IV) player, and a laptop portable computer, etc.

[0050] The electronic device 101 can generally refer to one of multiple electronic devices. In this embodiment, the electronic device 101 is taken as an example for illustration. Those skilled in the art can know that the number of the above-mentioned electronic devices can be more or less. For example, the above-mentioned electronic devices can be several, or the above-mentioned electronic devices can be dozens or hundreds, or a larger number. The embodiments of the present disclosure do not limit the number and type of the electronic devices.

[0051] The server 102 is at least one of a server, multiple servers, a cloud computing platform, and a virtualization center. Optionally, the number of the above-mentioned servers can be more or less, and the embodiments of the present disclosure do not limit this. Of course, the server 102 can also include other functional servers to provide more comprehensive and diversified services. In some embodiments, the server 102 undertakes the main computing work, and the electronic device 101 undertakes the secondary computing work; or, the server 102 undertakes the secondary computing work, and the electronic device 101 undertakes the main computing work; or, the server 102 and the electronic device 101 adopt a distributed computing architecture for collaborative computing. The server 102 can be connected to the electronic device 101 and other electronic devices through a wireless network or a wired network. Optionally, the number of the above-mentioned servers can be more or less, and the embodiments of the present disclosure do not limit this.

[0052] Figure 2 is a flowchart of a speech synthesis method shown according to an exemplary embodiment, as Figure 2 shown, this method is executed by an electronic device and includes the following steps:

[0053] In step S201, a first text is processed through a speech synthesis model to obtain a first speech signal.

[0054] In the embodiments of the present disclosure, the speech synthesis model (Text-to-Speech, TTS) is used to generate a corresponding speech signal according to the input text. That is to say, the speech synthesis model can receive the input text information, and then through a series of processes and operations, convert the text into a corresponding speech signal, so that the computer can "speak" the given text content. For example, common voice assistants. The speech synthesis model is any type of speech synthesis model.

[0055] The first text refers to the text information input into the speech synthesis model, and the first text includes paralinguistic tags. For example, the first text is "Hello <hahaha> World". Here, <hahaha> is the paralinguistic tag.

[0056] The first speech signal refers to the output result in the form of speech obtained after the first text is processed by the speech synthesis model, and it is an audio signal that can be played for people to hear.

[0057] In step S202, the speech segment corresponding to the paralinguistic tag in the first speech signal is masked to obtain the second speech signal.

[0058] In the embodiment of the present disclosure, since the first text contains paralinguistic tags, after being processed by the speech synthesis model to generate the first speech signal, the part of the speech content related to these paralinguistic tags is the speech segment corresponding to the paralinguistic tag.

[0059] The masking signal refers to a special audio signal. The role of the masking signal is to cover or hide a specific part of other audio signals. Optionally, the masking signal can be a noise signal (such as white noise, etc.), or other audio with specific characteristics (such as a silence signal, etc.), so that the speech segment covered by the masking signal is difficult to be clearly heard or recognized.

[0060] The second speech signal is a new speech signal obtained after the operation of masking the speech segment corresponding to the paralinguistic tag on the basis of the first speech signal.

[0061] In step S203, the first text and the second speech signal are processed by a speech editing model to obtain the third speech signal.

[0062] In the embodiment of the present disclosure, the speech editing model is used to generate paralinguistic segments. That is, the speech editing model is used to process the input text and speech signal, and can generate paralinguistic segments related to the paralinguistic tags to achieve the editing and modification of the speech signal, so that the speech is more in line with the expected expression effect. The paralinguistic segment refers to an audio segment containing specific speech features, and these features are indicated by paralinguistic tags, such as specific intonation changes, stress emphasis, speech rate adjustment, etc. The paralinguistic segment is the product generated by the speech editing model and is used to supplement or modify the relevant part of the speech signal.

[0063] The third speech signal is a new speech signal obtained after the first text and the second speech signal are processed by the speech editing model. The third speech signal includes paralinguistic segments corresponding to the paralinguistic tags. That is, the third speech signal restores or enhances the originally masked paralinguistic features.

[0064] Embodiments of the present disclosure provide a speech synthesis solution. This solution can work in cooperation with any type of speech synthesis model without modifying the original speech synthesis model, and generate a speech signal containing paralinguistic segments through a speech editing model, significantly improving the naturalness and expressiveness of speech synthesis and enhancing the effect of speech synthesis.

[0065] In some embodiments, masking the speech segment corresponding to the paralinguistic label in the first speech signal to obtain a second speech signal includes:

[0066] Determining the start frame and end frame of the speech segment corresponding to the paralinguistic label through a duration alignment model, where the duration alignment model is used to predict the duration of phonemes;

[0067] Replacing the speech segment between the start frame and the end frame in the first speech signal with a masking signal to obtain a second speech signal.

[0068] Precisely locating the start and end frames of the speech segment corresponding to the paralinguistic label through the duration alignment model provides an accurate basis for the replacement operation. Replacing a specific segment of the first speech signal with a masking signal to obtain a second speech signal enables the model to learn paralinguistic information and improves the training efficiency of the model.

[0069] In some embodiments, processing the first text and the second speech signal through a speech editing model to obtain a third speech signal includes:

[0070] Upsampling the first text to obtain a second text, where the length of the second text is the same as the number of frames of the second speech signal. The first text includes at least one morpheme, and the number of any morpheme in the first text in the second text is the same as the number of frames of that morpheme in the second speech signal;

[0071] Concatenating the second text and the second speech signal to obtain a concatenated signal;

[0072] Processing the concatenated signal through a speech editing model to obtain a third speech signal.

[0073] Upsampling the first text to obtain a second text with the same number of frames as the second speech signal ensures precise matching of the text and speech in the time dimension. Concatenating the two to form a concatenated signal provides richer information for the speech editing model. The speech editing model processes the concatenated signal to obtain a third speech signal, which can more precisely generate speech that conforms to the text semantics and speech characteristics, enhancing the naturalness and accuracy of speech expression.

[0074] The above Figure 2 shows a flowchart of a speech synthesis method of the present disclosure. The speech synthesis solution provided by the present disclosure will be further elaborated below. Figure 3is a flowchart of another speech synthesis method shown according to an exemplary embodiment. Refer to Figure 3 , this method is executed by an electronic device and includes the following steps:

[0075] In step S301, a first text is processed by a speech synthesis model to obtain a first speech signal.

[0076] In the embodiments of the present disclosure, the speech synthesis model is any existing speech synthesis model. The speech synthesis model is used to generate a corresponding speech signal according to the input text. Refer to the explanation of the speech synthesis model in the above step S201, which will not be elaborated here. The first text includes paralinguistic tags.

[0077] This step is the same as the above step S201 and will not be elaborated here.

[0078] In step S302, a duration alignment model is used to determine the start frame and end frame of the speech segment corresponding to the paralinguistic tag.

[0079] In the embodiments of the present disclosure, the duration alignment model is a model constructed based on bidirectional LSTM or Transformer. The duration alignment model is used to predict the duration of phonemes. That is, through learning and analyzing speech data, the duration alignment model can predict the duration of each phoneme in a specific speech environment. For example, according to factors such as different speech rates, intonations, and combinations of front and back phonemes, the model can estimate how long a certain phoneme should last. Among them, a phoneme is the smallest speech unit in speech, such as initial consonants like "b", "p", "m" in Chinese, vowels like "a", "o", "e", and " / p / ", " / b / ", " / m / " in English.

[0080] Since the first text contains paralinguistic tags, after the speech signal is generated, the part of the speech content associated with these paralinguistic tags constitutes the speech segment corresponding to the paralinguistic tag.

[0081] In the digital representation of the speech signal, speech is usually segmented into a series of frames (which can be understood as a small time segment of speech). The start frame refers to the frame at which the speech segment corresponding to the paralinguistic tag starts in the entire speech signal sequence. The start frame determines the start time position of the speech segment in the speech signal. Corresponding to the start frame, the end frame refers to the frame at which the speech segment corresponding to the paralinguistic tag ends in the entire speech signal sequence. The end frame determines the end time position of the speech segment in the speech signal.

[0082] In step S303, the speech segment between the start frame and the end frame in the first speech signal is replaced with a masking signal to obtain a second speech signal.

[0083] In an embodiment of the present disclosure, the first speech signal is a speech output obtained by processing a first text containing a paralinguistic tag by a speech synthesis model. First, a specific speech segment bounded by a start frame and an end frame is found in the first speech signal. This speech segment is usually related to the paralinguistic tag and represents a part with special speech features. Then, this part of the speech segment is removed from the first speech signal and replaced with a masking signal. The masking signal can be any one of Gaussian noise, white noise, or silence. After such a replacement operation, a new speech signal, that is, the second speech signal, is obtained.

[0084] In step S304, the first text is upsampled to obtain a second text.

[0085] In an embodiment of the present disclosure, upsampling is an operation to increase the number of data points in the field of signal processing. In the context of text processing, upsampling the first text means increasing the length or the number of elements of the text. That is, the original text elements are copied or extended according to certain rules. By performing upsampling processing on the first text, the length and content distribution of the text can be made to match the second speech signal. For example, operations such as copying and extending the morphemes in the first text are performed to increase the length of the text.

[0086] After upsampling, the length of the second text is consistent with the number of frames of the second speech signal. That is, if the second speech signal is segmented into 100 frames, the length of the second text also needs to be adjusted to correspond to the length of these 100 frames.

[0087] The first text includes at least one morpheme. For any morpheme in the first text, in the second text obtained after upsampling, the number of this morpheme should be consistent with the number of frames it corresponds to in the second speech signal. For example, for the morpheme "good" in the first text, if it corresponds to 5 frames in the second speech signal, then in the second text, the number of the morpheme "good" should also be 5.

[0088] Through such processing, a more accurate correspondence relationship between the text and the speech signal in the time dimension can be established, which is helpful for subsequent further speech processing and analysis tasks.

[0089] In step S305, the second text and the second speech signal are concatenated to obtain a concatenated signal.

[0090] In an embodiment of the present disclosure, the second text is a text sequence with a specific correspondence relationship in length and content distribution with the speech signal after processing, and is a one-dimensional sequence. The speech signal is also a one-dimensional sequence. Concatenation means combining two one-dimensional sequences into a new sequence structure. For example, if the second text is 1*N and the speech signal is also 1*N, then the concatenated signal is 1*N*2.

[0091] In step S306, the spliced signal is processed by a speech editing model to obtain a third speech signal.

[0092] In the embodiments of the present disclosure, the speech editing model is used to generate paralinguistic segments. Based on the function of generating paralinguistic segments, the speech editing model generates corresponding paralinguistic segments according to the information in the spliced signal (mainly the paralinguistic label information contained therein), and integrates these paralinguistic segments into the speech signal. After such processing, a new speech signal, that is, the third speech signal, is obtained. This third speech signal contains paralinguistic segments corresponding to the paralinguistic labels, thereby making the speech more rich and accurate in expression and capable of better conveying the non-verbal information that the text hopes to express.

[0093] To make the speech synthesis solution provided by the embodiments of the present disclosure easier to understand, refer to Figure 4 as shown Figure 4 is a schematic diagram of a speech synthesis solution provided according to an exemplary embodiment. As Figure 4 shown, the first text is: Hello <hahaha> world. The first speech signal 401 is obtained by processing the first text through TTS. The duration of <hahaha> is predicted through a duration alignment model (MFA, Montreal Forced Alignment) to obtain the start frame t_s and the end frame t_e. The speech segment between t_s and t_e is replaced by a masking signal to obtain the second speech signal 402. The first text is upsampled to obtain the second text 403. The second text and the second speech signal are spliced to obtain the spliced signal 404. The spliced signal 404 is processed by a speech editing model 405 to obtain the third speech signal 406.

[0094] The embodiments of the present disclosure provide a speech synthesis solution. This solution can work in cooperation with any type of speech synthesis model without modifying the original speech synthesis model, and generate a speech signal containing paralinguistic segments through a speech editing model, significantly improving the naturalness and expressiveness of speech synthesis and enhancing the effect of speech synthesis.

[0095] Figure 5 is a flowchart of a training method of a speech editing model shown according to an exemplary embodiment. As Figure 5 shown, this method is executed by an electronic device and includes the following steps:

[0096] In step S501, the sample text is processed by a speech synthesis model to obtain an intermediate speech signal.

[0097] In the embodiments of the present disclosure, the function of the speech synthesis model is to convert the input text information into corresponding speech signals. That is, the speech synthesis model is used to generate corresponding speech signals according to the input text. The speech synthesis model is a pre-trained model and can be any existing TTS (Text-to-Speech) model.

[0098] The sample text includes sample paralinguistic labels, which are special marks or symbols (or words in a specific format) added to the sample text. The paralinguistic labels are used to indicate that when the speech synthesis model generates speech signals, specific non-verbal information features are given at the corresponding positions, such as specific intonation changes (such as rising tone, falling tone), the position of stress (emphasizing certain words), the speed of speech, the duration of pauses, etc. Optionally, the sample text is: Hello <hahaha> World. Here, <hahaha> is the sample paralinguistic label.

[0099] The intermediate speech signal is the speech output result obtained after the speech synthesis model processes the sample text. Since existing TTS models cannot accurately model paralinguistic information, this intermediate speech signal usually lacks paralinguistic information.

[0100] In step S502, the sample text is upsampled to obtain an aligned text sequence.

[0101] In the embodiments of the present disclosure, upsampling refers to an operation of increasing the number of elements (such as characters, morphemes, etc.) in the text to expand the text length. Usually, according to certain rules or algorithms, the elements in the original text are copied or new elements are inserted to change the length and structure of the text, thereby obtaining a new text sequence, that is, the aligned text sequence.

[0102] The length of the aligned text sequence is consistent with the number of frames of the intermediate speech signal. If the intermediate speech signal is segmented into 100 frames, then the length of the aligned text sequence obtained after upsampling should also be adjusted to the corresponding length to ensure that the text and the speech are matched in terms of time span.

[0103] The sample text includes at least one morpheme. For any morpheme in the sample text, the number of occurrences of this morpheme in the aligned text sequence is consistent with the number of frames of this morpheme in the intermediate speech signal. For example, a certain morpheme "good" in the sample text corresponds to 5 frames in the intermediate speech signal, then in the aligned text sequence obtained after upsampling, the number of occurrences of the morpheme "good" should also be 5.

[0104] In step S503, the speech segment corresponding to the sample paralinguistic label in the intermediate speech signal is masked to obtain a masked speech signal.

[0105] In the embodiments of the present disclosure, first, the voice segments corresponding to the sample paralanguage labels are found in the intermediate voice signal. These voice segments originally contain the special voice features indicated by the sample paralanguage labels. Then, these voice segments are masked to generate a new voice signal, namely, the masked voice signal.

[0106] In step S504, the voice editing model is trained according to the aligned text sequence, the masked voice signal, and the intermediate voice signal.

[0107] In the embodiments of the present disclosure, the aligned text sequence provides text information that precisely corresponds to the voice signal in time, helping the model understand the relationship between text and voice.

[0108] The masked voice signal is obtained by replacing the voice segments corresponding to the sample paralanguage labels in the intermediate voice signal with the sample masking signal. The masked voice signal provides a special voice sample for model training, enabling the model to learn how to process and restore these masked voice features.

[0109] The intermediate voice signal is obtained by processing the sample text containing the sample paralanguage labels through the voice synthesis model. The intermediate voice signal contains the voice content corresponding to the sample text and the original voice features given by the sample paralanguage labels.

[0110] The voice editing model is used to generate paralanguage segments. During the training process, the voice editing model analyzes and learns these input data. By adjusting its own parameters, it gradually masters the ability to generate appropriate paralanguage segments according to different inputs (text and voice), thereby achieving effective editing of the voice signal.

[0111] By processing the sample text through the voice synthesis model to obtain the intermediate voice signal, the text-to-speech ability can be utilized to provide basic voice data for subsequent training. By upsampling the sample text to obtain the aligned text sequence, the text and the number of voice frames are precisely corresponded, enabling the model to better learn the association between text and voice. By replacing the voice segments corresponding to the sample paralanguage labels in the intermediate voice signal with the sample masking signal, the masked voice signal is obtained, enhancing the model's learning ability for paralanguage. By training the voice editing model with the aligned text sequence, the masked voice signal, and the intermediate voice signal, the model can accurately generate paralanguage segments, enhancing the naturalness and emotionality of voice expression.

[0112] In some embodiments, masking the voice segments corresponding to the sample paralanguage labels in the intermediate voice signal to obtain the masked voice signal includes:

[0113] Determine the start frame and end frame of the speech segment corresponding to the sample paralinguistic label through a duration alignment model, where the duration alignment model is used to predict the duration of phonemes;

[0114] Replace the speech segment between the start frame and the end frame in the intermediate speech signal with the sample masking signal to obtain a masked speech signal.

[0115] By accurately positioning with the help of the duration alignment model, it can ensure that the speech segment to be replaced is accurately found. Replacing the segment corresponding to the sample paralinguistic label in the intermediate speech signal with the sample masking signal to form a masked speech signal can simulate a speech interference scenario. This is beneficial for the model to learn paralinguistic information and improve the generation of paralinguistic segments by the model.

[0116] In some embodiments, train a speech editing model according to the aligned text sequence, the masked speech signal, and the intermediate speech signal, including:

[0117] Process the aligned text sequence and the masked speech signal through the speech editing model to obtain an output speech signal, where the output speech signal includes a paralinguistic segment corresponding to the paralinguistic label;

[0118] Determine the mean square error loss based on the intermediate speech signal and the output speech signal;

[0119] Train the speech editing model based on the mean square error loss.

[0120] Processing the aligned text sequence and the masked speech signal through the speech editing model to generate an output speech signal enables the model to learn to integrate multi-source information to generate paralinguistic segments. Using the intermediate and output speech signals to determine the mean square error loss provides a quantitative metric for evaluating the model's performance. Training the model based on this loss can optimize the model parameters, enabling it to generate speech that better fits the original features, improving the accuracy and naturalness of speech editing, and enhancing the model's practicality in speech processing tasks.

[0121] The above Figure 5 shows a flowchart of a method for training a speech editing model of the present disclosure. Next, the training scheme of the speech editing model provided by the present disclosure will be further elaborated. Figure 6 is a flowchart of another method for training a speech editing model shown according to an exemplary embodiment. Refer to Figure 6 , this method is executed by an electronic device and includes the following steps:

[0122] In step S601, process the sample text through a speech synthesis model to obtain an intermediate speech signal.

[0123] In an embodiment of the present disclosure, the speech synthesis model is used to generate a corresponding speech signal according to the input text, and the sample text includes sample paralinguistic tags. This step is the same as step S501 above. Refer to step S501 above and will not be elaborated here.

[0124] In step S602, the sample text is upsampled to obtain an aligned text sequence.

[0125] In an embodiment of the present disclosure, this step is the same as step S502 above. Refer to step S502 above and will not be elaborated here.

[0126] In step S603, the start frame and end frame of the speech segment corresponding to the sample paralinguistic tag are determined through the duration alignment model.

[0127] In an embodiment of the present disclosure, the duration alignment model can establish the corresponding relationship between each element (such as phonemes, syllables, etc.) in the speech and time through the analysis of the speech signal. That is, the duration alignment model is used to predict the duration of phonemes. Among them, a phoneme is the smallest speech unit in speech, such as the initial consonants "b", "p", "m", etc. in Chinese, and the finals "a", "o", "e", etc.; " / p / ", " / b / ", " / m / " in English, etc. Different phonemes have different pronunciation characteristics and durations.

[0128] When the speech signal is digitized, the speech signal is usually segmented into a series of short time segments, and each segment is called a frame. The start frame refers to the frame at which the speech segment corresponding to the sample paralinguistic tag starts in the entire speech signal sequence, determining the start time position of the speech segment in the speech signal. Corresponding to the start frame, the end frame refers to the frame at which the speech segment corresponding to the sample paralinguistic tag ends in the entire speech signal sequence, determining the end time position of the speech segment in the speech signal.

[0129] In step S604, the speech segment between the start frame and the end frame in the intermediate speech signal is masked to obtain a masked speech signal.

[0130] In an embodiment of the present disclosure, first, a specific speech segment defined by the start frame and the end frame is determined in the intermediate speech signal. This speech segment is usually related to the sample paralinguistic tag and has specific speech characteristics. Then, this part of the speech segment is removed from the intermediate speech signal and replaced with a sample masking signal. After such a replacement operation, a new speech signal, that is, a masked speech signal, is generated.

[0131] In step S605, the aligned text sequence and the sample masking signal are processed through the speech editing model to obtain an output speech signal.

[0132] In an embodiment of the present disclosure, first, an aligned text sequence and a sample masking signal are provided as inputs to a speech editing model. The speech editing model analyzes the input aligned text sequence, identifies the paralinguistic tag information therein, and at the same time combines the characteristics of the sample masking signal, and uses the knowledge and patterns learned by itself to generate paralinguistic segments corresponding to the paralinguistic tags. Then, the model incorporates the generated paralinguistic segments into the processing process and finally synthesizes a complete speech signal, that is, the output speech signal. The output speech signal includes paralinguistic segments corresponding to the paralinguistic tags. That is to say, the output speech signal not only contains the basic speech content corresponding to the text, but also contains the paralinguistic segments indicated by the paralinguistic tags, making the speech more rich and natural in expression and capable of better conveying the emotions, tones and key information contained in the text.

[0133] In step S606, based on the intermediate speech signal and the output speech signal, the mean squared error loss is determined.

[0134] In an embodiment of the present disclosure, the mean squared error loss (MSE) is a commonly used metric for measuring the difference between two signals.

[0135] First, the intermediate speech signal and the output speech signal are digitally represented. Usually, the speech signal is converted into a discrete numerical sequence. For example, through sampling and quantization operations, the continuous speech signal is converted into a series of digital samples.

[0136] For each corresponding sample point, the difference between them is calculated. That is, for the i-th sample value x_i in the intermediate speech signal and the i-th sample value y_i in the output speech signal, the difference d_i = x_i - y_i is calculated.

[0137] Then, these differences are squared to obtain d_i^2. This is to eliminate the positive and negative effects of the differences and highlight the effects of larger differences.

[0138] Then, the average value of all squared differences is calculated to obtain the mean squared error loss L, and the formula is:

[0139]

[0140] where n is the total number of sample points.

[0141] In step S607, based on the mean squared error loss, the speech editing model is trained.

[0142] In the embodiments of the present disclosure, the mean squared error loss can reflect the degree of difference between the output speech signal generated by the speech editing model and the original intermediate speech signal. The smaller the loss value, the closer the output speech signal is to the intermediate speech signal, and the better the performance of the speech editing model; conversely, the larger the loss value, the greater the difference between the two, indicating that the model performance needs to be improved. By minimizing the mean squared error loss, the parameters of the speech editing model can be optimized to make the generated speech closer to the expected intermediate speech signal, thereby improving the accuracy and robustness of the model.

[0143] It should be noted that the above speech editing model can be a FlowMatching model, which learns the conditional flow from the masked speech to the real paralinguistic segment, models the gradual change process of the data distribution through continuous normalizing flow (CNF), and uses the conditional flow matching loss function (CFM loss) to optimize the vector field of the CNF. During training, there is no need to explicitly access the gradient of the entire path or the target distribution, and it is directly calculated through the path generated by the samples.

[0144] Optionally, the speech editing model can also be a diffusion model, which generates paralinguistic through iterative denoising. Compared with the FlowMatching model, the computational cost is relatively high.

[0145] It should also be noted that for the above model training process, only a small amount of labeled data (1 hour of paralinguistic audio + corresponding text labels) is required to train a high-quality paralinguistic model, and the data requirement is reduced by 90% compared with the traditional method. If semi-supervised learning is used for training and unlabeled speech data is utilized to extract paralinguistic features through a paralinguistic predictor, complex self-supervised tasks need to be designed.

[0146] In addition, by adjusting the position and density of the paralinguistic labels, the frequency and intensity of the generated paralinguistic can be dynamically controlled.

[0147] To make the above model training process easier to understand, refer to Figure 7 shown in Figure 7 is a schematic diagram of a training scheme for a speech editing model provided according to an exemplary embodiment. As shown in Figure 7As shown, the sample text is: Hello <hahaha> World. The intermediate speech signal 701 is obtained by processing the sample text through TTS. The sample text is upsampled to obtain the aligned text sequence 702. Through the duration alignment model, the duration of <hahaha> is predicted to obtain the start frame t_s and the end frame t_e. The speech segment between t_s and t_e is replaced by the masking signal to obtain the masked speech signal 703. The aligned text sequence and the masked speech signal are concatenated to obtain the concatenated signal 704. The concatenated signal 704 is processed by the speech editing model 705 to obtain the output speech signal 706. Based on the intermediate speech signal and the output speech signal, the mean square error loss is determined, and the speech editing model is trained based on this mean square error loss.

[0148] Processing the sample text through the speech synthesis model to obtain the intermediate speech signal can provide the basic speech data for subsequent training by virtue of its text-to-speech ability. Upsampling the sample text to obtain the aligned text sequence makes the text correspond precisely to the speech frames, enabling the model to better learn the association between text and speech. By replacing the speech segment corresponding to the sample paralinguistic label in the intermediate speech signal with the sample masking signal, the masked speech signal is obtained, enhancing the model's ability to learn paralinguistics. By training the speech editing model with the aligned text sequence, the masked speech signal, and the intermediate speech signal, the model can accurately generate paralinguistic segments, enhancing the naturalness and emotionality of speech expression.

[0149] Figure 8 It is a block diagram of a speech synthesis device shown according to an exemplary embodiment. As Figure 8 shown, the device includes: a text processing unit 801, a signal masking unit 802, and a signal editing unit 803.

[0150] The text processing unit 801 is configured to process the first text through the speech synthesis model to obtain the first speech signal. The speech synthesis model is used to generate the corresponding speech signal according to the input text, and the first text includes paralinguistic labels;

[0151] The signal masking unit 802 is configured to mask the speech segment corresponding to the paralinguistic label in the first speech signal to obtain the second speech signal;

[0152] The signal editing unit 803 is configured to process the first text and the second speech signal through the speech editing model to obtain the third speech signal. The speech editing model is used to generate paralinguistic segments, and the third speech signal includes paralinguistic segments corresponding to the paralinguistic labels.

[0153] In some embodiments, the signal masking unit 802 is configured to determine the start frame and the end frame of the speech segment corresponding to the paralinguistic label through a duration alignment model, where the duration alignment model is used to predict the duration of phonemes; and replace the speech segment between the start frame and the end frame in the first speech signal with a masking signal to obtain a second speech signal.

[0154] In some embodiments, the signal editing unit 803 is configured to upsample the first text to obtain a second text, where the length of the second text is the same as the number of frames of the second speech signal, the first text includes at least one morpheme, and the number of any morpheme in the first text in the second text is the same as the number of frames of the morpheme in the second speech signal; splice the second text and the second speech signal to obtain a spliced signal; and process the spliced signal through a speech editing model to obtain a third speech signal.

[0155] The embodiments of the present disclosure provide a speech synthesis device, which can work in cooperation with any type of speech synthesis model without modifying the original speech synthesis model, and generate a speech signal including a paralinguistic segment through a speech editing model, significantly improving the naturalness and expressiveness of speech synthesis and enhancing the effect of speech synthesis.

[0156] Figure 9 is a block diagram of a training device for a speech editing model shown according to an exemplary embodiment. As Figure 9 shown, the device includes: a text processing unit 901, an upsampling unit 902, a signal masking unit 903, and a model training unit 904.

[0157] The text processing unit 901 is configured to process a sample text through a speech synthesis model to obtain an intermediate speech signal, where the speech synthesis model is used to generate a corresponding speech signal according to the input text, and the sample text includes a sample paralinguistic label;

[0158] The upsampling unit 902 is configured to upsample the sample text to obtain an aligned text sequence, where the length of the aligned text sequence is the same as the number of frames of the intermediate speech signal, the sample text includes at least one morpheme, and the number of any morpheme in the sample text in the aligned text sequence is the same as the number of frames of the morpheme in the intermediate speech signal;

[0159] The signal masking unit 903 is configured to mask the speech segment corresponding to the sample paralinguistic label in the intermediate speech signal to obtain a masked speech signal;

[0160] The model training unit 904 is configured to train a speech editing model according to the aligned text sequence, the masked speech signal, and the intermediate speech signal, where the speech editing model is used to generate a paralinguistic segment.

[0161] In some embodiments, the signal masking unit 903 is configured to determine the start frame and the end frame of the speech segment corresponding to the sample paralinguistic label through a duration alignment model, where the duration alignment model is used to predict the duration of phonemes; and replace the speech segment between the start frame and the end frame in the intermediate speech signal with a sample masking signal to obtain a masked speech signal.

[0162] In some embodiments, the model training unit 904 is configured to process the aligned text sequence and the masked speech signal through a speech editing model to obtain an output speech signal, where the output speech signal includes a paralinguistic segment corresponding to the paralinguistic label; determine a mean squared error loss based on the intermediate speech signal and the output speech signal; and train the speech editing model based on the mean squared error loss.

[0163] The embodiments of the present disclosure provide a training device for a speech editing model. By processing a sample text through a speech synthesis model to obtain an intermediate speech signal, the text-to-speech ability can be utilized to provide basic speech data for subsequent training. By upsampling the sample text to obtain an aligned text sequence, the text and the speech frame numbers are accurately corresponded, enabling the model to better learn the association between the text and the speech. By replacing the speech segment corresponding to the sample paralinguistic label in the intermediate speech signal with a sample masking signal to obtain a masked speech signal, the learning ability of the model for paralinguistic information is improved. By training the speech editing model with the aligned text sequence, the masked speech signal, and the intermediate speech signal, the model can accurately generate paralinguistic segments, enhancing the naturalness and emotionality of speech expression.

[0164] It should be noted that for the speech synthesis device and the training device for the speech editing model provided in the above embodiments, only the division of the above functional units is used for illustration. In practical applications, the above functions can be allocated to different functional units according to needs, that is, the internal structure of the electronic device is divided into different functional units to complete all or part of the functions described above. In addition, the speech synthesis device provided in the above embodiments and the method embodiments of the speech synthesis method belong to the same concept, and the specific implementation process is detailed in the method embodiments and will not be repeated here. The training device for the speech editing model provided in the above embodiments and the method embodiments of the training method for the speech editing model belong to the same concept, and the specific implementation process is detailed in the method embodiments and will not be repeated here.

[0165] Regarding the speech synthesis device and the training device for the speech editing model in the above embodiments, the specific manners in which each module performs operations have been described in detail in the embodiments of the related method, and will not be elaborated here.

[0166] In the embodiments of the present disclosure, the electronic device may be a terminal or a server. When the electronic device is a terminal, the terminal serves as the execution entity to implement the technical solutions provided by the embodiments of the present disclosure; when the electronic device is a server, the server serves as the execution entity to implement the technical solutions provided by the embodiments of the present disclosure; or, the technical solutions provided by the present disclosure are implemented through the interaction between the terminal and the server. The embodiments of the present disclosure do not limit this.

[0167] Figure 10 FIG. 4 is a block diagram of an electronic device shown according to an exemplary embodiment. Generally, the electronic device 1000 includes: a processor 1001 and a memory 1002.

[0168] The processor 1001 may include one or more processing cores, such as a 4-core processor, an 8-core processor, etc. The processor 1001 may be implemented in at least one hardware form of DSP (Digital Signal Processing), FPGA (Field-Programmable Gate Array), and PLA (Programmable Logic Array). The processor 1001 may also include a main processor and a coprocessor. The main processor is a processor for processing data in the wake state, also known as the CPU (Central Processing Unit); the coprocessor is a low-power processor for processing data in the standby state. In some embodiments, the processor 1001 may be integrated with a GPU (Graphics Processing Unit), and the GPU is responsible for rendering and drawing the content to be displayed on the display screen. In some embodiments, the processor 1001 may further include an AI (Artificial Intelligence) processor, and the AI processor is used to process computing operations related to machine learning.

[0169] The memory 1002 may include one or more computer-readable storage media, and the computer-readable storage media may be non-transitory. The memory 1002 may further include high-speed random access memory and non-volatile memory, such as one or more disk storage devices and flash storage devices. In some embodiments, the non-transitory computer-readable storage medium in the memory 1002 is used to store at least one program code, and the at least one program code is used to be executed by the processor 1001 to implement the speech synthesis method provided by the method embodiments in the present disclosure, or the training method of the speech editing model.

[0170] In some embodiments, the electronic device 1000 may further optionally include: a peripheral device interface 1003 and at least one peripheral device. The processor 1001, the memory 1002, and the peripheral device interface 1003 may be connected through a bus or signal lines. Each peripheral device may be connected to the peripheral device interface 1003 through a bus, signal lines, or a circuit board. Specifically, the peripheral device includes at least one of: a radio frequency circuit 1004, a display screen 1005, a camera assembly 1006, an audio circuit 1007, and a power supply 1008.

[0171] The peripheral device interface 1003 can be used to connect at least one peripheral device related to I / O (Input / Output) to the processor 1001 and the memory 1002. In some embodiments, the processor 1001, the memory 1002, and the peripheral device interface 1003 are integrated on the same chip or circuit board; in some other embodiments, any one or two of the processor 1001, the memory 1002, and the peripheral device interface 1003 can be implemented on a separate chip or circuit board, and this embodiment does not limit this.

[0172] The radio frequency circuit 1004 is used to receive and transmit RF (Radio Frequency) signals, also known as electromagnetic signals. The radio frequency circuit 1004 communicates with a communication network and other communication devices through electromagnetic signals. The radio frequency circuit 1004 converts an electrical signal into an electromagnetic signal for transmission, or converts a received electromagnetic signal into an electrical signal. Optionally, the radio frequency circuit 1004 includes: an antenna system, an RF transceiver, one or more amplifiers, a tuner, an oscillator, a digital signal processor, a codec chipset, a subscriber identity module card, and so on. The radio frequency circuit 1004 can communicate with other electronic devices through at least one wireless communication protocol. The wireless communication protocol includes but is not limited to: a metropolitan area network, various generations of mobile communication networks (2G, 3G, 4G, and 5G), a wireless local area network, and / or a WiFi (Wireless Fidelity) network. In some embodiments, the radio frequency circuit 1004 may further include a circuit related to NFC (Near Field Communication), and this disclosure does not limit this.

[0173] The display screen 1005 is used to display the UI (User Interface). The UI may include graphics, text, icons, videos, and any combination thereof. When the display screen 1005 is a touch display screen, the display screen 1005 also has the ability to collect touch signals on or above the surface of the display screen 1005. The touch signals can be input as control signals to the processor 1001 for processing. At this time, the display screen 1005 can also be used to provide virtual buttons and / or a virtual keyboard, also known as soft buttons and / or a soft keyboard. In some embodiments, there may be one display screen 1005, which is provided on the front panel of the electronic device 1000; in other embodiments, there may be at least two display screens 1005, which are respectively provided on different surfaces of the electronic device 1000 or are in a foldable design; in still other embodiments, the display screen 1005 may be a flexible display screen, which is provided on the curved surface or the folding surface of the electronic device 1000. Even, the display screen 1005 can also be set to an irregular non-rectangular shape, that is, a special-shaped screen. The display screen 1005 can be prepared from materials such as LCD (Liquid Crystal Display) and OLED (Organic Light-Emitting Diode).

[0174] The camera module 1006 is used to capture images or videos. Optionally, the camera module 1006 includes a front camera and a rear camera. Generally, the front camera is provided on the front panel of the electronic device, and the rear camera is provided on the back of the electronic device. In some embodiments, there are at least two rear cameras, which are any one of a main camera, a depth camera, a wide-angle camera, and a telephoto camera respectively, to implement functions such as the combination of the main camera and the depth camera to achieve the background blurring function, the combination of the main camera and the wide-angle camera to achieve panoramic shooting and VR (Virtual Reality) shooting functions, or other combined shooting functions. In some embodiments, the camera module 1006 may also include a flash. The flash can be a single-color temperature flash or a two-color temperature flash. A two-color temperature flash refers to the combination of a warm light flash and a cold light flash, which can be used for light compensation under different color temperatures.

[0175] The audio circuit 1007 may include a microphone and a speaker. The microphone is used to collect sound waves of the user and the environment, and convert the sound waves into electrical signals for input to the processor 1001 for processing, or input to the radio frequency circuit 1004 to achieve voice communication. For the purpose of stereo collection or noise reduction, there may be multiple microphones, which are respectively arranged at different parts of the electronic device 1000. The microphone may also be an array microphone or an omnidirectional collection microphone. The speaker is used to convert the electrical signals from the processor 1001 or the radio frequency circuit 1004 into sound waves. The speaker may be a traditional thin film speaker or a piezoelectric ceramic speaker. When the speaker is a piezoelectric ceramic speaker, it can not only convert electrical signals into sound waves audible to humans, but also convert electrical signals into sound waves inaudible to humans for uses such as ranging. In some embodiments, the audio circuit 1007 may further include a headphone jack.

[0176] The power supply 1008 is used to supply power to each component in the electronic device 1000. The power supply 1008 may be alternating current, direct current, a disposable battery or a rechargeable battery. When the power supply 1008 includes a rechargeable battery, the rechargeable battery may support wired charging or wireless charging. The rechargeable battery may also be used to support fast charging technology.

[0177] Those skilled in the art can understand that Figure 10 the structure shown in

[0178] does not constitute a limitation on the electronic device 1000, and may include more or fewer components than shown in the figure, or combine certain components, or adopt different component arrangements.

[0179] A computer program product includes a computer program which, when executed by a processor, implements the above-mentioned voice synthesis method, or implements the above-mentioned training method for the voice editing model.

[0180] After considering the specification and practicing the invention disclosed herein, those skilled in the art will readily conceive of other embodiments of the present disclosure. The present disclosure is intended to cover any variations, uses, or adaptations of the present disclosure, which follow the general principles of the present disclosure and include common general knowledge or conventional technical means in the technical field not disclosed in the present disclosure. The specification and the embodiments are only regarded as exemplary, and the true scope and spirit of the present disclosure are pointed out by the following claims.

[0181] It should be understood that the present disclosure is not limited to the exact structures described above and shown in the drawings, and various modifications and changes can be made without departing from its scope. The scope of the present disclosure is only limited by the appended claims.

Claims

1. A voice synthesis method, characterized in that, The method includes: Processing a first text through a speech synthesis model to obtain a first speech signal, where the speech synthesis model is used to generate a corresponding speech signal according to the input text, and the first text includes a paralinguistic tag; Masking a speech segment corresponding to the paralinguistic tag in the first speech signal to obtain a second speech signal; Processing the first text and the second speech signal through a speech editing model to obtain a third speech signal, where the speech editing model is used to generate a paralinguistic segment, and the third speech signal includes a paralinguistic segment corresponding to the paralinguistic tag.

2. The speech synthesis method according to claim 1, wherein The masking a speech segment corresponding to the paralinguistic tag in the first speech signal to obtain a second speech signal includes: Determining a start frame and an end frame of the speech segment corresponding to the paralinguistic tag through a duration alignment model, where the duration alignment model is used to predict the duration of phonemes; Replacing the speech segment between the start frame and the end frame in the first speech signal with a masking signal to obtain the second speech signal.

3. The speech synthesis method according to claim 1, wherein The processing the first text and the second speech signal through a speech editing model to obtain a third speech signal includes: Upsampling the first text to obtain a second text, where the length of the second text is the same as the number of frames of the second speech signal, the first text includes at least one morpheme, and the number of any morpheme in the first text in the second text is the same as the number of frames of the morpheme in the second speech signal; Concatenating the second text and the second speech signal to obtain a concatenated signal; Processing the concatenated signal through the speech editing model to obtain the third speech signal.

4. A training method for a speech editing model, characterized in that, The method includes: Processing a sample text through a speech synthesis model to obtain an intermediate speech signal, where the speech synthesis model is used to generate a corresponding speech signal according to the input text, and the sample text includes a sample paralinguistic tag; Upsampling the sample text to obtain an aligned text sequence, where the length of the aligned text sequence is the same as the number of frames of the intermediate speech signal, the sample text includes at least one morpheme, and the number of any morpheme in the sample text in the aligned text sequence is the same as the number of frames of the morpheme in the intermediate speech signal; Masking a speech segment corresponding to the sample paralinguistic tag in the intermediate speech signal to obtain a masked speech signal; Training a speech editing model according to the aligned text sequence, the masked speech signal, and the intermediate speech signal, where the speech editing model is used to generate a paralinguistic segment.

5. The training method of the speech editing model according to claim 4, characterized in that, The masking a speech segment corresponding to the sample paralinguistic tag in the intermediate speech signal to obtain a masked speech signal includes: Determining a start frame and an end frame of the speech segment corresponding to the sample paralinguistic tag through a duration alignment model, where the duration alignment model is used to predict the duration of phonemes; Replacing the speech segment between the start frame and the end frame in the intermediate speech signal with the sample masking signal to obtain the masked speech signal.

6. The training method of the speech editing model according to claim 4, characterized in that, Training a speech editing model according to the aligned text sequence, the masked speech signal, and the intermediate speech signal includes: Processing the aligned text sequence and the masked speech signal through a speech editing model to obtain an output speech signal, where the output speech signal includes a paralinguistic segment corresponding to the paralinguistic label; Determining a mean squared error loss based on the intermediate speech signal and the output speech signal; Training the speech editing model based on the mean squared error loss.

7. A voice synthesis device, characterized in that, The apparatus includes: A text processing unit configured to process a first text through a speech synthesis model to obtain a first speech signal, where the speech synthesis model is used to generate a corresponding speech signal according to the input text, and the first text includes a paralinguistic label; A signal masking unit configured to mask a speech segment corresponding to the paralinguistic label in the first speech signal to obtain a second speech signal; A signal editing unit configured to process the first text and the second speech signal through a speech editing model to obtain a third speech signal, where the speech editing model is used to generate a paralinguistic segment, and the third speech signal includes a paralinguistic segment corresponding to the paralinguistic label.

8. A training device for a speech editing model, characterized in that, The method includes: A text processing unit configured to process a sample text through a speech synthesis model to obtain an intermediate speech signal, where the speech synthesis model is used to generate a corresponding speech signal according to the input text, and the sample text includes a sample paralinguistic label; An upsampling unit configured to upsample the sample text to obtain an aligned text sequence, where the length of the aligned text sequence is the same as the number of frames of the intermediate speech signal, the sample text includes at least one morpheme, and the number of any morpheme in the aligned text sequence is the same as the number of frames of the morpheme in the intermediate speech signal; A signal masking unit configured to mask a speech segment corresponding to the sample paralinguistic label in the intermediate speech signal to obtain a masked speech signal; A model training unit configured to train a speech editing model according to the aligned text sequence, the masked speech signal, and the intermediate speech signal, where the speech editing model is used to generate a paralinguistic segment.

9. An electronic device, characterized in that, The electronic device includes: One or more processors; A memory for storing program code executable by the processor; Wherein, the processor is configured to execute the program code to implement the speech synthesis method according to any one of claims 1 to 3, or to implement the training method of the speech editing model according to any one of claims 4 - 6.

10. A computer-readable storage medium, characterized in that, When the instructions in the computer-readable storage medium are executed by the processor of the electronic device, the electronic device is enabled to execute the speech synthesis method according to any one of claims 1 to 3, or to execute the training method of the speech editing model according to any one of claims 4 - 6.

11. A computer program product, characterized in that, The computer program product includes a computer program which, when executed by a processor, implements the speech synthesis method according to any one of claims 1 to 3, or executes the training method of the speech editing model according to any one of claims 4 to 6.

Images