Speech synthesis method and apparatus

Abstract

The application is a divisional application of 202311870114.7, which discloses a speech synthesis method, comprising: dividing a target text into multiple target text segments according to the text sequence; inputting a first target text segment in the multiple target text segments into a speech synthesis model to obtain a first target audio segment; splicing a text-audio alternating sequence according to the first target text segment and the first target audio segment; obtaining a next target text segment and adding the next target text segment to the text-audio alternating sequence; inputting the text-audio alternating sequence into the speech synthesis model to obtain a next target audio segment of the next target text segment, and adding the next target audio segment to the text-audio alternating sequence until a last target audio segment corresponding to a last target text segment is obtained; and splicing the target audio segments to generate a target audio. The application can utilize the context of text and audio for character understanding, capture the rhythm, tone and emotion in the scene.

Description

[0001] This application is a divisional application, the original application has the application number of 202311870114.7, the application date is December 29, 2023, and the invention name is "speech synthesis model training method, speech synthesis method, electronic device and storage medium". TECHNICAL FIELD

[0002] The present application relates to the technical field of multimedia content processing, and in particular to a speech synthesis method, a speech synthesis device, an electronic device and a storage medium. BACKGROUND

[0003] Speech synthesis technology is a technology for converting text into speech, and can realize text-to-speech function through a neural network model. In current text-to-speech models, only the current text input is converted into speech, and the generated speech is difficult to match the overall text semantics, prosody and emotion, resulting in a speaker voice that does not conform to the scene and causing problems such as incoherent pronunciation.

[0004] The description of this background technology is intended to help understand the related technology in the relevant field, and does not mean to admit that the background technology content belongs to the prior art. SUMMARY

[0005] Therefore, the embodiments of the present application aim to provide a speech synthesis method and related electronic device and storage medium. Through the scheme of the embodiments of the present application, the prosody, rhythm and emotion of the current scene can be captured based on the context understanding of the text, so that the generated speech is more consistent with the real scene.

[0006] In a first aspect, the embodiments of the present application provide a speech synthesis model training method, comprising the following steps:

[0007] Obtaining initial training data, wherein the initial training data comprises coherent text and corresponding coherent audio;

[0008] Selecting a plurality of first text segments, a plurality of second text segments adjacent to the first text segments before the first text segments, and a plurality of third text segments adjacent to the first text segments after the first text segments from the coherent text, and obtaining a plurality of first audio segments, second audio segments and third audio segments corresponding to the plurality of first text segments, second text segments and third text segments from the coherent audio;

[0009] Detecting whether the associated first text segment, second text segment, third text segment, first audio segment, second audio segment and third audio segment meet the splicing condition;

[0010] The first text segment, the second text segment, the third text segment, the first audio segment, the second audio segment, and the third audio segment that meet the splicable condition are spliced in a text and audio alternating manner according to a text sequence to obtain combined training data, so as to generate a combined training data set including a plurality of the combined training data.

[0011] According to the combined training data set, an initial speech synthesis model is trained to obtain a trained speech synthesis model, and in the training, prosody, rhythm, and / or emotional features in the second text segment and / or the third text segment are extracted.

[0012] In some embodiments of the present application, the detection of whether the associated first text segment, the second text segment, the third text segment, the first audio segment, the second audio segment, and the third audio segment meet the splicable condition comprises:

[0013] The combined text length of the first text segment, the second text segment, and the third text segment is obtained, and the combined audio duration of the first audio segment, the second audio segment, and the third audio segment is obtained, and when the combined text length is less than a length threshold and the combined audio duration is less than a duration threshold, the text and speech corresponding to the first text segment, the second text segment, the third text segment, the first audio segment, the second audio segment, and the third audio segment meet a first splicable condition.

[0014] In some embodiments of the present application, the detection of whether the associated first text segment, the second text segment, the third text segment, the first audio segment, the second audio segment, and the third audio segment meet the splicable condition comprises:

[0015] The voiceprint feature approximation degree between the first audio segment, the second audio segment, and the third audio segment is obtained, and when the voiceprint feature approximation degree is greater than a voiceprint approximation threshold, the text and speech corresponding to the first text segment, the second text segment, the third text segment, the first audio segment, the second audio segment, and the third audio segment meet a second splicable condition.

[0016] In some embodiments of the present application, the detection of whether the associated first text segment, the second text segment, the third text segment, the first audio segment, the second audio segment, and the third audio segment meet the splicable condition comprises:

[0017] When the interval between the sound production time points of adjacent audio segments in the first audio segment, the second audio segment, and the third audio segment is less than an interval threshold, the text and speech corresponding to the first text segment, the second text segment, the third text segment, the first audio segment, the second audio segment, and the third audio segment meet a third splicable condition.

[0018] In some embodiments of the present application, the splicing to obtain the combined training data in a text and audio alternating manner according to a text sequence comprises:

[0019] performing word segmentation processing on the first text segment, the second text segment, and the third text segment to generate a first text sequence, a second text sequence, and a third text sequence, and performing vector conversion on the first text sequence, the second text sequence, and the third text sequence to generate a first text vector, a second text vector, and a third text vector;

[0020] performing discretization processing on the first audio segment, the second audio segment, and the third audio segment and extracting audio features to generate a first audio feature sequence, a second audio feature sequence, and a third audio feature sequence, and performing vector conversion on the first audio feature sequence, the second audio feature sequence, and the third audio feature sequence to generate a first audio vector, a second audio vector, and a third audio vector;

[0021] in the order of the second text vector, the second audio feature sequence, the first text vector, the first audio feature sequence, the third text vector, and the third audio feature sequence, performing vector splicing on the second text vector, the second audio feature sequence, the first text vector, the first audio feature sequence, the third text vector, and the third audio feature sequence to generate the combined training data.

[0022] In some embodiments of the present application, the training of the initial speech synthesis model according to the combined training data set to obtain the trained speech synthesis model comprises:

[0023] iteratively inputting the combined training data in the combined training data set into the initial speech synthesis model, obtaining a first output audio segment corresponding to the first text segment, calculating a first loss value according to the audio features of the first output audio segment and the first audio segment, adjusting the model parameters of the initial speech synthesis model according to the first loss value, until the initial speech synthesis model converges, and taking the converged initial speech synthesis model as the trained speech synthesis model.

[0024] In some embodiments of the present application, the training of the initial speech synthesis model according to the combined training data set to obtain the trained speech synthesis model comprises:

[0025] inputting the combined training data in the combined training data set into the initial speech synthesis model, obtaining a second output audio segment corresponding to the second text segment, calculating a second loss value according to the audio features of the second output audio segment and the second audio segment, adjusting the model parameters of the initial speech synthesis model according to the second loss value, iteratively updating until the initial speech synthesis model converges, and taking the converged initial speech synthesis model as the trained speech synthesis model.

[0026] In some embodiments of the present application, the training of the initial speech synthesis model according to the combined training data set to obtain the trained speech synthesis model comprises:

[0027] The combined training data in the combined training data set is iteratively input to the initial speech synthesis model to obtain a third output audio segment corresponding to the third text segment, a third loss value is calculated according to the audio features of the third output audio segment and the third audio segment, the model parameters of the initial speech synthesis model are adjusted according to the third loss value, and the initial speech synthesis model is converged until the initial speech synthesis model is converged, and the converged initial speech synthesis model is taken as the trained speech synthesis model.

[0028] In some embodiments of the present application, the training of the initial speech synthesis model according to the combined training data set to obtain the trained speech synthesis model comprises:

[0029] The combined training data in the combined training data set is iteratively input to the initial speech synthesis model to obtain a first output audio segment corresponding to the first text segment, a second output audio segment corresponding to the second text segment, and a third output audio segment corresponding to the third text segment; a first loss value is calculated according to the audio features of the first output audio segment and the first audio segment, a second loss value is calculated according to the audio features of the second output audio segment and the second audio segment, a third loss value is calculated according to the audio features of the third output audio segment and the third audio segment, and a fourth loss value is obtained according to the first loss value, the second loss value, and the third loss value; the model parameters of the initial speech synthesis model are adjusted according to the fourth loss value, and the initial speech synthesis model is converged until the initial speech synthesis model is converged, and the converged initial speech synthesis model is taken as the trained speech synthesis model.

[0030] In a second aspect, the embodiments of the present application provide a speech synthesis method, which can comprise:

[0031] Obtaining a target text, and dividing the target text into a plurality of target text segments according to a text sequence;

[0032] Inputting a first target text segment in the plurality of target text segments into a trained target speech synthesis model to obtain a first target audio segment;

[0033] Splicing a text-audio alternating sequence according to the first target text segment and the first target audio segment;

[0034] performing the following steps in a loop until a last target audio segment corresponding to a last target text segment is obtained: obtaining a next target text segment, adding the next target text segment to the text-audio alternating sequence to update the text-audio alternating sequence; inputting the text-audio alternating sequence into the trained speech synthesis model to obtain a next target audio segment corresponding to the next target text segment and extracting prosody, rhythm and / or emotional acoustic features of other target text segments, adding the next target audio segment to the text-audio alternating sequence to update the text-audio alternating sequence;

[0035] splicing each target audio segment to generate a target audio.

[0036] In some embodiments of the present application, another feasible speech synthesis method is provided, which can include:

[0037] obtaining a target text, dividing the target text into a plurality of target text segments according to text order;

[0038] inputting a last target text segment in the plurality of target text segments into a trained target speech synthesis model to obtain a last target audio segment;

[0039] splicing the last target text segment and the last target audio segment to obtain a text-audio alternating sequence;

[0040] performing the following steps in a loop until a first target audio segment corresponding to a first target text segment is obtained: obtaining a previous target text segment, adding the previous target text segment to the text-audio alternating sequence to update the text-audio alternating sequence; inputting the text-audio alternating sequence into the trained target speech synthesis model to obtain a previous target audio segment corresponding to the previous target text segment and extracting prosody, rhythm and / or emotional acoustic features of other target text segments, adding the previous target audio segment to the tail of the text-audio alternating sequence to update the text-audio alternating sequence;

[0041] splicing each target audio segment to generate a target audio.

[0042] In a third aspect, embodiments of the present application provide a speech synthesis model, including an encoder and a decoder, wherein the speech synthesis model is trained by the training method in any embodiment of the present application.

[0043] In a fourth aspect, embodiments of the present application provide a speech synthesis model training device, including a training set data acquisition module, a preprocessing module, a splicing detection module, a vector splicing module and a training module, wherein,

[0044] The training set data acquisition module is configured to acquire training set data, wherein the training set data comprises continuous text and corresponding continuous audio.

[0045] The preprocessing module is configured to select a plurality of first text segments, a plurality of second text segments located before and adjacent to the first text segments, and a plurality of third text segments located after and adjacent to the first text segments from the continuous text, and acquire a plurality of first audio segments, a plurality of second audio segments, and a plurality of third audio segments corresponding to the plurality of first text segments, the plurality of second text segments, and the plurality of third text segments from the continuous audio.

[0046] The concatenation detection module is configured to detect whether the text and the speech corresponding to the plurality of first text segments, the plurality of second text segments, the plurality of third text segments, the plurality of first audio segments, the plurality of second audio segments, and the plurality of third audio segments satisfy a concatenation condition.

[0047] The vector concatenation module is configured to concatenate the plurality of first text segments, the plurality of second text segments, the plurality of third text segments, the plurality of first audio segments, the plurality of second audio segments, and the plurality of third audio segments that satisfy the concatenation condition in a text and audio alternating manner according to a text sequence to generate a combined training data set, wherein the combined training data set comprises a plurality of combined training data.

[0048] The training module is configured to train the initial speech synthesis model according to the combined training data set to obtain a trained speech synthesis model.

[0049] In a fifth aspect, an embodiment of the present application provides a speech synthesis device, comprising a target text acquisition module, an initial generation module, a text-audio concatenation module, a cycle generation module, and an audio concatenation module, wherein

[0050] The target text acquisition module is configured to acquire target text, and divide the target text into a plurality of target text segments according to a text sequence.

[0051] The initial generation module is configured to acquire a first target text segment from the plurality of target text segments, input a vector corresponding to the first target text segment into a trained target speech synthesis model, and obtain a first target audio segment.

[0052] The text-audio concatenation module is configured to concatenate a text-audio alternating sequence according to the first target text segment and the first target audio segment.

[0053] The cycle generation module is configured to cyclically acquire a next target text segment, add the next target text segment to a tail of the text-audio alternation sequence to update the text-audio alternation sequence, input a vector corresponding to the text-audio alternation sequence into the trained target speech synthesis model, acquire a next target audio segment, and add the next target audio segment to the tail of the text-audio alternation sequence to update the text-audio alternation sequence, until a last target audio segment corresponding to a last target text segment is acquired.

[0054] The audio splicing module is configured to splice the target audio segments to generate target audio.

[0055] In a sixth aspect, an embodiment of the present application provides a computer readable storage medium having a computer program stored thereon, wherein the program, when executed by a processor, implements the method of any of the embodiments of the present application.

[0056] In a seventh aspect, an embodiment of the present application provides an electronic device, comprising a processor and a memory having a computer program stored thereon, wherein the processor is configured to execute the method of any of the embodiments of the present application when the computer program is run.

[0057] The technical solution provides a speech synthesis model training method, which groups the previous text and the next text of the training text and corresponding training speech data into a sample group, judges whether the sample group can be spliced, converts the sample group into a vector, inputs the vector into an initial speech synthesis model, trains the initial speech synthesis model, and generates a trained speech synthesis model. In the speech synthesis process, the generated trained speech synthesis model can use the context of the text to understand the text, capture the rhythm and emotion in the scene, and also understand the speech corresponding to the context, obtain the rhythm and emotion of the scene, so that the finally output generated speech information is more consistent with the scene, and can also be smoothly connected with the speech of the context. The training efficiency is improved by splicing the shorter audio in the training data together. The data is fully utilized. The technical solution also provides a speech recognition method, which divides the target text into multiple segments, first uses the trained speech synthesis model to synthesize speech for the first segment of text, then combines the generated first audio segment with the first segment of text to form a text-audio sequence, and iteratively performs speech synthesis on subsequent text segments. The current text segment to be recognized is added to the text-audio sequence to obtain the corresponding audio segment. The latest acquired audio segment is also added to the text-audio sequence to provide context text and audio for the next segment of text. Until the last segment of text generates an audio segment, then the generated audio segments are spliced, and finally the target audio is generated. In some embodiments of the present application, speech synthesis can also start from the last segment, then the generated audio segments are spliced, until the first segment of text generates an audio segment to generate the target audio, then the generated audio segments are spliced, and finally the target audio is generated. In the speech synthesis process of the embodiments of the present application, the context text understanding and speech understanding are used, and the generated audio can carry the rhythm, rhythm and emotion of the current scene, so that the speech synthesis is more consistent with the scene, and the realism of the character speaking or dialogue is improved. The scheme in the embodiments of the present application can use the existing speech synthesis model framework for speech model training and speech synthesis, and has high reliability, is easy to operate, and has high practical application value.

[0058] Some of the other optional features and technical effects of the embodiments of the present application are described below, and some can be understood by reading this document. BRIEF DESCRIPTION OF DRAWINGS

[0059] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the following will briefly introduce the drawings needed to be used. It needs to be clear that the drawings in the following description only cover some embodiments of the present application, and those skilled in the art can obtain other drawings from these drawings without creative work, and the purpose of these drawings is to better illustrate the technical details to help understand the embodiments of the present application, wherein:

[0060] Figure 1A data flow conversion schematic diagram of the speech synthesis model training method of the embodiment of the present application is shown.

[0061] Figure 2 A schematic flow chart of the speech synthesis model training method of the embodiment of the present application is shown.

[0062] Figure 3 A text segment and audio segment division schematic diagram in the speech synthesis model training method of the embodiment of the present application is shown.

[0063] Figure 4 A text segment and audio segment vectorization flow schematic diagram in the speech synthesis model training method of the embodiment of the present application is shown.

[0064] Figure 5 A model training flow schematic diagram in the speech synthesis model training method of the embodiment of the present application is shown.

[0065] Figure 6 Another model training flow schematic diagram in the speech synthesis model training method of the embodiment of the present application is shown.

[0066] Figure 7 Another model training flow schematic diagram in the speech synthesis model training method of the embodiment of the present application is shown.

[0067] Figure 8 Another model training flow schematic diagram in the speech synthesis model training method of the embodiment of the present application is shown.

[0068] Figure 9 A work flow schematic diagram of the speech synthesis model training method of the embodiment of the present application is shown.

[0069] Figure 10 A schematic flow chart of the speech synthesis method of the embodiment of the present application is shown.

[0070] Figure 11 A schematic flow chart of the speech synthesis method of the embodiment of the present application is shown.

[0071] Figure 12 An exemplary structural diagram of the speech synthesis model training apparatus of the embodiment of the present application is shown.

[0072] Figure 13 An exemplary structural diagram of the speech synthesis apparatus of the embodiment of the present application is shown.

[0073] Figure 14 An exemplary structural diagram of an electronic device capable of implementing the method according to the embodiment of the present application is shown. DETAILED DESCRIPTION

[0074] The exemplary embodiments of this application will be described in detail below with reference to the drawings. The same or similar components are denoted by the same or similar reference numerals, and the components thereof will not be repeatedly described below. The implementations described in the following exemplary embodiments are not meant to represent all implementations consistent with one or more embodiments of the present description, but merely represent some aspects of the apparatuses and methods encompassed by the present description as detailed in the claims of this application.

[0075] The term "comprises" and variations thereof herein are intended to cover a wide variety of compositions, including for example, "consisting of", unless otherwise stated. The term "or" means "and / or" unless otherwise stated, the term "based on" means "at least partially based on", the term "one example embodiment" and "an embodiment" means "at least one example embodiment", and the term "another embodiment" means "at least one different example embodiment". The terms "first", "second", and the like, can refer to different or same objects. Further explicit and implicit definitions thereof can be included below.

[0076] In embodiments of the present application, the speech synthesis process is to generate the speech audio corresponding to the target text field according to the input target text field and the context text content and the context audio content. It can be implemented by a speech synthesis (Text to Speech, TTS) model, or it can be implemented by a large language model.

[0077] In embodiments of the present application, "prosody" is one of the attributes of speech, including pitch, rhythm, etc. "Semantic" is one of the attributes of speech, which identifies the content of the speech.

[0078] In embodiments of the present application, "model" has the conventional meaning in the field of machine learning, for example, the model can be a machine learning or deep learning model, for example, a machine learning or deep learning model including or consisting of the above network.

[0079] In embodiments of the present application, "loss function" and "loss value" have the conventional meaning in the field of machine learning.

[0080] In embodiments of the present application, "embedding" refers to a data processing step that converts discrete text data into continuous data vectors so that the model can calculate it.

[0081] In embodiments of the present application, "phoneme" is the smallest unit of speech divided according to the natural attributes of speech, and one pronunciation action forms one phoneme. "Phoneme sequence" is a sequence composed of several phonemes.

[0082] Embodiments of the present application provide a speech synthesis model training method and device, a speech synthesis method, a speech synthesis model, a training device, a generating device, a storage medium, and an electronic device. The method and device / model can be implemented by one or more computers. In some embodiments, the device / model can be implemented by software, hardware, or a combination of software and hardware. In some embodiments, the electronic device or computer can be implemented by a computer or other electronic device capable of performing the corresponding functions as described herein.

[0083] The inventors of the present application have found that the speech of a normal person is influenced by many factors, not only depending on the text to be expressed, but also influenced by the situation and context. However, in the current TTS model, the model cannot obtain the context information of the speech segment to be synthesized, resulting in the synthesized speech not matching the situation, causing the connection with the context to be not smooth. At the same time, the model cannot control the tone and other attributes of the speech according to the needs during synthesis. Some improvement schemes propose to introduce additional inputs in addition to text in the input content of the speech synthesis process, such as style descriptions corresponding to the text, and add a prompt word processing module outside some TTS models. However, the inventors have found that this way of speech synthesis requires the creation of a special training data set, where each piece of data needs to contain audio, text, and style description, and the data annotation is difficult, resulting in additional costs in actual application and promotion, and it is difficult to expand the data scale to improve the effect. At the same time, the newly added prompt word processing module needs to fine-tune a BERT model, resulting in a long chain and increasing the error.

[0084] To this end, the speech synthesis model training method and device, the speech synthesis method and device, the speech synthesis model, and the storage medium and the electronic device provided by the scheme of the embodiments of the present application propose a method of using context for speech synthesis, controlling the prosody of the speech through context, thereby controlling the emotion of the speech, and in addition, making the generated speech more coherent.

[0085] As Figure 1As shown, in the model training process in the embodiment of the application, a piece of continuous text and corresponding audio is selected from the training set, and according to the part to be synthesized, the text is divided into three parts of pre-text, current text and post-text. After the text of each part is converted into a digital sequence through word segmentation and then converted into training data that can be input into the model through embedding, the corresponding audio data is discretized and then converted into training data through embedding. It is detected whether the three parts of data can meet the conditions for splicing: (1) the total length cannot be too long; (2) each part needs to come from the same speaker; (3) the positions of each part in the original audio cannot be too far apart and need to have continuity. If at least one of the above splicing conditions is met, the text and audio of the three parts are spliced together respectively and separated by special symbols. Then the spliced training data is input into the model for training to obtain the corresponding speech synthesis model. In the embodiment of the application, if there is noise in each piece of audio, the noise can be retained as the ambient sound input into the model.

[0086] Figure 2 As shown in the embodiment, a speech synthesis model training method is provided, and a trained speech synthesis model is obtained through training.

[0087] As shown Figure 2 The training method in the embodiment of the application includes the following steps.

[0088] S110: obtaining initial training data, wherein the initial training data includes continuous text and corresponding continuous audio.

[0089] In the embodiment of the application, the initial training data can be obtained from conventional training data, and the conventional training data can include multiple continuous texts and multiple continuous audios corresponding thereto, without the need for additional labeling of the training data.

[0090] S120: selecting multiple first text segments, multiple second text segments adjacent to the first text segments before the first text segments, and multiple third text segments adjacent to the first text segments after the first text segments from the continuous text, and obtaining multiple first audio segments, multiple second audio segments and multiple third audio segments corresponding to the multiple first text segments, the multiple second text segments and the multiple third text segments from the continuous audio.

[0091] In the embodiment of the application, the text data of the context is selected and input into the model together with the voice data for training, which improves the scene understanding ability of the speech synthesis model.

[0092] In some embodiments of the application, as shown Figure 3As shown, this application selects a first text, a second text preceding the first text, and a third text following the first text from a coherent text, and correspondingly extracts a first audio, a second audio, and a third audio from the coherent speech.

[0093] In this embodiment of the application, the division of the audio segment end line is based on the end of the text pronunciation in the corresponding text segment.

[0094] S130: Detect whether the associated first text segment, second text segment, third text segment, first audio segment, second audio segment, and third audio segment meet the concatenation condition.

[0095] In this embodiment of the application, in order to ensure the continuity of the spliced ​​data, the splicing detection is performed on the selected consecutive segments of text and audio.

[0096] In this embodiment of the invention, when the data is concatenated and input into the model, the data can be converted from discrete data into a continuous data vector through "embedding", so that the model can perform calculations on it.

[0097] In some embodiments of this application, for example, in step S130 above, it can be detected whether three text segments or three audio segments meet a first concatenation condition. The first concatenation condition can be a threshold value for the total length of the three text segments or three audio segments. Specifically, the combined text length of the first text segment, the second text segment, and the third text segment can be obtained, and the combined audio duration of the first audio segment, the second audio segment, and the third audio segment can be obtained. When the combined text length is less than the length threshold value and the combined audio duration is less than the duration threshold value, the text and speech corresponding to the first text segment, the second text segment, the third text segment, the first audio segment, the second audio segment, and the third audio segment meet the first concatenation condition.

[0098] In one specific embodiment of this application, reference continues to be made to... Figure 3 The length threshold can be set to 50 characters, and the duration threshold to 30 seconds. The second text segment is "A stroll on the riverbank during the holiday," with a length of 8 characters and a corresponding second audio duration of 3.5 seconds. The first text segment is "Seeing many children flying kites," with a length of 10 characters and a corresponding first audio duration of 6.1 seconds. The third text segment is "Long fuses," with a length of 8 characters and a corresponding third audio duration of 5.0 seconds. Therefore, the total text length is 8 + 10 + 8 = 26 characters, which is less than the length threshold (50 characters). The total sample duration is 3.5 + 6.1 + 5.0 = 14.6 seconds, which is less than the duration threshold (30 seconds). Thus, the text and audio corresponding to the first, second, and third text segments, as well as the first, second, and third audio segments, satisfy the first concatenation condition.

[0099] In some embodiments of the present application, in order to ensure the coherence of the model output speech, the audio input to the training model needs to be spoken by the same person, for example, whether it is the same person speaking can be determined by voiceprint features. In some embodiments of the present application, for example, in the above step S130, the voiceprint feature approximation degree between the first audio segment, the second audio segment and the third audio segment can be obtained, and when the voiceprint feature approximation degree is greater than a voiceprint approximation threshold, the first text segment, the second text segment, the third text segment, the first audio segment, the second audio segment and the third audio segment corresponding text and speech satisfy the second splicable condition.

[0100] In some embodiments of the present application, after discrete sampling of the first audio segment, the second audio segment and the third audio segment, the time domain signal is converted into a frequency domain signal, and the frequency feature is extracted, and whether it is the same person speaking is determined according to the similarity of the frequency feature. The embodiments of the present application set a voiceprint approximation threshold, for example, it can be 80%, and when the voiceprint feature approximation degree between each two audios is greater than 80%, the first audio, the second audio and the third audio are spoken by the same person. The first text segment, the second text segment, the third text segment, the first audio segment, the second audio segment and the third audio segment corresponding text and speech satisfy the second splicable condition.

[0101] In some embodiments of the present application, in order to determine whether the multiple audio segments are spoken by the same person, a voiceprint model (such as Gaussian mixture model (GMM)) or a deep learning model (such as CNN and RNN) can also be used for recognition.

[0102] In some embodiments of the present application, for example, in the above step S130, in order to improve the coherence of the audio output by the speech synthesis model, and to avoid selecting audio segments that are too different and do not have coherence characteristics, the embodiments of the present application also detect the interval between the audio segments when detecting the splicing condition. When the interval between the adjacent audio segments in the first audio segment, the second audio segment and the third audio segment is less than an interval threshold, the first text segment, the second text segment, the third text segment, the first audio segment, the second audio segment and the third audio segment corresponding text and speech satisfy the third splicable condition.

[0103] In the training sample generation process, when the speaker reads the text continuously, there will be a short pause after the punctuation. In order to ensure the coherence experience of the listener, the pause between the punctuations should not be too long, otherwise, the listener experience will be poor. In the embodiments of the present application, the interval between the adjacent audio segments can be understood as the interval between the end of the speech of the previous audio segment and the start time of the speech of the next audio segment. In the embodiments of the present application, based on the pause after the punctuation, there is usually a blank sound period before the start of the speech of each audio segment, for example, Figure 3The blank sound period is a blank part in the shadow part, and the length of the blank sound period is the interval between the sound time points of adjacent audio segments. In some embodiments of the present application, the interval threshold can be selected as 2-4 seconds.

[0104] In one specific embodiment of the present application, with continued reference to Figure 3 The interval threshold can be set as 3 seconds, wherein the second text segment is "turning on the beach during the holiday", the text length is 8 characters, the corresponding second audio duration is 3.5 seconds, and there is no blank sound period; the first text segment is "I saw many children flying kites", the text length is 10 characters, the corresponding first audio duration is 6.1, and the blank sound period is 0.8 seconds; the third text segment is "a long lead", the text length is 8 characters, the corresponding third audio duration is 5 seconds, and the blank sound period is 1.4 seconds. The interval between the sound time points of the second audio segment and the first audio segment is 0.8 seconds, and the interval between the sound time points of the first audio segment and the third audio segment is 1.4 seconds, both of which are less than the interval threshold. Therefore, the first text segment, the second text segment, the third text segment, the first audio segment, the second audio segment and the third audio segment satisfy the third splicing condition.

[0105] In the detection of whether the splicing condition is satisfied, the above detection conditions can be used alone or in combination.

[0106] S140: The first text segment, the second text segment, the third text segment, the first audio segment, the second audio segment and the third audio segment that satisfy the splicing condition are spliced in the order of text to obtain combined training data in the form of alternating text and audio, to generate a combined training data set including a plurality of combined training data.

[0107] In the embodiment of the present application, when the combined training data is spliced in the form of alternating text and audio, the text and audio data can be converted into continuous text and audio data vectors by "embedding" the discretely right data.

[0108] In some embodiments of the present application, as Figure 4 shown, the process of splicing the combined training data in the order of text to obtain combined training data in the form of alternating text and audio includes:

[0109] S141: The first text segment, the second text segment and the third text segment are segmented and processed to generate a first text sequence, a second text sequence and a third text sequence. The first text sequence, the second text sequence and the third text sequence are vector converted to generate a first text vector, a second text vector and a third text vector.

[0110] In the embodiments of the present application, in order to perform word segmentation processing on the text segment to generate a text sequence, a word segmentation algorithm can be used for processing, such as a method based on string matching or a rule-based method (for example, a forward maximum matching word segmentation algorithm, a reverse maximum matching word segmentation algorithm, a statistical-based method (for example, a Jieba word segmentation algorithm, etc.). In some embodiments of the present application, before the word segmentation, the text in the text segment can also be subjected to text normalization processing, including converting a character set (such as using Unicode encoding uniformly), eliminating differences in case, and removing irrelevant characters (such as HTML tags).

[0111] In some embodiments of the present application, the vector conversion process of the sequence can be processed by using embedding, and the embedding processing process can include:

[0112] a. Constructing a vocabulary

[0113] Word frequency statistics: Calculate the frequency of each token appearing in the entire data set.

[0114] Vocabulary screening: Screen out words in the vocabulary based on frequency and relevance. Remove high-frequency stop words (such as “of”, “and”) and low-frequency words.

[0115] Index assignment: Assign a unique numerical index to each token in the vocabulary.

[0116] b. Token encoding

[0117] One-Hot encoding: Convert each token into a high-dimensional sparse vector, where only the index position corresponding to the token is 1, and the rest is 0.

[0118] Limiting the vector size: In order to avoid high dimension, the size of the vocabulary can be limited, and special tokens can be used instead of infrequent words, such as <unk>`(unknown).

[0119] c. Learning Dense Vector Representation

[0120] Selection of Model: Use pre-trained models like Word2Vec, GloVe, FastText, or BERT, or train a custom model based on the specific task.

[0121] Training Process: Learn vector representations of tokens through context. For example, in Word2Vec, this can be done by predicting the context of a word (CBOW model), or by using a word to predict its context (Skip-gram model).

[0122] Vector Properties: These vectors capture complex relationships between words, such as semantic similarity, grammatical rules, etc.

[0123] d. Optimization of Vector Space

[0124] Dimensionality Reduction: Apply techniques like Principal Component Analysis (PCA) or t-SNE to reduce the dimensionality of the vector space, optimizing it so that similar words are closer in the vector space.

[0125] Fine-Tuning: Further fine-tune for specific tasks to optimize the performance of word vectors in that task.

[0126] S142: Discretize the first, second, and third audio segments and extract audio features to generate first, second, and third audio feature sequences, and perform vector conversion on the first, second, and third audio feature sequences to generate first, second, and third audio vectors.

[0127] In the embodiments of the present application, the audio segments can be discretized, specifically, frequency features can be extracted after sampling the audio segments, and spectral feature processing can be performed to generate audio feature sequences. The audio features can include mel-spectrogram, mel-frequency cepstral coefficients, etc. In some embodiments of the present application, such as step S142 described above, embedding can also be used to discretize the first, second, and third audio segments and extract audio features.

[0128] S143: Vector splice the second text vector, second audio feature sequence, first text vector, first audio feature sequence, third text vector, and third audio feature sequence in the order of second text vector, second audio feature sequence, first text vector, first audio feature sequence, third text vector, and third audio feature sequence to generate the combined training data.

[0129] In an embodiment of the present application, for example, the first text vector is T1, the second text vector is T2, the third text vector is T3, the first audio vector is A1, the second audio vector is A2, and the third audio vector is A3. The generated combined training vector is {T2, A2, T1, A1, T3, A3}, and each vector is separated by a special symbol.

[0130] S150: training the initial speech synthesis model according to the combined training data set to obtain a trained speech synthesis model.

[0131] In an embodiment of the present application, the prosody, rhythm, and / or emotional features in the second text segment and / or the third text segment can be extracted in the training.

[0132] The embodiments of the present application can input the context text and speech audio into the model together for training, can use the context of the text for character understanding, capture the prosody and emotion in the scene, and can also understand the speech corresponding to the context, obtain the rhythm and emotion of the scene, so that the finally output generated speech information is more consistent with the scene and can also be smoothly connected with the context speech.

[0133] In an embodiment of the present application, for example, in the above step S150, in the training process, the prosody, rhythm, and / or emotional features in the second text segment and / or the third text segment are extracted and used for training of the speech synthesis model, so that the trained speech synthesis model can control the prosody of the output speech and control the emotion of the speech by comprehensively considering the context, i.e., the above features of the second text segment and / or the third text segment.

[0134] In an embodiment of the present application, the speech synthesis model includes various speech synthesis models (for example, Tacotron, WaveNet, Fastspeech, etc.). The method in the embodiment of the present application can be used without adjusting the model structure, which can adapt to various existing models and has high practical application value.

[0135] In some embodiments of the present application, as shown in the method for training a speech synthesis model, the method can further include: Figure 5

[0136] S151a: inputting the combined training data in the combined training data set into the initial speech synthesis model iteratively to obtain a first output audio segment corresponding to the first text segment;

[0137] ​S152a: calculate a first loss value according to the audio features of the first output audio segment and the first audio segment;

[0138] S153a: adjust the model parameters of the initial speech synthesis model according to the first loss value until the initial speech synthesis model converges, and take the converged initial speech synthesis model as the trained speech synthesis model.

[0139] In the embodiments of the present application, the loss value includes but is not limited to Mel-Spectrogram Loss, DurationLoss, CTC Loss, Angular Margin Loss, etc. Mel-Spectrogram Loss is used to compare the Mel spectrogram of the generated audio and the input audio to ensure the accuracy of the sound features; DurationLoss is used for attention-based models to ensure that the predicted pronunciation duration is consistent with the real duration; CTC Loss is used for speech recognition models to process non-aligned input and output sequences; Angular Margin Loss is used to enhance the model's discriminative ability for speaker features in speaker verification and identification tasks.

[0140] In some embodiments of the present application, the loss value can be calculated according to the intermediate audio segment in the second audio segment, the first audio segment, and the third audio segment, i.e., the first audio segment and the corresponding first output audio segment, which can fully utilize the front part of the text and speech and the latter part of the text and speech to fully understand the prosody and emotion of the middle part of the text, and the model's understanding of semantics and scene is more sufficient.

[0141] In some embodiments of the present application, the loss value can also be calculated using the front segment of the second audio segment, the first audio segment, and the third audio segment, which can utilize the emotion and scene of the latter speech and text to train the speech synthesis model, making the speech output more coherent. Specifically, as shown in Figure 6 The training process can include:

[0142] S151b: input the combined training data in the combined training data set into the initial speech synthesis model to obtain a second output audio segment corresponding to the second text segment.

[0143] S152b: calculate a second loss value according to the audio features of the second output audio segment and the second audio segment.

[0144] S153b: adjust the model parameters of the initial speech synthesis model according to the second loss value, and iteratively update until the initial speech synthesis model converges, and take the converged initial speech synthesis model as the trained speech synthesis model.

[0145] In some embodiments of the present application, the loss value can also be calculated using the latter of the second audio segment, the first audio segment, and the third audio segment, so that the speech synthesis model can be trained using the emotions and scenes of the previous speech and text, and the speech output is more coherent. Specifically, as shown in Figure 7 The training process can include:

[0146] S151c: iteratively inputting the combined training data in the combined training data set into the initial speech synthesis model to obtain a third output audio segment corresponding to the third text segment.

[0147] S152c: calculating a third loss value according to the audio features of the third output audio segment and the third audio segment.

[0148] S153c: adjusting the model parameters of the initial speech synthesis model according to the third loss value until the initial speech synthesis model converges, and taking the converged initial speech synthesis model as the trained speech synthesis model.

[0149] In the embodiments of the present application, the target training vectors are iteratively or cyclically taken from the target training vector set, input into the initial speech synthesis model, the loss value is calculated, the model parameters are adjusted, and the model converges until the loss value is less than a set threshold, and the training ends.

[0150] As shown in Figure 9 In the embodiments of the present application, the speech synthesis model 500 can be abstracted into an encoder 510 and a decoder 520, and during the training process, a discriminator can also be set to calculate the loss value and perform automatic parameter adjustment training.

[0151] The initial speech synthesis model in the embodiments of the present application outputs audio segments including a first output audio segment, a second output audio segment, and a third output audio segment during the training process, and according to the need to calculate the loss value, the audio segments participating in the calculation of the loss value are selected from the output audio segments.

[0152] In addition to the neural network part, the speech synthesis model in the embodiments of the present application can also include a frequency domain to time domain processing module, and the neural network part outputs an audio segment and then passes through the frequency domain to time domain processing module to convert into a sound signal.

[0153] In the running process of the speech synthesis model in the present application, the text and audio sequence first pass through the embedding module, then enter the encoder, and then output to the decoder, and finally the output audio segment after the frequency domain to time domain signal is generated. The last sound signal is generated.

[0154] In some embodiments of the present application, in order to further improve the accuracy of the output of the speech output model, the output audio segment of each segment is involved in the calculation process of the loss value, and the degree of understanding of the context by the model output is improved, as shown in Figure 8 As shown, the training process can further include:

[0155] S151d: iteratively inputting the combined training data in the combined training data set into the initial speech synthesis model to obtain a first output audio segment corresponding to the first text segment, a second output audio segment corresponding to the second text segment, and a third output audio segment corresponding to the third text segment.

[0156] S152d: calculating a first loss value according to the audio features of the first output audio segment and the first audio segment, a second loss value according to the audio features of the second output audio segment and the second audio segment, and a third loss value according to the audio features of the third output audio segment and the third audio segment, and obtaining a fourth loss value according to the first loss value, the second loss value, and the third loss value.

[0157] In the process of calculating the fourth loss value, the embodiments of the present application can set the weights of the first loss value, the second loss value, and the third loss value. For example, the first loss value is L1, the second loss value is L2, and the third loss value is L3, and the fourth loss value is L4=a*L1+b*L2+c*L3, a, b, and c are the weights of the respective loss values.

[0158] In the process of calculating the fourth loss value, the embodiments of the present application can adjust the weights of the respective loss values according to the attention degrees. For example, the trained model is more based on the front and middle texts and speeches to generate the rear speech, so the weight of the third loss value is set to be larger, if it is more based on the front and rear texts and speeches to generate the middle speech, the weight of the first loss value is set to be larger, and if it is more based on the middle and rear texts and speeches to generate the front speech, the weight of the second loss value is set to be larger.

[0159] S153d: adjusting the model parameters of the initial speech synthesis model according to the fourth loss value until the initial speech synthesis model converges, and taking the converged initial speech synthesis model as the trained speech synthesis model.

[0160] The target speech synthesis model in the embodiments of the present application can be a TTS model, a large number of annotated speech-text pairs are used for supervised learning in the training process, the learning rate, the batch size, and other hyperparameters are adjusted, and techniques such as Dropout and BatchNormalization are applied to prevent overfitting. The model can also be fine-tuned and tuned according to the model performance, for example, the performance on the validation set is monitored, the model is fine-tuned, and the depth, the number of hidden units, and the attention mechanism of the model can be adjusted.

[0161] The target speech synthesis model in this embodiment can be a large language model, trained using a large-scale speech dataset. Self-supervised learning training can be performed to adjust the large language model to handle speech synthesis in different scenarios, such as classroom speech synthesis, recitation speech synthesis, or dialogue speech synthesis. After the initial model is generated, model evaluation and optimization can be performed. Performance is evaluated using an independent validation set, and model parameters, such as the number of layers, the number of hidden units, and the learning rate, are adjusted to optimize performance.

[0162] The speech synthesis model training method in this application embodiment forms a sample group with the previous and next segments of the training text and the corresponding training speech data. After determining that they can be concatenated, the sample group is converted into a vector and input into the initial speech synthesis model. The initial speech synthesis model is trained to generate a target speech synthesis model. During the speech synthesis process, the generated target speech synthesis model can use the context of the text to understand the text, capture the rhythm and emotion in the scene, and also understand the speech corresponding to the context to obtain the rhythm and emotion of the scene. As a result, the generated speech information output is more in line with the context and can be smoothly connected with the speech in the context.

[0163] like Figure 10 As shown, embodiments of this application provide a speech synthesis method, including:

[0164] S210: Obtain the target text and divide the target text into multiple target text segments according to the text order.

[0165] In some embodiments of this application, the target text is divided into multiple target text segments according to the text order, including: performing sentence segmentation on the target text to obtain multiple target text segments.

[0166] In one specific embodiment of this application, the target text segment is, for example, "On a holiday, I went for a stroll on the riverbank and saw many children flying kites. Long strings, one end tied to the sky and the other to the ground, the children and kites swayed between the sky and the ground, and even my heart was swayed and mesmerized, as if I had returned to my childhood." Based on sentence segmentation, the target text segment is divided into the following sub-segments: First sub-segment T1 ("On a holiday, I went for a stroll on the riverbank"), Second sub-segment T2 ("I saw many children flying kites"), Third sub-segment T3 ("Long strings"), Fourth sub-segment T4 ("One end tied to the sky"), Fifth sub-segment T5 ("One end tied to the ground"), Sixth sub-segment T6 ("The children and kites swayed between the sky and the ground"), Seventh sub-segment T7 ("Even my heart was swayed and mesmerized"), and Eighth sub-segment T8 ("As if I had returned to my childhood").

[0167] S220: input the first target text segment in the plurality of target text segments into the trained target speech synthesis model to obtain a first target audio segment.

[0168] In an embodiment of the present application, for example, the first text segment T1 ("Holiday to the beach turn") is input into the target speech synthesis model to obtain the first target audio segment A1.

[0169] In some embodiments of the present application, the trained target speech synthesis model is trained by any of the training methods of the present application.

[0170] In an embodiment of the present application, the first audio segment can also be generated without using the target speech synthesis model. The first audio segment can be generated by recording the sound emitted by the user after reading the first text segment, so as to carry the characteristics of the user's voice in subsequent speech synthesis, such as carrying the user's emotion, accent, and other information.

[0171] S230: concatenating a text-audio alternating sequence according to the first target text segment and the first target audio segment.

[0172] In an embodiment of the present application, for example, referring to the above example, the text-audio alternating sequence {T1, A1} is generated by concatenation.

[0173] S240: cyclically performing the following steps until the last target audio segment corresponding to the last target text segment is obtained: obtaining a next target text segment, adding the next target text segment to the tail of the text-audio alternating sequence to update the text-audio alternating sequence; inputting the text-audio alternating sequence into the trained speech synthesis model to obtain a next target audio segment corresponding to the next target text segment, and adding the next target audio segment to the text-audio alternating sequence to update the text-audio alternating sequence.

[0174] In an embodiment of the present application, for example, in the above step S240, the next target audio segment corresponding to the next target text segment extracts the prosody, rhythm, and / or emotional acoustic features of other target text segments. Wherein, the other target text segments refer to the text segments other than the current corresponding next target text segment, i.e. the other text segments other than the text segment corresponding to the current audio segment for synthesis. Preferably, the other text segments can be or include the previous target text segment of the next target text segment. In an embodiment of the present application, the prosody of the next target audio segment can be controlled by the prosody, rhythm, and / or emotional acoustic features of the context, especially the "previous" context, so as to control the emotion of the speech.

[0175] In an embodiment of the present application, for example, continuing the above example, a second text segment is added to the text-audio alternation sequence {T1, A1}, the text-audio alternation sequence is updated to {T1, A1, T2}, the updated text-audio alternation sequence {T1, A1, T2} is input into the trained speech synthesis model to obtain a second target audio segment A2, and then according to a next text segment T3, the text-audio alternation sequence is updated to {T1, A1, T2, A2, T3}, the trained speech synthesis model is input to obtain a corresponding third target audio segment A3, and the above steps are repeatedly executed until the last eighth target audio segment A8 is generated.

[0176] In some embodiments of the present application, as shown in Figure 11 the text 1 is input at the beginning of speech synthesis, the speech model generates the audio 1, and the audio 1 is spliced into a text-audio sequence, then a new text segment is input in sequence and spliced to the end of the sequence, and then the entire sequence is input into the trained speech synthesis model to synthesize the next audio segment, thereby fully utilizing the context information during synthesis. When the audio n is generated, the audio n is added to the text-audio sequence, and the text-audio sequence is updated to {text 1, audio 1, text 2, …, audio n}. When the text n+1 is input, the text n+1 is added to the text-audio sequence, and the text-audio sequence is updated to {text 1, audio 1, text 2, …, audio n, text n+1}. The updated text-audio sequence is input into the trained speech synthesis model to obtain the audio n+1. The above steps are repeatedly executed until all text segments corresponding to the audio are generated.

[0177] S250: Splice each target audio segment to generate a target audio.

[0178] In an embodiment of the present application, for example, in the above step S250, each target audio segment can be spliced according to the text order to generate a target audio.

[0179] In an embodiment of the present application, for example, continuing the above example, the last obtained target audio segments are {A1, A2, A3, A4, A5, A6, A7, A8}, and after splicing, the target audio A_target is generated.

[0180] In another alternative embodiment of the present application, a speech synthesis method is also provided, in which the speech is synthesized in reverse order of the text, and the synthesized speech can obtain the prosody, rhythm and / or emotional acoustic features in the context, especially the "context". Specifically, the method can include:

[0181] obtaining a target text, and dividing the target text into a plurality of target text segments according to a text order;

[0182] inputting a last target text segment in the plurality of target text segments into the trained target speech synthesis model to obtain a last target audio segment;

[0183] splicing a text-audio alternation sequence according to the last target text segment and the last target audio segment;

[0184] performing the following steps in a loop until a first target audio segment corresponding to a first target text segment is obtained: obtaining a previous target text segment, adding the previous target text segment to the text-audio alternation sequence to update the text-audio alternation sequence; inputting the text-audio alternation sequence into the trained target speech synthesis model to obtain a previous target audio segment corresponding to the previous target text segment, and adding the previous target audio segment to a tail of the text-audio alternation sequence to update the text-audio alternation sequence;

[0185] splicing the target audio segments to generate a target audio. In this way, the target audio segments can be spliced according to the text sequence to generate the target audio.

[0186] In the embodiments of the present application, for example, in the above embodiments, the obtained previous target audio segment corresponding to the previous target text segment extracts the prosody, rhythm and / or emotional acoustic features of other target text segments. The other target text segments refer to text segments other than the current corresponding previous target text segment, that is, other text segments other than the text segment corresponding to the current audio segment. Preferably, the other text segments can be or include the next target text segment of the previous target text segment. In the embodiments of the present application, the prosody of the previous target audio segment can be controlled through the prosody, rhythm and / or emotional acoustic features of the context, especially the "subsequent context", so as to control the emotion of the speech.

[0187] In some embodiments of the present application, the splicing of the target audio segments to generate the target audio further includes smoothing processing at the connection of the audio segments.

[0188] In the embodiments of the present application, when the target speech synthesis model is a TTS (Text-to-Speech) or a large speech model, after the target audio segment is generated, smoothing processing can be performed when the target audio segments are spliced, so as to ensure the coherence and naturalness of the synthesized audio in the sense of hearing.

[0189] In the embodiments of the present application, the smoothing processing when the target audio segments are spliced is as follows.

[0190] 1. Precise selection of splicing points

[0191] Rhythm-based Selection: Automatically analyze the rhythmic features (pitch, intensity, tempo, etc.) of the audio and select points that are rhythmically similar or naturally connected to each other as splice points.

[0192] 2. Digital Signal Processing (DSP)

[0193] Time Alignment and Pitch Correction: Use digital signal processing techniques like WSOLA (Waveform Similarity Overlap-Add) to stretch / compress the audio segments in time and match the pitch to ensure continuity at the splice points.

[0194] 3. Crossfade

[0195] Smooth Transition: Implement crossfade at the splice points so that one segment gradually fades out while the other gradually fades in, achieving a smooth transition.

[0196] 4. Spectral Continuity Preservation

[0197] Spectral Matching: Use equalizers or other spectral processing tools to ensure that the spectral characteristics of the audio before and after splicing remain consistent, especially in the low and high frequency ranges.

[0198] 5. Harmonic and Formant Matching

[0199] Harmonic Matching: Analyze and match the harmonic structure before and after the splice points to ensure tonal continuity.

[0200] Formant Matching Adjustment: Fine-tune the formants to maintain the consistency and naturalness of the sound.

[0201] 6. Ambient Noise and Background Sound Processing

[0202] Noise Smoothing: If there is background noise, adjust the consistency of the noise through technical means or use noise reduction techniques to eliminate it.

[0203] 7. Post-Mixing and Mastering

[0204] Dynamic Range Processing: Use compressors and limiters to adjust the dynamic range, ensuring the volume consistency of the entire audio.

[0205] Fine-Tuning: Make fine adjustments and optimizations to the spliced audio, including equalization, reverb, etc.

[0206] 8. Quality Inspection and Feedback

[0207] Auditory Inspection: Conduct detailed auditory inspection after splicing to ensure there are no obvious signs of splicing.

[0208] Objective evaluation: use an objective evaluation tool (such as PESQ, Perceptual Evaluation of Speech Quality) to evaluate the quality of the spliced audio. Adjust the parameters of the previous steps according to the evaluation quality.

[0209] In the embodiments of the present application, during the speech synthesis process, the text and audio sequence first pass through the embedding module, then enter the encoder, and then output to the decoder, and finally generate the output audio segment (expressed using frequency features) after converting the frequency domain to the time domain signal, the final sound signal is generated.

[0210] The speech recognition method in the embodiments of the present application divides the target text into multiple segments, first uses the target speech synthesis model to synthesize speech for the first segment of text, then combines the generated first audio segment with the first segment of text to form a text-audio sequence, and iteratively performs speech synthesis for subsequent text segments. The current text segment to be recognized is added to the text-audio sequence, and the corresponding audio segment is obtained. The newly obtained audio segment is also added to the text-audio sequence to provide context text and audio for the next segment of text. Finally, the generated audio segment is spliced to generate the target audio. During the speech synthesis process, the context text understanding and speech understanding are utilized, and the generated audio can carry the rhythm, pitch, and emotion of the current scene, making the speech synthesis more consistent with the scene.

[0211] As shown in Figure 12 The speech synthesis model training device 1200 provided in the embodiments of the present application includes a training set data acquisition module 1210, a preprocessing module 1220, a splicing detection module 1230, a vector splicing module 1240, and a training module 1250, wherein,

[0212] The training set data acquisition module 1210 is configured to acquire training set data, wherein the training set data includes consecutive text and corresponding consecutive audio.

[0213] The preprocessing module 1220 is configured to select a plurality of first text segments, a plurality of second text segments adjacent to the first text segments before the first text segments, and a plurality of third text segments adjacent to the first text segments after the first text segments from the consecutive text, and to acquire a plurality of first audio segments, second audio segments, and third audio segments corresponding to the plurality of first text segments, second text segments, and third text segments from the consecutive audio.

[0214] The splicing detection module 1230 is configured to detect whether the text and speech corresponding to the plurality of first text segments, second text segments, third text segments, first audio segments, second audio segments, and third audio segments satisfy the splicable condition.

[0215] The vector splicing module 1240 is configured to splice the plurality of first text segments, second text segments, third text segments, first audio segments, second audio segments and third audio segments that meet the splicable condition in a text sequence in a manner of alternation of text and audio to generate a combined training data set, where the combined training data set includes a plurality of combined training data.

[0216] The training module 1250 is configured to train the initial speech synthesis model according to the combined training data set to obtain a trained speech synthesis model.

[0217] In some embodiments of the present application, the splicing detection module 1230 is further configured to:

[0218] obtain a combined text length of the first text segment, the second text segment and the third text segment, obtain a combined audio duration of the first audio segment, the second audio segment and the third audio segment, and when the combined text length is less than a length threshold and the combined audio duration is less than a duration threshold, the text and speech corresponding to the first text segment, the second text segment, the third text segment, the first audio segment, the second audio segment and the third audio segment meet a first splicable condition;

[0219] In some other embodiments of the present application, the splicing detection module 1230 is further configured to:

[0220] obtain a voiceprint feature approximation degree between the first audio segment, the second audio segment and the third audio segment, and when the voiceprint feature approximation degree is greater than a voiceprint approximation threshold, the text and speech corresponding to the first text segment, the second text segment, the third text segment, the first audio segment, the second audio segment and the third audio segment meet a second splicable condition;

[0221] In some other embodiments of the present application, the splicing detection module 1230 is further configured to:

[0222] when a speaking time point interval between adjacent audio segments in the first audio segment, the second audio segment and the third audio segment is less than an interval threshold, the text and speech corresponding to the first text segment, the second text segment, the third text segment, the first audio segment, the second audio segment and the third audio segment meet a third splicable condition.

[0223] In some embodiments of the present application, the vector splicing module 1240 is configured to:

[0224] perform word segmentation processing on the first text segment, the second text segment and the third text segment to generate a first text sequence, a second text sequence and a third text sequence, and perform vector conversion on the first text sequence, the second text sequence and the third text sequence to generate a first text vector, a second text vector and a third text vector;

[0225] discretize the first audio segment, the second audio segment and the third audio segment and extract audio features to generate a first audio feature sequence, a second audio feature sequence and a third audio feature sequence, and perform vector conversion on the first audio feature sequence, the second audio feature sequence and the third audio feature sequence to generate a first audio vector, a second audio vector and a third audio vector;

[0226] In an order of the second text vector, the second audio feature sequence, the first text vector, the first audio feature sequence, the third text vector and the third audio feature sequence, the second text vector, the second audio feature sequence, the first text vector, the first audio feature sequence, the third text vector and the third audio feature sequence are spliced to generate the combined training data.

[0227] In some embodiments of the present application, the target speech synthesis model includes a large language model.

[0228] In some embodiments of the present application, the training module 1250 is configured to:

[0229] The combined training data in the combined training data set is iteratively input to the initial speech synthesis model, a first output audio segment corresponding to the first text segment is obtained, a first loss value is calculated according to the audio features of the first output audio segment and the first audio segment, the model parameters of the initial speech synthesis model are adjusted according to the first loss value, and the initial speech synthesis model is converged until the initial speech synthesis model is converged, and the converged initial speech synthesis model is taken as the trained speech synthesis model.

[0230] In some embodiments of the present application, the training module 1250 is configured to:

[0231] The combined training data in the combined training data set is input to the initial speech synthesis model, a second output audio segment corresponding to the second text segment is obtained, a second loss value is calculated according to the audio features of the second output audio segment and the second audio segment, the model parameters of the initial speech synthesis model are adjusted according to the second loss value, and the initial speech synthesis model is iteratively updated until the initial speech synthesis model is converged, and the converged initial speech synthesis model is taken as the trained speech synthesis model.

[0232] In some embodiments of the present application, the training module 1250 is configured to:

[0233] inputting the combined training data in the combined training data set into the initial speech synthesis model iteratively, obtaining a third output audio segment corresponding to the third text segment, calculating a third loss value according to the audio features of the third output audio segment and the third audio segment, adjusting the model parameters of the initial speech synthesis model according to the third loss value, until the initial speech synthesis model converges, and taking the converged initial speech synthesis model as the trained speech synthesis model.

[0234] In some embodiments of the present application, the training module 1250 can also be configured to:

[0235] inputting the combined training data in the combined training data set into the initial speech synthesis model iteratively, obtaining a first output audio segment corresponding to the first text segment, a second output audio segment corresponding to the second text segment, and a third output audio segment corresponding to the third text segment; calculating a first loss value according to the audio features of the first output audio segment and the first audio segment, calculating a second loss value according to the audio features of the second output audio segment and the second audio segment, calculating a third loss value according to the audio features of the third output audio segment and the third audio segment, and obtaining a fourth loss value according to the first loss value, the second loss value, and the third loss value; adjusting the model parameters of the initial speech synthesis model according to the fourth loss value, until the initial speech synthesis model converges, and taking the converged initial speech synthesis model as the trained speech synthesis model.

[0236] As shown in Figure 13 The speech synthesis device 1300 provided by the embodiments of the present application includes a target text acquisition module 1310, an initial generation module 1320, a text-audio splicing module 1330, a cycle generation module 1340, and an audio splicing module 1350, wherein,

[0237] The target text acquisition module 410 is configured to acquire a target text, and divide the target text into a plurality of target text segments according to a text sequence.

[0238] The initial generation module 1320 is configured to acquire a first target text segment in the plurality of target text segments, input a vector corresponding to the first target text segment into a trained target speech synthesis model, and obtain a first target audio segment.

[0239] The text-audio splicing module 1330 is configured to splice and generate a text-audio alternating sequence according to the first target text segment and the first target audio segment.

[0240] The loop generation module 1340 is configured to loop through and obtain the next target text segment, add the next target text segment to the end of the text-audio alternation sequence to update the text-audio alternation sequence; input the vector corresponding to the text-audio alternation sequence into the trained target speech synthesis model, obtain the next target audio segment, add the next target audio segment to the end of the text-audio alternation sequence to update the text-audio alternation sequence, until the last target audio segment corresponding to the last target text segment is obtained;

[0241] The audio splicing module 1350 is configured to splice together the various target audio segments to generate the target audio.

[0242] In some embodiments of this application, the trained target speech synthesis model is obtained by performing the training method in any of the embodiments.

[0243] In some embodiments of this application, the target text acquisition module 1310 is configured as follows:

[0244] The target text is segmented to obtain multiple target text segments.

[0245] In some embodiments of this application, the audio splicing module 1350 is further configured to:

[0246] Smoothing is applied at the junctions of the various audio segments.

[0247] Figure 14 A schematic diagram is shown that can be used to implement the methods or electronic devices 1400 of the embodiments of this application. In some embodiments, the number of electronic devices may be more or less than the number shown. In some embodiments, it can be implemented using a single or multiple electronic devices. It can also be implemented using cloud or distributed electronic devices in some embodiments.

[0248] like Figure 14 As shown, the electronic device 1400 includes a processor 1410 and a memory 1420. The processor executes programs stored in the memory, which, when executed by a computer, can implement the methods, steps, or functions described in the above embodiments. The processor 1410 may include various types of processors, such as a central processing unit (CPU), a graphics processing unit (GPU), a neural network processor (NPU), a digital signal processor (DSP), etc. The processor 1410 and the memory 1420 are interconnected via a bus 1430. Input / output (I / O) interfaces may also be connected to the bus 1430.

[0249] Although not shown in the figures, the embodiments of the present application also provide a storage medium storing computer programs configured to perform the speech synthesis model training and the speech synthesis method related to any of the embodiments when executed.

[0250] It should be understood that all of the operations in the above-described methods are merely exemplary, and the present disclosure is not limited to any of the operations in the methods or the order of the operations, but should encompass all other equivalent transformations under the same or similar concepts.

[0251] It should also be understood that all of the modules in the above-described apparatuses can be implemented in various ways. The modules can be implemented as hardware, software, or a combination thereof. In addition, any of the modules can be further divided functionally into sub-modules or combined together.

[0252] Processors have been described in connection with various apparatuses and methods. These processors can be implemented using electronic hardware, computer software, or any combination thereof. Whether such processors are implemented as hardware or software depends upon the particular application and general design constraints imposed on the system. As an example, the processors given in the present disclosure, any portion of the processors, or any combination of the processors can be implemented as a microprocessor, a microcontroller, a digital signal processor (DSP), a field programmable gate array (FPGA), a programmable logic device (PLD), a state machine, gated logic, discrete hardware circuits, and other suitable processing components configured to perform the various functions described throughout the present disclosure. The functions of the processors given in the present disclosure, any portion of the processors, or any combination of the processors can be implemented as software executed by a microprocessor, a microcontroller, a DSP, or other suitable platform.

[0253] Software shall be construed broadly to mean instructions, instruction sets, code, code segments, program code, programs, subprograms, software modules, applications, software applications, software packages, routines, subroutines, objects, execution threads, procedures, functions, etc. The software can reside on a computer-readable medium. The computer-readable medium can include, for example, memory such as a magnetic storage device (e.g., hard disk), optical storage device, intelligent card, flash device, random access memory (RAM), read only memory (ROM), programmable ROM (PROM), erasable PROM (EPROM), electrically erasable PROM (EEPROM), a register, or a removable disk. Although the memory is shown separate from the processors in the various aspects given in the present disclosure, the memory can also be internal to the processors (e.g., cache or register).

[0254] The above description is provided as is to enable any person skilled in the art to practice the various aspects described herein. Various modifications to these aspects will be readily apparent to those skilled in the art, and the generic principles defined herein can be applied to other aspects. Thus, the claims are not intended to be limited to the aspects shown herein, but is to be accorded the full scope consistent with the language claims, wherein reference to an element in the claims is not meant to limit, but rather mean any element defined by any of the specialized dictionaries identified herein, and / or throughout the specification.< / unk>

Claims

1. A speech synthesis method characterized by, The method comprises the following steps: obtaining a target text, dividing the target text into a plurality of target text segments according to a text sequence; determining a first target text segment in the plurality of target text segments, and obtaining a sound emitted by a user when reading the first target text segment to generate a first target audio segment; splicing a text-audio alternating sequence according to the first target text segment and the first target audio segment; recursively performing the following steps until a last target audio segment corresponding to a last target text segment is obtained: obtaining a next target text segment, adding the next target text segment to the text-audio alternating sequence to update the text-audio alternating sequence; inputting the text-audio alternating sequence into a trained speech synthesis model to obtain a next target audio segment corresponding to the next target text segment, and adding the next target audio segment to the text-audio alternating sequence to update the text-audio alternating sequence; splicing each target audio segment to generate a target audio.

2. The method of claim 1, wherein, The training process of the speech synthesis model comprises the following steps: obtaining initial training data, wherein the initial training data comprises consecutive text and corresponding consecutive audio; selecting a plurality of first text segments, a plurality of second text segments adjacent to the first text segments, and a plurality of third text segments adjacent to the first text segments from the consecutive text, and obtaining a plurality of first audio segments, a plurality of second audio segments, and a plurality of third audio segments corresponding to the plurality of first text segments, the plurality of second text segments, and the plurality of third text segments from the consecutive audio; detecting whether the associated first text segment, the second text segment, the third text segment, the first audio segment, the second audio segment, and the third audio segment satisfy a splicable condition; splicing the first text segment, the second text segment, the third text segment, the first audio segment, the second audio segment, and the third audio segment that satisfy the splicable condition to obtain a combined training data in a text-audio alternating manner according to a text sequence to generate a combined training data set comprising a plurality of the combined training data; training an initial speech synthesis model according to the combined training data set to obtain a trained speech synthesis model, wherein prosody, rhythm, and / or emotional features in the second text segment and / or the third text segment are extracted and used for training of the speech synthesis model in the training.

3. The method of claim 2, wherein, The splicing the first text segment, the second text segment, and the third text segment to obtain the combined training data in the text-audio alternating manner according to the text sequence comprises: performing word segmentation on the first text segment, the second text segment, and the third text segment to generate a first text sequence, a second text sequence, and a third text sequence, and performing vector conversion on the first text sequence, the second text sequence, and the third text sequence to generate a first text vector, a second text vector, and a third text vector; Discretize the first audio segment, the second audio segment and the third audio segment and extract audio features to generate a first audio feature sequence, a second audio feature sequence and a third audio feature sequence, and perform vector conversion on the first audio feature sequence, the second audio feature sequence and the third audio feature sequence to generate a first audio vector, a second audio vector and a third audio vector; In the order of the second text vector, the second audio feature sequence, the first text vector, the first audio feature sequence, the third text vector and the third audio feature sequence, the second text vector, the second audio feature sequence, the first text vector, the first audio feature sequence, the third text vector and the third audio feature sequence are spliced to generate the combined training data.

4. The method of claim 2, wherein, The detection of whether the associated first text segment, the second text segment, the third text segment, the first audio segment, the second audio segment and the third audio segment satisfy the splicable condition comprises: Obtaining the combined text length of the first text segment, the second text segment and the third text segment, and obtaining the combined audio duration of the first audio segment, the second audio segment and the third audio segment, when the combined text length is less than the length threshold and the combined audio duration is less than the duration threshold, the text and voice corresponding to the first text segment, the second text segment, the third text segment, the first audio segment, the second audio segment and the third audio segment satisfy the first splicable condition.

5. The method of claim 2, wherein, The detection of whether the associated first text segment, the second text segment, the third text segment, the first audio segment, the second audio segment and the third audio segment satisfy the splicable condition comprises: Obtaining the voiceprint feature approximation degree between the first audio segment, the second audio segment and the third audio segment, when the voiceprint feature approximation degree is greater than the voiceprint approximation threshold, the text and voice corresponding to the first text segment, the second text segment, the third text segment, the first audio segment, the second audio segment and the third audio segment satisfy the second splicable condition.

6. The method of claim 2, wherein, The detection of whether the associated first text segment, the second text segment, the third text segment, the first audio segment, the second audio segment and the third audio segment satisfy the splicable condition comprises: When the speaking time point interval of adjacent audio segments in the first audio segment, the second audio segment and the third audio segment is less than the interval threshold, the text and voice corresponding to the first text segment, the second text segment, the third text segment, the first audio segment, the second audio segment and the third audio segment satisfy the third splicable condition.

7. The method of claim 2, wherein, The training of the initial speech synthesis model according to the combined training data set to obtain a trained speech synthesis model comprises: Iteratively inputting the combined training data in the combined training data set into the initial speech synthesis model, obtaining a first output audio segment corresponding to the first text segment, calculating a first loss value according to the audio features of the first output audio segment and the first audio segment, adjusting the model parameters of the initial speech synthesis model according to the first loss value, until the initial speech synthesis model converges, and taking the converged initial speech synthesis model as the trained speech synthesis model.

8. The method of claim 2, wherein, The training of the initial speech synthesis model according to the combined training data set to obtain a trained speech synthesis model comprises: inputting the combined training data in the combined training data set into the initial speech synthesis model, obtaining a second output audio segment corresponding to the second text segment, calculating a second loss value according to the audio features of the second output audio segment and the second audio segment, adjusting the model parameters of the initial speech synthesis model according to the second loss value, iteratively updating until the initial speech synthesis model converges, and taking the converged initial speech synthesis model as the trained speech synthesis model.

9. The method of claim 2, wherein, The training of the initial speech synthesis model according to the combined training data set to obtain the trained speech synthesis model comprises: inputting the combined training data in the combined training data set into the initial speech synthesis model, obtaining a third output audio segment corresponding to the third text segment, calculating a third loss value according to the audio features of the third output audio segment and the third audio segment, adjusting the model parameters of the initial speech synthesis model according to the third loss value, and converging the initial speech synthesis model, and taking the converged initial speech synthesis model as the trained speech synthesis model.

10. The method of claim 2, wherein, The training of the initial speech synthesis model according to the combined training data set to obtain the trained speech synthesis model comprises: inputting the combined training data in the combined training data set into the initial speech synthesis model, obtaining a first output audio segment corresponding to the first text segment, a second output audio segment corresponding to the second text segment, and a third output audio segment corresponding to the third text segment; calculating a first loss value according to the audio features of the first output audio segment and the first audio segment, calculating a second loss value according to the audio features of the second output audio segment and the second audio segment, calculating a third loss value according to the audio features of the third output audio segment and the third audio segment, and obtaining a fourth loss value according to the first loss value, the second loss value and the third loss value; adjusting the model parameters of the initial speech synthesis model according to the fourth loss value, and converging the initial speech synthesis model, and taking the converged initial speech synthesis model as the trained speech synthesis model.

11. The method of claim 1, wherein, The dividing of the target text into multiple target text segments according to the text sequence comprises: performing sentence breaking processing on the target text to obtain multiple target text segments.

12. The method of claim 1, wherein, The splicing processing of the target audio segments to generate the target audio further comprises: performing smoothing processing at the connection of the audio segments.

13. A speech synthesis apparatus characterized by comprising: The system comprises a target text acquisition module, an initial generation module, a text-audio splicing module, a loop generation module and an audio splicing module, wherein the target text acquisition module is configured to acquire a target text, and divide the target text into multiple target text segments according to the text sequence; the initial generation module is configured to acquire a first target text segment in the multiple target text segments, and acquire a sound emitted by a user when reading the first target text segment, and generate a first target audio segment; the text-audio splicing module is configured to splice the first target text segment and the first target audio segment to generate a text-audio alternating sequence; The cycle generation module is configured to cyclically acquire a next target text segment, add the next target text segment to a tail of the text-audio alternation sequence to update the text-audio alternation sequence, input a vector corresponding to the text-audio alternation sequence into the trained speech synthesis model, acquire a next target audio segment, and add the next target audio segment to the tail of the text-audio alternation sequence to update the text-audio alternation sequence, until a last target audio segment corresponding to a last target text segment is acquired; The audio splicing module is configured to splice the target audio segments to generate a target audio.

Images