Speech synthesis method, device, storage medium and electronic device

By converting text data into phoneme sequences and generating a Mel spectrum containing breathing sound tags, the problem of insufficient realism in the prior art is solved, and more realistic speech synthesis is achieved, which improves the user experience.

CN114093340BActive Publication Date: 2024-11-22BEIJING DA MI TECH CO LTD
View PDF 5 Cites 0 Cited by

Patent Information

Application Number
CN202010753313.X
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2020-07-30
Publication Date
2024-11-22
Estimated Expiration
2040-07-30

AI Technical Summary

Technical Problem

Existing voice synthesis technologies are difficult to generate voice with strong sense of realism, resulting in poor user experience.

Method used

By converting text data into phoneme sequences and processing based on a pretrained speech synthesis model, a Mel spectrum containing breathing sound tags is generated to synthesize speech with a real background environment.

Benefits of technology

It enhances the authenticity of voice synthesis, brings it closer to real-person voice, and improves the user experience.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN114093340B_ABST
    Figure CN114093340B_ABST
Patent Text Reader

Abstract

The embodiment of the present application discloses a speech synthesis method, device, storage medium and electronic device, which belongs to the field of computer technology. The method includes: a server converts text data into at least one phoneme sequence, the text data is provided with at least one breathing sound label, speech synthesis processing is performed on the at least one phoneme sequence based on a pre-trained speech synthesis model to obtain a Mel spectrum corresponding to the text data, and synthesized speech corresponding to the text data is obtained based on the Mel spectrum corresponding to the text data, the synthesized speech includes a breathing sound corresponding to at least one breathing sound label, and the realism of the synthesized speech is enhanced to make it closer to real person speech, thereby improving the user experience.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a speech synthesis method, device, storage medium and electronic device. Background Art

[0002] With the development of artificial intelligence technology, TTS (Text To Speech) has also been developed. Speech synthesis technology can be used to convert text data into natural speech, and its application scenarios are relatively wide, such as: used in voice dictionaries, news broadcasts, SMS broadcasts, e-book readings and other scenarios. However, in related technologies, the speech synthesis process is relatively complicated, and the generated speech is too stiff and there is a big difference from the sound of real people speaking, resulting in a poor user experience. Summary of the invention

[0003] The embodiments of the present application provide a speech synthesis method, device, storage medium and electronic device, which can solve the problem that speech with a strong sense of reality cannot be synthesized in the related art. The technical solution is as follows:

[0004] In a first aspect, an embodiment of the present application provides a speech synthesis method, the method comprising:

[0005] Converting text data into at least one phoneme sequence; wherein the text data is provided with at least one breathing sound label;

[0006] Performing speech synthesis processing on the at least one phoneme sequence based on a pre-trained speech synthesis model to obtain a Mel spectrum corresponding to the text data;

[0007] The synthesized speech corresponding to the text data is obtained based on the Mel-spectrogram corresponding to the text data; wherein the synthesized speech includes the breathing sound corresponding to the at least one breathing sound label.

[0008] In a second aspect, an embodiment of the present application provides a speech synthesis device, the speech synthesis device comprising:

[0009] A conversion module, used to convert text data into at least one phoneme sequence; wherein the text data is provided with at least one breathing sound label;

[0010] A first processing module, configured to perform speech synthesis processing on the at least one phoneme sequence based on a pre-trained speech synthesis model to obtain a Mel spectrum corresponding to the text data;

[0011] The second processing module is used to obtain the synthesized speech corresponding to the text data based on the Mel-frequency spectrum corresponding to the text data; wherein the synthesized speech includes the breathing sound corresponding to the at least one breathing sound label.

[0012] In a third aspect, an embodiment of the present application provides a computer storage medium, wherein the computer storage medium stores a plurality of instructions, wherein the instructions are suitable for being loaded by a processor and executing the above-mentioned method steps.

[0013] In a fourth aspect, an embodiment of the present application provides an electronic device, which may include: a processor and a memory; wherein the memory stores a computer program, and the computer program is suitable for being loaded by the processor and executing the above-mentioned method steps.

[0014] The beneficial effects brought about by the technical solutions provided by some embodiments of the present application include at least:

[0015] When the solution of the embodiment of the present application is executed, the server converts text data into at least one phoneme sequence, the text data is provided with at least one breathing sound label, and speech synthesis processing is performed on the at least one phoneme sequence based on a pre-trained speech synthesis model to obtain a Mel spectrum corresponding to the text data, and synthesized speech corresponding to the text data is obtained based on the Mel spectrum corresponding to the text data. The synthesized speech includes breathing sounds corresponding to at least one breathing sound label, thereby enhancing the realism of the synthesized speech, making it closer to real person speech, and thereby improving the user experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0017] Figure 1 It is a system architecture diagram provided by an embodiment of the present application;

[0018] Figure 2 It is a flowchart of a speech synthesis method provided in an embodiment of the present application;

[0019] Figure 3 is another flowchart of the speech synthesis method provided in an embodiment of the present application;

[0020] Figure 4 is a schematic diagram of the structure of a device provided in an embodiment of the present application;

[0021] Figure 5 It is a structural schematic diagram of a device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0022] In order to make the objectives, technical solutions and advantages of the present application more clear, the embodiments of the present application will be further described in detail below with reference to the accompanying drawings.

[0023] Figure 1 A schematic diagram of an exemplary system architecture 100 is shown to which the speech synthesis method or speech synthesis device according to the embodiments of the present application can be applied.

[0024] like Figure 1 As shown, the system architecture 100 may include one or more of terminal devices 101, 102, 103, a network 104, and a server 105. The network 104 is used to provide a medium for communication links between the terminal devices 101, 102, 103 and the server 105. Various communication client applications may be installed on the terminal devices 101, 102, 103, such as: video recording applications, video playback applications, voice interaction applications, search applications, instant messaging tools, email clients, social platform software, etc. The network 104 may include various connection types, such as wired, wireless communication links or optical fiber cables, etc.

[0025] Users can use terminal devices 101, 102, 103 to interact with server 105 through network 104 to receive or send messages, etc. Terminal devices 101, 102, 103 can be various electronic devices with display screens, including but not limited to smart phones, tablet computers, portable computers, desktop computers, etc. Network 104 can include various types of wired communication links or wireless communication links, for example: wired communication links include optical fibers, twisted pairs or coaxial cables, and wireless communication links include Bluetooth communication links, wireless fidelity (WIreless-FIdelity, Wi-Fi) communication links or microwave communication links, etc. Terminal devices 101, 102, 103 can be hardware or software. When terminal devices 101, 102, 103 are software, they can be installed in the electronic devices listed above. It can be implemented as multiple software or software modules (for example: used to provide distributed services), or it can be implemented as a single software or software module, which is not specifically limited here. When the terminal devices 101, 102, and 103 are hardware, a display device and a camera may be installed thereon. The display device may be any device capable of realizing a display function, and the camera is used to collect video streams; for example, the display device may be a cathode ray tube display (CR), a light-emitting diode display (LED), an electronic ink screen, a liquid crystal display (LCD), a plasma display panel (PDP), etc. The user may use the display device on the terminal devices 101, 102, and 103 to view displayed text, pictures, videos, and other information.

[0026] It should be noted that the speech synthesis method provided in the embodiment of the present application is usually performed by the server 105, and accordingly, the speech synthesis device is usually arranged in the server 105. The server 105 can be a server that provides various services, and the server 105 can be hardware or software. When the server 105 is hardware, it can be implemented as a distributed server cluster composed of multiple servers, or it can be implemented as a single server. When the server 105 is software, it can be implemented as multiple software or software modules (for example, for providing distributed services), or it can be implemented as a single software or software module, which is not specifically limited here.

[0027] The server 105 in the present application can be a terminal device that provides various services, such as: the server converts text data into at least one phoneme sequence, the text data is provided with at least one breathing sound label, speech synthesis processing is performed on the at least one phoneme sequence based on a pre-trained speech synthesis model to obtain a Mel spectrum corresponding to the text data, and synthesized speech corresponding to the text data is obtained based on the Mel spectrum corresponding to the text data, and the synthesized speech includes breathing sounds corresponding to at least one breathing sound label.

[0028] It should be noted here that the speech synthesis method provided in the embodiment of the present application can be executed by one or more of the terminal devices 101, 102, 103, and / or the server 105. Accordingly, the speech synthesis device provided in the embodiment of the present application is generally arranged in the corresponding terminal device, and / or the server 105, but the present application is not limited to this.

[0029] It should be understood that Figure 1 The number of terminal devices, networks and servers in the embodiment is only for illustration. Any number of terminal devices, networks and servers may be provided according to the implementation requirements.

[0030] The following will be combined with the attached Figure 2 To Attachment Figure 3 , the speech synthesis method provided in the embodiment of the present application is described in detail. Figure 2 , which is a flow chart of a speech synthesis method provided in an embodiment of the present application. Figure 2 As shown, the method of the embodiment of the present application may include the following steps:

[0031] S201, converting text data into at least one phoneme sequence.

[0032] Among them, text data refers to data presented in text form and containing content information, and at least one breathing sound label is set in the text data, and a phoneme sequence refers to phoneme elements arranged in a column; the text data can be text data of English words, and each English word in the text data corresponds to a phoneme sequence; the text data can also be Chinese words, and each word and / or breathing sound label in the text data corresponds to a phoneme sequence.

[0033] Generally, a user can send original text data containing content information to a server through a terminal. After obtaining the original text data, the server will analyze the original text data to obtain sentence segmentation information, and determine the position information corresponding to at least one breathing sound tag based on the sentence segmentation information. The position information is the position information of at least one breathing sound tag in the original text data. Based on the position information, at least one breathing sound tag is marked on the original text data to obtain text data. At least one breathing sound tag is set in the text data, and at least one breathing sound tag includes duration information corresponding to the breathing sound. Then, the server will convert each word and each breathing sound tag in the obtained text data into their corresponding phoneme sequences by querying the phoneme table, so as to facilitate subsequent processing of at least one phoneme sequence corresponding to the text data.

[0034] For example: the text data consists of Chinese words, and the content of the text data is "Good morning! [Breathing sound label 1] Have you eaten? [Breathing sound label 2]", the duration information included in the breathing sound label 1 is 2S, and the duration information included in the breathing sound label 2 is 3S. After querying the phoneme table, the server can convert the text data into 10 phoneme sequences: {z,a,o}, {s,h,a,n,g}, {h,a,o}, {Breathing sound 1}, {n,i}, {c,h,i}, {f,a,n}, {l,e}, {m,a}, {Breathing sound 2}. Each word and each breathing sound label in the text data corresponds to a phoneme sequence.

[0035] S202: Perform speech synthesis processing on at least one phoneme sequence based on a pre-trained speech synthesis model to obtain a Mel spectrum corresponding to the text data.

[0036] The Mel spectrum refers to a spectrum represented by the Mel scale, and the Mel spectrum contains the characteristics of the sound.

[0037] Generally, after the server obtains at least one phoneme sequence corresponding to the text data, it can perform speech synthesis processing on the at least one phoneme sequence through a pre-trained speech synthesis model to obtain a Mel spectrum corresponding to the text data. The Mel spectrum contains the sound features corresponding to the text data (including the features of breathing sounds). Based on the Mel spectrum, the speech corresponding to the text data can be determined.

[0038] The training process of the pre-trained speech synthesis model may include:

[0039] The server obtains sample text data and sample speech data, converts the sample text data into at least one phoneme sequence by querying a phoneme table, and converts the at least one phoneme sequence into at least one phoneme feature vector, performs dimensionality reduction processing on the at least one phoneme feature vector to obtain a first feature vector, performs position encoding processing on the first feature vector based on text sequence information of the sample text data to obtain a second feature vector, the text sequence information is used to represent at least one of the order and features of words in the sample text data, performs FFT fast Fourier transform processing on the second feature vector to obtain a phoneme sequence vector; and performs probability evaluation on the duration of at least one phoneme sequence in the sample text data to obtain a pronunciation duration of at least one phoneme sequence. long, performing time extraction processing on the phoneme sequence vector based on the pronunciation duration of at least one phoneme sequence to obtain a phoneme alignment vector; performing position encoding processing on the phoneme alignment vector based on the text sequence information of the sample text data to obtain a third feature vector, the text sequence information is used to represent at least one of the order and characteristics of words in the sample text data, performing FFT fast Fourier transform processing on the third feature vector to obtain a fourth feature vector, and processing the fourth feature vector based on a linear layer to obtain a Mel spectrum corresponding to the sample text data; calculating a loss value between the Mel spectrum corresponding to the sample text data and the Mel spectrum corresponding to the sample speech data, and generating a pre-trained speech synthesis model when the loss value is less than or equal to a preset threshold.

[0040] S203, obtaining a synthesized speech corresponding to the text data based on the Mel-spectrogram corresponding to the text data.

[0041] Among them, synthesized speech refers to speech that has been accurately synthesized and can more realistically reflect the user's voice characteristics, and the synthesized speech includes at least one breathing sound corresponding to a breathing sound label.

[0042] Generally, since the Mel spectrum corresponding to the text data contains the sound features corresponding to the text data, the Mel spectrum can be Fourier transformed based on the feature information in the Mel spectrum to obtain the synthesized speech corresponding to the text data. In order to make the final sound data more realistic, background noise data can be calculated based on a preset signal-to-noise ratio, and the background noise data can be added to the synthesized speech to obtain the speech corresponding to the text data, that is, speech with a real background environment can be obtained, and the speech also includes breathing sounds, which can better reflect the realism of the speech.

[0043] For example, the text data consists of Chinese words, and the text data content is "Good morning! [Breathing sound label 1] Have you eaten? [Breathing sound label 2]", the duration information included in the breathing sound label 1 is 2S, and the duration information included in the breathing sound label 2 is 3S. In the synthesized speech processed by the speech synthesis model, a 2S breathing sound speech segment is presented after the speech segment "Good morning!", and a 3S breathing sound speech segment is presented after the speech segment "Have you eaten?"

[0044] When the solution of the embodiment of the present application is executed, the server converts text data into at least one phoneme sequence, the text data is provided with at least one breathing sound label, and speech synthesis processing is performed on the at least one phoneme sequence based on a pre-trained speech synthesis model to obtain a Mel spectrum corresponding to the text data, and synthesized speech corresponding to the text data is obtained based on the Mel spectrum corresponding to the text data. The synthesized speech includes breathing sounds corresponding to at least one breathing sound label, thereby enhancing the realism of the synthesized speech, making it closer to real person speech, and thereby improving the user experience.

[0045] See also Figure 3 , a flow chart of a speech synthesis method is provided for an embodiment of the present application, and the speech synthesis method may include the following steps:

[0046] S301, obtaining original text data.

[0047] The original text data refers to the text data that has not been processed and contains content information.

[0048] Generally, the server can obtain the corresponding original text data by receiving the text information sent by the terminal and parsing the text information. Alternatively, the server can obtain the corresponding original text data by identifying the specified text information.

[0049] S302, analyzing the original text data to obtain sentence segmentation information.

[0050] The sentence segmentation information refers to the position information where pauses are required after the sentence is segmented according to the essence of the sentence, the semantics of the sentence and the punctuation marks of the sentence.

[0051] Generally, the server will analyze the original text data to obtain the semantics of the sentences in the original text data, and can obtain the essence of the sentences based on the semantics of the sentences, and each sentence contains punctuation marks. Real-person speech usually makes corresponding pauses according to the semantics of the sentences and the key points that need to be highlighted, and the pauses are accompanied by breathing sounds. In order to simulate real-person speech more realistically, the server can determine the position where the breathing sound label needs to be added based on the obtained sentence semantics, the essence of the sentences and the punctuation marks in the sentences, and add the breathing sound label at the corresponding position.

[0052] S303: Determine position information corresponding to at least one breathing sound tag based on the sentence segmentation information.

[0053] Among them, the breathing sound label refers to a label that includes the duration information corresponding to the breathing sound. After the breathing sound label is added to the text data, it can become a part of the text data, and the breathing sound label has its corresponding phoneme sequence. The duration information is used to indicate that the voice data corresponding to the text data needs to add the time length of the breathing sound at the position corresponding to the text data with the breathing sound label; the position information refers to the position of the breathing sound label in the sentence of the text data, and the voice corresponding to the text data also corresponds to the breathing sound at the position with the breathing sound label in the text data.

[0054] S304: annotate original text data with at least one breathing sound label based on the position information to obtain text data.

[0055] Among them, text data refers to data presented in text form and containing content information. The text data can be text data of English words, and each English word in the text data corresponds to a phoneme sequence; the text data can also be Chinese words, and each word in the text data corresponds to a phoneme sequence.

[0056] Generally, the server will mark the breathing sound label at the corresponding position of the original text data according to the position information corresponding to at least one breathing sound label. The corresponding breathing sound labels at different positions of the original text data may be different. According to the semantics of the sentence, the essential part of the sentence, and the type of sentence punctuation in the original text data, the position that needs to be highlighted or expressed in the original text data can be determined. The breathing sound corresponding to the position that needs to be highlighted or expressed in the original text data is longer in duration or heavier in volume. Therefore, different positions that need to be marked with the breathing sound label have different corresponding breathing sound labels. Different breathing sound labels can be reflected in different duration information and volume contained in the breathing sound labels.

[0057] S305: Convert the text data into at least one phoneme sequence.

[0058] Among them, text data refers to data presented in text form and containing content information, and at least one breathing sound label is set in the text data, and the phoneme sequence refers to the phoneme elements arranged in a column; the text data can be text data of English words, and each English word and / or breathing sound label in the text data corresponds to a phoneme sequence; the text data can also be Chinese words, and each word and / or breathing sound label in the text data corresponds to a phoneme sequence.

[0059] Generally, a user can send original text data containing content information to a server through a terminal. After obtaining the original text data, the server will analyze the original text data to obtain sentence segmentation information, and determine the position information corresponding to at least one breathing sound tag based on the sentence segmentation information. The position information is the position information of at least one breathing sound tag in the original text data. Based on the position information, at least one breathing sound tag is marked on the original text data to obtain text data. At least one breathing sound tag is set in the text data, and at least one breathing sound tag includes duration information corresponding to the breathing sound. Then, the server will convert each word and each breathing sound tag in the obtained text data into their corresponding phoneme sequences by querying the phoneme table, so as to facilitate subsequent processing of at least one phoneme sequence corresponding to the text data.

[0060] For example: the text data consists of Chinese words, and the content of the text data is "Good morning! [Breathing sound label 1] Have you eaten? [Breathing sound label 2]", the duration information included in the breathing sound label 1 is 2S, and the duration information included in the breathing sound label 2 is 3S. After querying the phoneme table, the server can convert the text data into 10 phoneme sequences: {z,a,o}, {s,h,a,n,g}, {h,a,o}, {Breathing sound 1}, {n,i}, {c,h,i}, {f,a,n}, {l,e}, {m,a}, {Breathing sound 2}. Each word and each breathing sound label in the text data corresponds to a phoneme sequence.

[0061] S306, obtaining sample text data and sample voice data.

[0062] The sample text data refers to data presented in text form and containing sample content information, and at least one breathing sound label is set in the sample text data. The sample voice data is the voice data corresponding to the sample text data. The sample voice data contains the sound features (Mel spectrum) that the user needs to synthesize, and the sample voice data also contains breathing sounds corresponding to the breathing label positions in the sample text data. The sample text data can be text data of English words, and each English word and / or breathing sound label in the sample text data corresponds to a phoneme sequence; the sample text data can also be Chinese words, and each word and / or breathing sound label in the sample text data corresponds to a phoneme sequence

[0063] Generally, a speech synthesis model that can synthesize speech based on text data needs to be trained on a large amount of sample data. The large amount of sample data includes a large amount of sample text data and sample speech data corresponding to each of the sample text data.

[0064] S307, performing encoding and decoding processing on the sample text data to obtain the Mel spectrum corresponding to the sample text data.

[0065] The Mel spectrum refers to a spectrum represented by the Mel scale, and the Mel spectrum contains the characteristics of the sound.

[0066] In general, the process of encoding the sample text data may include: obtaining the phoneme sequence corresponding to the sample text data by querying the phoneme table, each word / phrase in the sample text data corresponds to a phoneme sequence, and in order to process the obtained multiple phoneme sequences later, it is necessary to convert the obtained multiple phoneme sequences into the form of phoneme feature vectors, the number of phoneme feature vectors is the same as the number of phoneme sequences, each phoneme sequence corresponds to a phoneme feature vector, and the phoneme feature vector refers to a vector containing the features corresponding to the phoneme sequence obtained after the preliminary conversion of the phoneme sequence. Performing dimensionality reduction processing on at least one phoneme feature vector to obtain a first feature vector, the first feature vector is a vector obtained after the dimensionality reduction of the phoneme feature vector, and there is a dimension difference with the original dimensional phoneme feature vector. By performing position encoding processing on the first feature vector, the text sequence information in the sample text data can be added to the first feature vector, and a second feature vector that can reflect the time series can be obtained. The text sequence information is the relevant information of the words / phrases in the sample text data, and the text sequence information can be used to represent at least one of the order and features of the words in the sample text data; the second feature vector is a vector obtained after the first feature vector is processed by position encoding, which is different from the first feature vector. The second eigenvector can be processed by FFT based on a transformer feed-forward network composed of an FFT module, an attention mechanism, and a convolutional layer. The parameters contained in the second eigenvector are trained, and the information that needs attention is extracted to obtain a phoneme sequence vector. The phoneme sequence vector refers to a vector expressed in vector form after the phoneme sequence is converted multiple times.

[0067] The process of performing duration extraction processing on sample text data may include: performing probability evaluation on the duration of at least one phoneme sequence in the sample text data to obtain the pronunciation duration of at least one phoneme sequence, wherein the pronunciation duration refers to the sum of the pronunciation durations of each phoneme in the phoneme sequence, and each phoneme sequence corresponds to a pronunciation duration, which is also the duration information of the phoneme sequence. The existing method of extracting the duration information of each phoneme in the phoneme sequence is to extract it through a pre-trained model, which has a poor effect and only realizes the alignment of sentences, but does not realize the alignment of phonemes to phonemes; in this scheme, a statistical model (classical decoder) is used to process the phoneme sequence to realize the forced alignment of phonemes, specifically including: performing statistics on the pronunciation duration of the phoneme sequence corresponding to each word / phrase, and then performing probability evaluation on the pronunciation duration of each obtained phoneme sequence, and selecting the phoneme sequence with the largest probability from the probability evaluation result as the output result, thereby realizing the alignment of phonemes to phonemes, and then obtaining the phoneme alignment vector of the phoneme alignment, which refers to the vector obtained after the phoneme alignment based on the pronunciation duration of the phoneme sequence.

[0068] The process of decoding the sample text data may include: by performing position encoding on the phoneme alignment vector, the text sequence information in the sample text data may be added to the phoneme alignment feature vector, and a third feature vector that can reflect the time series may be obtained. The third feature vector is a vector obtained after the phoneme alignment vector is processed by position encoding, and is distinguished from the first feature vector and the second feature vector. The third feature vector may be subjected to FFT processing based on a transformer feedforward network composed of an FFT module and including an attention mechanism and a convolutional layer, and the parameters contained in the third feature vector may be trained, and after extracting the information that needs attention, a fourth feature vector may be obtained. The fourth feature vector is a vector obtained after the third feature vector is subjected to FFT processing, and is distinguished from the first feature vector, the second feature vector, and the third feature vector. The fourth feature vector may then be processed based on a linear layer to obtain a Mel spectrum α corresponding to the sample text data.

[0069] S308, generating a pre-trained speech synthesis model when the loss value is less than or equal to a preset threshold.

[0070] The loss value refers to the inconsistency between the Mel spectrum α corresponding to the sample text data and the comparison label. The smaller the loss value, the better the robustness of the trained model. The comparison label is the Mel spectrum β corresponding to the sample speech data. The preset threshold refers to the maximum loss value pre-set based on the model to be trained. When the loss value is less than or equal to the pre-set maximum loss value, it indicates that the model training is completed.

[0071] Generally, the sample speech data is processed to obtain the Mel spectrum β corresponding to the sample speech data, and the Mel spectrum β corresponding to the sample speech data is used as a comparison label, and then the Mel spectrum α corresponding to the sample text data is compared with the comparison label to calculate the loss value between the two. Before the loss value reaches a preset threshold, continuous iterative training is performed based on the above S307 step until the loss value is less than or equal to the preset threshold, indicating that the model training is completed, and the model at this time is a trained speech synthesis model. The server can perform speech synthesis processing on the input text data based on the speech synthesis model to obtain speech corresponding to the input text data.

[0072] S309: Perform speech synthesis processing on at least one phoneme sequence based on a pre-trained speech synthesis model to obtain a Mel spectrum corresponding to the text data.

[0073] Generally, after the server obtains at least one phoneme sequence corresponding to the text data, it can perform speech synthesis processing on the at least one phoneme sequence through a pre-trained speech synthesis model to obtain a Mel spectrum corresponding to the text data. The Mel spectrum contains the sound features corresponding to the text data (including the features of breathing sounds). Based on the Mel spectrum, the speech corresponding to the text data can be determined.

[0074] S310, obtaining a synthesized speech corresponding to the text data based on the Mel-spectrogram corresponding to the text data.

[0075] Among them, synthesized speech refers to speech that has been accurately synthesized and can more realistically reflect the user's voice characteristics, and the synthesized speech includes at least one breathing sound corresponding to a breathing sound label.

[0076] Generally, since the Mel spectrum corresponding to the text data contains the sound features corresponding to the text data, the Mel spectrum can be Fourier transformed based on the feature information in the Mel spectrum to obtain the synthesized speech corresponding to the text data. In order to make the final sound data more realistic, background noise data can be calculated based on a preset signal-to-noise ratio, and the background noise data can be added to the synthesized speech to obtain the speech corresponding to the text data, that is, speech with a real background environment can be obtained, and the speech also includes breathing sounds, which can better reflect the realism of the speech.

[0077] For example, the text data consists of Chinese words, and the text data content is "Good morning! [Breathing sound label 1] Have you eaten? [Breathing sound label 2]", the duration information included in the breathing sound label 1 is 2S, and the duration information included in the breathing sound label 2 is 3S. In the synthesized speech processed by the speech synthesis model, a 2S breathing sound speech segment is presented after the speech segment "Good morning!", and a 3S breathing sound speech segment is presented after the speech segment "Have you eaten?"

[0078] S311, calculating background noise data based on a preset signal-to-noise ratio.

[0079] The signal-to-noise ratio refers to the ratio of the signal to the noise in an electronic device or electronic system, and the preset signal-to-noise ratio is the ratio of the synthesized speech signal to the background noise that can be preset. The relevant data of the background noise can be obtained through the preset signal-to-noise ratio. The background noise data can be the power information corresponding to the noise that needs to be added.

[0080] S312, adding the background noise data to the synthesized speech to obtain the speech of the text data.

[0081] Generally, based on the power information of the synthesized speech and the power information of the background noise, the server can add noise to the synthesized speech to obtain speech with background noise, that is, a more realistic speech corresponding to the text data. The synthesized speech after adding background noise is more realistic and natural.

[0082] When the scheme of the embodiment of the present application is executed, the server obtains original text data, analyzes the original text data to obtain sentence segmentation information, determines the position information corresponding to at least one breathing sound label based on the sentence segmentation information, annotates the original text data with at least one breathing sound label based on the position information to obtain text data, converts the text data into at least one phoneme sequence, obtains sample text data and sample voice data, respectively encodes and decodes the sample text data to obtain a Mel spectrum corresponding to the sample text data, generates a pre-trained speech synthesis model when the loss value is less than or equal to a preset threshold, performs speech synthesis processing on at least one phoneme sequence based on the pre-trained speech synthesis model to obtain a Mel spectrum corresponding to the text data, obtains synthesized speech corresponding to the text data based on the Mel spectrum corresponding to the text data, calculates background noise data based on a preset signal-to-noise ratio, and adds the background noise data to the synthesized speech to obtain the speech of the text data. In this way, the realism of the synthesized speech can be enhanced, making it closer to real-life speech, thereby improving the user experience.

[0083] The following is an embodiment of the device of the present application, which can be used to execute the embodiment of the method of the present application. For details not disclosed in the embodiment of the device of the present application, please refer to the embodiment of the method of the present application.

[0084] See also Figure 4 , which shows a schematic diagram of the structure of a speech synthesis device provided by an exemplary embodiment of the present application. Hereinafter referred to as device 4, device 4 can be implemented as all or part of a terminal through software, hardware or a combination of both. Device 4 includes a conversion module 401, a first processing module 402, and a second processing module 403.

[0085] A conversion module 401, used to convert text data into at least one phoneme sequence; wherein the text data is provided with at least one breathing sound label;

[0086] A first processing module 402 is configured to perform speech synthesis processing on the at least one phoneme sequence based on a pre-trained speech synthesis model to obtain a Mel spectrum corresponding to the text data;

[0087] The second processing module 403 is configured to obtain a synthesized speech corresponding to the text data based on the Mel-frequency spectrum corresponding to the text data; wherein the synthesized speech includes a breathing sound corresponding to the at least one breathing sound label.

[0088] Optionally, the conversion module 401 includes:

[0089] A first acquisition unit, used to acquire original text data;

[0090] A first labeling unit is used to label the original text data with a breathing sound label to obtain text data; wherein the text data is provided with at least one breathing sound label;

[0091] The conversion unit is used to convert the text data into the at least one phoneme sequence.

[0092] Optionally, the conversion module 401 includes:

[0093] A first determining unit is used to determine the position information corresponding to the at least one breathing sound label; wherein the position information is the position information of the at least one breathing sound label in the original text data;

[0094] The second labeling unit is used to label the original text data with the at least one breathing sound label based on the position information to obtain the text data.

[0095] Optionally, the conversion module 401 includes:

[0096] An analysis unit, used for analyzing the original text data to obtain sentence segmentation information;

[0097] The second determining unit is used to determine the position information corresponding to the at least one breathing sound tag based on the sentence segmentation information.

[0098] Optionally, the device 4 further comprises:

[0099] A second acquisition unit, used to acquire sample text data and sample voice data;

[0100] A processing unit, used for performing encoding and decoding processing on the sample text data to obtain a Mel spectrum corresponding to the sample text data;

[0101] A generating unit, used to generate the pre-trained speech synthesis model when the loss value is less than or equal to a preset threshold; wherein the loss value is the loss value between the Mel spectrum corresponding to the sample text data and the Mel spectrum corresponding to the sample speech data.

[0102] Optionally, the second processing module 403 further includes:

[0103] A calculation unit, used to calculate background noise data based on a preset signal-to-noise ratio;

[0104] An adding unit is used to add the background noise data to the synthesized speech.

[0105] It should be noted that, when the device 4 provided in the above embodiment executes the speech synthesis method, only the division of the above functional modules is used as an example. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device is divided into different functional modules to complete all or part of the functions described above. In addition, the speech synthesis method embodiments provided in the above embodiment belong to the same concept, and the implementation process thereof is detailed in the method embodiment, which will not be repeated here.

[0106] Figure 5 A schematic diagram of the structure of a speech synthesis device provided in an embodiment of the present application, hereinafter referred to as device 5, can be integrated into the aforementioned server or terminal device, such as Figure 5 As shown, the device includes: a memory 502, a processor 501, an input device 503, an output device 504 and a communication interface.

[0107] The memory 502 may be an independent physical unit, and may be connected to the processor 501, the input device 503, and the output device 504 via a bus. The memory 502, the processor 501, the input device 503, and the output device 504 may also be integrated together and implemented by hardware.

[0108] The memory 502 is used to store programs for implementing the above method embodiments or various modules of the device embodiments, and the processor 501 calls the program to execute the operations of the above method embodiments.

[0109] The input device 502 includes but is not limited to a keyboard, a mouse, a touch panel, a camera and a microphone; the output device includes but is not limited to a display screen.

[0110] The communication interface is used to send and receive various types of messages, and the communication interface includes but is not limited to a wireless interface or a wired interface.

[0111] Optionally, when part or all of the distributed task scheduling method in the above embodiment is implemented by software, the device may also include only a processor. The memory for storing the program is located outside the device, and the processor is connected to the memory through a circuit / wire to read and execute the program stored in the memory.

[0112] The processor may be a central processing unit (CPU), a network processor (NP), or a combination of a CPU and a NP.

[0113] The processor may further include a hardware chip. The hardware chip may be an application-specific integrated circuit (ASIC), a programmable logic device (PLD) or a combination thereof. The PLD may be a complex programmable logic device (CPLD), a field-programmable gate array (FPGA), a generic array logic (GAL) or any combination thereof.

[0114] The memory may include volatile memory, such as random-access memory (RAM); the memory may also include non-volatile memory, such as flash memory, hard disk drive (HDD) or solid-state drive (SSD); the memory may also include a combination of the above types of memory.

[0115] The processor 501 calls the program code in the memory 502 to perform the following steps:

[0116] Converting text data into at least one phoneme sequence; wherein the text data is provided with at least one breathing sound label;

[0117] Performing speech synthesis processing on the at least one phoneme sequence based on a pre-trained speech synthesis model to obtain a Mel spectrum corresponding to the text data;

[0118] The synthesized speech corresponding to the text data is obtained based on the Mel-spectrogram corresponding to the text data; wherein the synthesized speech includes the breathing sound corresponding to the at least one breathing sound label.

[0119] In one or more embodiments, the processor 501 is further configured to:

[0120] Get the original text data;

[0121] The original text data is annotated with a breathing sound label to obtain text data; wherein the text data is provided with at least one breathing sound label;

[0122] The text data is converted into the at least one phoneme sequence.

[0123] In one or more embodiments, the processor 501 is further configured to:

[0124] Determine the position information corresponding to the at least one breathing sound tag; wherein the position information is the position information of the at least one breathing sound tag in the original text data;

[0125] The text data is obtained by labeling the original text data with the at least one breathing sound label based on the position information.

[0126] In one or more embodiments, the processor 501 is further configured to:

[0127] Analyze the original text data to obtain sentence segmentation information;

[0128] The position information corresponding to the at least one breathing sound tag is determined based on the sentence segmentation information.

[0129] In one or more embodiments, the processor 501 is further configured to:

[0130] Obtaining sample text data and sample voice data;

[0131] Performing encoding and decoding processing on the sample text data to obtain a Mel spectrum corresponding to the sample text data;

[0132] The pre-trained speech synthesis model is generated when the loss value is less than or equal to a preset threshold; wherein the loss value is the loss value between the Mel spectrum corresponding to the sample text data and the Mel spectrum corresponding to the sample speech data.

[0133] In one or more embodiments, the processor 501 is further configured to:

[0134] Calculate the background noise data based on the preset signal-to-noise ratio;

[0135] The background noise data is added to the synthesized speech.

[0136] It should be noted that, when the device 5 provided in the above embodiment executes the speech synthesis method, only the division of the above functional modules is used as an example. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device is divided into different functional modules to complete all or part of the functions described above. In addition, the speech synthesis method embodiments provided in the above embodiment belong to the same concept, and the implementation process thereof is detailed in the method embodiment, which will not be repeated here.

[0137] The serial numbers of the above-mentioned embodiments of the present application are for description only and do not represent the advantages or disadvantages of the embodiments.

[0138] The present application also provides a computer storage medium, which can store multiple instructions, and the instructions are suitable for being loaded and executed by a processor as described above. Figure 2-Figure 3 The method steps of the embodiment shown in the figure can be found in the specific implementation process. Figure 2-Figure 3 The specific description of the illustrated embodiment will not be repeated here.

[0139] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented in one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that include computer-usable program code.

[0140] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0141] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1The steps for the functions specified in one or more boxes.

Claims

1. A speech synthesis method, characterized in that: The method comprises: Convert each word and each breathing sound label in the text data into at least one phoneme sequence; wherein the text data is provided with at least one breathing sound label; and the at least one breathing sound label includes duration information corresponding to the breathing sound; Performing speech synthesis processing on the at least one phoneme sequence based on a pre-trained speech synthesis model to obtain a Mel spectrum corresponding to the text data; wherein the Mel spectrum contains a breathing sound feature; Obtaining a synthesized speech corresponding to the text data based on the Mel-frequency spectrum corresponding to the text data; wherein the synthesized speech includes a breathing sound corresponding to the at least one breathing sound label; The training process of the pre-trained speech synthesis model includes: Acquire sample text data including at least one breathing sound label and sample voice data including at least one breathing sound; The sample text data is respectively subjected to encoding processing, duration extraction processing and decoding processing to obtain a Mel spectrum corresponding to the sample text data; wherein the duration extraction processing comprises: performing probability evaluation on the duration of at least one phoneme sequence in the sample text data to obtain the pronunciation duration of the at least one phoneme sequence; The pre-trained speech synthesis model is generated when the loss value is less than or equal to a preset threshold; wherein the loss value is the loss value between the Mel spectrum corresponding to the sample text data and the Mel spectrum corresponding to the sample speech data.

2. The method according to claim 1, characterized in that The step of converting text data into at least one phoneme sequence comprises: Get the original text data; The original text data is annotated with a breathing sound label to obtain text data; wherein the text data is provided with at least one breathing sound label; The text data is converted into the at least one phoneme sequence.

3. The method according to claim 2, characterized in that The step of labeling the original text data with a breathing sound label to obtain text data includes: Determine the position information corresponding to the at least one breathing sound tag; wherein the position information is the position information of the at least one breathing sound tag in the original text data; The text data is obtained by labeling the original text data with the at least one breathing sound label based on the position information.

4. The method according to claim 3, characterized in that The determining the position information corresponding to the at least one breathing sound tag includes: Analyze the original text data to obtain sentence segmentation information; The position information corresponding to the at least one breathing sound tag is determined based on the sentence segmentation information.

5. The method according to claim 1, characterized in that After obtaining the synthesized speech corresponding to the text data based on the Mel spectrum corresponding to the text data, the method further includes: Calculate the background noise data based on the preset signal-to-noise ratio; The background noise data is added to the synthesized speech.

6. A speech synthesis device, characterized in that: The device comprises: A conversion module, used to convert each word and each breathing sound label in the text data into at least one phoneme sequence; wherein the text data is provided with at least one breathing sound label; and the at least one breathing sound label includes duration information corresponding to the breathing sound; A first processing module, configured to perform speech synthesis processing on the at least one phoneme sequence based on a pre-trained speech synthesis model to obtain a Mel spectrum corresponding to the text data; wherein the Mel spectrum contains a breathing sound feature; A second processing module, configured to obtain a synthesized speech corresponding to the text data based on the Mel spectrum corresponding to the text data; wherein the synthesized speech includes a breathing sound corresponding to the at least one breathing sound label; A model training module, used to obtain sample text data containing at least one breathing sound label and sample voice data containing at least one breathing sound; Performing encoding and decoding processing on the sample text data to obtain a Mel spectrum corresponding to the sample text data; The pre-trained speech synthesis model is generated when the loss value is less than or equal to a preset threshold; wherein the loss value is the loss value between the Mel spectrum corresponding to the sample text data and the Mel spectrum corresponding to the sample speech data.

7. A computer storage medium, characterized in that: The computer storage medium stores a plurality of instructions, and the instructions are suitable for being loaded by a processor and executing the method steps as claimed in any one of claims 1 to 5.

8. An electronic device, characterized in that: include: A processor and a memory; wherein the memory stores a computer program, and the computer program is suitable for being loaded by the processor and executing the method steps as claimed in any one of claims 1 to 5.

Citation Information

Patent Citations

  • End-to-end speech synthesis method and device, and storage medium

    CN110797006A

  • Voice synthesis method and device, computer readable medium and electronic equipment

    CN111292719A

  • Speech synthesis model training method and device thereof, electronic equipment and storage medium

    CN111402857A

  • Realistic Speech Synthesis System

    US20130289998A1

  • Inserting breath sounds into text-to-speech output

    US9508338B1