A speech synthesis method, computer device and readable storage medium
By using silent phonemes as splicing points when switching between cloud and terminal synthesis services, the problems of discontinuity and prosodic inconsistency in speech synthesis are solved, achieving continuity and prosodic consistency in speech synthesis and improving user experience.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- IFLYTEK CO LTD
- Filing Date
- 2022-12-20
- Publication Date
- 2026-05-15
AI Technical Summary
Existing technologies suffer from issues such as repeated or missed character synthesis when switching between cloud and terminal synthesis services, leading to discontinuous speech synthesis and inconsistent prosody.
By receiving the text to be synthesized and the audio stream from the cloud, the interruption point phonemes are determined, and silence phonemes are inserted at the terminal as splicing points. The audio streams synthesized from the cloud and the terminal are then spliced together to ensure the continuity of audio playback and the consistency of rhythm.
It achieves continuity and prosodic consistency in speech synthesis even under network anomalies, thus improving the user experience.
Smart Images
Figure CN116030784B_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of speech synthesis technology, and in particular to a speech synthesis method, computer device, and computer storage medium. Background Technology
[0002] Speech synthesis is a signal processing technology that converts text into speech, giving machines the ability to speak like humans. In recent years, with the development and implementation of artificial intelligence technology, human-computer interaction has become increasingly frequent, and voice-based interaction has gradually become the mainstream form in the field of human-computer interaction, highlighting the growing importance of speech synthesis technology. From a deployment perspective, speech synthesis capabilities can be divided into cloud-based synthesis and local terminal synthesis. When synthesizing speech, cloud-based synthesis services are preferred, while local terminal synthesis is used in environments with offline or weak network connections.
[0003] Existing technologies, when using both cloud-based and terminal-based synthesis services simultaneously, typically select characters as anomalies for splicing. However, since characters are composed of multiple phonemes, this can lead to partial duplicate synthesis or partial omission of a character. Furthermore, when using terminal synthesis services, existing technologies usually continue synthesis based on text that has not yet been synthesized in the cloud, failing to guarantee the consistency of the prosodic boundaries of the text to be synthesized, thus affecting the user experience. Summary of the Invention
[0004] This application provides a speech synthesis method, a computer device, and a computer storage medium, which can improve the continuity of speech synthesis and achieve prosodic consistency.
[0005] In a first aspect, this application provides a speech synthesis method, the method comprising:
[0006] Receive the text to be synthesized from the cloud and the first audio stream and phoneme information corresponding to the text to be synthesized, wherein the text to be synthesized includes several characters and the phoneme information includes phonemes;
[0007] Audio is played based on the first audio stream, and when a reception error occurs, the currently playing phoneme and the interruption phoneme are determined.
[0008] The text to be synthesized is subjected to speech synthesis to obtain a second audio stream, and the target phoneme is determined, wherein the target phoneme is a silence phoneme and its position is between the currently playing phoneme and the interruption point phoneme;
[0009] Using the target phoneme as the splicing point, the first audio stream and the second audio stream are spliced together to obtain the target audio stream, and audio is played based on the target audio stream.
[0010] Secondly, this application provides a speech synthesis method, the method comprising:
[0011] A text to be synthesized, consisting of several characters, is obtained, and the text to be synthesized is processed by speech synthesis in the cloud according to the character order to obtain a first audio stream and phoneme information corresponding to each character, wherein the phoneme information includes phonemes;
[0012] Audio is played based on the first audio stream, and when a reception error occurs, the currently playing phoneme and the interruption phoneme are determined.
[0013] The text to be synthesized is sent to the terminal for speech synthesis to obtain a second audio stream, and the target phoneme is determined, wherein the target phoneme is a silence phoneme and its position is between the currently playing phoneme and the interruption point phoneme;
[0014] Using the target phoneme as the splicing point, the first audio stream and the second audio stream are spliced together to obtain the target audio stream, and audio is played based on the target audio stream.
[0015] Thirdly, this application also provides a computer device, the computer device comprising:
[0016] Memory and processor;
[0017] The memory is connected to the processor and is used to store programs;
[0018] The processor is used to implement the steps of the speech synthesis method as described in any of the embodiments of this application by running the program stored in the memory.
[0019] Fourthly, this application also provides a computer-readable storage medium storing a computer program that, when executed by a processor, causes the processor to implement the steps of the speech synthesis method as described in any of the embodiments of this application.
[0020] The speech synthesis method, computer device, and storage medium disclosed in this application can receive the text to be synthesized from the cloud and a first audio stream based on the text to be synthesized from the cloud. In the event of a reception error, the method can synthesize the text to be synthesized. Since the text to be synthesized input from the terminal and the cloud is consistent, the consistency of the speech synthesis prosody is guaranteed. Furthermore, splicing points can be determined based on the phoneme information of the text to be synthesized, and the audio stream synthesized from the cloud and the audio stream synthesized by the terminal can be segmented and spliced based on these splicing points, thereby ensuring the continuity of audio playback.
[0021] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and do not limit this application. Attached Figure Description
[0022] To more clearly illustrate the technical solutions of the embodiments of this application, the drawings used in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.
[0023] Figure 1 This is a schematic diagram illustrating the steps of a speech synthesis method provided in an embodiment of this application;
[0024] Figure 2 This is a schematic diagram illustrating the specific steps for determining target phonemes based on preset rules, provided in an embodiment of this application.
[0025] Figure 3 This is a schematic diagram illustrating the specific steps for obtaining the target audio stream provided in an embodiment of this application;
[0026] Figure 4 This is a schematic diagram illustrating the steps of another speech synthesis method provided in an embodiment of this application;
[0027] Figure 5 This is a schematic block diagram of a computer device provided in an embodiment of this application;
[0028] Figure 6 This is a schematic diagram of the structure of a computer-readable storage medium provided in this application.
[0029] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and do not limit this application. Detailed Implementation
[0030] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.
[0031] The flowchart shown in the attached diagram is for illustrative purposes only and does not necessarily include all content and operations / steps, nor does it necessarily have to be performed in the order described. For example, some operations / steps can be broken down, combined, or partially merged, so the actual execution order may change depending on the actual situation.
[0032] It should be understood that the terminology used in this specification is for the purpose of describing particular embodiments only and is not intended to limit the scope of the application. As used in this specification and the appended claims, the singular forms “a,” “an,” and “the” are intended to include the plural forms unless the context clearly indicates otherwise.
[0033] It should be understood that, in order to clearly describe the technical solutions of the embodiments of this application, the terms "first" and "second" are used in the embodiments of this application to distinguish identical or similar items with essentially the same function and effect. For example, the first identification model and the second identification model are only used to distinguish different callback functions and do not limit their order. Those skilled in the art will understand that the terms "first" and "second" do not limit the quantity or execution order, and the terms "first" and "second" do not necessarily mean they are different.
[0034] It should also be understood that the term "and / or" as used in this application specification and the appended claims means any combination of one or more of the associated listed items and all possible combinations, and includes such combinations.
[0035] The following detailed description of some embodiments of this application is provided in conjunction with the accompanying drawings. Unless otherwise specified, the following embodiments and features can be combined with each other.
[0036] To facilitate understanding of the embodiments of this application, some background information related to the embodiments of this application will be briefly explained below.
[0037] Currently, when synthesizing speech, cloud-based synthesis services are preferred. The cloud can synthesize the text to be synthesized into an audio stream, which can then be played back on the playback side. However, in interactive scenarios, the time interval between speech synthesis and playback is short, and the cloud-synthesized audio stream is continuously fed into the playback stream. Therefore, if there is a reception anomaly, there may be no audio stream to play, resulting in stuttering. To avoid this problem, the local terminal is usually used to continue synthesizing the incomplete text. However, when existing technologies splice the audio synthesized from the terminal with the remaining incomplete text to be synthesized onto the cloud-synthesized audio, there are often problems with unnatural rhythm and discontinuous audio playback.
[0038] For example, for the text to be synthesized "Hello, I am your assistant", the character "am" consists of two phonemes "sh" and "i". If the breakpoint is between the two phonemes "sh" and "i", according to the existing method, "Hello, I" uses cloud synthesis audio, and "your voice assistant" uses terminal synthesis audio. After splicing the cloud synthesis audio and the audio synthesized by the terminal, a part of the phoneme "i" will be missed in the synthesis, thus unable to ensure the continuity of audio playback. Based on this, it is necessary to propose a voice synthesis method to improve the continuity of voice synthesis and achieve the consistency of prosody.
[0039] Please refer to Figure 1 , Figure 1 which is a schematic flowchart of the steps of a voice synthesis method provided by an embodiment of the present application. This method can be applied to a computer device to implement voice synthesis for the text to be synthesized.
[0040] As Figure 1 shown, the voice synthesis method includes steps S11 to S14.
[0041] Step S11: Receive the text to be synthesized from the cloud, the first audio stream corresponding to the text to be synthesized, and phoneme information. The text to be synthesized includes several characters, and the phoneme information includes phonemes.
[0042] Among them, the text to be synthesized consists of several characters in a certain order. Further, the cloud can synthesize the text to be synthesized in sequence according to the character order. For example, the text to be synthesized is "Hello, I am your assistant.", when the cloud performs voice synthesis based on the text to be synthesized, it starts voice synthesis with the character "Hello" as the starting point and synthesizes in sequence according to the character order until it synthesizes to the character "assistant", at this time the cloud successfully synthesizes the text to be synthesized.
[0043] The first audio stream is the audio stream obtained after the cloud synthesis is completed. After the cloud performs voice synthesis on the text to be synthesized, the synthesis result is presented in the form of an audio stream, and then the playback side can implement audio playback based on the audio stream.
[0044] Further, the cloud can also parse the text to be synthesized to obtain several phonemes corresponding to each character. It should be noted that a phoneme is the smallest speech unit divided according to the natural attributes of speech, and it is analyzed based on the pronunciation actions in a syllable. One action constitutes one phoneme.
[0045] For example, for the text to be synthesized "Hello, I am your assistant.", the phonemes corresponding to each character in the synthesis order are as follows: "你 (nǐ)" includes the two phonemes "n" and "i"; "好 (hǎo)" includes the two phonemes "h" and "ao"; "我 (wǒ)" includes the phoneme "uo"; "是 (shì)" includes the two phonemes "sh" and "i"; "你 (nǐ)" includes the two phonemes "n" and "i"; "的 (de)" includes the two phonemes "d" and "e"; "助 (zhù)" includes the two phonemes "zh" and "u"; "理 (lǐ)" includes the two phonemes "l" and "i".
[0046] It can be understood that since the text to be synthesized is composed of characters in a certain order, the corresponding phonemes obtained based on the characters also have a certain order. For example, for the text to be synthesized "Hello, I am your assistant.", the starting phoneme is "n", and the ending phoneme is "i".
[0047] Therefore, after performing speech synthesis on the text to be synthesized in the cloud, the text to be synthesized in the cloud, the corresponding first audio stream, and phoneme information can be received.
[0048] In a specific implementation, in speech synthesis, based on S11, the text to be synthesized in the cloud, the corresponding first audio stream, and phoneme information are received. The embodiments of the present disclosure can perform the next step of processing through step S12, which is specifically described as follows:
[0049] Step S12: Play the audio based on the first audio stream, and when an abnormal reception occurs, determine the currently played phoneme and the breakpoint phoneme.
[0050] In an interactive scenario, there is a short time interval between speech synthesis and speech playback. Therefore, when an abnormal reception occurs, the currently played phoneme and the breakpoint phoneme can be determined. It can be understood that the breakpoint phoneme is the last phoneme abnormally synthesized by the cloud.
[0051] It should be noted that this application does not limit the situation of abnormal reception. For example, it can be caused by the interruption of cloud synthesis or network interruption.
[0052] Optionally, the audio stream includes audio data. Before determining the currently played phoneme when an abnormal reception occurs, it includes: judging whether new audio data is received based on a preset time; if no new audio data is received, it is determined that the reception is abnormal.
[0053] It should be noted that this application does not limit the preset time. For example, it can be 1s. That is, if no new audio data is received after 1s, it can be determined that the reception is abnormal.
[0054] Optionally, the audio stream further includes phoneme durations, that is, the pronunciation times of phonemes. When receiving an exception, determining the breakpoint phoneme includes: when receiving an exception, adding up the phoneme durations corresponding to multiple characters in character order to obtain the added-up phoneme duration; when the first audio stream duration is the same as the added-up phoneme duration, determining the phoneme that is finally added up as the breakpoint phoneme.
[0055] It can be understood that if the added-up phoneme duration is the same as the first audio stream duration, it means that the cloud synthesis reaches the phoneme that is finally added up. Based on this, the phoneme that is finally added up is determined as the breakpoint phoneme.
[0056] Optionally, when receiving an exception, the currently played phoneme can be directly determined based on the currently played audio.
[0057] In the embodiments of the present application, it can be determined whether an exception occurs in receiving based on whether new audio data is received within a preset time. Further, when receiving an exception, the breakpoint phoneme can be determined based on the phoneme duration and the currently played phoneme can be determined based on the currently played audio. By subdividing the text to be synthesized into the attribute of phonemes, compared with directly performing speech synthesis based on characters, the continuity of speech synthesis can be ensured.
[0058] Step S13: Perform speech synthesis on the text to be synthesized to obtain a second audio stream, and determine the target phoneme, where the target phoneme is a silent phoneme and its position is between the currently played phoneme and the breakpoint phoneme.
[0059] Among them, the second audio stream is an audio stream synthesized by the terminal based on the text to be synthesized; the silent phoneme is a phoneme that does not make a sound in the text to be synthesized. For example, for the character "he", its pronunciation phonemes are "h" and "e", and there are also non-pronouncing phonemes for expressing tone and pause after the pronunciation phonemes, that is, the silent phonemes. Further, the silent phoneme includes a certain silent duration range. Therefore, the splicing of the audio streams of the terminal and the cloud based on the silent phoneme will not cause problems of missing reading or over-reading, thus ensuring the integrity of audio synthesis.
[0060] Further, determining the position of the target phoneme between the currently played phoneme and the breakpoint phoneme, at this time the cloud can synthesize to the position of the target phoneme, and since the playback side has not played to the position of the target phoneme, the second audio stream synthesized by the terminal can be spliced with the cloud based on the target phoneme, realizing the continuity of audio synthesis.
[0061] Optionally, if the interval between the currently played phoneme and the breakpoint phoneme is short, resulting in no silent phoneme between them, the breakpoint phoneme can be directly determined as the target phoneme, and then the audio stream can be spliced based on the target phoneme.
[0062] Please refer to Figure 2 , Figure 2 This is a schematic diagram illustrating the specific steps for determining the target phoneme provided in an embodiment of this application, as shown below. Figure 2 As shown, the target phoneme can be determined through steps S131 to S134.
[0063] Step S131: Determine the initial phoneme based on the currently played phoneme, wherein the position of the initial phoneme is after the currently played phoneme.
[0064] To ensure audio continuity, the terminal can synthesize the target phoneme when it is played on the playback side. An initial phoneme can be determined based on the currently playing phoneme. This initial phoneme is the one synthesized by the terminal in real-time that perfectly catches up with the currently playing phoneme. In other words, the terminal can synthesize any phoneme after the initial phoneme before it is played on the playback side.
[0065] Optionally, determining the initial phoneme based on the currently played phoneme includes: obtaining the duration of phonemes following the currently played phoneme; obtaining the real-time rate and first-sound time of the audio synthesis; constructing an objective function based on the duration of the subsequent phonemes, the real-time rate, and the first-sound time; and obtaining the variable values when the objective function is minimized to obtain the initial phoneme.
[0066] Specifically, the initial phoneme can be determined based on the currently playing phoneme using the following formula:
[0067]
[0068] Where x is the phoneme following the currently played phoneme, y is the initial phoneme, Di is the duration corresponding to the i-th phoneme; r is the real-time rate of the terminal synthesis; and fs is the first sound time.
[0069] Step S132: Predict the prosodic boundaries of the text to be synthesized to obtain the prosodic boundary prediction results, wherein the prosodic boundary prediction results include several silent phonemes and the duration corresponding to each silent phoneme.
[0070] Step S133: Prioritize several silence phonemes based on the duration corresponding to each silence phoneme.
[0071] Step S134: Starting from the initial phoneme, select the highest priority silent phoneme as the target phoneme according to the character order.
[0072] It should be noted that this application does not limit the method for predicting prosodic boundaries. For example, the prosodic boundaries of the text to be synthesized can be predicted using a BILSTM-CRF prediction model, a statistical model, etc. Based on the predicted prosodic boundaries, several silent phonemes of the text to be synthesized and the duration of each silent phoneme can be obtained.
[0073] Furthermore, based on the duration information corresponding to each silent phoneme, several silent phonemes can be prioritized, and starting from the initial phoneme, the silent phoneme with the highest priority can be selected as the target phoneme according to the character order.
[0074] Understandably, the longer the duration of a silence phoneme, the better it is for splicing the audio stream synthesized at the terminal with the audio stream synthesized in the cloud. Therefore, several silence phonemes can be prioritized based on the duration information corresponding to each silence phoneme. For example, silence phonemes with durations in the ranges of 1s-2s, 0.5s-1s, and 0.1s-0.5s can be prioritized from high to low.
[0075] Furthermore, starting with the initial phoneme, the target phoneme can be selected by choosing the first silent phoneme whose duration falls within the 1-2s time interval according to the character order. If no silent phoneme falls within the 1-2s time interval, other time intervals are selected according to priority, and the above steps are repeated until the target phoneme is determined.
[0076] Step S14: Using the target phoneme as the splicing point, splice the first audio stream and the second audio stream to obtain the target audio stream, and play the audio based on the target audio stream.
[0077] After determining the target phoneme, the first audio stream and the second audio stream can be spliced together using the target phoneme as the splicing point to obtain the target audio stream, and audio playback can be performed based on the target audio stream.
[0078] Understandably, since the target phoneme is a silent phoneme and its location is between the initial phoneme and the breakpoint phoneme, the splicing of the first and second audio streams based on the target phoneme ensures the continuity of audio playback and avoids issues of missed or over-reading.
[0079] Optionally, please refer to Figure 3 , Figure 3 This is a schematic diagram illustrating the specific steps involved in obtaining the target audio stream, as provided in an embodiment of this application. Figure 3 As shown, the target audio stream can be obtained through steps S141 and S143.
[0080] Step S141: Using the target phoneme as the endpoint, select a third audio stream based on the first audio stream.
[0081] Step S142: Starting from the target phoneme, select the fourth audio stream based on the second audio stream.
[0082] Step S143: Concatenate the third audio stream with the fourth audio stream to obtain the target audio stream.
[0083] The third audio stream is an audio stream extracted from the target phonemes by the cloud; the fourth audio stream is an audio stream extracted from the target phonemes by the terminal.
[0084] Specifically, a third audio stream can be selected from the first audio stream synthesized in the cloud, with the target phoneme as the endpoint, and a fourth audio stream can be selected from the second audio stream synthesized at the terminal, with the target phoneme as the starting point. Furthermore, the third and fourth audio streams are concatenated to obtain the target audio stream, and audio playback can be performed based on the target audio stream.
[0085] Optionally, the fourth audio stream consists of several fourth audio sampling points. Starting from the target phoneme, after selecting the fourth audio stream based on the second audio stream, the process includes: determining the maximum value of the third audio energy of the third audio stream and the maximum value of the fourth audio energy of the fourth audio stream; adjusting each fourth audio sampling point based on the maximum value of the third audio energy and the maximum value of the fourth audio energy to obtain several fifth audio sampling points; obtaining the fifth audio stream based on the several fifth audio sampling points; and concatenating the third audio stream and the fifth audio stream to obtain the final audio stream.
[0086] Among them, the fourth audio sampling point is the sampling point that constitutes the fourth audio stream; the maximum value of the third audio energy value is the maximum value of the audio energy corresponding to the third audio stream; the maximum value of the fourth audio energy value is the maximum value of the audio energy corresponding to the fourth audio stream; the fifth audio sampling point is the sampling point obtained after adjusting the fourth audio sampling point; and the fifth audio stream is an audio stream composed of several fifth audio sampling points.
[0087] Since the target audio stream is spliced from the terminal and the cloud, inconsistencies in audio energy may occur, leading to an unnatural listening experience. To address this issue, each fourth audio sampling point can be adjusted based on the maximum values of the third and fourth audio energy values to obtain several fifth audio sampling points. A fifth audio stream is then obtained based on these fifth audio sampling points and spliced with the first audio stream to obtain the final audio stream.
[0088] Optionally, each fourth audio sampling point is adjusted based on the maximum value of the third audio energy value and the maximum value of the fourth audio energy value to obtain several fifth audio sampling points, including: determining the ratio of the maximum value of the third audio energy value to the maximum value of the fourth audio energy value; multiplying the ratio by each fourth audio sampling point to obtain several fifth audio sampling points.
[0089] Specifically, the following formula can be used to adjust each fourth audio sampling point based on the maximum value of the third and fourth audio energy values to obtain several fifth audio sampling points:
[0090]
[0091] Wherein, V1 is the fifth audio sampling point, V'max is the third maximum audio energy value, and V''max is the fourth maximum audio energy value.
[0092] It should be noted that the method for determining the maximum audio energy value in this application embodiment can be calculated using any method in the prior art, and no specific limitation is made here. For example, the audio energy can be obtained by confirming the signal waveform and taking the logarithm of the corresponding peak value. Furthermore, the maximum audio energy value can be determined based on the peak value of the audio.
[0093] In this embodiment, a target audio stream is obtained by splicing the first audio stream and the second audio stream based on the target phoneme as the splicing point, and then played back based on the target audio stream. Since the target phoneme is a silent phoneme located between the initial phoneme and the interruption point phoneme, the splicing between the first and second audio streams based on the target phoneme ensures the continuity of audio playback and avoids issues of missed or over-reading. Furthermore, to prevent inconsistent audio energy that could lead to an unnatural listening experience, each fourth audio sampling point can be adjusted based on the maximum values of the third and fourth audio energy values to obtain a fifth audio sampling point and a fifth audio stream, thus resulting in the final audio stream.
[0094] Please see Figure 4 , Figure 4 This is a schematic diagram illustrating the steps of another speech synthesis method provided in an embodiment of this application. For example... Figure 4 As shown, the speech synthesis method includes steps S21 to S24.
[0095] Step S21: Obtain the text to be synthesized, which consists of several characters, and perform speech synthesis on the text to be synthesized in the order of characters through the cloud to obtain the first audio stream and the phoneme information corresponding to each character, wherein the phoneme information includes phonemes.
[0096] Step S22: Play audio based on the first audio stream, and when a reception error occurs, determine the currently playing phoneme and the interruption phoneme.
[0097] Step S23: Send the text to be synthesized to the terminal for speech synthesis to obtain the second audio stream, and determine the target phoneme, wherein the target phoneme is the silence phoneme and its position is between the currently playing phoneme and the interruption point phoneme.
[0098] Step S24: Using the target phoneme as the splicing point, splice the first audio stream and the second audio stream to obtain the target audio stream, and play the audio based on the target audio stream.
[0099] The first audio stream is an audio stream synthesized from the text to be synthesized by the cloud; the second audio stream is an audio stream synthesized from the text to be synthesized by the terminal.
[0100] In this embodiment, the text to be synthesized can be speech-synthesized via the cloud to obtain a first audio stream and phoneme information corresponding to the text. Furthermore, in the event of a reception error, the currently playing phoneme and the interruption point phoneme can be determined. The text to be synthesized is then sent to the terminal for speech synthesis to obtain a second audio stream, and a target phoneme is determined based on preset rules. After determining the target phoneme, the first audio stream and the second audio stream can be concatenated using the target phoneme as the splicing point to obtain a target audio stream, and audio playback is performed based on the target audio stream.
[0101] It should be noted that this application does not impose restrictions on preset rules. Please refer to steps S131 to S134 for details. To avoid repetition, these will not be elaborated here.
[0102] Understandably, since the target phoneme is a silent phoneme and its location is between the initial phoneme and the breakpoint phoneme, the splicing of the first and second audio streams based on the target phoneme ensures the continuity of audio playback and avoids issues of missed or over-reading.
[0103] The speech synthesis method, computer device, and storage medium disclosed in this application can perform speech synthesis on the text to be synthesized via the cloud, and send the text to be synthesized to the terminal for speech synthesis in case of reception abnormalities. Since the text to be synthesized input by the terminal and the cloud is consistent, the consistency of the speech synthesis prosody is guaranteed. In addition, splicing points can be determined based on the phoneme information of the text to be synthesized, and the audio stream synthesized in the cloud and the audio stream synthesized by the terminal can be segmented and spliced based on the splicing points, thereby ensuring the continuity of audio playback.
[0104] The methods and apparatus of this application can be used in a wide variety of general-purpose or special-purpose computing system environments or configurations. For example: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer terminal devices, network PCs, minicomputers, mainframe computers, distributed computing environments including any of the above systems or devices, etc.
[0105] For example, the above-described method and apparatus can be implemented as a computer program, which can be used in, for example... Figure 5 It runs on the computer device shown.
[0106] Please see Figure 5 , Figure 5This is a schematic diagram of a computer device provided in an embodiment of this application. The computer device 300 may be a server. Figure 5 As shown, the computer device 300 includes a processor 301, a memory 302, and a network interface connected via a system bus. The memory 302 may include volatile storage media, non-volatile storage media, and internal memory. The non-volatile storage media may store an operating system and computer programs. The computer programs include program instructions that, when executed, cause the processor 301 to perform any speech synthesis method.
[0107] The processor 301 provides computing and control capabilities to support the operation of the entire computer device 300.
[0108] The internal memory provides an environment for the execution of computer programs in non-volatile storage media. When the computer program is executed by the processor 301, the processor 301 can execute any speech synthesis method.
[0109] This network interface is used for network communication, such as sending assigned tasks. Those skilled in the art will understand that the structure of this computer device 300 is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device 300 to which the present application applies. A specific computer device 300 may include more or fewer components than shown in the figure, or combine certain components, or have different component arrangements.
[0110] It should be understood that processor 301 can be a Central Processing Unit (CPU), but it can also be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. Among these, a general-purpose processor can be a microprocessor or any conventional processor.
[0111] In some embodiments, the processor 301 is used to run a computer program stored in the memory 302 to perform the following steps: receiving text to be synthesized from the cloud and a first audio stream and phoneme information corresponding to the text to be synthesized, wherein the text to be synthesized includes several characters and the phoneme information includes phonemes; playing audio based on the first audio stream, and determining the currently playing phoneme and the interruption point phoneme when receiving an abnormality; performing speech synthesis on the text to be synthesized to obtain a second audio stream, and determining a target phoneme, wherein the target phoneme is a silence phoneme and is located between the currently playing phoneme and the interruption point phoneme; using the target phoneme as a splicing point, splicing the first audio stream and the second audio stream to obtain a target audio stream, and playing audio based on the target audio stream.
[0112] In some embodiments, the processor 301 is further configured to determine, based on a preset time, whether new audio data has been received; if no new audio data has been received, then a reception anomaly is determined.
[0113] In some embodiments, the processor 301 is further configured to, when the reception is abnormal, sum the phoneme durations corresponding to multiple characters in character order to obtain a summed phoneme duration; and when the duration of the first audio stream is the same as the summed phoneme duration, determine the last phoneme to be summed as the interruption point phoneme.
[0114] In some embodiments, the processor 301 is further configured to: determine an initial phoneme based on the currently played phoneme, wherein the position of the initial phoneme is after the currently played phoneme; predict the prosodic boundary of the text to be synthesized to obtain a prosodic boundary prediction result, wherein the prosodic boundary prediction result includes a plurality of silent phonemes and the duration corresponding to each silent phoneme; prioritize the plurality of silent phonemes based on the duration information corresponding to each silent phoneme; and select the silent phoneme with the highest priority as the target phoneme according to the character order, starting from the initial phoneme.
[0115] In some embodiments, the processor 301 is further configured to obtain the duration of the phoneme following the currently played phoneme; obtain the real-time rate and first-sound time of the audio synthesis; construct an objective function based on the duration of the following phoneme, the real-time rate, and the first-sound time; and obtain the initial phoneme by finding the variable value when the objective function is minimized.
[0116] In some embodiments, the processor 301 is further configured to select a third audio stream based on the first audio stream, with the target phoneme as the endpoint; select a fourth audio stream based on the second audio stream, with the target phoneme as the starting point; and concatenate the third audio stream and the fourth audio stream to obtain the target audio stream.
[0117] In some embodiments, the processor 301 is further configured to determine the maximum value of the third audio energy value of the third audio stream and the maximum value of the fourth audio energy value of the fourth audio stream; adjust each of the fourth audio sampling points based on the maximum value of the third audio energy value and the maximum value of the fourth audio energy value to obtain a plurality of fifth audio sampling points; obtain a fifth audio stream based on the plurality of fifth audio sampling points; and concatenate the third audio stream and the fifth audio stream to obtain a final audio stream.
[0118] In some embodiments, the processor 301 is further configured to determine the ratio of the maximum value of the third audio energy value to the maximum value of the fourth audio energy value; and multiply the ratio by each of the fourth audio sampling points to obtain a plurality of the fifth audio sampling points.
[0119] In some embodiments, the processor 301 is further configured to acquire a text to be synthesized consisting of several characters, and to perform speech synthesis on the text to be synthesized in the order of characters via the cloud to obtain a first audio stream and phoneme information corresponding to each character, wherein the phoneme information includes phonemes; to play audio based on the first audio stream, and to determine the currently playing phoneme and the interruption point phoneme when a reception error occurs; to send the text to be synthesized to the terminal for speech synthesis to obtain a second audio stream, and to determine a target phoneme based on a preset rule, wherein the target phoneme is a silence phoneme and is located between the currently playing phoneme and the interruption point phoneme; to concatenate the first audio stream and the second audio stream using the target phoneme as a splicing point to obtain a target audio stream, and to play audio based on the target audio stream.
[0120] This application also provides a computer-readable storage medium storing a computer program, the computer program including program instructions, which, when executed, implement the speech synthesis method provided in this application.
[0121] The computer-readable storage medium may be the internal storage unit of the computer device 300 described in the foregoing embodiments, such as the hard disk or memory of the computer device.
[0122] Please see Figure 6 , Figure 6This is a schematic diagram of a computer-readable storage medium provided for this application. The storage medium 40 of this application stores a computer program capable of implementing all the aforementioned speech synthesis methods. This computer program can be stored in the storage medium 40 in the form of a software product, including several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) or processor to execute all or part of the steps of the methods of various embodiments of this application. The aforementioned storage devices include various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks, or devices such as computers, servers, mobile phones, and tablets.
[0123] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any person skilled in the art can easily conceive of various equivalent modifications or substitutions within the technical scope disclosed in this application, and these modifications or substitutions should all be covered within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.
Claims
1. A speech synthesis method, characterized in that, The method includes: Receive the text to be synthesized from the cloud and the first audio stream and phoneme information corresponding to the text to be synthesized, wherein the text to be synthesized includes several characters and the phoneme information includes phonemes; Audio is played based on the first audio stream, and when a reception error occurs, the currently playing phoneme and the interruption phoneme are determined. The text to be synthesized is subjected to speech synthesis to obtain a second audio stream, and the target phoneme is determined, wherein the target phoneme is a silence phoneme and its position is between the currently playing phoneme and the interruption point phoneme; Using the target phoneme as the splicing point, the first audio stream and the second audio stream are spliced together to obtain the target audio stream, and audio is played based on the target audio stream.
2. The method according to claim 1, characterized in that, The audio stream includes audio data, and in the event of a reception error, before determining the currently playing phoneme and the interrupted phoneme, it includes: Based on a preset time, determine whether new audio data has been received; If no new audio data is received, a reception error is identified.
3. The method according to claim 1, characterized in that, The phoneme information also includes phoneme duration, and the step of determining the interruption point phoneme when a reception anomaly occurs includes: When a reception error occurs, the phoneme durations corresponding to multiple characters are summed in character order to obtain the summed phoneme durations. When the duration of the first audio stream is the same as that of the summed phonemes, the last phoneme to be summed is determined as the breakpoint phoneme.
4. The method according to claim 1, characterized in that, The determination of the target phoneme includes: An initial phoneme is determined based on the currently played phoneme, wherein the position of the initial phoneme is after the currently played phoneme; The prosodic boundaries of the text to be synthesized are predicted to obtain prosodic boundary prediction results, wherein the prosodic boundary prediction results include several silent phonemes and the duration corresponding to each silent phoneme. The silence phonemes are prioritized based on the duration of each silence phoneme. Starting from the initial phoneme, the highest priority silent phoneme is selected as the target phoneme according to the character order.
5. The method according to claim 4, characterized in that, The process of determining the initial phoneme based on the currently played phoneme includes: Get the duration of the phoneme following the currently playing phoneme; Obtain the real-time rate and first-sound time of the audio synthesis; A target function is constructed based on the duration of the subsequent phonemes, the real-time rate, and the first sound time. The initial phonemes are obtained by finding the minimum value of the variable when the objective function is minimized.
6. The method according to claim 1, characterized in that, The step of splicing the first audio stream and the second audio stream together using the target phoneme as the splicing point to obtain the target audio stream includes: Using the target phoneme as the endpoint, a third audio stream is selected based on the first audio stream; Starting from the target phoneme, a fourth audio stream is selected based on the second audio stream; The target audio stream is obtained by concatenating the third audio stream with the fourth audio stream.
7. The method according to claim 6, characterized in that, The fourth audio stream is composed of several fourth audio sampling points. The step of selecting the fourth audio stream based on the second audio stream, starting from the target phoneme, includes: Determine the maximum value of the third audio energy of the third audio stream and the maximum value of the fourth audio energy of the fourth audio stream; Based on the maximum value of the third audio energy value and the maximum value of the fourth audio energy value, each of the fourth audio sampling points is adjusted to obtain a number of fifth audio sampling points; A fifth audio stream is obtained based on several of the aforementioned fifth audio sampling points; The third audio stream and the fifth audio stream are concatenated to obtain the final audio stream.
8. The method according to claim 7, characterized in that, The adjustment of each fourth audio sampling point based on the maximum value of the third audio energy value and the maximum value of the fourth audio energy value yields several fifth audio sampling points, including: Determine the ratio between the maximum value of the third audio energy value and the maximum value of the fourth audio energy value; Multiplying the ratio by each of the fourth audio sampling points yields a plurality of the fifth audio sampling points.
9. A speech synthesis method, characterized in that, The method further includes: A text to be synthesized, consisting of several characters, is obtained, and the text to be synthesized is processed by speech synthesis in the cloud according to the character order to obtain a first audio stream and phoneme information corresponding to each character, wherein the phoneme information includes phonemes; Audio is played based on the first audio stream, and when a reception error occurs, the currently playing phoneme and the interruption phoneme are determined. The text to be synthesized is sent to the terminal for speech synthesis to obtain a second audio stream, and the target phoneme is determined, wherein the target phoneme is a silence phoneme and its position is between the currently playing phoneme and the interruption point phoneme; Using the target phoneme as the splicing point, the first audio stream and the second audio stream are spliced together to obtain the target audio stream, and audio is played based on the target audio stream.
10. A computer device, characterized in that, The computer device includes: Memory and processor; The memory is connected to the processor and is used to store programs; The processor is used to implement the steps of the speech synthesis method as described in any one of claims 1-9 by running the program stored in the memory.
11. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, causes the processor to implement the steps of the speech synthesis method as described in any one of claims 1-9.