Speech synthesis method and device, electronic equipment and storage medium

Abstract

The invention provides a speech synthesis method and device, electronic equipment and a storage medium, and belongs to the technical field of speech synthesis, and the method comprises the steps: determining a first standard audio based on a first text; determining first data based on the first text and the first instruction information, and determining a first text identifier based on the first data; the first data comprises context information or voice information of the first text; and determining a target voice based on the first text identifier and the first standard audio. According to the speech synthesis method and device, the electronic equipment and the storage medium provided by the invention, the quality of synthesized speech can be improved.

Description

Technical Field

[0001] The present disclosure belongs to the technical field of speech synthesis, and more specifically, relates to a speech synthesis method and device, electronic equipment, and storage medium. Background Art

[0002] With the rapid development of artificial intelligence and deep learning technologies, text-to-speech (TTS) technology has been widely used in multiple fields such as intelligent assistants, automatic customer service, and in-vehicle systems, greatly improving the interactive capabilities and user experience of these systems.

[0003] However, despite significant progress, existing speech synthesis technologies still have many shortcomings. For example, these technologies are still computationally complex, which makes it difficult to achieve efficient real-time processing in practical applications. In addition, the quality of synthesized speech is often unsatisfactory and cannot reach the level of natural fluency and emotion, which greatly limits its application in a wider range of scenarios. Therefore, how to further improve the performance of speech synthesis technology, reduce computational complexity, and improve the quality of synthesized speech has become an important topic in current research and development. Summary of the invention

[0004] The purpose of the present disclosure is to provide a speech synthesis method and device, an electronic device, and a storage medium to improve the quality of synthesized speech.

[0005] According to a first aspect of the present disclosure, a speech synthesis method is provided, comprising: A first standard audio is determined based on the first text.

[0006] The first data is determined based on the first text and the first instruction information, and the first text identifier is determined based on the first data. The first data includes context information or voice information of the first text.

[0007] A target voice is determined based on the first text identifier and the first standard audio.

[0008] According to a second aspect of the embodiments of the present disclosure, there is provided a speech synthesis device, including: The first calculation module is used to determine a first standard audio based on the first text.

[0009] The second calculation module is used to determine the first data based on the first text and the first instruction information, and determine the first text identifier based on the first data. The first data includes context information or voice information of the first text.

[0010] The speech synthesis module is used to determine the target speech based on the first text identifier and the first standard audio.

[0011] According to a third aspect of an embodiment of the present disclosure, an electronic device is provided, including a memory, a processor, and a computer program stored in the memory and running on the processor, wherein the processor implements the steps of the above-mentioned speech synthesis method when executing the computer program.

[0012] According to a fourth aspect of the embodiments of the present disclosure, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the above-mentioned speech synthesis method are implemented.

[0013] The beneficial effects of a speech synthesis method and device, electronic device, and storage medium provided by the embodiments of the present disclosure are: on the one hand, the embodiments of the present disclosure ensure the baseline quality of speech synthesis by determining the first standard audio based on the first text, so that the generated speech is more in line with the standards in terms of intonation, speaking speed, etc., thereby improving the accuracy and efficiency of speech synthesis.

[0014] On the other hand, the disclosed embodiment combines the first text and the first instruction information to determine the first data, and generates the first text identifier accordingly. This step makes full use of the context information of the text and possible voice requirements, so that the speech synthesis can better understand and express the complete meaning of the text, and enhance the semantic accuracy and expressiveness of the speech.

[0015] On the other hand, by combining the first text identifier and the first standard audio to determine the target voice, the method achieves refined control over voice synthesis, which not only improves the flexibility of voice synthesis, but also enables the generated voice to better adapt to different application scenarios and requirements, providing strong support for the application and development of voice technology.

[0016] In summary, the embodiments of the present disclosure have shown significant beneficial effects in terms of accuracy, efficiency, naturalness, and flexibility of speech synthesis. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] In order to more clearly illustrate the technical solutions in the embodiments of the present disclosure, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present disclosure. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0018] Figure 1 A flowchart of a speech synthesis method provided by an embodiment of the present disclosure; Figure 2 A structural block diagram of a speech synthesis device provided by an embodiment of the present disclosure; Figure 3 A schematic block diagram of an electronic device provided by an embodiment of the present disclosure. DETAILED DESCRIPTION

[0019] In the following description, specific details such as specific system structures and technologies are provided for the purpose of illustration rather than limitation, so as to provide a thorough understanding of the embodiments of the present disclosure. However, it should be clear to those skilled in the art that the present disclosure may be implemented in other embodiments without these specific details. In other cases, detailed descriptions of well-known systems, devices, circuits, and methods are omitted to avoid obstructing the description of the present disclosure with unnecessary details.

[0020] In order to make the purpose, technical solutions and advantages of the present disclosure more clear, specific embodiments will be described below in conjunction with the accompanying drawings.

[0021] Please refer to Figure 1 , Figure 1 A flowchart of a speech synthesis method provided in one embodiment of the present disclosure, the method may include S101 to S103.

[0022] S101: Determine a first standard audio based on a first text.

[0023] In this embodiment, the first text may include the original text content of the speech to be synthesized. The first standard audio may be understood as a reference audio preliminarily determined based on the first text. The first standard audio may include audio selected from an existing audio library that matches the first text in terms of content, style, etc. Alternatively, the first standard audio may also be an audio template, which serves as a starting point for further synthesizing the target speech. For the first text, parameters such as text content, grammatical structure, and semantic information may be included. For example, specific vocabulary, sentence length, paragraph structure, topic classification, etc.

[0024] The parameters of the first standard audio may include basic audio parameters (such as sampling rate, frequency range, duration, volume, etc.), parameters related to speech features (such as fundamental frequency, resonance peak, timbre characteristics, etc.), and parameters related to prosody (such as intonation, stress, rhythm, etc.).

[0025] Exemplarily, the first text is analyzed to extract key feature information, such as text content, grammatical structure, theme, emotional tendency, etc. According to the key feature information, a search and match is performed in the audio library to select the audio corresponding to the first text as the first standard audio, or an audio corresponding to the first text is generated as the first standard audio through a specific algorithm.

[0026] S102: Determine first data based on the first text and the first instruction information, and determine a first text identifier based on the first data. The first data includes context information or voice information of the first text.

[0027] In this embodiment, the first instruction information is used to indicate the speech synthesis type, and the first instruction information may include a question-answering instruction or a reading instruction; different instructions correspond to different speech synthesis types. Different speech synthesis types correspond to different types of input data.

[0028] The first data is the input source data, and the target speech can be synthesized based on the source data. The context information in the first data may include parameters such as the context of the first text and its semantic relationship, theme consistency, etc.; the speech information may include the speech information and characteristic parameters emitted by the user during the question-and-answer process, such as pitch range, speech speed parameters, emotional intensity, etc.

[0029] The first text identifier can be understood as a classification or label of the first text, which is used to indicate the method and style of speech synthesis. The first text identifier can include different type parameters, such as news, story, technology, conference, teaching, etc., and corresponding style parameters, such as formal, lively, etc.

[0030] For example, in the intelligent education software, when students select a reading instruction, the context information, such as the theme of the article and the logical relationship between the preceding and following texts, is extracted based on the first text that is identified as the first data. Based on these data, the text is identified as a popular science article, and then a voice with a clear and smooth tone is synthesized for reading. If it is a question-and-answer instruction, the voice information sent by the user is first extracted for recognition and analysis, and then the reply text (i.e., the first text) of the generated voice to be synthesized is identified, and the two are combined to synthesize a voice answer with appropriate emotional emphasis.

[0031] S103: Determine a target voice based on the first text identifier and the first standard audio.

[0032] In this embodiment, the specific style, emotional tendency, speech speed and other characteristic requirements of the target speech can be determined according to the first text mark. The first standard audio is used as the initial audio template based on the first standard audio. By analyzing the difference between the characteristics represented by the first text mark and the first standard audio, the first standard audio is adjusted and optimized so that it gradually approaches the requirements of the target speech in terms of timbre, rhythm, emotional expression, etc. Finally, the target speech that meets the specific requirements is obtained.

[0033] Exemplarily, audio processing algorithms and machine learning techniques may be used. According to the characteristic parameters determined by the first text identifier, the audio parameters of the first standard audio are adjusted, such as changing the pitch, adjusting the speech speed, increasing or decreasing the volume of a specific frequency, etc. The deep learning model is used to learn the relationship between different text identifiers and audio features, and the first standard audio is automatically optimized to meet the characteristics of the target speech.

[0034] For example, in an audio novel application, the first text identifier indicates that a certain chapter is a tense plot paragraph. Based on a common first standard audio, according to the first text identifier, the audio speed is accelerated, the pitch is slightly higher, and the tense sound effect background is added, so that the synthesized target voice can better convey the tense atmosphere and make the audience feel more involved. In the intelligent customer service system, the first text identifier is determined according to different question types, the first standard audio is adjusted, and answer voices with different styles are synthesized to improve the user experience.

[0035] From the above, it can be concluded that, on the one hand, this embodiment provides a high-quality starting point for subsequent speech synthesis by accurately matching or generating the first standard audio that matches the content and style of the first text, thereby ensuring the accuracy and naturalness of the synthesized speech. This can not only improve the user experience, but also achieve more realistic voice interaction in a variety of application scenarios.

[0036] On the other hand, this embodiment combines the first text and the first instruction information to determine the first data, and generates the first text identifier accordingly. This step enables the speech synthesis to fully understand and express the complete meaning and context of the text. This helps to enhance the semantic accuracy and expressiveness of the speech, making the synthesized speech more in line with the needs of actual application scenarios.

[0037] On the other hand, this embodiment adjusts and optimizes the first standard audio to meet the feature requirements represented by the first text identifier, thereby achieving refined control of speech synthesis. This control method not only improves the flexibility and adaptability of speech synthesis, but also enables the generated speech to better convey specific emotions, styles and atmospheres, further enhancing the practical value and user experience of speech synthesis.

[0038] In summary, the embodiments of the present disclosure have shown significant beneficial effects in terms of accuracy, efficiency, naturalness, and flexibility of speech synthesis.

[0039] In one embodiment of the present disclosure, determining a first standard audio based on a first text includes: A plurality of text blocks are determined based on the first text, and index sequences corresponding to the plurality of text blocks are generated.

[0040] One or more standard audio blocks corresponding to each text block are determined from the standard speech data set based on the index sequence. The standard speech data set includes multiple standard audio blocks.

[0041] A first standard audio is determined based on one or more standard audio blocks corresponding to each text block.

[0042] In this embodiment, the text block is a smaller text unit obtained by dividing the first text according to certain rules, and the relevant parameters of the text block may include text content, length, position in the first text, etc.

[0043] An index sequence is a sequence used to identify text blocks. An index sequence may contain parameters such as digital numbers and specific codes for quick positioning and matching.

[0044] The standard speech dataset is a pre-collected and organized collection of a large number of standard audio blocks, which can contain audio with different text content, style, speaking speed, and emotion.

[0045] A standard audio chunk is an audio segment corresponding to a text chunk in a standard speech dataset.

[0046] Exemplarily, in the intelligent voice assistant scenario, the first text is the user's problem description. Divide it into text blocks based on words. For example, "How is the weather today" can be divided into text blocks such as "today", "weather", and "how is it", and generate a corresponding index sequence. From the standard voice data set, find the standard audio block corresponding to each text block according to the index sequence. For example, "weather" may correspond to multiple audio blocks with different pronunciation styles. Finally, combine the audio blocks corresponding to each text block to determine the first standard audio.

[0047] In this embodiment, if the corresponding audio block is missing in the standard speech data set, determining the first standard audio based on the first text may also include: determining multiple text blocks based on the first text, generating low-resolution audio features corresponding to each text block using a hierarchical generation model, and determining an initial audio block based on the low-resolution audio features. Determine the first standard audio based on one or more initial audio blocks corresponding to each text block.

[0048] Exemplarily, for each text block, its text features are used as input, and low-resolution audio features are generated through the bottom layer of the hierarchical generative model. These low-resolution audio features can be some basic audio contours, such as the initial pitch range, rhythm pattern, etc. In the subsequent stage, the hierarchical structure of the generative adversarial network can be used to take context information, instruction information, etc. as additional inputs, and these low-resolution features can be gradually refined through the generator. The discriminator is used to determine whether the generated high-resolution audio features are real and reasonable, and the generation process is continuously optimized through adversarial training, and finally a high-resolution audio block for determining the target speech is obtained.

[0049] In this embodiment, when determining the standard audio block corresponding to each text block from the standard speech data set, the new audio representation technology can be used to more efficiently extract the features of the audio in the standard speech data set. The characteristics of the audio are described by methods such as Mel spectrum analysis technology or linear envelope analysis, so that when matching text blocks and audio blocks, comparison and selection can be performed based on more effective audio features. This helps to improve the accuracy of selecting appropriate audio blocks, thereby improving the quality of the first standard audio.

[0050] Exemplarily, the audio signal in the standard speech data set is processed by applying Mel spectrum analysis technology or linear envelope analysis algorithm to extract audio features. These audio features may include information such as frequency distribution and energy change of the audio. When determining the standard audio block corresponding to the text block, the audio features corresponding to the semantics, grammar and other information of the text block are matched with the extracted features of the audio in the standard speech data set. For example, if the text block contains an emphasized word, the audio features extracted by the new audio representation technology can find a standard audio block with corresponding emphasized audio features (such as higher pitch, louder volume).

[0051] This embodiment processes in units of text blocks, which can synthesize speech more finely and improve accuracy and naturalness. The efficiency of speech synthesis can be improved by using data matching. The use of new audio representation technology in standard speech data sets can more efficiently extract audio features, improve matching accuracy and first standard audio quality. When the corresponding audio block is missing in the data set, the hierarchical generation model can generate audio features, ensuring that speech can be synthesized even in the absence of data, enhancing the applicability and robustness of the method, and providing more stable and reliable speech synthesis services for scenarios such as intelligent voice assistants.

[0052] In an embodiment of the present disclosure, the first instruction information includes a reading instruction or a question-and-answer instruction.

[0053] Determining a first standard audio based on the first text and the first instruction information includes: If the first instruction information is a reading instruction, context information associated with the first text is determined based on the first text, and the context information is used as the first data.

[0054] If the first instruction information is a question-and-answer instruction, context information associated with the first text and voice information corresponding to the first text are determined based on the first text, and the context information and the voice information are used as the first data.

[0055] In this embodiment, the question-answering instruction can be used to instruct the following process: translating the question (in voice form) and generating an answer text, and performing voice synthesis on the answer text. The reading instruction can be used to instruct the following process: synthesizing the corresponding voice based on the given text content.

[0056] In this embodiment, for reading instructions and question-and-answer instructions, in the process of determining the first standard audio, an improved attention mechanism is used to focus on the important features of the text, and the audio features are extracted in combination with the new audio representation technology to improve the quality and efficiency of the synthesized speech. For reading instructions, by analyzing the context information, the improved attention mechanism can better understand the semantic coherence of the text, and the new audio representation technology ensures that the generated reading speech is more natural in terms of timbre, rhythm, etc. For question-and-answer instructions, in addition to context information, voice information is also considered, and the improved attention mechanism can deal with the key parts of the question in a targeted manner.

[0057] Exemplarily, when determining the context information associated with the first text, an adaptive multi-head attention mechanism is used. The attention weight change signal of the historical reading text is analyzed, and the number of attention heads and the weight range are dynamically adjusted in combination with reinforcement learning. For example, in the reading of history, a higher attention weight is given to descriptive words. Then, when similar words are encountered in the current text, the number of attention heads and weights for these words are adaptively increased. At the same time, a local attention mechanism is adopted, combined with a sliding window, and the appropriate window size is adjusted so that the attention head only focuses on the current text segment and the context within a certain range before and after it, thereby improving the efficiency of attention calculation.

[0058] Exemplarily, when synthesizing answering speech, new audio representation technology is used to ensure the clarity and accuracy of the speech. When extracting features from the answering text, effective audio features are extracted in combination with technologies such as Mel spectrum analysis, so that the synthesized speech is more in line with the requirements of the question in terms of frequency distribution and energy change. At the same time, according to the type and emotional tendency of the question, audio parameters such as pitch and speaking speed are adjusted to make the answering speech more expressive and targeted. For example, for an urgent question, the synthesized speech can have a faster speaking speed and a higher pitch to convey a sense of urgency.

[0059] This embodiment divides speech synthesis scenarios into two types and uses instruction information to distinguish and guide. For reading instructions, by analyzing the context information associated with the text to determine the first standard audio, the synthesized speech can better match the text semantics and emotions, and improve the naturalness and fluency of reading. For question-and-answer instructions, comprehensive consideration of context and voice information helps to understand the questions more accurately and generate targeted, clear and accurate answer voices, improve the quality and efficiency of voice interaction, and meet the speech synthesis needs of different scenarios.

[0060] In one embodiment of the present disclosure, determining the first text identifier based on the first data includes: A text content type of the first text is determined based on the first data.

[0061] A corresponding first text identifier is determined based on the text content type.

[0062] In this embodiment, the first text identifier is used to classify and mark the first text. The content indicated by the first text identifier may include text subject category (such as technology, literature, news, etc.), emotional tendency (positive, negative, etc.), style type (formal, colloquial, humorous, etc.).

[0063] The text content type is a specific classification description of the first text, such as story, description, science, technology, news, etc. The text content type is divided based on the text's structural characteristics (such as the chronological order of narrative texts, the arguments and evidence of argumentative texts, etc.), language characteristics (the number of professional terms, vocabulary complexity, etc.), and subject areas (medicine, history, entertainment, etc.).

[0064] Exemplarily, by analyzing the context information or voice information in the first data, key features are extracted. These features are used to determine the characteristics of the first text in terms of theme, emotion, style, etc., so as to determine its text content type. According to a preset corresponding relationship, a corresponding first text identifier is assigned to the text content type.

[0065] Exemplarily, quantization and knowledge distillation can be used to refine complex knowledge from a large-scale "teacher" model into a relatively simple model, and combined with quantization technology to reduce the model's computational complexity and storage requirements, thereby more efficiently processing the first data while ensuring the accuracy and performance of text content type judgment.

[0066] Exemplarily, audio feature extraction technology can be used to capture audio-related features of the first text. These features can assist in determining the text content type, and then more accurately determine the first text identifier so that the identifier can better reflect the characteristics of the text and the appropriate speech synthesis method, thereby improving the quality and effect of subsequent speech synthesis.

[0067] This embodiment can accurately classify text by analyzing the first data to determine the text content type and identification, providing a clear direction for subsequent speech synthesis. By using quantization, knowledge distillation and audio feature extraction technology, the processing efficiency is improved, the judgment accuracy is guaranteed, and the identification can better reflect the characteristics of the text. This helps to improve the quality of speech synthesis, meet the synthesis requirements of different text types, bring users a better voice experience, and adapt to a variety of application scenarios.

[0068] In one embodiment of the present disclosure, determining a target voice based on a first text identifier and a first standard audio includes: A target audio parameter feature is determined based on the first text identifier.

[0069] The first standard audio is updated based on the target audio parameter feature to obtain the second audio.

[0070] The second audio is speech-concatenated based on the first text identifier to obtain a target speech.

[0071] In this embodiment, the target audio parameter feature is an audio characteristic description for synthesizing the target speech determined according to the first text identifier. The target audio parameter feature may include pitch (high, medium, low, etc.), speech speed (fast, medium, slow, etc.), volume, emotional color (joy, sadness, seriousness, etc.), rhythm, etc.

[0072] The second audio is an intermediate audio product obtained by updating the first standard audio based on the target audio parameter characteristics. The second audio contains content and parameters similar to the first standard audio, but is closer to the requirements of the target speech in terms of audio parameter characteristics, such as adjusted sampling rate, frequency range, duration, timbre, etc.

[0073] Exemplarily, an audio processing algorithm or machine learning technology may be used. For example, the first standard audio may be updated by adjusting the frequency, duration, amplitude and other parameters of the audio. For speech splicing, the splicing point and splicing method may be determined according to the first text identifier, such as performing smooth splicing at a specific emotional expression.

[0074] For example, in an audiobook application, the first text identifier indicates a tense plot section in the story. According to this identifier, the target audio parameter characteristics are determined to be a higher pitch, a faster speaking speed, and a tense emotional color. The first standard audio is updated to obtain the second audio, so that it has these characteristics. At the plot turning point, voice splicing is performed according to the first text identifier, such as adding tense sound effects, to generate the final target voice, so that the audience can better immerse themselves in the tense atmosphere of the story.

[0075] In this embodiment, in the process of updating the first standard audio to obtain the second audio, a hierarchical generation model can be used to implement it. According to the target audio parameter features determined by the first text identifier, the hierarchical generation model can extract low-resolution target audio parameter features from the first standard audio, and then gradually refine these features so that the updated second audio can better meet the requirements of the target speech. This hierarchical approach can more finely control the generation process of the audio, reduce unnecessary calculations, reduce the computational complexity, and improve the quality of the generated speech.

[0076] Exemplarily, relevant audio parameter features are extracted from the first standard audio and used as input of the hierarchical generation model to generate low-resolution target audio parameter features through the bottom layer. According to the more detailed requirements contained in the first text identifier (such as specific emotional expression, style, etc.), the low-resolution features are gradually refined into high-resolution target audio parameter features using the generator. In this process, adversarial training with the discriminator is used to ensure that the generated high-resolution features can effectively update the first standard audio to obtain high-quality second audio. In the speech splicing stage, the hierarchically generated and refined audio parameter features can be used to better fuse audio clips and generate the final target speech.

[0077] In this embodiment, when updating the first standard audio, the first standard audio can also be input into a feature extraction model based on deep learning, and the convolution layer is used to extract local features of the audio, such as pronunciation details of specific words, changes in tone, etc. These local features are integrated and strengthened by enhancing connection strategies, such as residual connections or dense connections. According to the entertainment content characteristics required by the first text identifier (such as emotionally rich narration, vivid role-playing, etc.), the relevant features are adjusted to update the first standard audio to obtain the second audio. In the process of speech splicing, the use of finely processed audio features can ensure that the target speech after splicing can better convey the content and provide a better user experience.

[0078] This embodiment updates the standard audio through target audio parameter features and utilizes technologies such as hierarchical generation models to more finely control audio generation and improve voice quality, while reducing computational complexity, improving synthesis efficiency, meeting different application requirements, and bringing better user experience and technological progress to the field of speech synthesis.

[0079] In one embodiment of the present disclosure, the first text identifier includes an audiobook identifier.

[0080] Performing voice splicing on the second audio based on the first text identifier includes: If the first text identifier is an audio book identifier, a sound effect block is determined based on the second audio, and voice splicing is performed on the second audio and the sound effect block.

[0081] In this embodiment, voice splicing is performed on the second audio and the sound effect block, including: The target audio block is determined based on the sound effect block, and the audio parameter features of the target audio block and the sound effect block are updated to obtain the updated second audio and the sound effect block. The target audio block is the audio block corresponding to the sound effect block in the second audio.

[0082] Perform voice concatenation on the updated second audio and sound effect blocks.

[0083] In this embodiment, the book listening flag is used to clearly indicate that the current speech processing task is for a book listening scenario. The book listening flag can be used to indicate a book type parameter (such as novel, biography, history, popular science, etc.) to determine a suitable overall sound effect style, or an emotional atmosphere parameter (such as relaxed, tense, sad, etc.) to select a sound effect with a corresponding emotional tendency, or a speech speed preference parameter (such as slow, medium, fast, etc.) to adjust a speech speed suitable for book listening.

[0084] The sound effect block is a specific audio clip that matches the content of the audiobook and is used to enhance the auditory experience of the audiobook. The content or parameters that the sound effect block may include are: sound effect type, such as background music (classical, pop, light music, etc.), environmental sound effects (wind, rain, birdsong, etc.), special event sound effects (book turning, door closing, footsteps, etc.); audio feature parameters, such as volume, pitch, rhythm, sound effect duration, etc.; trigger conditions or tags of sound effects, such as association with specific plots, scenes or keywords.

[0085] The target audio block is a portion of the second audio corresponding to the sound effect block, and is an audio segment that needs to be fused with the sound effect block.

[0086] The target audio block may include the following contents or parameters: basic audio parameters consistent with the second audio, such as sampling rate, bit depth, etc.; feature parameters such as semantics and emotions based on the second audio content, which are used to match and adjust the audio parameter features of the sound effect block.

[0087] Exemplarily, when an audiobook identifier is identified, a sound effect block that matches the second audio is searched based on the content and features of the second audio. A target audio block corresponding to the sound effect block is determined in the second audio. The audio parameter features of the target audio block and the sound effect block, such as volume, pitch, rhythm, etc., are analyzed. According to preset rules or algorithms, the audio parameter features of the target audio block and the sound effect block are updated to make them more coordinated and natural when fused. Finally, the updated second audio and the sound effect block are spliced ​​in chronological order to form a complete audiobook audio.

[0088] In this embodiment, updating the audio parameter features of the target audio block and the sound effect block may include: updating the volume parameters, pitch parameters, rhythm parameters, timbre parameters and time parameters of the target audio block and the sound effect block.

[0089] Exemplarily, the volume is adjusted according to the importance of the target audio block and the sound effect block. For example, in a book listening scenario, if it is an important conversation (target audio block) and the background sound effect is a gentle wind (sound effect block), the conversation volume can be appropriately increased and the wind volume can be reduced to ensure that the conversation is clearly audible while retaining the atmosphere of the ambient sound effect. For target audio blocks with strong emotional expressions, such as exciting speeches, when integrating sound effect blocks (such as exciting background music), the dynamic range of the overall volume can be appropriately expanded to make the climax more prominent and enhance the appeal.

[0090] For example, the target audio block is a male voiceover with a low pitch, while the sound effect block is a melodious flute sound (with a higher pitch). The pitch of the flute sound can be appropriately lowered, or the base frequencies of the two can be made closer through audio processing algorithms to avoid a sense of abruptness. When the target audio block carries a sad emotion, the sound effect block (such as a slow piano piece) fused with it can lower the pitch accordingly and reduce the fluctuation of the pitch to create a depressing atmosphere.

[0091] For example, if the target audio block is a fast narrative (fast speaking speed), the rhythm of the background music (sound effect block) integrated with it should also be appropriately accelerated, such as adjusting from a soothing rhythm to a brisk rhythm, so that the two are synchronized in rhythm. For rhythmic target audio blocks, such as poetry recitation, the rhythm of the sound effect block (such as rhythmic percussion) should match the rhythm of the poem, such as strengthening the rhythm points of the percussion at the rhyme of the poem to enhance the sense of rhythm.

[0092] For example, if the target audio block is a soft female voice and the sound effect block is an environmental sound with a metallic texture (such as a mechanical running sound), the timbre of the mechanical running sound can be softened and reduced through filtering and other means, so that it can be better integrated with the female voice. Effects such as reverberation can be used to adjust the spatial sense of the target audio block and the sound effect block. For example, for the dialogue (target audio block) and the sound of water drops (sound effect block) in the cave scene, the same reverberation effect can be added to simulate the spatial environment of the cave so that the two are coordinated in spatial timbre.

[0093] For example, the start and end time of the sound effect block can be finely adjusted to match the semantics and emotional expression of the target audio block. For example, before a mysterious scene in the story appears, a mysterious sound effect block is introduced a little earlier, and the sound effect ends slightly after the scene ends to enhance the suspense effect. If the target audio block is long and the sound effect block is short, the sound effect block can be looped or stretched (without changing the pitch) to better match the target audio block in duration, avoiding audio gaps or uncoordinated repetitions.

[0094] This embodiment can accurately match the appropriate sound effect style and speech speed through the detailed classification of audio book identifiers, thereby improving the adaptability of audio books. The multi-parameter adjustment and fusion of the target audio block and the sound effect block can make the synthesized audio of the audio book more coordinated and natural in terms of volume, pitch, rhythm, timbre, etc., and enhance the auditory experience. The fine time parameter processing and sound effect matching can better fit the plot and emotional expression, create an immersive atmosphere for audio books, improve the user's understanding and enjoyment of the audio book content, and meet the needs of different users in various audio book scenarios.

[0095] Corresponding to the speech synthesis method of the above embodiment, Figure 2 This is a structural block diagram of a speech synthesis device provided by an embodiment of the present disclosure. For ease of description, only the parts related to the embodiment of the present disclosure are shown. Figure 2 The speech synthesis device 20 includes: a first calculation module 21, a second calculation module 22 and a speech synthesis module 23.

[0096] The first calculation module 21 is used to determine the first standard audio based on the first text.

[0097] The second calculation module 22 is used to determine first data based on the first text and the first instruction information, and determine a first text identifier based on the first data. The first data includes context information or voice information of the first text.

[0098] The speech synthesis module 23 is used to determine the target speech based on the first text identifier and the first standard audio.

[0099] In an embodiment of the present disclosure, the first calculation module 21 is specifically configured to determine a plurality of text blocks based on the first text, and generate index sequences corresponding to the plurality of text blocks.

[0100] One or more standard audio blocks corresponding to each text block are determined from the standard speech data set based on the index sequence. The standard speech data set includes multiple standard audio blocks.

[0101] A first standard audio is determined based on one or more standard audio blocks corresponding to each text block.

[0102] In an embodiment of the present disclosure, the first instruction information includes a reading instruction or a question-and-answer instruction.

[0103] The second calculation module 22 is specifically configured to determine context information associated with the first text based on the first text if the first instruction information is a reading instruction, and use the context information as the first data.

[0104] If the first instruction information is a question-and-answer instruction, context information associated with the first text and voice information corresponding to the first text are determined based on the first text, and the context information and the voice information are used as the first data.

[0105] In an embodiment of the present disclosure, the second calculation module 22 is specifically configured to determine the text content type of the first text based on the first data.

[0106] A corresponding first text identifier is determined based on the text content type.

[0107] In an embodiment of the present disclosure, the speech synthesis module 23 is specifically configured to determine a target audio parameter feature based on the first text identifier.

[0108] The first standard audio is updated based on the target audio parameter feature to obtain the second audio.

[0109] The second audio is speech-concatenated based on the first text identifier to obtain a target speech.

[0110] In one embodiment of the present disclosure, the first text identifier includes an audio book identifier. The speech synthesis module 23 is further configured to, if the first text identifier is an audio book identifier, determine a sound effect block based on the second audio, and perform speech splicing on the second audio and the sound effect block.

[0111] In one embodiment of the present disclosure, the speech synthesis module 23 is further configured to determine a target audio block based on the sound effect block, update audio parameter features of the target audio block and the sound effect block, and obtain an updated second audio and sound effect block. The target audio block is an audio block corresponding to the sound effect block in the second audio.

[0112] Perform voice concatenation on the updated second audio and sound effect blocks.

[0113] See also Figure 3 , Figure 3 A schematic block diagram of an electronic device provided by an embodiment of the present disclosure. Figure 3 The electronic device 300 in the embodiment shown may include: one or more processors 301, one or more input devices 302, one or more output devices 303 and one or more memories 304. The processors 301, input devices 302, output devices 303 and memories 304 communicate with each other via a communication bus 305. The memory 304 is used to store computer programs, which include program instructions. The processor 301 is used to execute the program instructions stored in the memory 304. The processor 301 is configured to call the program instructions to execute the functions of the modules in the above-mentioned device embodiments, such as Figure 2 The functions of modules 21 to 23 are shown.

[0114] It should be understood that in the embodiment of the present disclosure, the processor 301 may be a central processing unit (CPU), and the processor may also be other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc.

[0115] The input device 302 may include a touch panel, a fingerprint collection sensor (for collecting the user's fingerprint information and fingerprint direction information), a microphone, etc., and the output device 303 may include a display (LCD, etc.), a speaker, etc.

[0116] The memory 304 may include a read-only memory and a random access memory, and provide instructions and data to the processor 301. A portion of the memory 304 may also include a non-volatile random access memory. For example, the memory 304 may also store information about the device type.

[0117] In a specific implementation, the processor 301, input device 302, and output device 303 described in the embodiments of the present disclosure can execute the implementation methods described in the first and second embodiments of a speech synthesis method provided in the embodiments of the present disclosure, and can also execute the implementation methods of the electronic device 300 described in the embodiments of the present disclosure, which will not be repeated here.

[0118] In another embodiment of the present disclosure, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores a computer program, wherein the computer program includes program instructions, and when the program instructions are executed by the processor, all or part of the processes in the above-mentioned embodiment method are implemented, and the computer program can also be completed by instructing the relevant hardware through the computer program. The computer program can be stored in a computer-readable storage medium, and when the computer program is executed by the processor, the steps of each of the above-mentioned method embodiments can be implemented. Among them, the computer program includes computer program code, and the computer program code can be in source code form, object code form, executable file or some intermediate form, etc. The computer-readable medium can include: any entity or device capable of carrying computer program code, recording medium, U disk, mobile hard disk, magnetic disk, optical disk, computer memory, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), electric carrier signal, telecommunication signal and software distribution medium, etc.

[0119] The computer-readable storage medium may be an internal storage unit of the electronic device of any of the aforementioned embodiments, such as a hard disk or memory of the electronic device. The computer-readable storage medium may also be an external storage device of the electronic device, such as a plug-in hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (SecureDigital, SD) card, a flash card (Flash Card), etc. equipped on the electronic device. Furthermore, the computer-readable storage medium may also include both an internal storage unit of the electronic device and an external storage device. The computer-readable storage medium is used to store computer programs and other programs and data required by the electronic device. The computer-readable storage medium may also be used to temporarily store data that has been output or is to be output.

[0120] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described in terms of function in the above description. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this disclosure.

[0121] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the electronic devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0122] In the several embodiments provided in the present application, it should be understood that the disclosed electronic devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic, for example, the division of units is only a logical function division, and there may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces or units, or it can be an electrical, mechanical or other form of connection.

[0123] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the embodiments of the present disclosure.

[0124] In addition, each functional unit in each embodiment of the present disclosure may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0125] The above are only specific embodiments of the present disclosure, but the protection scope of the present disclosure is not limited thereto. Any technician familiar with the technical field can easily think of various equivalent modifications or replacements within the technical scope disclosed in the present disclosure, and these modifications or replacements should be included in the protection scope of the present disclosure. Therefore, the protection scope of the present disclosure should be based on the protection scope of the claims.

Claims

1. A speech synthesis method, characterized in that: include: determining a first standard audio based on the first text; Determine first data based on the first text and the first instruction information, and determine a first text identifier based on the first data; the first data includes context information or voice information of the first text; A target voice is determined based on the first text identifier and the first standard audio.

2. The speech synthesis method according to claim 1, characterized in that: The determining of the first standard audio based on the first text comprises: Determine a plurality of text blocks based on the first text, and generate index sequences corresponding to the plurality of text blocks; Determining one or more standard audio blocks corresponding to each text block from a standard speech data set based on the index sequence; the standard speech data set includes a plurality of standard audio blocks; The first standard audio is determined based on one or more standard audio blocks corresponding to each text block.

3. The speech synthesis method according to claim 1, characterized in that: The first instruction information includes a reading instruction or a question-and-answer instruction; The determining the first data based on the first text and the first instruction information includes: If the first instruction information is a reading instruction, determining context information associated with the first text based on the first text, and using the context information as the first data; If the first instruction information is a question-and-answer instruction, context information associated with the first text and voice information corresponding to the first text are determined based on the first text, and the context information and the voice information are used as the first data.

4. The speech synthesis method according to claim 1, wherein: The determining the first text identifier based on the first data includes: determining a text content type of the first text based on the first data; A corresponding first text identifier is determined based on the text content type.

5. The speech synthesis method according to claim 1, wherein: The determining the target voice based on the first text identifier and the first standard audio includes: Determining a target audio parameter feature based on the first text identifier; Update the first standard audio based on the target audio parameter feature to obtain a second audio; The second audio is speech-concatenated based on the first text identifier to obtain the target speech.

6. The speech synthesis method according to claim 5, characterized in that: The first text identifier includes an audiobook identifier; The performing voice splicing on the second audio based on the first text identifier includes: If the first text identifier is the audiobook identifier, a sound effect block is determined based on the second audio, and voice splicing is performed on the second audio and the sound effect block.

7. The speech synthesis method according to claim 6, characterized in that: The voice splicing of the second audio and the sound effect block includes: Determine a target audio block based on the sound effect block, update audio parameter features of the target audio block and the sound effect block, and obtain an updated second audio and sound effect block; the target audio block is an audio block corresponding to the sound effect block in the second audio; Perform voice splicing on the updated second audio and sound effect block.

8. A speech synthesis device, characterized in that: include: A first calculation module, configured to determine a first standard audio based on the first text; A second calculation module is used to determine first data based on the first text and the first instruction information, and determine a first text identifier based on the first data; the first data includes context information or voice information of the first text; A speech synthesis module is used to determine a target speech based on the first text identifier and the first standard audio.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and running on the processor, characterized in that: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

Images