Speech generation method, device, electronic device and storage medium

Abstract

The embodiments of the present disclosure disclose a speech generation method, device, electronic device and storage medium. The method comprises the following steps: obtaining content to be synthesized for performing content constraints during speech generation, extracting content embedding of the content to be synthesized; obtaining style prompt speech for performing style constraints during speech generation, extracting multiple original speech tags in the style prompt speech; inputting the content embedding and the multiple original speech tags into a first large language model for prediction, and obtaining multiple target speech tags; obtaining timbre prompt speech for performing timbre constraints during speech generation, extracting a first speaking object embedding of the timbre prompt speech; performing speech generation based on the first speaking object embedding and the multiple target speech tags, and obtaining a target speech, thereby improving the authenticity and flexibility of the generated target speech. The method can be widely used in cloud technology, artificial intelligence, smart transportation and other scenarios.

Description

Technical Field

[0001] The present disclosure relates to the field of artificial intelligence technology, and in particular to a speech generation method, device, electronic device and storage medium. Background Art

[0002] With the development of artificial intelligence technology, the application of zero-shot speech generation has become more and more widespread, for example, it has been widely used in scenarios such as voice assistants, audio books, map navigation, etc. However, the speech generated by the current zero-shot speech generation system is relatively fixed in timbre and style, and its authenticity and flexibility need to be improved. Summary of the invention

[0003] The following is a summary of the subject matter of the detailed description of the present disclosure. This summary is not intended to limit the scope of the claims.

[0004] The embodiments of the present disclosure provide a speech generation method, device, electronic device, and storage medium, which can improve the authenticity and flexibility of the generated target speech.

[0005] In one aspect, an embodiment of the present disclosure provides a method for generating speech, comprising:

[0006] Acquire content to be synthesized for content constraint during speech generation, and extract content embedding of the content to be synthesized;

[0007] Acquire style prompt speech for performing style constraints during speech generation, and extract multiple original speech tags from the style prompt speech;

[0008] Inputting the content embedding and the plurality of original speech tags into a first large language model for prediction to obtain a plurality of target speech tags;

[0009] Acquire a timbre prompt speech for performing timbre constraint during speech generation, and extract a first speaker embedding of the timbre prompt speech;

[0010] Speech generation is performed based on the first speaker embedding and the plurality of target speech tags to obtain a target speech.

[0011] On the other hand, the present disclosure also provides a speech generating device, including:

[0012] A content processing module, used to obtain content to be synthesized for content constraint during speech generation, and extract content embedding of the content to be synthesized;

[0013] A style processing module, used to obtain style prompt speech for performing style constraints during speech generation, and extract multiple original speech tags in the style prompt speech;

[0014] A prediction module, configured to input the content embedding and the plurality of original speech tags into a first large language model for prediction to obtain a plurality of target speech tags;

[0015] A timbre processing module, used for acquiring a timbre prompt voice for timbre constraint during speech generation, and extracting a first speaker embedding of the timbre prompt voice;

[0016] A generation module is used to generate speech based on the first speaker embedding and the plurality of target speech tags to obtain a target speech.

[0017] Furthermore, the style processing module is specifically used for:

[0018] Inputting the content to be synthesized into the second largest language model for prediction, obtaining a first style description text for describing the style of the speaker of the content to be synthesized, and a first emotion description text for describing the emotion of each content segment in the content to be synthesized;

[0019] Building a first global embedding of the content fragment based on the embedding of the first style description text, and building a first detail embedding of the content fragment based on the embedding of the first emotion description text;

[0020] Based on the first global embedding and the first detail embedding, a style prompt speech corresponding to the content segment is retrieved from a preset knowledge base, wherein the style prompt speech is used to perform style constraints during speech generation.

[0021] Furthermore, the knowledge base stores a reference speech segment and a second global embedding and a second detail embedding associated with the reference speech segment, the second global embedding is used to indicate the style of the speaker of the reference speech segment, and the second detail embedding is used to indicate the emotion of the reference speech segment, and the style processing module is further used to:

[0022] Retrieving a plurality of reference speech segments from the knowledge base according to the similarity between the first global embedding and the second global embedding;

[0023] According to the similarity between the first detail embedding and the second detail embedding, the style prompt speech corresponding to the content segment is retrieved from the plurality of reference speech segments.

[0024] Furthermore, before retrieving the style prompt voice corresponding to the content segment from a preset knowledge base based on the first global embedding and the first detail embedding, the style processing module is further used to:

[0025] Acquire original speech, and segment the original speech into speaking objects to obtain multiple speech blocks;

[0026] Performing speech recognition on each of the speech blocks respectively to obtain speech block text in each of the speech blocks;

[0027] Segmenting the speech block text to obtain a plurality of reference text segments, and segmenting the speech block into reference speech segments corresponding to the reference text segments;

[0028] The second global embedding is constructed based on the plurality of speech block texts, the second detail embedding is constructed based on the reference text segment, and the reference speech segment, the second global embedding and the second detail embedding are associated and stored in the knowledge base.

[0029] Furthermore, the style processing module is also used for:

[0030] Constructing a first prompt text based on the plurality of speech block texts, inputting the first prompt text into the second largest language model for prediction, and obtaining a second style description text for describing the style of the speaker of the plurality of speech block texts, wherein the first prompt text is used to prompt the second largest language model to output the second style description text based on the plurality of speech block texts;

[0031] Building the second global embedding based on the embedding of the second style description text;

[0032] Constructing a second prompt text based on the second style description text, the reference text segment, and the plurality of speech block texts, inputting the second prompt text into the second large language model for prediction, and obtaining a second emotion description text for describing the emotion of the reference text segment, wherein the second prompt text is used to prompt the second large language model to output the second emotion description text based on the second style description text, the reference text segment, and the plurality of speech block texts;

[0033] The second detail embedding is constructed based on the embedding of the second emotion description text.

[0034] Furthermore, the style processing module is also used for:

[0035] Retrieving a plurality of candidate speech segments from a plurality of the reference speech segments according to the similarity between the first detail embedding and the second detail embedding;

[0036] Determine the contextual relevance between each of the candidate speech segments and the content segment, and determine the candidate speech segment with the highest contextual relevance as the style prompt speech corresponding to the content segment, or, again determine the similarity between the second global embedding and the first global embedding of each of the candidate speech segments, and determine the candidate speech segment with the highest similarity as the style prompt speech corresponding to the content segment, or, perform speech quality evaluation on each of the candidate speech segments, and determine the candidate speech segment with the highest speech quality as the style prompt speech corresponding to the content segment.

[0037] Furthermore, the style processing module is also used for:

[0038] Acquire a first relevant description text for supplementary description when retrieving the style prompt voice;

[0039] The embedding of the first style description text and the embedding of the first related description text are fused to obtain a first global embedding of the content segment.

[0040] Furthermore, the style processing module is also used for:

[0041] The first global embedding and the embedding of the first emotion description text are fused to obtain a first detail embedding of the content segment.

[0042] Furthermore, the content to be synthesized is speech to be synthesized, the content segment is a speech segment, and the style processing module is further used for:

[0043] Extracting speech emotion embedding of the speech segment;

[0044] The first global embedding, the embedding of the first emotion description text, and the speech emotion embedding are fused to obtain a first detail embedding of the content segment.

[0045] Furthermore, the style processing module is also used for:

[0046] Inputting the first global embedding and the first detail embedding into a classifier for classification to obtain a style emotion classification result of the content segment, wherein the style emotion classification result is used to indicate that the style emotion of the content segment is neutral, or is used to indicate that the style emotion of the content segment is rich;

[0047] When the style emotion classification result indicates that the style emotion of the content segment is rich, based on the first global embedding and the first detail embedding, the style prompt speech corresponding to the content segment is retrieved from a preset knowledge base.

[0048] Furthermore, the style processing module is also used for:

[0049] When the style emotion classification result indicates that the style emotion of the content segment is neutral, based on the first global embedding, the style prompt speech corresponding to the content segment is retrieved from the knowledge base.

[0050] Furthermore, the above prediction module is specifically used for:

[0051] A second speaker embedding of the style cue speech is extracted, and the content embedding, the second speaker embedding, and the plurality of original speech tags are input into a first large language model for prediction to obtain a plurality of target speech tags.

[0052] Furthermore, the above prediction module is also used for:

[0053] The sequence start marker, the second speaker embedding, the content embedding, the voice conversion marker and the plurality of original voice markers are sequentially concatenated and input into the first large language model for prediction to obtain a plurality of target voice markers.

[0054] Furthermore, the above generation module is specifically used for:

[0055] The acoustic features of the style prompt speech or the timbre prompt speech are extracted, and speech generation is performed based on the acoustic features, the first speaker embedding, and the plurality of target speech tags to obtain a target speech.

[0056] On the other hand, an embodiment of the present disclosure further provides an electronic device, including a memory and a processor, wherein the memory stores a computer program, and the processor implements the above-mentioned speech generation method when executing the computer program.

[0057] On the other hand, an embodiment of the present disclosure further provides a computer-readable storage medium, wherein the storage medium stores a computer program, and the computer program is executed by a processor to implement the above-mentioned speech generation method.

[0058] On the other hand, the embodiment of the present disclosure further provides a computer program product, which includes a computer program, and the computer program is stored in a computer-readable storage medium. The processor of the computer device reads the computer program from the computer-readable storage medium, and the processor executes the computer program, so that the computer device executes the above-mentioned speech generation method.

[0059] The disclosed embodiments include at least the following beneficial effects: by acquiring content to be synthesized for content constraint during speech generation, extracting content embedding of the content to be synthesized, and being able to constrain the content in the target speech through the content embedding during speech generation; in addition, by acquiring style prompt speech for style constraint during speech generation, extracting multiple original speech tags in the style prompt speech, inputting the content embedding and the multiple original speech tags into the first language model for prediction, being able to predict multiple target speech tags similar to the style of the style prompt speech based on the first language model; on this basis, by acquiring style prompt speech for timbre constraint during speech generation, The constrained timbre prompt speech extracts the first speaker embedding of the timbre prompt speech, and then generates speech based on the first speaker embedding and multiple target speech tags, thereby obtaining a target speech that is similar in timbre to the timbre prompt speech and in style to the style prompt speech, thereby achieving control of style and timbre, making the target speech more natural, and thus improving the authenticity of the target speech. In addition, since the content to be synthesized, the style prompt speech, and the timbre prompt speech are respectively introduced at different stages of the generation process, the decoupling of the content, style, and timbre of the target speech is achieved, and the flexible combination of any content, style, and timbre is supported, thereby improving the flexibility of the target speech.

[0060] Other features and advantages of the present disclosure will be set forth in the following description, and in part will be apparent from the description, or may be learned by practicing the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0061] The accompanying drawings are used to provide further understanding of the technical solution of the present disclosure and constitute a part of the specification. Together with the embodiments of the present disclosure, they are used to explain the technical solution of the present disclosure and do not constitute a limitation on the technical solution of the present disclosure.

[0062] Figure 1 A schematic diagram of an optional implementation environment provided for an embodiment of the present disclosure;

[0063] Figure 2 An optional schematic diagram of applying the speech generation method provided in the embodiment of the present disclosure to a virtual human voice assistant scenario;

[0064] Figure 3 An optional schematic diagram of the application of the speech generation method provided in the embodiment of the present disclosure in a live broadcast chat robot scenario;

[0065] Figure 4 An optional flow chart of a speech generation method provided in an embodiment of the present disclosure;

[0066] Figure 5 An optional schematic diagram of the raw data processing process provided by the embodiment of the present disclosure;

[0067] Figure 6 An optional schematic diagram of a first prompt text and a second prompt text provided in an embodiment of the present disclosure;

[0068] Figure 7 An optional schematic diagram of the overall technical architecture of the speech generation method provided in the embodiment of the present disclosure;

[0069] Figure 8 An optional schematic diagram of the internal processing process of the overall technical architecture of the speech generation method provided in the embodiment of the present disclosure;

[0070] Fig. 9 An optional schematic diagram of a speech generation module framework based on a large language model provided in an embodiment of the present disclosure;

[0071] Fig.10 An optional schematic diagram of a chat robot configuration process for a live broadcast room provided in an embodiment of the present disclosure;

[0072] Fig.11 A schematic diagram of an optional structure of a speech generating device provided in an embodiment of the present disclosure;

[0073] Fig.12 A partial structural block diagram of a terminal provided in an embodiment of the present disclosure;

[0074] Fig.13 A partial structural block diagram of a server provided in an embodiment of the present disclosure. DETAILED DESCRIPTION

[0075] In order to make the purpose, technical solution and advantages of the present disclosure more clear, the present disclosure is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present disclosure and are not used to limit the present disclosure.

[0076] It should be noted that in various specific embodiments of the present disclosure, when it comes to the need to perform relevant processing based on data related to the characteristics of the target object such as the target object attribute information or attribute information set, the permission or consent of the target object will be obtained first, and the collection, use and processing of these data will comply with relevant laws, regulations and standards. Among them, the target object can be a user. In addition, when the embodiment of the present disclosure needs to obtain the attribute information of the target object, the separate permission or separate consent of the target object will be obtained through a pop-up window or by jumping to a confirmation page. After clearly obtaining the separate permission or separate consent of the target object, the necessary target object-related data used to enable the normal operation of the embodiment of the present disclosure will be obtained.

[0077] In the embodiments of the present disclosure, the term "module" or "unit" refers to a computer program or a part of a computer program that has a predetermined function and works together with other related parts to achieve a predetermined goal, and can be implemented in whole or in part by using software, hardware (such as processing circuits or memories), or a combination thereof. Similarly, a processor (or multiple processors or memories) can be used to implement one or more modules or units. In addition, each module or unit can be part of an overall module or unit that includes the function of the module or unit.

[0078] To facilitate understanding of the technical solution provided by the embodiments of the present disclosure, some key terms used in the embodiments of the present disclosure are explained here:

[0079] Large Language Model (LLM) refers to a deep learning model trained with a large amount of text data that can generate natural language text or understand the meaning of language text. Large language models can handle a variety of natural language tasks, such as text classification, question answering, and dialogue.

[0080] Prompt text represents a command or instruction, which is used to instruct the large language model to perform an action or generate an output, that is, to instruct the model what action should be taken or what output should be generated when performing a specific task.

[0081] With the development of artificial intelligence technology, the application of zero-shot speech generation has become more and more widespread, for example, it has been widely used in scenarios such as voice assistants, audio books, map navigation, etc. However, the speech generated by the current zero-shot speech generation system is relatively fixed in timbre and style, and its authenticity and flexibility need to be improved.

[0082] Based on this, the embodiments of the present disclosure provide a speech generation method, device, electronic device and storage medium, which can improve the authenticity and flexibility of the generated target speech.

[0083] Reference Figure 1 , Figure 1 A schematic diagram of an optional implementation environment provided for an embodiment of the present disclosure, the implementation environment includes a terminal 101 and a server 102, wherein the terminal 101 and the server 102 are connected via a communication network.

[0084] Exemplarily, the server 102 may obtain the content to be synthesized for content constraint during speech generation, extract the content embedding of the content to be synthesized; obtain the style prompt speech for style constraint during speech generation, extract multiple original speech tags in the style prompt speech; input the content embedding and multiple original speech tags into the first large language model for prediction, and obtain multiple target speech tags; obtain the timbre prompt speech for timbre constraint during speech generation, extract the first speaker embedding of the timbre prompt speech; perform speech generation based on the first speaker embedding and multiple target speech tags, and obtain the target speech. Among them, the content to be synthesized, the style prompt speech, and the timbre prompt speech may be sent by the terminal 101 to the server 102, and the server 102 may send the generated target speech to the terminal 101.

[0085] In addition, the above process of generating the target speech may also be performed only on the terminal 101 side.

[0086] Server 102 may be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN (Content Delivery Network), and big data and artificial intelligence platforms. In addition, server 102 may also be a node server in a blockchain network.

[0087] The terminal 101 may be a mobile phone, a computer, an intelligent voice interaction device, an intelligent wearable device, an intelligent home appliance, a vehicle-mounted terminal, etc., but is not limited thereto. The terminal 101 and the server 102 may be directly or indirectly connected via wired or wireless communication, which is not limited in the embodiments of the present disclosure.

[0088] For example, the speech generation method in the embodiment of the present disclosure is applicable to a variety of specific application scenarios, such as virtual human voice assistant, live chat robot, audio and video conference, system AI intelligent assistant, in-vehicle voice interaction system and other projects and product applications. Among them, virtual human voice assistant and live chat robot are used as examples for illustration:

[0089] (1) Virtual human voice assistant.

[0090] Reference Figure 2 , Figure 2An optional schematic diagram of the application of the speech generation method provided in the embodiment of the present disclosure in the scenario of a virtual human voice assistant is shown in the figure, which is a virtual human voice assistant named "Little Assistant", which is designed to provide a variety of services such as information query, schedule management, and entertainment interaction. By adopting the above-mentioned speech generation method, the target object can ask the "Little Assistant" "What is the weather like today", and after the "Little Assistant" receives the request, it extracts the content to be synthesized from the request and extracts the content embedding. Then, according to the preferences of the target object or the atmosphere of the current scene, the "Little Assistant" can select a suitable style prompt voice. In order to maintain the personalized characteristics of the "Little Assistant", it selects a specific timbre as the timbre prompt voice, and finally the target voice with a timbre similar to the timbre prompt voice and a style similar to the style prompt voice makes the "Little Assistant"'s reply more natural and real, enhances the target object's sense of immersion and trust, and makes the "Little Assistant" a more personalized voice assistant, so that it can interact with the target object more effectively and improve the user experience.

[0091] (2) Chatbot in the live broadcast room.

[0092] Reference Figure 3 , Figure 3 An optional schematic diagram of the application of the voice generation method provided by the embodiment of the present disclosure in the chat robot scenario of the live broadcast room. In the live broadcast room, the chat robot not only needs to respond quickly to the audience's questions, but also needs to communicate in a natural and attractive way. Using the above-mentioned voice generation method, the chat robot in the live broadcast room can reply with a voice similar to the style of the anchor or the style of the target object, which enhances the immersion and interactivity of the live broadcast room; the style prompt voice and timbre prompt voice can also be adjusted, and the live broadcast room can have a unique "sound logo" to enhance the brand image and memory point. And the voice generation method can quickly process a large amount of text input and convert it into natural and smooth voice output, ensuring that the chat robot can respond to the audience's questions immediately. Therefore, whether it is a formal product launch conference live broadcast or a relaxed entertainment live broadcast, the chat robot can adjust the voice style according to the scene to maintain consistency with the live broadcast atmosphere. In addition, during the peak period of audience questions, the chat robot can share part of the reply work, allowing the anchor to have more energy to focus on the presentation of the live broadcast content while maintaining interaction with the audience. Therefore, applying the above-mentioned speech generation method to the live broadcast chat robot not only improves the interactivity and personalization of the live broadcast room, but also improves the response efficiency of the robot, bringing a more natural and smooth live broadcast experience to the audience.

[0093] The method provided by the embodiments of the present disclosure can also be applied to different scenarios, including but not limited to cloud technology, artificial intelligence, smart transportation, assisted driving and the like.

[0094] Reference Figure 4 , Figure 4 An optional flow chart of a speech generation method provided in an embodiment of the present disclosure, the speech generation method can be executed by a server or a terminal alone, or by a terminal and a server in cooperation, and the speech generation method includes but is not limited to the following steps 401 to 405.

[0095] Step 401, obtaining content to be synthesized for content constraint during speech generation, and extracting content embedding of the content to be synthesized;

[0096] In the above step 401, the content to be synthesized is the text information, voice information or image information that is desired to be converted during the speech generation process, and is used to constrain the content during speech generation, and the content to be synthesized can be obtained through the input of the target object, or extracted from the currently available information. Among them, when the content to be synthesized is the text to be synthesized, it can be the text information that the target object wants to be read aloud or converted into speech, and these texts can be sentences, paragraphs, articles or any form of text content; when the content to be synthesized is the speech to be synthesized, it can be an existing voice segment, or a voice segment input by the target object; when the content to be synthesized is the image to be synthesized, it can be a picture input by the target object, or a video frame selected from the video input by the target object, or a live screen captured from the current live scene, etc.

[0097] Content embedding is to convert the content to be synthesized into a numerical representation or feature vector. By extracting the content embedding of the content to be synthesized, the features in the content to be synthesized can be mapped to a high-dimensional vector space. It should be noted that content embedding can be obtained by processing the content to be synthesized through a deep learning model, such as by embedding the content to be synthesized through a word embedding model (Word2Vec).

[0098] Step 402, obtaining style prompt speech for performing style constraints during speech generation, and extracting a plurality of original speech tags in the style prompt speech;

[0099] In the above step 402, the style hint speech can represent the style that the target object wants to imitate in the generated speech, and is used to impose style constraints during speech generation. This style can include the speaker's intonation, speaking speed, pronunciation characteristics, emotional expression, and other aspects, providing a specific style reference for speech generation.

[0100] The original speech marker is a kind of speech token. There are multiple original speech markers. They can be a series of acoustic features or speech units extracted from the style cue speech. These units provide stylistic references and materials for the subsequent speech generation process. It should be noted that the original speech marker is a discretized representation of the style cue speech. They can be any recognizable units in the style cue speech. These units can reflect the style characteristics of the style cue speech. These units can be phonemes, syllables, words or other acoustic features. The original speech marker can be obtained by processing the style cue speech through a deep learning model, such as by processing the style cue speech through a speech recognition model, or by extracting the style cue speech through an acoustic feature extraction algorithm, or by tokenizing the style cue speech through a speech tokenizer.

[0101] Step 403, inputting the content embedding and the multiple original speech tags into the first language model for prediction to obtain multiple target speech tags;

[0102] In the above step 403, the first large language model is a pre-trained large language model. In the embodiment of the present disclosure, content embedding and original speech tags can be used as inputs of the first large language model. These input information together constitute the context of model prediction. The first large language model predicts multiple target speech tags similar to the style of the style hint speech based on the input context information.

[0103] The target speech tag is also a speech token. There are multiple target speech tags, which are the output of the first language model. They represent speech features similar to the style of the style-cued speech. These tags are learned during the model training process and can capture the subtle differences in speech, such as intonation, speaking speed, pronunciation habits, etc. The introduction of target speech tags makes the speech generation process more flexible and controllable. In the embodiment of the present disclosure, by adjusting the input of the first language model, such as content embedding, style-cued speech, etc., speech with different content and style can be generated to meet the needs of various application scenarios.

[0104] Step 404, obtaining a timbre prompt speech for performing timbre constraint during speech generation, and extracting a first speaker embedding of the timbre prompt speech;

[0105] In the above step 404, the timbre prompt speech is a timbre sample that the target object hopes to imitate in speech generation. It can be a speech uttered by a specific speaker, which can characterize the unique timbre characteristics of the speaker to be imitated and provide a specific timbre reference for speech generation.

[0106] The first speaker embedding is a numerical representation or feature vector extracted from the timbre prompt speech, which captures the characteristic information of the speaker in the timbre prompt speech. It should be noted that the first speaker embedding can be obtained by processing the timbre prompt speech through a deep learning model, such as by processing the timbre prompt speech through a speaker recognition model, or by encoding the timbre prompt speech through a speaker encoder.

[0107] Step 405 , generating speech based on the first speaker embedding and multiple target speech tags to obtain target speech.

[0108] In the above step 405, the target speech is a speech signal finally generated by the speech generation method in the embodiment of the present disclosure, which is generated based on multiple factors such as the content to be synthesized, the style prompt speech and the timbre prompt speech provided by the target object, aiming to achieve flexible control of content, style and timbre. The target speech has style features similar to those of the style prompt speech, timbre features similar to those of the timbre prompt speech, and contains information in the content to be synthesized, so the target speech is more natural, realistic and flexible, and can meet the speech generation needs of the target object in different scenarios.

[0109] It should be noted that the disclosed embodiment can embed the first speaking object and multiple target speech tags as input features and input them into a pre-trained generative model or framework. The model will comprehensively consider these input features, including content, style, timbre, and acoustic environment, to generate high-quality speech signals and obtain the required target speech. The speech signal generated in this way is more natural and realistic, and can flexibly adapt to different content and style requirements.

[0110] Exemplarily, the disclosed embodiment may embed the first speaker and multiple target speech tags into a pre-trained flow matching module, but it should be noted that the flow matching module is only an example, and other conditionally generated artificial intelligence generated content (AIGC) models may be used in addition, such as generation models based on diffusion or other technologies, which are capable of generating high-quality speech signals based on input features.

[0111] In summary, by executing steps 401 to 405, the embodiment of the present disclosure can obtain the content to be synthesized for content constraint during speech generation, extract the content embedding of the content to be synthesized, and constrain the content in the target speech through the content embedding during speech generation. In addition, by obtaining the style prompt speech for style constraint during speech generation, extracting multiple original speech tags in the style prompt speech, inputting the content embedding and the multiple original speech tags into the first language model for prediction, multiple target speech tags similar to the style of the style prompt speech can be predicted based on the first language model. On this basis, by obtaining the style prompt speech for style constraint during speech generation, A timbre prompt speech with timbre constraints is performed, the first speaker embedding of the timbre prompt speech is extracted, and speech is generated based on the first speaker embedding and multiple target speech tags, so as to obtain a target speech with a timbre similar to that of the timbre prompt speech and a style similar to that of the style prompt speech, so as to achieve control of style and timbre, make the target speech more natural, thereby improving the authenticity of the target speech, and, because the content to be synthesized, the style prompt speech and the timbre prompt speech are respectively introduced at different stages of the generation process, the decoupling of the content, style and timbre of the target speech is achieved, and the flexible combination of any content, style and timbre is supported, thereby improving the flexibility of the target speech.

[0112] Above, the details of the overall process of the speech generation method in the embodiment of the present disclosure are introduced through steps 401 to 405. Below, the contents further included in the steps are described in detail.

[0113] In a possible implementation, the style hint speech may be retrieved from a preset knowledge base. The above-mentioned acquisition of the style hint speech for performing style constraints during speech generation may specifically include inputting the content to be synthesized into the second largest language model for prediction to obtain a first style description text for describing the style of the speaker of the content to be synthesized, and a first emotion description text for describing the emotions of each content segment in the content to be synthesized; constructing a first global embedding of the content segment based on the embedding of the first style description text, and constructing a first detail embedding of the content segment based on the embedding of the first emotion description text; and retrieving the style hint speech corresponding to the content segment from the preset knowledge base based on the first global embedding and the first detail embedding, wherein the style hint speech is used for performing style constraints during speech generation.

[0114] The knowledge base is a database or data set storing a variety of style prompt voices and related information, which can be used to retrieve the style prompt voices matching the input description text according to the input description text. It should be pointed out that the various style prompt voices pre-stored in the knowledge base can be set as needed, for example, it can include various types of style prompt voices, such as those with different types of intonation, speech speed, pronunciation characteristics, emotional expression, etc. Each style type can have multiple different instances, and these voice instances are marked or indexed so that one or more of the best matches can be quickly retrieved according to the input description text, and the embodiments of the present disclosure do not impose specific restrictions on this.

[0115] The second largest language model is a pre-trained large language model. In the embodiment of the present disclosure, the content to be synthesized can be used as the input of the second largest language model, and the content to be synthesized constitutes the context of model prediction. The second largest language model predicts a first style description text for describing the style of the speaker of the content to be synthesized and a first emotion description text for describing the emotions of each content segment in the content to be synthesized based on the input context information.

[0116] The first style description text is a text describing the style of the speaker of the content to be synthesized. It may contain vocabulary of various style types such as intonation, speaking speed, pronunciation characteristics, emotional expression, etc. These vocabulary reflect the overall speaking style or intonation adopted by the speaker. The first emotion description text is a text describing the emotions of each content segment in the content to be synthesized. The first emotion description text of different content segments may be different. The first emotion description text can specifically reflect the emotional state expressed in the content segment, such as happiness, sadness, surprise, anger, etc., and can provide more detailed guidance for the speech generation process.

[0117] The first global embedding is an embedding representation based on the first style description text, which captures the overall style characteristics of the speaker in the content to be synthesized. For example, the first style description text contains words or phrases describing the style of the speaker, such as "formal", "casual", "cheerful", etc. These texts are input into an embedding layer, such as a Word2Vec module, to generate corresponding embedding vectors, and then these embedding vectors are combined into a global embedding vector, namely the first global embedding.

[0118] The first detail embedding is an embedding representation based on the first emotion description text, which captures the specific emotion characteristics of each content segment in the content to be synthesized. A content segment is a segment in the content to be synthesized. For example, when the content to be synthesized is a text, the content segment can be a sentence in the content to be synthesized. Similar to the first global embedding, the first emotion description text is also input into the embedding layer to generate the corresponding embedding vector. However, unlike the global embedding, the detail embedding is generated separately for each content segment or emotion description, or combined into multiple detail embedding vectors through sequence models, attention mechanisms, etc., to reflect the emotion changes in different parts of the content, and finally the first detail embedding of each content segment can be obtained.

[0119] It should be noted that when retrieving style-cue speech, the first global embedding and the first detail embedding are used as query vectors in the disclosed embodiment to search the knowledge base for speech instances that best match these query vectors, such as matching based on the similarity between the embedding vector and each speech instance, or matching after scoring each speech instance based on the embedding vector. Ultimately, by comprehensively considering the global style and detail emotion information, the disclosed embodiment can retrieve the style-cue speech that best matches the content to be synthesized in terms of both overall style and specific emotion, so as to perform style constraints during speech generation.

[0120] Furthermore, the knowledge base stores a reference speech segment and a second global embedding and a second detail embedding associated with the reference speech segment, the second global embedding is used to indicate the speaking style of the reference speech segment, and the second detail embedding is used to indicate the emotion of the reference speech segment.

[0121] The reference speech segments are speech sample instances stored in a preset knowledge base, and these speech samples may have different speaking styles and emotions. In the disclosed embodiment, the reference speech segments are used as candidates for style-cued speech so as to be selected according to the style requirements of the content to be synthesized.

[0122] The second global embedding is an embedding vector associated with the reference speech segment, which is used to indicate the speaker style of the reference speech segment. It should be noted that in the knowledge base, each reference speech segment is associated with a second global embedding, which captures the overall style characteristics of the speech segment, and multiple reference speech segments can be associated with the same second global embedding. The second detail embedding is another embedding vector associated with the reference speech segment, which is used to indicate the emotion of the reference speech segment. Similar to the second global embedding, each reference speech segment is associated with a second detail embedding, which captures the emotional characteristics in the speech segment. At this time, a reference speech segment can correspond to a content segment.

[0123] In a possible implementation, the style prompt speech corresponding to the content segment is retrieved from a preset knowledge base based on the first global embedding and the first detail embedding. Specifically, according to the similarity between the first global embedding and the second global embedding, multiple reference speech segments are retrieved from the knowledge base; according to the similarity between the first detail embedding and the second detail embedding, the style prompt speech corresponding to the content segment is retrieved from multiple reference speech segments.

[0124] It should be noted that the disclosed embodiment can retrieve multiple reference speech segments similar in style to the content to be synthesized by calculating the similarity between the first global embedding and the second global embedding of each reference speech segment in the knowledge base, such as cosine similarity, Euclidean distance, etc. These reference speech segments match the requirements of the content to be synthesized in terms of the style of the speaking object. Subsequently, among the multiple reference speech segments retrieved in the first step, by calculating the similarity between the first detail embedding and the second detail embedding of each reference speech segment, reference speech segments that match the emotional requirements of the content to be synthesized can be further screened out, ensuring that the style cue speech finally selected not only matches the content to be synthesized in terms of the speaking object style, but also matches the requirements of the content to be synthesized in terms of emotional details, thereby retrieving the style cue speech more accurately and finely.

[0125] Exemplarily, in the disclosed embodiment, after calculating the similarity between the first global embedding and the second global embedding of each reference speech segment in the knowledge base, the first similarity value between the current first global embedding and each second global embedding can be obtained. Further, the disclosed embodiment can sort the various first similarity values, and select the first M first similarity values ​​in order from large to small, and use the reference speech segments to which the second global features corresponding to the first M first similarity values ​​belong as multiple reference speech segments similar in style to the content to be synthesized, where the size of M can be set according to actual needs; or, the disclosed embodiment can set a first similarity threshold, and select a first similarity value greater than or equal to the first similarity threshold, and use the reference speech segments to which the second global features corresponding to these first similarity values ​​belong as multiple reference speech segments similar in style to the content to be synthesized, where the size of the first similarity threshold can be set according to actual needs.

[0126] Similarly, in the embodiment of the present disclosure, after calculating the similarity between the first detail embedding and the second detail embedding of the multiple reference speech segments retrieved in the first step, the second similarity value between the current first detail embedding and each second detail embedding can be obtained. Further, the embodiment of the present disclosure can sort the various second similarity values, and select the first K second similarity values ​​in order from large to small, and use the reference speech segments to which the second detail embeddings corresponding to the first K second similarity values ​​belong as the style prompt speech corresponding to the content segment, where the size of K can be set according to actual needs; or, the embodiment of the present disclosure can set a second similarity threshold, and select a second similarity value greater than or equal to the second similarity threshold, and use the reference speech segments to which the second detail features corresponding to these second similarity values ​​belong as the style prompt speech corresponding to the content segment, where the size of the second similarity threshold can be set according to actual needs.

[0127] In a possible implementation, before retrieving the style cue speech corresponding to the content segment from the preset knowledge base based on the first global embedding and the first detail embedding, the disclosed embodiment may also obtain the original speech, segment the original speech into speaking objects, and obtain multiple speech blocks; perform speech recognition on each speech block respectively to obtain the speech block text in each speech block; segment the speech block text to obtain multiple reference text segments, and segment the speech blocks into reference speech segments corresponding to each reference text segment; construct a second global embedding based on the multiple speech block texts, construct a second detail embedding based on the reference text segments, and store the reference speech segments, the second global embedding, and the second detail embedding in association with each other in the knowledge base.

[0128] The original speech is a piece of speech data to be processed, which may include dialogues, monologues or speeches of multiple speakers. The original speech may be obtained from a recording file, an audio stream, etc., or may be obtained based on the input of a target object. After obtaining the original speech, the embodiment of the present disclosure may segment the original speech containing multiple speakers into multiple independent speech blocks, each of which contains only the speech of one speaker. In addition, the embodiment of the present disclosure may also perform other audio processing on the original speech, including noise reduction, scoring and screening, etc., to improve the quality of the segmented speech blocks.

[0129] After obtaining each speech block, the embodiment of the present disclosure can perform speech recognition on each speech block respectively. In the speech recognition process, the audio features of each speech block can be analyzed to extract the text content contained therein, and each speech block can be converted into a text corresponding to the speech audio contained therein, and finally the speech block text in each speech block is obtained. Further, in the embodiment of the present disclosure, speech recognition of each speech block can be realized based on a preset speech recognition model, and the embodiment of the present disclosure does not make any specific restrictions on this.

[0130] After obtaining the text of each speech block, the embodiment of the present disclosure can segment each speech block text respectively, and split the speech block text containing multiple sentences or paragraphs into smaller text units, so as to obtain multiple reference text segments, each of which can represent a sentence or a paragraph, or each reference text segment can represent at least two of multiple sentences or multiple paragraphs. Similarly, the embodiment of the present disclosure can segment each speech block respectively to obtain reference speech segments corresponding to each reference text segment.

[0131] Finally, for multiple speech block texts, the embodiment of the present disclosure can construct a corresponding second global embedding through word embedding or sentence embedding to capture the overall semantic information of the speech block text through the constructed second global embedding. In addition, a more fine-grained analysis is performed on the speech block text, and a similar method is used to construct a detail embedding for each reference text segment to capture the local semantic information of the text segment through the constructed second detail embedding. Subsequently, the embodiment of the present disclosure associates each reference speech segment with its corresponding second global embedding and second detail embedding, and stores them in a knowledge base, so that each reference speech segment in the knowledge base has a second global embedding and a second detail embedding corresponding to it, so as to be retrieved and used in the subsequent speech generation process.

[0132] The above embodiments are described below with reference to the accompanying drawings.

[0133] Reference Figure 5 , Figure 5An optional schematic diagram of the raw data processing process provided by the embodiment of the present disclosure. In this embodiment, after obtaining the original speech, the original speech can be subjected to noise reduction processing, and the speech can be subjected to language recognition after noise reduction to determine the language to which it belongs, so as to facilitate subsequent text recognition, and the speech can be subjected to speaker diarization processing after language recognition. The original speech can be segmented by the speaker diarization processing, and the start time and end time of each speech segment in the original speech can be determined to obtain the speech blocks corresponding to each speaker ID after segmentation. Subsequently, this embodiment can cut each speech block based on voice activity detection (VAD), and the score of each speech block can be obtained based on VAD, and the speech block with a score higher than the preset score can be selected for subsequent processing. Next, this embodiment obtains the text data corresponding to each speech block through automatic speech recognition (ASR) of different languages, such as obtaining the speech block text under Chinese recognition or English recognition, and saves the speech block text into json structured data after word error rate (WER) screening.

[0134] Next, this embodiment divides (Chunking) the speech block text and the corresponding speech corpus into smaller blocks, obtains multiple reference text segments and reference speech segments corresponding to each reference text segment, and finally constructs a second global embedding based on multiple speech block texts, constructs a second detail embedding based on the reference text segment, and stores the reference speech segment, the second global embedding and the second detail embedding in association to the knowledge base. Among them, Chunking is particularly important for improving the accuracy of emotional style retrieval, refining the retrieval granularity and avoiding length problems. Further, in this embodiment, there are three different levels of text segmentation methods for Chunking based on text. For example, each speech block text can be marked by a tag-level segmentation method, and then segmented according to the marking result, which is simple and clear; or, a semantic-level segmentation method can be used to determine the breakpoints of the speech block text using a large language model obtained by pre-training, so that more context can be combined for segmentation; or, a sentence-level segmentation method can be used to divide the corresponding segments of each sentence, so as to achieve a balance between retaining text semantics and being concise and efficient.

[0135] In a possible implementation, the second global embedding is constructed based on multiple speech block texts, and the second detail embedding is constructed based on a reference text segment. Specifically, a first prompt text is constructed based on multiple speech block texts, and the first prompt text is input into a second largest language model for prediction to obtain a second style description text for describing the style of the speaking object of the multiple speech block texts, wherein the first prompt text is used to prompt the second largest language model to output a second style description text based on the multiple speech block texts; a second global embedding is constructed based on the embedding of the second style description text; a second prompt text is constructed based on the second style description text, a reference text segment, and multiple speech block texts, and the second prompt text is input into a second largest language model for prediction to obtain a second emotion description text for describing the emotion of the reference text segment, wherein the second prompt text is used to prompt the second largest language model to output a second emotion description text based on the second style description text, the reference text segment, and multiple speech block texts; and a second detail embedding is constructed based on the embedding of the second emotion description text.

[0136] Among them, the first prompt text is a prompt text constructed based on multiple voice block texts, which is used to prompt the second largest language model to generate a text describing the style of the speaker based on the multiple voice block texts. Therefore, the first prompt text may contain some guiding vocabulary or sentence structures to guide the second largest language model to generate.

[0137] After obtaining the first prompt text, the embodiment of the present disclosure can input the first prompt text and the multiple speech block texts into the trained second largest language model, and the second largest language model can generate a text describing the style of the speaker of the multiple speech block texts, that is, the second style description text. Finally, the embodiment of the present disclosure converts the second style description text into an embedding form, and this embedding is the second global embedding representing the overall style of this group of speech blocks.

[0138] The second prompt text is similar to the first prompt text. It is a prompt text constructed based on the second style description text, the reference text fragment and multiple voice block texts. It is used to prompt the second largest language model to generate a text describing the emotion of the reference text fragment based on the second style description text, the reference text fragment and the original voice block text. Therefore, the second prompt text may also contain some guiding vocabulary or sentence structures.

[0139] After obtaining the second prompt text, the disclosed embodiment can input the second prompt text, the second style description text, the reference text segment and the original speech block text into the second largest language model. The second largest language model generates a text describing the emotion of the reference text segment based on the input second prompt text and other relevant information, that is, the second emotion description text. Finally, the disclosed embodiment converts the second emotion description text into an embedded form, which is the second detail embedding representing the specific emotion of the reference text segment.

[0140] Through the above steps, each reference speech clip in the knowledge base is assigned a second global embedding and a second detail embedding. These embedding information will be used to retrieve and match the style cue speech in the subsequent speech generation process, thereby ensuring that the generated speech has the desired style and emotional characteristics.

[0141] The first prompt text and the second prompt text in the above embodiment are illustrated below:

[0142] Reference Figure 6 , Figure 6 An optional schematic diagram of the first prompt text and the second prompt text provided for the embodiment of the present disclosure. In this embodiment, in addition to including multiple voice block texts, the first prompt text may also include some guiding vocabulary or sentence structures. For example, the first prompt text may be "Given the following conversation between the given speaker {conversation text content} Based on the above conversation, what kind of style and personality do you think the speaker {speakerID} has? (Note that the answer is limited to 150 words.)", where the conversation text content is multiple voice block texts, and speakerID is the corresponding speaker ID. Similarly, in addition to the second style description text, the reference text fragment and multiple voice block texts, the second prompt text may also include some guiding vocabulary or sentence structures. For example, the second prompt text may be "system: You are an expert in analyzing the sentence emotions of the speaker in the conversation. Given the personality style description of the speaker {speaker ID}: {speaker persona}; given the following conversation as the context {conversation}", "user: Based on the above conversation and the speaker's personality style description, what is the emotional label of the sentence {utterance}?" and "assistant: The emotional label of the sentence {utterance} is {emotion}", where speaker persona is the second style description text, conversation is multiple voice block texts, utterance is a reference text fragment, and emotion is the second emotion description text to be output.

[0143] In addition, there are other ways to construct a second global embedding based on multiple speech block texts, and to construct a second detail embedding based on a reference text segment. Furthermore, the disclosed embodiment can also directly extract the corresponding second global embedding and second detail embedding based on the original speech. For example, the disclosed embodiment can use audio feature extraction technology to capture the acoustic characteristics of the original speech to reflect the style characteristics of the speaker, such as the sound quality, speaking speed, and intonation, and then input the extracted features into a pre-trained neural network model, which can be a convolutional neural network (CNN), a recurrent neural network (RNN), or a Transformer, etc. The task of the model is to map these features into a low-dimensional embedding space. The vectors in this embedding space can represent the style of the speaker. Through training, the model can learn the difference between the styles of different speakers and generate the corresponding second global embedding. Similarly, in addition to the above-mentioned acoustic features, the embodiments of the present disclosure can also extract emotion-related features, such as rhythmic changes, timbre changes, pauses and stresses of speech, and generate corresponding second detail embeddings based on the extracted features. The generation process of the second detail embedding is similar to that of the second global embedding, and the embodiments of the present disclosure do not make specific limitations here.

[0144] In a possible implementation, the above-mentioned method of retrieving the style cue speech corresponding to the content segment from multiple reference speech segments based on the similarity between the first detail embedding and the second detail embedding may specifically be retrieving multiple candidate speech segments from multiple reference speech segments based on the similarity between the first detail embedding and the second detail embedding; determining the context relevance between each candidate speech segment and the content segment, and determining the candidate speech segment with the highest context relevance as the style cue speech corresponding to the content segment, or, again determining the similarity between the second global embedding and the first global embedding of each candidate speech segment, and determining the candidate speech segment with the highest similarity as the style cue speech corresponding to the content segment, or, performing speech quality evaluation on each candidate speech segment, and determining the candidate speech segment with the highest speech quality as the style cue speech corresponding to the content segment.

[0145] Furthermore, after retrieving multiple reference speech segments from the knowledge base according to the similarity between the first global embedding and the second global embedding, the disclosed embodiment can also retrieve multiple candidate speech segments from the multiple reference speech segments according to the similarity between the first detail embedding and the second detail embedding. Specifically, for each reference speech segment in the knowledge base, the disclosed embodiment can calculate the similarity between its second detail embedding and the first detail embedding of the content to be synthesized, and then set a similarity threshold, and regard those reference speech segments whose similarity exceeds the threshold as candidate speech segments, or, sort all reference speech segments from high to low according to similarity, and select the top N as candidate speech segments.

[0146] Next, there are many ways to select the style prompt voice corresponding to the content segment from multiple candidate voice segments, for example:

[0147] The disclosed embodiments can determine the contextual relevance between each candidate voice segment and the content segment, so as to better understand the contextual relevance between the candidate voice segment and the content segment, and better understand the subtle differences in context, thereby providing higher sorting accuracy. For example, text analysis is performed on the content segment to extract key information, themes or emotions. Similarly, a similar text analysis is performed on each candidate voice segment to obtain its semantic content and emotional expression, and then the semantic similarity or contextual coherence between the content segment and the candidate voice segment is evaluated. Subsequently, based on the calculation result of the contextual relevance, the candidate voice segment with the highest relevance is selected as the style cue speech corresponding to the content segment, thereby ensuring that the style cue speech not only matches the style of the content to be synthesized, but also maintains consistency in semantics and context.

[0148] Alternatively, the disclosed embodiment may further determine the similarity between the second global embedding and the first global embedding of each candidate speech segment, perform global consistency rearrangement, and select the candidate speech segment with the highest similarity as the style cue speech based on the similarity calculation result, thereby ensuring that the style of the speaker of the style cue speech is highly consistent with the style of the speaker of the content to be synthesized.

[0149] Alternatively, the disclosed embodiment may also perform voice quality assessment on each candidate voice segment, and use a voice quality assessment tool or model to assess the voice quality of each candidate voice segment, which involves analyzing the acoustic features of the voice segment, such as spectrum, fundamental frequency, noise level, etc. For example, a WER assessment may be performed on each candidate voice segment to obtain a voice quality assessment result of each candidate voice segment. Subsequently, the candidate voice segment with the highest voice quality is determined as the style cue voice corresponding to the content segment, thereby ensuring that the style cue voice is optimal in terms of voice quality, thereby helping to improve the authenticity and naturalness of the target voice.

[0150] In a possible implementation, the above-mentioned construction of the first global embedding of the content fragment based on the embedding of the first style description text may specifically be obtaining a first relevant description text for supplementary description when retrieving the style prompt voice; fusing the embedding of the first style description text and the embedding of the first relevant description text to obtain the first global embedding of the content fragment.

[0151] Among them, the first relevant description text is a text used to provide additional information or supplementary description when retrieving style hint speech, which can further enhance the understanding of the style of the speaker and help more accurately locate the style hint speech that matches the content to be synthesized. Exemplarily, the first relevant description text may include attributes describing the speaker, such as the region, education level, etc. of the speaker, which have an important impact on determining the speaking style, intonation, word choice, etc. of the speaker. In addition, the first relevant description text may also include relevant information such as the application scenario of the speaker, the gender of the speaker, style emotion, age, region, etc., which is not specifically limited in the embodiments of the present disclosure.

[0152] After obtaining the first relevant description text, the embodiment of the present disclosure can fuse the embedding of the first style description text and the embedding of the first relevant description text to obtain a first global embedding of the content segment. Specifically, the first style description text and the first relevant description text can be converted into an embedding representation, and then the two embedding representations can be merged into a global embedding to obtain the required first global embedding.

[0153] Exemplarily, there are multiple ways to fuse the embedding of the first style description text and the embedding of the first related description text. For example, the embedding of the first style description text and the embedding of the first related description text can be averaged and fused, and the two embedding vectors can be added or averaged to obtain the first global embedding; or, the embedding of the first style description text and the embedding of the first related description text can be weighted fused, including giving different weights according to the importance of each embedding, and then summing them to obtain the required first global embedding; or, the embedding of the first style description text and the embedding of the first related description text can be spliced ​​and fused, and the two embedding vectors can be spliced ​​into a longer vector to obtain the required first global embedding; or, the weights of the embedding of the first style description text and the embedding of the first related description text can be determined through a pre-built attention mechanism, and then summed to obtain the required first global embedding.

[0154] In a possible implementation, the first detail embedding of the content fragment is constructed based on the embedding of the first emotion description text, and specifically, the first global embedding and the embedding of the first emotion description text are integrated to obtain the first detail embedding of the content fragment.

[0155] It should be noted that the embodiment of the present disclosure can fuse the first global embedding and the embedding of the first emotion description text, combine the two embedding representations to form a new embedding representation, obtain the first detail embedding of the content segment, and thus construct an embedding representation that includes both the overall style of the speaker and the emotional details of the content segment, which helps to more accurately simulate the style and emotional state of the speaker in the speech generation process, thereby generating more natural and realistic speech. Among them, the fusion process may include but is not limited to vector concatenation, weighted averaging, attention mechanism, etc., aiming to effectively combine the information in the two embeddings, and the embodiment of the present disclosure does not make specific restrictions on this.

[0156] In addition, in the embodiment of the present disclosure, the embedding of the first emotion description text can also be directly embedded as the first detail embedding of the content segment. It should be noted that when the embedding of the first emotion description text is directly embedded as the first detail embedding, it means that when retrieving the style cue speech, the main basis is the emotion description of the content segment, and the overall style characteristics of the speaker are no longer considered. Therefore, the embodiment of the present disclosure is suitable for those scenes where the emotion expression dominates the speech style, or when the overall style characteristics of the speaker have little effect on the speech generation.

[0157] In a possible implementation, the content to be synthesized is speech to be synthesized, and the content segment is a speech segment. The first detail embedding of the content segment is constructed based on the embedding of the first emotion description text. Specifically, the speech emotion embedding of the speech segment can be extracted; the first global embedding, the embedding of the first emotion description text and the speech emotion embedding are integrated to obtain the first detail embedding of the content segment.

[0158] Furthermore, when the content to be synthesized is speech to be synthesized and the content segment is a speech segment, the embodiment of the present disclosure can extract speech emotion embedding of the speech segment. The speech emotion embedding refers to an embedding representation extracted from the speech segment that can reflect its emotional characteristics. The embedding can be generated by analyzing the intonation, rhythm, volume and other features in the speech segment to capture the emotional information in the speech.

[0159] After obtaining the speech emotion embedding of the speech segment, the embodiment of the present disclosure can fuse the first global embedding, the embedding of the first emotion description text and the speech emotion embedding, and can obtain a more comprehensive and accurate first detail embedding of the content segment, so that the first detail embedding not only contains the overall style characteristics of the speaker, but also contains the emotional details in the content segment and the emotional information in the speech, so as to provide a richer and more accurate reference when retrieving the style hint speech from the knowledge base later. Among them, the fusion process can include but is not limited to vector concatenation, weighted average, attention mechanism, etc., aiming to effectively combine the information in the first global embedding, the embedding of the first emotion description text and the speech emotion embedding, and the embodiment of the present disclosure does not make specific restrictions on this.

[0160] In addition, if the content to be synthesized is an image to be contracted, the technical means for constructing the first detail embedding of the content segment in the above embodiment is also applicable. Specifically, when the content to be synthesized is an image to be contracted and the content segment is an image block, the embodiment of the present disclosure can extract the image emotion embedding of the image block. The image emotion embedding refers to the embedding representation extracted from the image block that can reflect its emotional characteristics. The embedding can be generated by analyzing the color, texture, character characteristics or environmental characteristics of the image block to capture the emotional information in the image, and then the first global embedding, the embedding of the first emotional description text and the image emotion embedding are fused to obtain the first detail embedding of the content segment, which will not be described in detail in the embodiment of the present disclosure.

[0161] It can be seen that in the disclosed embodiments, the first detail embedding may be of multiple types depending on the modality of the content to be synthesized. When the content to be synthesized is a text to be synthesized, the first global embedding and the embedding of the first emotion description text are integrated to obtain the first detail embedding of the content segment. When the content to be synthesized is a speech to be synthesized, the first global embedding, the embedding of the first emotion description text (the text obtained after speech recognition of the speech to be synthesized) and the speech emotion embedding are integrated to obtain the first detail embedding of the content segment. When the content to be synthesized is an image to be synthesized, the first global embedding, the embedding of the first emotion description text (the text obtained after image recognition of the image to be synthesized) and the image emotion embedding are integrated to obtain the first detail embedding of the content segment. Accordingly, different types of second detail embeddings can be associated with reference speech segments in the knowledge base, stored in different partitions, and the second detail embeddings can be retrieved from the corresponding partitions according to the specific modality of the content to be synthesized.

[0162] In a possible implementation, the style prompt voice corresponding to the content fragment is retrieved from a preset knowledge base based on the first global embedding and the first detail embedding. Specifically, the first global embedding and the first detail embedding can be input into a classifier for classification to obtain a style emotion classification result of the content fragment, wherein the style emotion classification result is used to indicate that the style emotion of the content fragment is neutral, or to indicate that the style emotion of the content fragment is rich; when the style emotion classification result indicates that the style emotion of the content fragment is rich, the style prompt voice corresponding to the content fragment is retrieved from the preset knowledge base based on the first global embedding and the first detail embedding.

[0163] Among them, the classifier is a machine learning or deep learning model, and its main task is to assign input data to predefined categories. In the embodiment of the present disclosure, the classifier is used to analyze the first global embedding and the first detail embedding of the content segment, and output a style sentiment classification result, which indicates whether the style sentiment of the content segment is neutral (insufficient) or rich (sufficient). Neutral style sentiment means that the content segment has no obvious characteristics or emotional expression in style, while rich style sentiment indicates that the content segment has rich characteristics or emotional expression in style.

[0164] It should be noted that the classifier can be one of multiple types of machine learning models, including but not limited to support vector machines, decision trees, random forests, neural networks, and their variants, which are not specifically limited in the embodiments of the present disclosure. During the training phase, the classifier is trained using a large amount of labeled data, which contains embedded representations of content segments and their corresponding style and emotion categories. Through training, the classifier learns how to embed and map new content segments to the correct style and emotion categories.

[0165] When the style emotion classification result indicates that the style emotion of the content segment is rich, the embodiment of the present disclosure needs to perform subsequent retrieval for the expression of the style emotion. Based on this, the embodiment of the present disclosure uses the first global embedding and the first detail embedding as retrieval conditions to search in a preset knowledge base, and finds the style prompt speech that best matches these embedding representations in the knowledge base, and uses it in the subsequent speech generation step to ensure that the generated speech matches the content segment in style and emotion.

[0166] In a possible implementation, based on the first global embedding and the first detail embedding, the style prompt voice corresponding to the content clip is retrieved from a preset knowledge base. Specifically, when the style emotion classification result indicates that the style emotion of the content clip is neutral, based on the first global embedding, the style prompt voice corresponding to the content clip is retrieved from the knowledge base.

[0167] Furthermore, when the style sentiment classification result indicates that the style sentiment of the content segment is neutral, it means that the content does not have particularly significant or diverse expressions in style or sentiment. In this case, because the first global embedding summarizes the overall style of the speaker, the process of retrieving the style hint speech will mainly rely on the first global embedding. Based on this, the embodiment of the present disclosure uses the first global embedding as a retrieval condition, searches in a preset knowledge base, and finally retrieves the style hint speech corresponding to the content segment, thereby ensuring that the generated speech is consistent with the overall style of the content to be synthesized in style, while avoiding the introduction of unnecessary emotional diversity.

[0168] However, not all the content to be synthesized needs to retrieve the style hint speech, especially when the style emotion of the content to be synthesized is classified as neutral. In this case, because the content itself does not have a strong style or emotional tendency, the disclosed embodiment can be considered that using any style hint speech in the preset knowledge base will not have much impact on the generated speech, so the additional step of retrieving a specific style hint speech is unnecessary, or because the system has a default style setting, speech can be generated without a specific style hint. Therefore, in the disclosed embodiment, when the style emotion classification result indicates that the style emotion of the content segment is neutral, the style hint speech is no longer retrieved, thereby improving the efficiency of speech generation.

[0169] In a possible implementation, the content embedding and the multiple original speech tags are input into the first large language model for prediction to obtain multiple target speech tags. Specifically, the second speaking object embedding of the style cue speech can be extracted, and the content embedding, the second speaking object embedding and the multiple original speech tags are input into the first large language model for prediction to obtain multiple target speech tags.

[0170] The second speaker embedding is a numerical representation or feature vector extracted from the style cue speech, which captures the characteristic information of the speaker in the style cue speech. It should be noted that the second speaker embedding can be obtained by processing the style cue speech through a deep learning model, such as by processing the style cue speech through a speaker recognition model, or by encoding the style cue speech through a speaker encoder.

[0171] It should be noted that the disclosed embodiment extracts the second speaking object embedding of the style cue speech and multiple original speech tags in the style cue speech, and inputs the content embedding, the second speaking object embedding and the multiple original speech tags into the first large language model for prediction. It is able to predict multiple target speech tags similar to the style of the style cue speech based on the first large language model, and due to the addition of the second speaking object embedding, it is possible to further predict multiple target speech tags that are more similar to the style of the style cue speech.

[0172] In a possible implementation, the content embedding, the second speaking object embedding, and the multiple original speech tags are input into the first large language model for prediction to obtain multiple target speech tags. Specifically, the sequence start tag, the second speaking object embedding, the content embedding, the speech conversion tag, and the multiple original speech tags may be sequentially concatenated and then input into the first large language model for prediction to obtain multiple target speech tags.

[0173] The sequence start marker is a special marker used to tell the first language model that the input sequence has started, helping the model to identify the starting point of the input sequence and better process the input data. The speech conversion marker is also a special marker used to instruct the first language model to convert from content embedding to speech tag generation, helping the model understand that the current task is the conversion from text content to speech tags.

[0174] In the disclosed embodiment, a sequence start marker, a second speaker embedding, a content embedding, a speech conversion marker, and a plurality of original speech markers may be sequentially concatenated to form a complete input sequence, which is then input into the first large language model for prediction. The first large language model predicts a series of target speech markers based on the input sequence in combination with its pre-trained knowledge and the current task, so that they can be used in subsequent speech generation steps, and ultimately generate target speech that meets the requirements.

[0175] In a possible implementation, the target speech is generated based on the first speaking object embedding and the multiple target speech tags. Specifically, the acoustic features of the style prompt speech or the timbre prompt speech are extracted, and the target speech is generated based on the acoustic features, the first speaking object embedding and the multiple target speech tags.

[0176] Among them, the acoustic feature is used to simulate the acoustic environment of the style prompt speech or the timbre prompt speech. Exemplarily, the acoustic feature may include a masked Mel feature or a Mel frequency feature. Among them, the masked Mel feature is a Mel spectrum feature processed by a masking operation, which can convert the style prompt speech or the timbre prompt speech into a representation method in the frequency domain to simulate different acoustic environments or emphasize specific sound features; the Mel frequency feature can map the spectrum of the style prompt speech or the timbre prompt speech to the Mel frequency scale, and then extract the coefficients of the spectrum envelope, so the spectral information of the style prompt speech or the timbre prompt speech can be captured through the Mel frequency feature. Furthermore, the embodiments of the present disclosure will be described later using the acoustic feature as a masked Mel feature, and the embodiments of the present disclosure will not be specifically limited to this.

[0177] The acoustic features in the embodiments of the present disclosure are extracted from the style prompt speech or the timbre prompt speech. Among them, by extracting the acoustic features of the style prompt speech, the acoustic environment of the style prompt speech can be simulated, and these features are helpful to generate a speech with a style similar to that of the style prompt speech; by extracting the acoustic features of the timbre prompt speech, the acoustic environment of the timbre prompt speech can be simulated, and these features are helpful to generate a speech with a timbre similar to that of the timbre prompt speech. It should be noted that in actual applications, the target object can choose to generate a speech that has both a specific style and a specific timbre. Therefore, the embodiments of the present disclosure allow the target object to control these two aspects separately by providing style prompt speech and timbre prompt speech. Furthermore, the acoustic features can also be extracted from the style prompt speech and the timbre prompt speech, such as extracting the sub-acoustic features of the style prompt speech or the timbre prompt speech respectively, and then obtaining the required acoustic features based on the fusion of these two sub-acoustic features. The embodiments of the present disclosure do not make specific restrictions on this.

[0178] It should be noted that, in the embodiments of the present disclosure, by extracting the acoustic features of the style prompt speech or the timbre prompt speech, the acoustic environment of the style prompt speech or the timbre prompt speech can be simulated through the acoustic features, and then speech generation is performed based on the acoustic features, the first speaking object embedding and multiple target speech markers, so as to obtain a target speech with a timbre similar to that of the timbre prompt speech and a style similar to that of the style prompt speech, thereby achieving control of style and timbre.

[0179] In one possible implementation, the style prompt voice or timbre prompt voice can also be subjected to a quality test on the acoustic environment. When the quality test result shows that the acoustic environment quality is poor (for example, loud reverberation and noise), the recording function can be called to obtain the current near-end voice to extract acoustic features, or a pre-stored template voice can be read to extract acoustic features, etc., thereby improving the stability and reliability of acoustic feature extraction.

[0180] The speech generation method in the embodiment of the present disclosure is described below with examples in conjunction with specific embodiments.

[0181] Reference Figure 7 , Figure 7 An optional schematic diagram of the overall technical architecture of the speech generation method provided in the embodiment of the present disclosure. As shown in the figure, the speech generation model architecture in the embodiment of the present disclosure may include three parts, namely, a retrieval knowledge base module, a style retrieval module, and a speech generation module based on a large language model. The style retrieval module may also be referred to as a retrieval augmented generation (RAG) style retrieval module. Among them, the retrieval knowledge base module is responsible for constructing a multimodal knowledge base based on text and speech corpus materials, and the knowledge base contains style and emotional information that can be used for retrieval and corresponding speech and text data; the RAG style retrieval module is responsible for analyzing the style and emotional information according to the input content to be synthesized, retrieving the most relevant speech and text data from the multimodal knowledge base, and arranging the output as the input prompt of the next module; the speech generation module based on the large language model is deployed with a first large language model, which is responsible for generating a target speech that conforms to the style and timbre based on the style prompts, the content to be synthesized, and the timbre prompts from the RAG style retrieval module.

[0182] Next, we will further describe the implementation and processing details of the specific technologies of the three modules. Figure 8 , Figure 8 An optional schematic diagram of the internal processing process of the overall technical architecture of the speech generation method provided in the embodiment of the present disclosure.

[0183] Among them, in the retrieval knowledge base module, when constructing a multimodal knowledge base, it is necessary to decide how to pre-process and segment the original speech into blocks, select which embedded semantic representations to use, and select a suitable vector database to efficiently store embedded vectors. Specifically, after obtaining the original speech, the original speech can be subjected to noise reduction processing, and the speech can be subjected to language recognition after noise reduction to determine the language in which it is located, so as to facilitate subsequent text recognition, and the speech can be subjected to speaker diarization processing after language recognition. The original speech can be segmented by the speaker diarization processing, and the start time and end time of each speech segment in the original speech can be determined to obtain the speech blocks corresponding to each speaker ID after segmentation. Subsequently, this embodiment can cut each speech block based on voice activity detection (VAD), and the score of each speech block can be obtained based on VAD, and the speech block with a score higher than the preset score can be selected for subsequent processing. Next, this embodiment obtains text data corresponding to each speech block through automatic speech recognition (Automatic Speech Recognition, ASR) in different languages, such as obtaining speech block text in Chinese or English, and saves it into json structured data after word error rate (Word Error Rate, WER) screening.

[0184] Next, this embodiment divides (Chunking) the speech block text and the corresponding speech corpus into smaller blocks, and obtains multiple reference text segments and reference speech segments corresponding to each reference text segment. Among them, Chunking is particularly important for improving the accuracy of emotional style retrieval, refining the retrieval granularity and avoiding length problems. Furthermore, in this embodiment, there are three different levels of text chunking methods based on text chunking. For example, each speech block text can be marked by a tag-level chunking method, and then chunked according to the marking result, which is simple and clear; or, a semantic-level chunking method can be used to use a large language model obtained by pre-training to determine the breakpoints of the speech block text, so that it can be better combined with the context for chunking; or, a sentence-level chunking method can be used to divide the corresponding chunks of each sentence, so as to achieve a balance between retaining the text semantics and being concise and efficient.

[0185] In the process of constructing the second global embedding based on multiple speech block texts and constructing the second detail embedding based on the reference text fragment, in order to extract the embedding semantics that can represent style and emotion information, this embodiment fine-tunes a model called PersoERC based on LLM and uses it as the second largest language model. It uses the context of the dialogue text block to perform style profiling on each speaker, and then combines the style profiling to perform emotion recognition on each sentence spoken by the speaker. Specifically, this embodiment constructs a first prompt text based on multiple speech block texts, inputs the first prompt text into the second largest language model for prediction, obtains a second style description text for describing the style of the speaker of the multiple speech block texts, and constructs a second global embedding based on the embedding of the second style description text; and also constructs a second prompt text based on the second style description text, the reference text fragment and the multiple speech block texts, inputs the second prompt text into the second largest language model for prediction, obtains a second emotion description text for describing the emotion of the reference text fragment, and constructs a second detail embedding based on the embedding of the second emotion description text, including using emotion2vec to extract the embedding of the second emotion description text and use it as the second detail embedding. Finally, the reference speech fragment, the second global embedding and the second detail embedding are associated and stored in the knowledge base.

[0186] In addition, in addition to the embedding of the style of the speaking object mentioned above, the present embodiment can also obtain the first relevant description text for supplementary description when retrieving the style prompt voice, and then fuse the embedding of the first style description text and the embedding of the first relevant description text to obtain the first global embedding of the content fragment. Among them, the first relevant description text may include attributes describing the speaking object, such as the region, education level, etc. of the speaking object. This information has an important influence on determining the speaking style, intonation, word choice, etc. of the speaking object. In addition, the first relevant description text may also include relevant information such as the application scenario of the speaking object, the gender of the speaking object, style emotion, age, region, etc., which is not specifically limited in the present embodiment. After obtaining the first relevant description text, the present embodiment can use the LLM Embedder embedding model to compress the above text information into an embedding vector of fixed length.

[0187] It should be noted that, among the embeddings extracted from the above-mentioned style profile of the speaker and the first relevant description text, the text information of the speaker's style profile itself is used to calculate an embedding representing global semantics, that is, the second global embedding, which is used for the subsequent RAG retrieval; in addition, as global information, the second global embedding can also be used to fine-tune the first language model or the second language model to improve the accuracy of emotion recognition by fine-grained text emotion embedding at the sentence level of the dialogue text.

[0188] Based on the above information, a multimodal knowledge base for retrieval can be constructed in different ways. For example, the second global embedding and the second detail embedding are concatenated; or the second global embedding and the second detail embedding are added and then L2 normalized to 1, and the obtained embedding is used as the vector representation of the corresponding speech corpus in the retrieval knowledge base.

[0189] The multimodal knowledge base can be constructed by different vector database tools, including but not limited to Milvus, Weaviate, Faiss, Chroma, Qdrant, etc. In this embodiment, Milvus is used to index the knowledge base, and a search based on the maximum inner product search (Max Inner Product Search) is used.

[0190] Among them, in the RAG style retrieval module, it is necessary to determine whether retrieval enhancement is needed based on the content to be synthesized input by the target object to improve system efficiency. It should be noted that the first language model itself has certain general generation capabilities, continuation capabilities and expressiveness. Although the RAG style retrieval module can make the generated speech expression more accurate, more real and natural, and more personalized, frequent retrieval will increase response time and computing cost. Therefore, Figure 8 As shown, the present embodiment first needs to judge the content to be synthesized to determine whether retrieval is required. For example, taking the content to be synthesized as the text to be synthesized as an example, when the target object provides not only the text to be synthesized but also a clear reference sample sound, there is no need to perform retrieval enhancement. At this time, the reference sample sound provided by the target object can be directly used as the prompt voice input to the first large voice model for generation; on the contrary, if the target object provides a query description or the first related description text, the next further retrieval enhancement process will be entered.

[0191] The purpose of the retrieval enhancement process is to select the top k corpora that are most relevant to the query vector from the pre-built knowledge base based on the similarity of the embeddings. First, the query needs to be rewritten based on the information input by the target object to improve the query so as to better match the relevant corpora. Specifically, the second largest language model can be used to perform style and emotion profiling on each speaker of the synthesized text to obtain a first style description text, and build a first global embedding of the content fragment based on the embedding of the first style description text. Then, the style profiling is combined to perform emotion recognition on each sentence spoken by the speaker, and the first detail embedding of the content fragment is built based on the embedding of the first emotion description text.

[0192] If the target object provides the first relevant description text related to the query, the HyDE method can be used to generate a pseudo document based on the first relevant description text, and calculate the query description embedding to obtain the embedding of the first style description text to represent other text information related to these retrievals.

[0193] After the above-mentioned query rewriting step, it is necessary to make another judgment on "whether subsequent retrieval enhancement is needed". The specific method is to train a classifier, input the first global embedding and the first detail embedding into the classifier for classification, and obtain the style sentiment classification result of the content fragment. The style sentiment classification result indicates whether the style sentiment of the content fragment is neutral (insufficient) or rich (sufficient). The neutral style sentiment in this embodiment does not need to be retrieved.

[0194] If the style emotion classification result indicates that the style emotion of the content segment is rich, a subsequent search is required for the expression of the style emotion. This embodiment first retrieves multiple reference voice segments from the knowledge base based on the similarity between the first global embedding and the second global embedding, and obtains the top M reference voice segments with the highest similarity as the candidate global set. Next, referring to the second detail embeddings corresponding to the M reference voice segments in the knowledge base, the similarity between the first detail embedding and these second detail embeddings is calculated, and a detail search is performed to obtain the top K reference voice segments with the highest similarity.

[0195] Optionally, this embodiment can use reranker to perform fine sorting. The purpose of this reranking step is to enhance the relevance of the retrieved files and ensure that the most relevant information appears at the top of the list. Specifically, this embodiment can determine the contextual relevance between each candidate voice segment and the content segment, so as to better understand the contextual relevance between the candidate voice segment and the content segment, and better understand the contextual subtle differences, thereby providing higher sorting accuracy. Subsequently, according to the calculation result of the contextual relevance, the candidate voice segment with the highest relevance is selected as the style prompt voice corresponding to the content segment; or, this embodiment can also determine the similarity between the second global embedding and the first global embedding of each candidate voice segment again, perform global consistency reranking (i.e., reranking according to the similarity between the second global embedding and the first global embedding), and select the candidate voice segment with the highest similarity as the style prompt voice according to the calculation result of the similarity; or, this embodiment can also perform voice quality evaluation on each candidate voice segment, and use a voice quality evaluation tool or model to evaluate the voice quality of each candidate voice segment, such as performing WER evaluation on each candidate voice segment to obtain the voice quality evaluation result of each candidate voice segment, and then determine the candidate voice segment with the highest voice quality as the style prompt voice corresponding to the content segment.

[0196] In the above rough sorting or fine sorting process, this embodiment may also provide an interactive interface to allow the target object to manually select and sort the candidate voice segments.

[0197] Finally, after determining the style cue speech, the corpus files of the retrieved style cue speech can be organized into an aligned structured form by repackaging, and in the repackaging step, since no style retrieval is performed when the style sentiment of the content segment is "neutral", continuous sentences of this type can be merged at this time, and then the "neutral" corpus files that meet the global consistency (i.e., the similarity between the second global embedding and the first global embedding) can be retrieved and completed, so that all retrieved corpus files can be aligned one by one with each speaker sentence in the target text to be synthesized.

[0198] Among them, in the speech generation module based on the large language model, refer to Fig. 9 , Fig. 9 An optional schematic diagram of a speech generation module framework based on a large language model provided for an embodiment of the present disclosure. As shown in the figure, a first large language model is deployed in the speech generation module based on a large language model, which is responsible for generating a target speech that conforms to the style and timbre based on the style prompt speech from the RAG style retrieval module, the content to be synthesized, and the timbre prompt speech. The target object can also input other prompts for speech generation according to actual needs. The dotted line in the figure represents the data flow required only in the training phase.

[0199] In this module, the speech generation task is divided into two major modeling blocks. One block is modeled as the generation task of the autoregressive speech tag sequence based on the first language model, that is, the generation task of the target speech tag. The other block is modeled based on the target speech tag sequence generated above, and the speech mel spectrum of the target speech is generated by the flow matching module.

[0200] In the LLM training phase, the input sequence is constructed as ,in , and They represent the sequence start marker, the speech transition marker between content embedding and speech marker, and the sequence end marker respectively. The speaker embedding vector extracted by the speaker encoder of the style cue speech, i.e., the second speaker embedding; It represents the content embedding obtained by passing the text to be synthesized T through a text encoder. is the sequence number of the embedded element in the content embedding, is the total number of embedded elements in the content embed; It represents the tag sequence of the tag speech X extracted by the speech tokenizer. The tag speech is the speech used as the label in the loss calculation. is the sequence number of the label speech mark in the label speech mark sequence, is the total number of label speech tags in the label speech tag sequence; Represents the acoustic features extracted from the timbre cue speech so that the acoustic environment of the generated speech sounds similar to the acoustic environment in the timbre cue speech. The objective function for training the first language model uses cross entropy loss :

[0201] ;

[0202] in, Represents the prediction of the first language model and softmax layer The posterior probability of .

[0203] like Fig. 9 As shown, the input of the Flow Matching module includes the second speaker embedding, acoustic features, and the target speech tag output by the first language model. Given these input conditions, the parameters of the Flow Matching module are trained to match the vector field of the speech data distribution.

[0204] During inference, the input sequence of the first language model is ,in, Represents the raw speech tokens extracted by the Speech Tokenizer from the style-cued speech. is the serial number of the original speech mark, is the total number of original speech tokens; represents the speaker embedding vector extracted from the style cue speech, i.e., the second speaker embedding; Represents the embedding vector of the timbre cue speech speaker extracted by the Speaker Encoder, that is, the first speaker embedding. The input of the Flow Matching module includes the first speaker embedding, the acoustic features, and the target speech tag output by the first language model. It should be emphasized that when the acoustic features are masked Mel features, in addition to the masked Mel features extracted from the timbre cue speech, the acoustic features can also be masked Mel features extracted from the style cue speech. Then, the acoustic environment of the generated speech sounds similar to the acoustic environment in the style cue speech. When the actual system is deployed, you can choose which prompt speech to extract the Mel representation from according to business needs.

[0205] This embodiment can be applied to a variety of projects and product applications such as virtual human voice assistants, live chat robots, audio and video conferences, system AI smart assistants, and in-vehicle voice interaction systems. Fig.10 , Fig.10 An optional schematic diagram of the live chatbot configuration process provided for the disclosed embodiment, as shown in the figure, this embodiment can provide a configuration box on one side of the interface, such as a live chatbot script configuration box. In the live chatbot script configuration box, you can select the re-talk function or other column functions, where you can enter the content that needs to be spoken by the live chatbot in the live chatbot function. After entering a text such as "Dear friends, welcome to my live chatbot!", click the generate button below, and the input text can be used as the text to be synthesized, and the speech generation method in the above embodiment can be used to generate speech. Finally, after determining that the generated target speech meets the requirements, you can click the delivery button at the bottom of the interface, and the target speech can be used in the live broadcast, and the chatbot will send out the target speech to achieve the interaction of the live chatbot.

[0206] In this embodiment, the Mel representation extracted from the timbre prompt speech is used as the acoustic feature. In addition, in this embodiment, the Text Encoder can be obtained by training and fine-tuning the SentencePiece Tokenizer and Text Encoder in the Non-Language Left Behind (NLLB) open source pre-trained large model; the Speech Tokenizer can be obtained by training and fine-tuning the w2v-bert 2.0 open source pre-trained large model; the first large language model is obtained by training and fine-tuning the pre-trained large language model. In addition, the Speech Tokenizer in this embodiment can also be obtained by training and fine-tuning the supervised Speech Tokenizer in cosyvoice, and accordingly, the Text Encoder is obtained by training the Byte PairEncoded (BPE) tokenizer, and the Text Encoder is used to align the semantic space of the text token and the speech token.

[0207] It should be noted that the Text Encoder used in this embodiment, in addition to the Text Encoder based on the NLLB pre-trained large model or the Text Encoder based on CosyVoice mentioned in the above technical solution, can also use other Text Encoders as alternatives; the Speech Tokenizer used in this embodiment, in addition to the Speech Tokenizer based on the w2v-bert 2.0 open source pre-trained large model and fine-tuned or the Speech Tokenizer based on CosyVoice's supervised Speech Tokenizer mentioned in the above technical solution, can also use other Speech Tokenizer as an alternative, and the alternative needs to meet the requirements of only extracting, encoding and tokenizing the semantic content of the speech, without including other acoustic information (such as the timbre of the speaker, ambient sound), etc.

[0208] The first large language model in this embodiment may also adopt other large language models, and the model size may be selected, distilled, and optimized according to the resource conditions of the actual application scenario.

[0209] The speech generation module based on Flow Matching adopted in this embodiment can be replaced by other AIGC models based on conditional generation, including SOTA or future extended models based on Diffusion or Flow Matching.

[0210] The multimodal knowledge base construction and RAG style retrieval module described in this embodiment include text modality and voice modality. Other alternatives include but are not limited to voice and text modalities, and may also include multimodal knowledge base and retrieval including vision and images.

[0211] In summary, this embodiment successfully decouples the content, style and timbre of the target speech, supports flexible combination of any style and timbre of any speaking object, and the generated speech timbre is highly similar to the target timbre. The style and emotion of the generated speech are natural, real, and appropriate, and its expressiveness can be personalized and controlled, and can achieve a high degree of consistency with the target style and emotion. The speech generation quality, accuracy and stability all meet the requirements of practical applications.

[0212] In addition, this embodiment has simple training and deployment, extremely high friendliness, supports speech generation under zero-shot conditions, and does not require relatively troublesome training and deployment such as additional fine-tuning data and fine-tuning models. Moreover, only a very short (e.g., 3 seconds) reference style and reference timbre are required as prompts to generate highly expressive speech with highly consistent style and timbre, which is very friendly.

[0213] This embodiment supports automatic, flexible, high-precision, and personalized matching of appropriate style and emotion for the text content to be synthesized, achieving real, natural, fluent, appropriate, personalized, richly varied, flexible and not rigid voice expression. Moreover, the cost of this beneficial effect is extremely low (almost zero cost), unlike traditional technical solutions that require huge manpower and financial costs, extremely long time, hiring voice actors for customized recording, fine-tuning training models, etc., but through the innovative RAG+LLM framework design of this embodiment, it is possible to fully, automatically, and intelligently utilize the massive, rich, and free data and knowledge resources in the Internet and media era to achieve better results.

[0214] It is to be understood that, although the steps in the above-mentioned various flow charts are shown in sequence according to the indication of the arrows, these steps are not necessarily performed in sequence according to the order indicated by the arrows. Unless there is a clear explanation in the present embodiment, the execution of these steps does not have a strict order restriction, and these steps can be performed in other orders. Moreover, at least a portion of the steps in the above-mentioned flow charts can include a plurality of steps or a plurality of stages, and these steps or stages are not necessarily performed at the same time, but can be performed at different times, and the execution order of these steps or stages is not necessarily performed in sequence, but can be performed in turn or alternately with at least a portion of the steps or stages in other steps or other steps.

[0215] Reference Fig.11 , Fig.11 This is an optional structural diagram of a speech generation device provided in an embodiment of the present disclosure. The speech generation device 1100 can execute the speech generation method in any of the above embodiments. The device includes:

[0216] The content processing module 1101 is used to obtain the content to be synthesized for content constraint during speech generation, and extract the content embedding of the content to be synthesized;

[0217] A style processing module 1102 is used to obtain a style prompt speech for performing style constraints during speech generation, and extract a plurality of original speech tags in the style prompt speech;

[0218] A prediction module 1103 is used to input the content embedding and the multiple original speech tags into the first language model for prediction to obtain multiple target speech tags;

[0219] The timbre processing module 1104 is used to obtain the timbre prompt speech for performing timbre constraint during speech generation, and extract the first speaker embedding of the timbre prompt speech;

[0220] The generating module 1105 is configured to generate speech based on the first speaker embedding and a plurality of target speech tags to obtain a target speech.

[0221] Furthermore, the style processing module 1102 is specifically used for:

[0222] Inputting the content to be synthesized into the second largest language model for prediction, obtaining a first style description text for describing the style of the speaker of the content to be synthesized, and a first emotion description text for describing the emotion of each content segment in the content to be synthesized;

[0223] Building a first global embedding of the content fragment based on the embedding of the first style description text, and building a first detail embedding of the content fragment based on the embedding of the first emotion description text;

[0224] Based on the first global embedding and the first detail embedding, a style prompt speech corresponding to the content segment is retrieved from a preset knowledge base, wherein the style prompt speech is used to perform style constraints during speech generation.

[0225] Furthermore, the knowledge base stores a reference speech segment and a second global embedding and a second detail embedding associated with the reference speech segment, the second global embedding is used to indicate the speaker style of the reference speech segment, and the second detail embedding is used to indicate the emotion of the reference speech segment. The style processing module 1102 is also used to:

[0226] Retrieving a plurality of reference speech segments from a knowledge base according to the similarity between the first global embedding and the second global embedding;

[0227] According to the similarity between the first detail embedding and the second detail embedding, the style cue speech corresponding to the content segment is retrieved from the plurality of reference speech segments.

[0228] Furthermore, before retrieving the style prompt voice corresponding to the content segment from the preset knowledge base based on the first global embedding and the first detail embedding, the style processing module 1102 is further used to:

[0229] Acquire original speech, segment the original speech into speaking objects, and obtain multiple speech blocks;

[0230] Performing speech recognition on each speech block respectively to obtain the speech block text in each speech block;

[0231] Segmenting the speech block text to obtain multiple reference text segments, and segmenting the speech block into reference speech segments corresponding to each reference text segment;

[0232] A second global embedding is constructed based on multiple speech block texts, a second detail embedding is constructed based on a reference text segment, and the reference speech segment, the second global embedding, and the second detail embedding are associated and stored in a knowledge base.

[0233] Furthermore, the style processing module 1102 is also used for:

[0234] Constructing a first prompt text based on the multiple speech block texts, inputting the first prompt text into the second largest language model for prediction, and obtaining a second style description text for describing the style of the speaker of the multiple speech block texts, wherein the first prompt text is used to prompt the second largest language model to output a second style description text based on the multiple speech block texts;

[0235] Building a second global embedding based on the embedding of the second style description text;

[0236] Constructing a second prompt text based on the second style description text, the reference text segment, and the plurality of speech block texts, inputting the second prompt text into the second largest language model for prediction, and obtaining a second emotion description text for describing the emotion of the reference text segment, wherein the second prompt text is used to prompt the second largest language model to output a second emotion description text based on the second style description text, the reference text segment, and the plurality of speech block texts;

[0237] The second detail embedding is constructed based on the embedding of the second sentiment description text.

[0238] Furthermore, the style processing module 1102 is also used for:

[0239] Retrieving a plurality of candidate speech segments from a plurality of reference speech segments according to a similarity between the first detail embedding and the second detail embedding;

[0240] Determine the contextual relevance between each candidate speech segment and the content segment, and determine the candidate speech segment with the highest contextual relevance as the style prompt speech corresponding to the content segment, or, again determine the similarity between the second global embedding and the first global embedding of each candidate speech segment, and determine the candidate speech segment with the highest similarity as the style prompt speech corresponding to the content segment, or, perform speech quality evaluation on each candidate speech segment, and determine the candidate speech segment with the highest speech quality as the style prompt speech corresponding to the content segment.

[0241] Furthermore, the style processing module 1102 is also used for:

[0242] Acquire a first relevant description text for supplementary description when retrieving the style hint voice;

[0243] The embedding of the first style description text and the embedding of the first related description text are fused to obtain a first global embedding of the content segment.

[0244] Furthermore, the style processing module 1102 is also used for:

[0245] The first global embedding and the embedding of the first sentiment description text are fused to obtain the first detail embedding of the content segment.

[0246] Furthermore, the content to be synthesized is speech to be synthesized, and the content segment is a speech segment. The style processing module 1002 is further used for:

[0247] Extract speech emotion embeddings of speech clips;

[0248] The first global embedding, the embedding of the first emotion description text, and the speech emotion embedding are integrated to obtain the first detail embedding of the content segment.

[0249] Furthermore, the style processing module 1102 is also used for:

[0250] Inputting the first global embedding and the first detail embedding into a classifier for classification, and obtaining a style emotion classification result of the content segment, wherein the style emotion classification result is used to indicate that the style emotion of the content segment is neutral, or is used to indicate that the style emotion of the content segment is rich;

[0251] When the style emotion classification result indicates that the style emotion of the content segment is rich, based on the first global embedding and the first detail embedding, the style prompt speech corresponding to the content segment is retrieved from a preset knowledge base.

[0252] Furthermore, the style processing module 1102 is also used for:

[0253] When the style emotion classification result indicates that the style emotion of the content segment is neutral, based on the first global embedding, the style prompt speech corresponding to the content segment is retrieved from the knowledge base.

[0254] Furthermore, the prediction module 1103 is specifically used for:

[0255] A second speaker embedding of the style cue speech is extracted, and the content embedding, the second speaker embedding, and a plurality of original speech tags are input into the first large language model for prediction to obtain a plurality of target speech tags.

[0256] Furthermore, the prediction module 1103 is also used for:

[0257] The sequence start marker, the second speaker embedding, the content embedding, the voice conversion marker and the multiple original voice markers are sequentially concatenated and input into the first large language model for prediction to obtain multiple target voice markers.

[0258] Furthermore, the generation module 1105 is specifically used for:

[0259] The acoustic features of the style prompt speech or the timbre prompt speech are extracted, and speech generation is performed based on the acoustic features, the first speaker embedding, and a plurality of target speech tags to obtain a target speech.

[0260] In summary, the speech generation device 1100 in the embodiment of the present disclosure can constrain the content in the target speech through the content embedding by acquiring the content to be synthesized for performing content constraints during speech generation, and can also extract the content embedding of the content to be synthesized during speech generation. In addition, by acquiring the style prompt speech for performing style constraints during speech generation, a plurality of original speech tags in the style prompt speech are extracted, and the content embedding and the plurality of original speech tags are input into the first language model for prediction, and can predict a plurality of target speech tags similar to the style of the style prompt speech based on the first language model. On this basis, by acquiring the style prompt speech for performing style constraints during speech generation, the target speech tags similar to the style of the style prompt speech are predicted based on the first language model. The timbre-constrained timbre prompt speech extracts the first speaker embedding of the timbre prompt speech, and then generates speech based on the first speaker embedding and multiple target speech tags, thereby obtaining a target speech that is similar in timbre to the timbre prompt speech and in style to the style prompt speech, thereby achieving control of style and timbre, making the target speech more natural, and thus improving the authenticity of the target speech. In addition, since the content to be synthesized, the style prompt speech, and the timbre prompt speech are respectively introduced at different stages of the generation process, the decoupling of the content, style, and timbre of the target speech is achieved, and the flexible combination of any content, style, and timbre is supported, thereby improving the flexibility of the target speech.

[0261] The electronic device for executing the above-mentioned speech generation method provided in the embodiment of the present disclosure may be a terminal. Fig.12 , Fig.12 This is a partial structural block diagram of a terminal provided in an embodiment of the present disclosure, and the terminal includes: a camera assembly 1210, a first memory 1220, an input unit 1230, a display unit 1240, a sensor 1250, an audio circuit 1260, a wireless fidelity (WiFi) module 1270, a first processor 1280, and a first power supply 1290. Those skilled in the art can understand that Fig.12 The terminal structure shown in the figure does not constitute a limitation on the terminal, and may include more or less components than shown in the figure, or combine certain components, or arrange the components differently.

[0262] The camera assembly 1210 can be used to capture images or videos. Optionally, the camera assembly 1210 includes a front camera and a rear camera. Typically, the front camera is disposed on the front panel of the terminal, and the rear camera is disposed on the back of the terminal. In some embodiments, there are at least two rear cameras, which are any one of a main camera, a depth of field camera, a wide-angle camera, and a telephoto camera, so as to realize the fusion of the main camera and the depth of field camera to realize the background blur function, the fusion of the main camera and the wide-angle camera to realize the panoramic shooting and VR (Virtual Reality) shooting function or other fusion shooting functions.

[0263] The first memory 1220 may be used to store software programs and modules. The first processor 1280 executes various functional applications and data processing of the terminal by running the software programs and modules stored in the first memory 1220 .

[0264] The input unit 1230 may be used to receive input digital or character information and generate key signal input related to the terminal's settings and function control. Specifically, the input unit 1230 may include a touch panel 1231 and other input devices 1232 .

[0265] The display unit 1240 may be used to display input information or provided information and various menus of the terminal. The display unit 1240 may include a display panel 1241 .

[0266] The audio circuit 1260 , the speaker 1261 , and the microphone 1262 may provide an audio interface.

[0267] The first power source 1290 may be alternating current, direct current, a disposable battery, or a rechargeable battery.

[0268] The number of sensors 1250 may be one or more, and the one or more sensors 1250 include but are not limited to: acceleration sensors, gyroscope sensors, pressure sensors, optical sensors, etc. Among them:

[0269] The acceleration sensor can detect the magnitude of acceleration on the three coordinate axes of the coordinate system established by the terminal. For example, the acceleration sensor can be used to detect the components of gravity acceleration on the three coordinate axes. The first processor 1280 can control the display unit 1240 to display the user interface in a horizontal view or a vertical view according to the gravity acceleration signal collected by the acceleration sensor. The acceleration sensor can also be used for collecting game or user motion data.

[0270] The gyroscope sensor can detect the body direction and rotation angle of the terminal, and the gyroscope sensor can cooperate with the acceleration sensor to collect the user's 3D actions on the terminal. The first processor 1280 can implement the following functions based on the data collected by the gyroscope sensor: motion sensing (such as changing the UI according to the user's tilt operation), image stabilization during shooting, game control, and inertial navigation.

[0271] The pressure sensor can be set in the side frame of the terminal and / or the lower layer of the display unit 1240. When the pressure sensor is set in the side frame of the terminal, the user's holding signal of the terminal can be detected, and the first processor 1280 performs left and right hand recognition or shortcut operation according to the holding signal collected by the pressure sensor. When the pressure sensor is set in the lower layer of the display unit 1240, the first processor 1280 controls the operability controls on the UI interface according to the user's pressure operation on the display unit 1240. The operability control includes at least one of a button control, a scroll bar control, an icon control, and a menu control.

[0272] The optical sensor is used to collect the ambient light intensity. In one embodiment, the first processor 1280 can control the display brightness of the display unit 1240 according to the ambient light intensity collected by the optical sensor. Specifically, when the ambient light intensity is high, the display brightness of the display unit 1240 is increased; when the ambient light intensity is low, the display brightness of the display unit 1240 is decreased. In another embodiment, the first processor 1280 can also dynamically adjust the shooting parameters of the camera assembly 1210 according to the ambient light intensity collected by the optical sensor.

[0273] In this embodiment, the first processor 1280 included in the terminal can execute the speech generation method of the previous embodiment.

[0274] The electronic device for executing the above-mentioned speech generation method provided in the embodiment of the present disclosure may also be a server. Fig.13 , Fig.13 This is a partial structural block diagram of a server provided in an embodiment of the present disclosure. The server may have relatively large differences due to different configurations or performances, and may include one or more second processors 1310 and a second memory 1330, and one or more storage media 1340 (e.g., one or more mass storage devices) storing application programs 1343 or data 1342. Among them, the second memory 1330 and the storage medium 1340 may be short-term storage or permanent storage. The program stored in the storage medium 1340 may include one or more modules (not shown in the figure), and each module may include a series of instruction operations on the server. Furthermore, the second processor 1310 may be configured to communicate with the storage medium 1340 to execute a series of instruction operations in the storage medium 1340 on the server.

[0275] The server may also include one or more second power supplies 1320, one or more wired or wireless network interfaces 1350, one or more input and output interfaces 1360, and / or one or more operating systems 1341, such as Windows ServerTM, Mac OS XTM, UnixTM, LinuxTM, FreeBSDTM, etc.

[0276] The second processor 1310 in the server may be configured to execute the speech generation method.

[0277] The embodiments of the present disclosure also provide a computer-readable storage medium, which is used to store a computer program, and the computer program is used to execute the speech generation method of each of the aforementioned embodiments.

[0278] The present disclosure also provides a computer program product, which includes a computer program stored in a computer-readable storage medium. A processor of a computer device reads the computer program from the computer-readable storage medium, and the processor executes the computer program, so that the computer device executes the above-mentioned speech generation method.

[0279] The terms "first", "second", "third", "fourth", etc. (if any) in the specification of the present disclosure and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate to describe the embodiments of the present disclosure, such as being able to be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0280] It should be understood that in the present disclosure, "at least one (item)" means one or more, and "plurality" means two or more. "And / or" is used to describe the association relationship of associated objects, indicating that three relationships may exist. For example, "A and / or B" can mean: only A exists, only B exists, and A and B exist at the same time, where A and B can be singular or plural. The character " / " generally indicates that the objects associated before and after are in an "or" relationship. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single or plural items. For example, at least one of a, b or c can mean: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, c can be single or multiple.

[0281] It should be understood that in the description of the embodiments of the present disclosure, the meaning of multiple (or multiple items) is more than two, greater than, less than, exceed, etc. are understood as not including the number, and above, below, within, etc. are understood as including the number.

[0282] In the several embodiments provided in the present disclosure, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of units is only a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or units, which can be electrical, mechanical or other forms.

[0283] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0284] In addition, each functional unit in each embodiment of the present disclosure may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0285] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present disclosure is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the various embodiments of the present disclosure. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (Read-Only Memory, referred to as ROM), random access memory (Random Access Memory, referred to as RAM), disk or optical disk and other media that can store program codes.

[0286] It should also be understood that the various implementations provided in the embodiments of the present disclosure can be combined arbitrarily to achieve different technical effects.

[0287] The above is a specific description of the preferred implementation of the present disclosure, but the present disclosure is not limited to the above-mentioned implementation mode. Technical personnel familiar with the field can also make various equivalent deformations or substitutions under the shared conditions without violating the spirit of the present disclosure. These equivalent deformations or substitutions are all included in the scope defined by the claims of the present disclosure.

Claims

1. A speech generation method, characterized in that: include: Acquire content to be synthesized for content constraint during speech generation, and extract content embedding of the content to be synthesized; The content to be synthesized is input into the second largest language model for prediction, and a first style description text for describing the style of the speaker of the content to be synthesized and a first emotion description text for describing the emotion of each content segment in the content to be synthesized are obtained; a first global embedding of the content segment is constructed based on the embedding of the first style description text, and a first detail embedding of the content segment is constructed based on the embedding of the first emotion description text; based on the first global embedding and the first detail embedding, a style prompt speech corresponding to the content segment is retrieved from a preset knowledge base, and a plurality of original speech markers in the style prompt speech are extracted, wherein the first style description text includes words or phrases for describing the style of the speaker, the first emotion description text is used to reflect the emotional state expressed in the content segment, and the style prompt speech is used to perform style constraints during speech generation; Inputting the content embedding and the plurality of original speech tags into a first large language model for prediction to obtain a plurality of target speech tags; Acquire a timbre prompt speech for performing timbre constraint during speech generation, and extract a first speaker embedding of the timbre prompt speech; Speech generation is performed based on the first speaker embedding and the plurality of target speech tags to obtain a target speech.

2. The speech generation method according to claim 1, characterized in that: The knowledge base stores a reference voice segment and a second global embedding and a second detail embedding associated with the reference voice segment, wherein the second global embedding is used to indicate the style of the speaker of the reference voice segment, and the second detail embedding is used to indicate the emotion of the reference voice segment. The method of retrieving the style prompt voice corresponding to the content segment from the preset knowledge base based on the first global embedding and the first detail embedding includes: Retrieving a plurality of reference speech segments from the knowledge base according to the similarity between the first global embedding and the second global embedding; According to the similarity between the first detail embedding and the second detail embedding, the style cue speech corresponding to the content segment is retrieved from the plurality of reference speech segments.

3. The speech generation method according to claim 2, characterized in that: Before retrieving the style prompt voice corresponding to the content segment from a preset knowledge base based on the first global embedding and the first detail embedding, the voice generation method further includes: Acquire original speech, and segment the original speech into speaking objects to obtain multiple speech blocks; Performing speech recognition on each of the speech blocks respectively to obtain speech block text in each of the speech blocks; Segmenting the speech block text to obtain a plurality of reference text segments, and segmenting the speech block into reference speech segments corresponding to the reference text segments; The second global embedding is constructed based on the plurality of speech block texts, the second detail embedding is constructed based on the reference text segment, and the reference speech segment, the second global embedding and the second detail embedding are associated and stored in the knowledge base.

4. The speech generation method according to claim 3, characterized in that: The step of constructing the second global embedding based on the plurality of speech block texts and constructing the second detail embedding based on the reference text segment comprises: Constructing a first prompt text based on the plurality of speech block texts, inputting the first prompt text into the second largest language model for prediction, and obtaining a second style description text for describing the style of the speaker of the plurality of speech block texts, wherein the first prompt text is used to prompt the second largest language model to output the second style description text based on the plurality of speech block texts; Building the second global embedding based on the embedding of the second style description text; Constructing a second prompt text based on the second style description text, the reference text segment, and the plurality of speech block texts, inputting the second prompt text into the second large language model for prediction, and obtaining a second emotion description text for describing the emotion of the reference text segment, wherein the second prompt text is used to prompt the second large language model to output the second emotion description text based on the second style description text, the reference text segment, and the plurality of speech block texts; The second detail embedding is constructed based on the embedding of the second emotion description text.

5. The speech generation method according to claim 2, characterized in that: The retrieving the style prompt speech corresponding to the content segment from the plurality of reference speech segments according to the similarity between the first detail embedding and the second detail embedding, comprises: Retrieving a plurality of candidate speech segments from a plurality of the reference speech segments according to the similarity between the first detail embedding and the second detail embedding; Determine the contextual relevance between each of the candidate speech segments and the content segment, and determine the candidate speech segment with the highest contextual relevance as the style prompt speech corresponding to the content segment, or, again determine the similarity between the second global embedding and the first global embedding of each of the candidate speech segments, and determine the candidate speech segment with the highest similarity as the style prompt speech corresponding to the content segment, or, perform speech quality evaluation on each of the candidate speech segments, and determine the candidate speech segment with the highest speech quality as the style prompt speech corresponding to the content segment.

6. The speech generation method according to claim 1, characterized in that: The constructing a first global embedding of the content fragment based on the embedding of the first style description text includes: Acquire a first relevant description text for supplementary description when retrieving the style prompt voice; The embedding of the first style description text and the embedding of the first related description text are fused to obtain a first global embedding of the content segment.

7. The speech generation method according to claim 1, characterized in that: The step of constructing the first detail embedding of the content segment based on the embedding of the first emotion description text includes: The first global embedding and the embedding of the first emotion description text are fused to obtain a first detail embedding of the content segment.

8. The speech generation method according to claim 7, characterized in that: The content to be synthesized is speech to be synthesized, the content segment is a speech segment, and the embedding of the first emotion description text based on the first detail embedding of the content segment is constructed, including: Extracting speech emotion embedding of the speech segment; The first global embedding, the embedding of the first emotion description text, and the speech emotion embedding are fused to obtain a first detail embedding of the content segment.

9. The speech generation method according to claim 1, characterized in that: The retrieving the style prompt voice corresponding to the content segment from a preset knowledge base based on the first global embedding and the first detail embedding includes: Inputting the first global embedding and the first detail embedding into a classifier for classification to obtain a style emotion classification result of the content segment, wherein the style emotion classification result is used to indicate that the style emotion of the content segment is neutral, or is used to indicate that the style emotion of the content segment is rich; When the style emotion classification result indicates that the style emotion of the content segment is rich, based on the first global embedding and the first detail embedding, the style prompt speech corresponding to the content segment is retrieved from a preset knowledge base.

10. The speech generation method according to claim 9, characterized in that: The retrieving the style prompt voice corresponding to the content segment from a preset knowledge base based on the first global embedding and the first detail embedding also includes: When the style emotion classification result indicates that the style emotion of the content segment is neutral, based on the first global embedding, the style prompt speech corresponding to the content segment is retrieved from the knowledge base.

11. The speech generation method according to claim 1, characterized in that: The embedding of the content and the plurality of original speech tags are input into a first large language model for prediction to obtain a plurality of target speech tags, including: A second speaker embedding of the style cue speech is extracted, and the content embedding, the second speaker embedding, and the plurality of original speech tags are input into a first large language model for prediction to obtain a plurality of target speech tags.

12. The speech generation method according to claim 11, characterized in that: The step of inputting the content embedding, the second speaker embedding, and the plurality of original speech tags into a first large language model for prediction to obtain a plurality of target speech tags includes: The sequence start marker, the second speaker embedding, the content embedding, the voice conversion marker and the plurality of original voice markers are sequentially concatenated and input into the first large language model for prediction to obtain a plurality of target voice markers.

13. The speech generation method according to claim 1, characterized in that: The generating speech based on the first speaker embedding and the plurality of target speech tags to obtain the target speech includes: The acoustic features of the style prompt speech or the timbre prompt speech are extracted, and speech generation is performed based on the acoustic features, the first speaker embedding, and the plurality of target speech tags to obtain a target speech.

14. A speech generating device, characterized in that: include: A content processing module, used to obtain content to be synthesized for content constraint during speech generation, and extract content embedding of the content to be synthesized; A style processing module, used for inputting the content to be synthesized into the second largest language model for prediction, obtaining a first style description text for describing the style of the speaker of the content to be synthesized, and a first emotion description text for describing the emotion of each content segment in the content to be synthesized; constructing a first global embedding of the content segment based on the embedding of the first style description text, and constructing a first detail embedding of the content segment based on the embedding of the first emotion description text; based on the first global embedding and the first detail embedding, retrieving the style prompt speech corresponding to the content segment from a preset knowledge base, and extracting a plurality of original speech markers in the style prompt speech, wherein the first style description text includes words or phrases for describing the style of the speaker, the first emotion description text is used to reflect the emotional state expressed in the content segment, and the style prompt speech is used to perform style constraints during speech generation; A prediction module, configured to input the content embedding and the plurality of original speech tags into a first large language model for prediction to obtain a plurality of target speech tags; A timbre processing module, used for acquiring a timbre prompt voice for timbre constraint during speech generation, and extracting a first speaker embedding of the timbre prompt voice; A generation module is used to generate speech based on the first speaker embedding and the plurality of target speech tags to obtain a target speech.

15. An electronic device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the speech generation method according to any one of claims 1 to 13 is implemented.

16. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the speech generation method according to any one of claims 1 to 13 is implemented.

17. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the speech generation method according to any one of claims 1 to 13 is implemented.

Images